Techniques and systems for minimizing excess egress data in a communication session

Abstract

Techniques for reducing excess egress data in a communication session including transmitting, with a client device, a request for a file to a plurality of server devices, and determining an operating characteristic of each communication path of a plurality of communication paths that are each between the client device and a respective server device. The techniques include determining a window credit value for each server device based at least in part on the respective operating characteristic, a size of the file, an amount of the file that has been received by the client device, and a bytes-in-flight (BIF) value corresponding to data that has been requested by the client device but has not yet been received by the client device. The techniques include transmitting the respective window credit value to each of the server devices to limit data transmitted by the server devices to the client device.

Description

D24017W001TECHNIQUES AND SYSTEMS FOR MINIMIZING EXCESS EGRESS DATA IN A COMMUNICATION SESSIONCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority from U.S. Provisional Application No. 63 / 682,258 filed on 12 August 2024, and U.S. Provisional Application No. 63 / 707,375 filed on 15 October 2024, each of which is incorporated by reference herein in its entirety.TECHNICAL FIELD

[0002] The present disclosure relates generally to management of downloading of data on a client device from multiple communication paths to minimize (e.g., reduce or eliminate) excess egress data during the download process.SUMMARY

[0003] Client devices that are used to stream media (e.g., smart televisions, smart phones, computers, and other electronic devices) may use multipath downloading to improve streaming throughput and robustness and provide smooth downloading experience on the client side.When source content is coded, multipath delivery becomes even more powerful as the client device only needs to download enough coded packets from any one or a combination of the sources and then decode the coded packets to retrieve the original source file (e.g., because the coded packets from any source are interchangeable and allow for decoding of the source file if an amount of coded packets corresponding to the size of the source file have been received). Downloading of coded packets from multiple sources (e.g., server devices) can be achieved, for example, by using HyperText Transfer Protocol (HTTP), where the client device sends a request to each source (e.g., server device) and starts downloading a coded file from each source. The client device then stops requesting additional data when the client device receives 100% of coded information corresponding to the size of the requested file. FIG. 1 is a diagram that illustrates an example multipath downloading process of a client device 102. The client device 102 receives coded packets 105 from each of three different server devices 110A, HOB, 110C via respective communication paths 115 A, 115B, 115C (which also may be referred to as communication channels, communication links, etc.). While the amount of data shown as the coded packets 105 and the data from the communication paths 115 in FIG. 1 is shown as a single bar that is not subdivided, it should be understood that the bars that represents the coded packets 105 and the data from the communication paths 115 represent a number of different packets 105.D24017W001In other words, although not shown as subdivided packets, the bars that represent data in FIG. 1 represent coded packets that are separately transmitted from the server devices 110 and received by the client device 102.

[0004] In some instances, the server devices 110 continue sending requested data (e.g., 100% of the coded file) unless they receive a cancellation signal from the client device 102. The goal of the client device 102 is to receive 100% of the coded file from multiple sources 110. In this case, at most 300% of the file can be downloaded by the client (e.g., if each server device 110 transmits 100% of the coded file to the client device 102. In other words, if the client device 102 does not cancel the downloading process when it receives a total of 100% of the coded file from any combination of the server devices 110, the server devices 110 will keep sending data to the client device 102 and eventually the client will end up receiving 300% of the coded information even though only 100% of the coded information is needed to decode the original file. Thus, the amount of excess egress data generated by downloading this file is 200% in this instance. Although using this downloading approach maximizes throughput by not canceling the downloading request on any of the communication paths 115, this downloading approach also maximizes the excess egress data. Moreover, the excess egress data may block the downloading process of next / future objects / files, and may negatively affect the downloading speed of the next object in the downloading session. Excess egress data unnecessarily increases network resources being used for the download (e.g., downloading excess data on one or more communication paths 115 that is not useful to the client device 102 instead of allowing the communication paths 115 to be free sooner for downloading of additional data by the client device 102 or other client devices), unnecessarily wastes power, memory space, and processing resources of the client device 102 by receiving excess data, and unnecessarily increases the cost of download.Accordingly, there is a technological problem with the downloading process on client devices. The technological problem involves the balance between the desire to have high throughput / fast downloading speeds and the desire to reduce or eliminate excess egress data that wastes network resources and client devices resources. The techniques, systems, and devices described herein address this technological problem by minimizing (e.g., reducing or eliminating) excess egress data while minimally affecting throughput or by not affecting throughput at all. The techniques, systems, and devices described herein also allow for different downloading modes, for example, to maximize throughput (sometimes at the expense of excess egress data), to minimize excess egress data (sometimes at the expense of throughput), or to reduce excess egress data compared to current downloading approaches without sacrificing throughput. In some instances, theD24017W001 techniques, systems, and devices described herein leverage features of existing communication protocols (e.g., HTTP / 2, HTTP / 3 over Quick User Datagram Protocol (UDP) Internet Connections (QUIC), etc.) to reduce or eliminate excess egress data using the client device to limit the amount of data that is transmitted to the client device by each server device 110 during a multipath downloading process.

[0005] In the Transmission Control Protocol / Intemet Protocol (TCP / IP), the server device performs flow control to avoid folding a receiving client device. Correspondingly, the client device informs the server device of its receiving buffer size. For example, when the client device sends an acknowledgement (Ack) packet to the server device, the client device also sends its buffer size along with the Ack packet. Thus, when the server receives the Ack packet, the server device checks the size of the client device buffer. If the buffer size is zero, the client device is fully occupied and cannot receive more data. Accordingly, the server device will wait before transmitting additional data and will periodically send probing packets to the client device. The server device will resume sending TCP packets to the client device when the client's buffer size becomes greater than zero. This process is generally known as "TCP Flow Control," and it is implemented in the transport layer of the TCP. In the transport layer, the sender (i.e., the server device) waits for the receiver (i.e., the client device), as the data sent by the server device is the minimum of the client device’s receiver buffer “rwnd” or congestion window “cwnd.” Moreover, the information exchange rate between the client device and the server device is at its theoretical maximum, as the server device almost immediately sends data in response to receiving an Ack packet. The TCP flow control process allows the server device to transmit data to the client device without overwhelming the client device.

[0006] Given the TCP flow control process explained above, a simple cancellation technique (i.e., having the client device send a cancellation request to the server devices in response to the client device receiving 100% of a requested file / object) does not work well to reduce or eliminate excess egress data. For example, FIG. 2 is a diagram that illustrates an example multipath downloading process of the client device 102 where a cancellation request / command is used by the client device 102 when 100% of the file is received by the client device 102. In a cancellation-based approach, the client device 102 may send only one request (e.g., a request to download a file) per content delivery network (CDN) at the start of the downloading process. Moreover, the speed of the downloading from each communication path 115 is at its maximum depending on the network conditions of each communication path 115. When the client device 102 receives 100% of the file, the client device 102 sends a HTTP cancellation command to theD24017W001 server devices 110, and the server devices 110 will stop sending the data. However, due to the delay between the time when client device 102 sends the cancellation command and when the cancellation command is received by the server devices 110 (and executed by the server devices 110), there will be excess data 205 being sent to the client device 102 anyway (i.e., momentum data).

[0007] The reasons that this momentum data continues to be sent include that the server devices 110 do not know files are being delivered using multiple communication paths 115. As a result, the server devices 110 try to send 100% of the file or requested data as quickly as possible. In other words, current implementations of HTTP and TCP alone are not designed with multipath usage in mind. Additionally, there is an extra layer of delay added between the HTTP layer and TCP layer (e.g., when the HTTP layer receives the stop command, the TCP layer is still trying to send data). Thus, the control information exchange rate between the server devices 110 and the client device 102 is not maximized, which causes delay. FIG. 2 shows an example indication of excess egress data 205 sent by each server device 110. The portion of data in each channel to the left of the excess egress data 205 indicates data that was sent to the client device 102 and used by the client device 102 to reach 100% of the requested data (e.g., 20% of the file / data from the third server device 110C and 40% from of the file / data from each of the first and second server devices 110A and HOB results in 100% of the file / data at the client device 102).

[0008] Based on the above explanations of the TCP flow control process and momentum data, one approach to minimizing the excess egress data may include sending control / state information as quickly as possible (e.g., fusing the control and state information into the transport layer by modifying the TCP packets). This will ensure that the server device 110 can receive the control signal sent by the client device 102 at a higher rate. Another approach to minimizing the excess egress data may include making the server-side TCP and HTTP protocol smarter. For example, the server-side HTTP protocol may be configured to not send more than the amount of data that is desired by the client device 102. However, there are several difficulties of implementing these approaches. One such difficulty is that TCP packets cannot be readily modified on the client side. Moreover, the behavior of the server-side TCP protocol cannot be modified in some instances. Thus, the above-noted approaches are challenging to implement.

[0009] In contrast, as described herein, other client-side capabilities incorporated into existing protocols can be creatively leveraged to reduce or eliminate excess egress data. ForD24017W001 example, as described herein, a client-side flow control algorithm using HTTP / 2 can be used to reduce or eliminate excess egress data in a manner that does not involve the above-noted difficulties associated with TCP.

[0010] In one aspect of the present disclosure, there is provided a method for reducing excess egress data in a communication session. The method may include transmitting, with a client device, a request for a file to a plurality of server devices. The method may also include determining, with the client device, an operating characteristic of each communication path of a plurality of communication paths. Each communication path may be between the client device and a respective server device of the plurality of server devices. The method may also include determining, with the client device, a window credit value for each server device based at least in part on the respective operating characteristic, a size of the file, an amount of the file that has been received by the client device on all of the plurality of communication paths, and a bytes-in- flight (BIF) value corresponding to data that has been requested by the client device but has not yet been received by the client device. The method may also include transmitting, with the client device, the respective window credit value to each of the server devices. The respective window credit value may indicate an amount of the file that each server device is configured to transmit to the client device. The method may also include receiving, with the client device, a portion of the file from each of the server devices. A size of the portion of the file from each server device may be limited by the respective window credit value of the respective server device.

[0011] In addition to any combination of features described above, determining the operating characteristic of each communication path may includes determining, with the client device, a bandwidth of each communication path and a round-trip time (RTT) of each communication path, and determining, with the client device, a bandwidth delay product (BDP) of each communication path by determining a product of the bandwidth and the RTT of each communication path. The method may also include determining, with the client device, the amount of the file that has been received by the client device based on the portions of the file received from each server device. The method may also include determining, with the client device, the BIF value for each communication path by subtracting the amount of the file that has been received by the client device on each communication path from a total amount of window credit values that have been transmitted to the respective server device corresponding to each communication path. The method may also include determining, with the client device, whether a difference between the size of the file and the total amount of window credit values that haveD24017W001 been transmitted to all of the plurality of server devices is greater than a predetermined threshold. The method may also include, in response to determining that the difference between the size of the file and the total amount of window credit values that have been transmitted to all of the plurality of server devices is greater than the predetermined threshold, determining, with the client device, the window credit value for each server device based on the BDPs of the communication paths over which the server devices communicate with the client device.

[0012] In addition to any combination of features described above, the method may also include setting, with the client device, a maximum window credit value for each server device as a multiple of the BDP of the communication path over which the respective server device communicates with the client device. The method may also include setting, with the client device, a minimum window credit value for each server device to be equal to the BDP of the communication path over which the respective server device communicates with the client device. Determining the window credit value for each server device based on the BDPs of the communication paths over which the server devices communicate with the client device and in response to determining that the difference between the size of the file and the total amount of window credit values that have been transmitted to all of the plurality of server devices is greater than the predetermined threshold may include determining, with the client device, the window credit value for each server device such that the window credit value for each server device is in a range between and including its respective minimum window credit value and its respective maximum window credit value.

[0013] In addition to any combination of features described above, determining the window credit value for each server device based on the BDPs of the communication paths over which the server devices communicate with the client device may include determining a first window credit value for a first server device and a second window credit value for a second server device, wherein the first window credit value and the second window credit value are set to different values.

[0014] In addition to any combination of features described above, the method may also include determining, with the client device and while the difference between the size of the file and the total amount of window credit values that have been transmitted to all of the plurality of server devices is greater than the predetermined threshold, an adjusted window credit value for at least one of the server devices based on a change of the BDP of the communication path over which the at least one of the server devices communicates with the client device.D24017W001

[0015] In addition to any combination of features described above, determining the adjusted window credit value for the at least one of the server devices may include determining, with the client device, the adjusted window credit value for the at least one of the server devices such that the adjusted window credit value is proportionate to the change of the BDP of the at least one of the server devices.

[0016] In addition to any combination of features described above, determining the bandwidth of each communication path includes: determining an inter-chunk arrival (ICA) time of each chunk of data received, and discarding one or more chunks of data from a bandwidth determination in response to determining that the ICA time for the one or more chunks of data is greater than a predetermined threshold. Determining the bandwidth of each communication path may also include determining, with the client device, an amount of time elapsed to receive a specified amount of data from the respective server device associated with each communication path. Determining the bandwidth of each communication path may also include determining the bandwidth of each communication path by dividing the specified amount of data by the amount of time elapsed to receive the specified amount of data. The specified amount of data may exclude the discarded one or more chunks of data, and the time elapsed to receive the specified amount of data may exclude time associated with receipt of the discarded one or more chunks of data.

[0017] In addition to any combination of features described above, the method may further include determining, with the client device, that the difference between the size of the file and the total amount of window credit values that have been transmitted to all of the plurality of server devices is less than the predetermined threshold. The method may further include implementing, by the client device and in response to determining that the difference between the size of the file and the total amount of window credit values that have been transmitted to all of the plurality of server devices is less than the predetermined threshold, a load balanced downloading approach where the client device determines an adjusted window credit value for at least one of the server devices.

[0018] In addition to any combination of features described above, implementing the load balanced downloading approach may include determining, with the client device, the adjusted window credit value for the at least one of the server devices based at least in part on the operating characteristic and the difference between the size of the file and the total amount of window credit values that have been transmitted to all of the plurality of server devices. Implementing the load balanced downloading approach may also include transmitting, with theD24017W001 client device, the adjusted window credit value to the at least one of the server devices. Implementing the load balanced downloading approach may also include receiving, with the client device, a second portion of the file from the at least one of the server devices. A size of the second portion of the file from each server device may be limited by the adjusted window credit value of the at least one of the server devices.

[0019] In addition to any combination of features described above, implementing the load balanced downloading approach may include determining a reduced window credit value for each of the server devices of the plurality of server devices. The reduced window credit value for each server device may be included in repeated transmissions from the client device to the respective server device until the total amount of window credit values that have been transmitted to all of the plurality of server devices is greater than or equal to the size of the file.

[0020] In addition to any combination of features described above, the method may also include determining, with the client device, whether any one of the plurality of communication paths has become idle due to the client device downloading all data to which its respective server device transmitted in accordance with the respective adjusted window credit value. The method may also include determining, with the client device, whether an amount of the file that has been received by the client device on all of the plurality of communication paths is greater than or equal to the size of the file. The method may also include, in response to determining that one of the plurality of communication paths has become idle and that the amount of the file that has been received by the client device on all of the plurality of communication paths is less than the size of the file, transmitting, with the client device, the adjusted window credit value or a further adjusted window credit value to a server device corresponding to the idle communication path in order to allow the server device corresponding to the idle communication path to transmit additional data of the file to the client device.

[0021] In addition to any combination of features described above, the method may also include waiting for a predetermined period of time after the idle communication path has become idle to determine whether the amount of the file that has been received by the client device on all of the plurality of communication paths is greater than or equal to the size of the file.

[0022] In addition to any combination of features described above, the method may also include determining, with the client device, an estimated downloading time for the data from each of the server devices based on the operating characteristic of the respective communicationD24017W001 path and the respective adjusted window credit transmitted to each server device. The method may also include determining, with the client device, a maximum estimated downloading time as a highest estimated downloading time from among the estimated downloading times of the server devices. The method may also include determining, with the client device, a minimum estimated downloading time as a lowest estimated downloading time from among the estimated downloading times of the server devices. The method may also include determining, with the client device, whether a difference between the maximum estimated downloading time and the minimum estimated downloading time is greater than a predetermined downloading time difference threshold. The method may also include in response to determining that the difference between the maximum estimated downloading time and the minimum estimated downloading time is greater than the predetermined downloading time difference threshold, transmitting, with the client device, the adjusted window credit value or a further adjusted window credit value to a server device corresponding to the communication path with the lowest estimated downloading time in order to allow the server device corresponding to the communication path with the lowest estimated downloading time to transmit additional data of the file to the client device.

[0023] In addition to any combination of features described above, the further adjusted window credit value may correspond to an amount of the file remaining to be received by the client device or may correspond to a predetermined maximum amount of excess egress data allowed by the client device.

[0024] In addition to any combination of features described above, the predetermined threshold may corresponds to a sum of the maximum window credit values of the plurality of server devices.

[0025] In addition to any combination of features described above, the client device may communicate with the server devices using HyperText Transfer Protocol-2 (HTTP-2), and the respective window credit value may be transmitted by the client device to each of the server devices as a WINDOW_UPDATE frame parameter included as a flow control mechanism that is native to HTTP-2.

[0026] In addition to any combination of features described above, the client device may communicate with the server devices using Quick User Datagram Protocol (UDP) Internet Connections (QUIC), and the respective window credit value may be transmitted by the clientD24017W001 device to each of the server devices as a MAX_STREAM_DATA or MAX_DATA frame parameter included as a flow control mechanism that is native to QUIC.

[0027] In addition to any combination of features described above, a sum of the window credit values transmitted to all of the server devices may be equal to the size of the file such that none of the server devices transmit excess egress data to the client device.

[0028] In addition to any combination of features described above, a sum of the window credit values transmitted to all of the server devices may be less than or equal to a predetermined excess egress percentage of the size of the file such that the server devices transmit a limited amount of excess egress data to the client device. The excess egress data less than or equal to the predetermined excess egress percentage of the size of the file.

[0029] In another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing instructions which, when executed by a computing apparatus, cause the computing apparatus to perform the method and any of the combination of features described above.

[0030] In another aspect of the present disclosure, there is provided a client device that includes a computing apparatus, including an electronic processor, and a memory storing instructions, which when executed by the electronic processor, cause the computing apparatus to perform the method and any of the combination of features described above.

[0031] Other aspects of the embodiments will become apparent by consideration of the detailed description and accompanying drawings.BRIEF DESCRIPTION OF THE DRAWINGS

[0032] FIG. 1 is a diagram that illustrates an example multipath downloading process of a client device, in accordance with some embodiments.

[0033] FIG. 2 is a diagram that illustrates an example multipath downloading process of a client device and shows an example indication of excess egress data sent by each server device, in accordance with some embodiments.

[0034] FIG. 3 is a hardware block diagram of a client device, in accordance with some embodiments.D24017W001

[0035] FIG. 4 is a flowchart of a method for minimizing excess egress data in a communication session that is performed by the client device of FIG. 3, in accordance with some embodiments.

[0036] FIG. 5 is a graph showing the relationship between accumulated file size and download time during a controlled downloading experiment, in accordance with some embodiments.

[0037] FIG. 6 is a graph showing the relationship between throughput and download time during the controlled downloading experiment and shows an example of the calculated throughput of the downloading session of FIG. 5 each time a HyperText Transfer Protocol (HTTP) Data Frame is received, in accordance with some embodiments.

[0038] FIG. 7 is a graph showing the relationship between throughput and download time during the controlled downloading experiment and shows that the calculated throughput will gradually reach a steady state over time, in accordance with some embodiments.

[0039] FIG. 8 is a graph showing inter-chunk arrival (ICA) time values of data received by the client device over time during the controlled downloading experiment, in accordance with some embodiments.

[0040] FIG. 9 is a graph showing throughput and bandwidth calculations with respect to download time during the controlled downloading experiment, in accordance with some embodiments.

[0041] FIGS. 10A and 10B illustrate example use cases of transmitting data from a server device to a client device with different HTTP / 2 window sizes relative to Transmission Control Protocol (TCP) window sizes, in accordance with some embodiments.

[0042] FIG. 11 is a diagram that illustrates an example multipath downloading process of the client device of FIG. 3 while implementing a load balanced download process, in accordance with some embodiments.

[0043] FIG. 12 illustrates an example use case of the load balanced download process with three communication paths, in accordance with some embodiments.

[0044] FIG. 13 is a graph of accumulated file size versus download time for multiple server devices in an example use case, in accordance with some embodiments.D24017W001

[0045] FIGS. 14A-14B illustrates pseudocode of an algorithm that is performed by the client device of FIG. 3 while downloading a file / object, in accordance with some embodiments.DETAILED DESCRIPTION

[0046] The following description sets forth exemplary methods, parameters, and the like. It should be recognized, however, that such description is not intended as a limitation on the scope of the present disclosure but is instead provided as a description of example embodiments.

[0047] Although the following description uses terms “first,” “second,” etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, a first touch could be termed a second touch, and, similarly, a second touch could be termed a first touch, without departing from the scope of the various described embodiments. The first touch and the second touch are both touches, but they are not the same touch.

[0048] The terminology used in the description of the various described embodiments herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various described embodiments and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms “includes,” “including,” “comprises,” and / or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0049] The term “if’ is, optionally, construed to mean “when” or “upon” or “in response to determining” or “in response to detecting,” depending on the context. Similarly, the phrase “if it is determined” or “if [a stated condition or event] is detected” is, optionally, construed to mean “upon determining” or “in response to determining” or “upon detecting [the stated condition or event]” or “in response to detecting [the stated condition or event],” depending on the context. The term “based on” is to be read as “based at least in part on.” The term “one example implementation” and “an example implementation” are to be read as “at least one example implementation.” The term “another implementation” is to be read as “at least one other implementation.” The terms “determined,” “determines,” or “determining” are to be read asD24017W001 obtaining, receiving, computing, calculating, estimating, predicting, or deriving. In addition, in the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skills in the art to which this disclosure belongs.

[0050] FIG. 3 is a hardware block diagram of the client device 102 according to some example embodiments. In some instances, the client device 102 includes a media player. The client device 102 may be a playback device configured to output audio and / or video media content to a user (either natively or via hardware capture devices communicatively coupled to the device). Example client devices 102 include, but are not limited to, televisions, smartphones, computers, tablet personal computers, audio and / video rendering devices, and / or the like.

[0051] In the example illustrated in FIG. 3, the client device 102 includes an electronic processor 310 (for example, a microprocessor or other electronic device). The electronic processor 310 includes input and output interfaces (not shown) and is electrically coupled to a memory 315, a network interface 320, a speaker(s) 325 (e.g., one or more audio output devices), and a display screen 330. In some instances, the client device 102 includes fewer or additional components in configurations different from that illustrated in FIG. 3 and / or optionally combines two or more components shown in FIG. 3. For example, the client device 102 may include more than one speaker 125. As another example, the client device 105 may additionally include one or a combination of a microphone(s), a camera(s), and a sensor(s). As yet another example, the electronic processor 310 may include multiple electronic processors 310 within the client device 102 that together function to control various aspects of the client device 102. In some instances, the client device 102 is implemented within a distributed system including one or more components located in different locations. For example, in some embodiments, the client device 102 (e.g., the electronic processor 310) includes local hardware components and one or more remote hardware components (e.g., cloud-based components, hosted on public and / or private cloud infrastructure, Software as a Service (SaaS), Platform as a Service (PaaS), etc.). In other words, the electronic processor 310 may include any one or a combination of electronic processors located within a single device (e.g., the client device 102) or distributed among various devices and / or systems (e.g., the client device 102, a cloud-based device, etc.). For example, the electronic processor 310 may include a first electronic processor of the client device 102, a remote electronic processor configured to communicate with the client device 102, or both the first electronic processor and the remote electronic processor. Thus, in the claims, if an apparatus or system is claimed, for example, as including an electronic processor or otherD24017W001 element configured in a certain manner, for example, to make multiple determinations, the claim or claim element should be interpreted as meaning one or more electronic processors (or other element) where any one of the one or more electronic processors (or other element) is configured as claimed, for example, to make some or all of the multiple determinations. To reiterate, those electronic processors and processing may be distributed within a single device or across multiple devices.

[0052] In some instances, the client device 102 performs functionality other than the functionality described below. The various components of the client device 102 described herein are implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing and / or application- specific integrated circuits.

[0053] The memory 315 may include read only memory (ROM), random access memory (RAM), other non-transitory computer-readable media, or a combination thereof. The electronic processor 310 is configured to receive instructions and data from the first memory 315 and execute, among other things, the instructions. In particular, the electronic processor 310 executes instructions stored in the memory 315 to perform the methods described herein.

[0054] The network interface 320 sends data to and receives data from other devices (e.g., other client devices 102 such as smart phones, cloud-based devices, server devices 110, etc.). In some instances, the network interface 320 includes one or more transceivers for wirelessly communicating with the other devices. Alternatively or in addition, the network interface 320 may include a connector or port for receiving a wired connection to one or more of the other devices, such as connector to receive a coaxial cable. The electronic processor 310 may receive media data (for example, audio data, video / image data, and / or the like) from one or more server devices 110 via the network interface 320. The electronic processor 310 may output the media data via the speaker 325, the display screen 330, or a combination thereof. In some instances, the speaker 325 may include a plurality of speakers (e.g., multiple built-in speakers and / or multiple external speakers communicatively coupled to the client device 102).

[0055] The display screen 330 is configured to display images, video, text, and / or data to the user. The display screen 330 may be a liquid crystal display (LCD) screen, an organic light emitting display (OLED) display screen, a waveguide display, a quantum dot display, or the like. In some embodiments, a touch sensitive input interface may be incorporated into the display screen 330 as well, allowing the user to interact with content provided on the display screen 330. In some embodiments, the display screen 330 includes future-developed display technologies.D24017W001In some embodiments, the speaker 325 and the display screen 330 are referred to as output devices that output media data (e.g., image / video and audio) and other information to a user of the client device 102 as explained herein. The display screen 330 may be referred to as a user interface that may act as an input device (e.g., to receive user inputs) and / or an output device (e.g., to provide media data to the user). In some instances, the client device 102 optionally additionally or alternatively includes one or more physical user-interface devices, such as a physical keyboard, a mouse, and / or a joystick and typically supports a variety of applications.

[0056] Attention is now directed towards embodiments of techniques for minimizing (e.g., reducing or eliminating) excess egress data in a communication session that are optionally implemented on, for example, the client device 102 described previously herein. As described previously herein, there is a technological problem with the downloading process on client devices during coded, multipath downloading of data. The techniques, systems, and devices described herein address this technological problem by minimizing excess egress data while minimally affecting throughput or by not affecting throughput at all, and, in some instances, may actually improve throughput (e.g., compared to a single communication path downloading approach). In some instances, the techniques, systems, and devices described herein leverage features of existing communication protocols (e.g., HTTP / 2, HTTP / 3 over QUIC, etc.) to reduce or eliminate excess egress data using the client device 102 to limit the amount of data that is transmitted to the client device 102 by each server device 110 during a multipath downloading process.

[0057] In some instances, a WINDOW_UPDATE frame that is native to the HTTP / 2 specification may be used by the client device 102 to limit the amount of data server devices 110 can send to the client device 102. The HTTP / 2 specification indicates that the client device 102 can perform its own flow control algorithm. Additionally, the HTTP / 2 specification does not restrict the form of flow control implemented. The HTTP / 2 specification also specifies that the server device 110 must respect the “window size” defined by the client device 102 and cannot send more data that exceeds the “window size” defined by the client device 102. As described herein, these properties of HTTP / 2 can be used to make sure the server devices 110 do not send more data than needed by the client device 102, and can be used to enable multipath downloads that maximize the throughput (or minimally affect throughput) while minimizing (e.g., reducing or eliminating) excess egress data from be sent by the server devices 110.

[0058] For example, the WINDOW_UPDATE frame sent by the client device 102 to a server device 110 indicates an additional amount of data that is to be sent by the server deviceD24017W001110 to the client device 102. A first WINDOW_UPDATE frame may provide the server device with a 64 kilobyte (KB) window to allow the server device to transmit 64 KB of data to the client device 102. A second WINDOW_UPDATE frame adds onto the initial 64 KB window by the amount specified in the second WINDOW_UPDATE frame. For example, a second WINDOW_UPDATE frame of 100 KB allows the server device 110 to transmit 100 KB of additional data and 164 KB total (i.e., a total window size of 164 KB). A third WINDOW_UPDATE frame of 20 KB allows the server device 110 to transmit 20 KB of further additional data and 184 KB total. Thus, the overall window credit for each server device 110 can only be increased, but it can be increased by any amount in accordance with the WINDOW_UPDATE frame. In some instances, the total window size may indicate a total payload data size without including minor overhead data that is included, for example, in packet headers. The value included in the WINDOW_UPDATE frame (e.g., a WINDOW_UPDATE frame parameter) that indicates the amount of data to be transmitted to the client device 102 may be referred to as a window credit value or window credit. For example, the values 64 KB, 100 KB, and 20 KB in the above example are window credit values. The sum of the WINDOW_UPDATE frame parameters (e.g., a total amount of window credit values transmitted to the server device 110) equals the total window size for the server device 110 (e.g., the total amount of data that the server device 110 is able to transmit to the client device 102).

[0059] While use of the WINDOW_UPDATE frame of HTTP / 2 (or similar client-side window update functionality) can make sure the server devices 110 do not send more data than necessary for the client device 102 to decode the original file, the WINDOW_UPDATE frame should be precisely controlled by the client device 102 to maximize the throughput (or minimally affect throughput) while minimizing (e.g., reducing or eliminating) excess egress data. Specifically, the client device 102 is programmed to dynamically determine (i) when the WINDOW_UPDATE frame is transmitted to each server device 110 and (ii) the size of the WINDOW_UPDATE frame that should be transmitted. The client device 102 may be programmed to make these determinations in a manner that keeps the server devices 110 busy (e.g., properly timing transmission of the WINDOW_UPDATE frame and fully utilizing available bandwidth on a respective communication path 115 of each server device 110) by avoiding idle time for the server devices 110. In some instances, the client device 102 is programmed to make these determinations to optimally load balance the downloading process based on the throughput of each communication path 115. The client device 102 may be configured to make the above-noted determinations in combination with using existing network protocols or network infrastructures (e.g., without modification to HTTP or TCP protocols) andD24017W001 may run completely on top of the application layer. Thus, the techniques described herein are plug-and-play and work for devices that can run for example, HTTP / 2. Implementation of the techniques described herein put adequate pressure on the TCP layer to ensure that the TCP layer stays busy and will not be bottle-necked by the HTTP / 2 layer’s flow control algorithm. In certain selectable modes (e.g., in a performance mode), the techniques described herein allow for sacrificing excess egress data to increase the throughput of downloading. In some instances, the techniques described herein utilize HTTP / 2 frame measurements to perform estimation on the bandwidth of communication paths 115, thus inferring the TCP layer’s behavior. In some instances, the techniques described herein allow the client device 102 to send multiple byte range requests to download a file if the file is being downloaded from a CDN having a load balancer in-place. When CDN servers have a load-balancer in place, using byte-range requests can further lower excess egress data reported by the CDN servers.

[0060] For sake of simplicity, the techniques described herein are described with respect to utilizing HTTP / 2. However, it should be understood that other implementations consistent with these techniques may be possible with other existing or future protocols including similar functionality. For example, the client device 102 may communicate with the server devices 110 using Quick User Datagram Protocol (UDP) Internet Connections (QUIC), and the respective window credit values may be transmitted by the client device 102 to each of the server devices 110 as a MAX_STREAM_DATA or MAX_DATA frame parameter included as a flow control mechanism that is native to QUIC.

[0061] FIG. 4 illustrates a flowchart of a method 400 for minimizing excess egress data in a communication session. The method 400 is described as being performed by client device 102 (e.g., by the electronic processor 310). While a particular order of processing steps, message receptions, and / or message transmissions is indicated in FIG. 4 as an example, timing and ordering of such steps, receptions, and transmissions may vary where appropriate without negating the purpose and advantages of the examples set forth in detail throughout the remainder of this disclosure.

[0062] At block 405, the method 400 begins when the client device 102 transmits a request for a file / object to a plurality of server devices 110 (e.g., for multipath delivery of coded packets including data corresponding to the file). In some instances, the client device 102 performs block 405 in response to a user input selecting a video (e.g., a file / object) to be streamed for output on the client device 102. In some instances, the requests for the file include socket connection commands or are sent in conjunction with socket connection commands that are usedD24017W001 to set up socket connections with each of the server devices 110. In some instances, the client device 102 waits and determines whether any of the socket connections have been established. In response to a first socket connection being established, the client device 102 sends an HTTP “get request” to the first socket connection that has been established. The requests for the file are sent to multiple server devices 110 to allow the CDNs to race to see which server device 110 responds the quickest to the HTTP “get request.” In other words, the client device 102 can receive a response very quickly since all of the server devices 110 are individually attempting to respond as quickly as possible. This racing by the CDNs (a) improves robustness in case some server devices 110 do not respond and (b) gets a response header to the client device 102 as quickly as possible in order to allow the client device 102 to obtain / determine a size of the file.

[0063] When the client device 102 does not receive a response to any of the HTTP “get requests,” the client device 102 may transmit one or more additional HTTP “get requests” on new socket connections as well until a response is received. Once any one of the server devices 110 has responded to the HTTP “get request” and the client device 102 receives the response, the size of the file is determined from header information included in the response. In some instances, the size of the file referred to herein is the size of the payload of the file and may not include additional size caused by, for example, header information that is not part of the payload. In some instances the size of the file corresponds to minimum amount of the file needed for decoding a coded file to determine original file content. Examples of transmitting requests for a file are included in the example use cases at the end of the detailed description. In some instances, once the size of the file (e.g., file size) is determined and the socket connections are established, the method 400 proceeds to block 410.

[0064] At block 410, the client device 102 determines an operating characteristic of each communication path 115 of the plurality of communication paths 115 between the client device 102 and respective server devices 110 of the plurality of server devices 110. The operating characteristic may include one or more operating characteristics such as throughput, bandwidth, round trip time (RTT), a bandwidth delay product (BDP), etc., and combinations thereof as described in greater detail herein. The operating characteristic(s) of each communication path is determined to allow the client device 102 to request certain amounts of the file from each server device 110 in accordance with one or more operating characteristics as described herein. Accordingly, the client device 102 may attempt to reduce or eliminate excess egress data while minimally affecting throughput, for example, by requesting data from each server device 110D24017W001 based on their respective operating characteristics in a manner that attempts to have all received data from each communication path 115 finish downloading at the same time, in some instances.

[0065] In some instances, the electronic processor 310 of the client device 102 implements a throughput and bandwidth calculator at block 410. The calculated throughput and bandwidth values may be used to indicate the network condition that the client device 102 experiences on each communication path 115. Thus, the throughput and bandwidth values may be used by a multipath flow controller implemented by the electronic processor 310 to determine how much of the file to request from each of the server devices 110 at any given time.

[0066] To illustrate how the throughput and bandwidth estimator works, consider a controlled downloading experiment shown in FIG. 5. In FIG. 5, the amount of data downloaded on one communication path 115 has controlled network conditions: a fixed delay of 100ms and fixed bandwidth of 30Mbit / s (3.75Mbyte / s). The throughput on communication path i, in accordance with some embodiments, may be calculated using Equation 1 below.F ' Equation 1: Thpt= — ti

[0067] where Ftis the accumulated file size that the communication path i has downloaded by the time tt, and ttis the difference between the current and request times. Equation 1 works for a single file. In the case of multiple back-to-back requests for more than one file, the accumulated throughput can be calculated using Equation 2 below.Equation 2:

[0068] In Equation 2, the throughput is calculated using the sum of files on this communication path z divided by the time it took to download those files. In some instances, a sliding window technique is used to calculate the throughput based on most recent measurements of network conditions. FIG. 6 shows an example of the calculated throughput of the downloading session of FIG. 5 each time an HTTP Data Frame is received. If the client device 102 keeps downloading the file at the controlled network conditions, the calculated throughput will gradually reach a steady state as shown in FIG. 7.

[0069] In some instances, the electronic processor 310 of the client device 102 implements a bandwidth calculator / estimator to estimate the capacity of the network (e.g., the capacity of each communication path 115). In FIG. 5, each dot on the graph represents a HTTP Data Frame thatD24017W001 has been received by the client device 102. The bandwidth of a communication path 115, which is indicated by the maximum HTTP Data Frame receiving rate of the communication path 115, may be calculated by determining the slope of the graph of FIG. 5. To calculate the average slope of the graph of FIG. 5, the horizontal gap between each batch of HTTP Frame is first removed. This removal is achieved by determining the inter-chunk- arrival time (ICA) between consecutively received HTTP Data Frames as shown in FIG. 8. As shown in FIG. 8, when there is a high value of ICA it indicates that (i) a new batch of HTTP Data Frames will arrive next or (ii) there is a sudden change of network condition (e.g., loss of packets, head of line blocking of TCP packets on the TCP layer, etc.). Thus, by removing the high (i.e., abnormal) ICA values and only using common ICA values (e.g., that are within one or two standard deviations of each other), the slope shown in FIG. 5 may be calculated more accurately for the purposes of determining bandwidth of the communication path 115.

[0070] In other words, the client device 102 may determine the bandwidth of each communication path by determining an inter-chunk arrival (ICA) time of each chunk of data received on the respective communication path, and discarding one or more chunks of data from a bandwidth determination in response to determining that the ICA time for the one or more chunks of data is greater than a predetermined threshold (e.g., abnormal as indicated by the ICA times being more than one or two standard deviations away from other ICA times of the respective communication path 115). The client device 102 may determine an amount of time elapsed to receive a specified amount of data from the respective server device 110 associated with each communication path 115, and determine the bandwidth of each communication path 115 by dividing the specified amount of data by the amount of time elapsed to receive the specified amount of data. In some instances, the specified amount of data excludes the discarded one or more chunks of data, and the time elapsed to receive the specified amount of data excludes time associated with receipt of the discarded one or more chunks of data. More specific details of this bandwidth calculation / estimation are included below.

[0071] The throughput shown in Equation 1 can be represented by the sum of ICA values using Equation 3 below where ICAanrepresents the abnormal or high ICA value, and lCAnrepresents the normal ICA value.Equation 3:

[0072] When implementing the bandwidth calculator, the client device 102 clusters the ICA value into two types: normal and abnormal by using a moving window technique. The clientD24017W001 device 102 also may be configured to analyze past ICA values in a given window size and then determine the normal range for a next ICA value. In some instances, the normal range for the next ICA value is determined using Equation 4 below.Equation 4:

[0073] If the next ICA value is larger than the predetermined threshold defined in Equation 4, then the next ICA value is classified as abnormal. An example graph of classification results for ICA values is illustrated in FIG. 8 that shows an inter-chunk arrival plot of the downloading session shown in FIG. 5.

[0074] In some instances, once ICA values are grouped into normal and abnormal characterizations, the client device 102 may calculate the bandwidth for the communication path 115 using Equation 5 below.Equation 5:

[0075] In Equation 5, wtis used to gradually reduce the contribution of abnormal ICA values. In some instances, the value of wtis determined by using Equation 6 below where t is the current time, ttis when the ICA time out value is calculated, rtt is the estimated round trip time value, and m is a user-defined tuning parameter / constant.Equation 6:

[0076] By using Equation 5, the effect of the time-out value will diminish as time passes, and the bandwidth calculated by the bandwidth calculator will converge to the actual bandwidth of the communication path 115 much quicker than the using just the throughput value.Accordingly, an accurate bandwidth of each communication path 115 is determined more quickly using the above-noted calculations than using other methods that involve using the determined throughput without discarding abnormal ICA values. For example, BWt> Thptas shown in FIG. 9 that shows the calculated bandwidth and throughput values of the example downloading session shown in FIG. 5. In other words, as shown in FIG. 9, the bandwidth value is larger than the throughput value, and the bandwidth value converges to the true bandwidth of the communication path 115 at a much quicker rate. In some instances, the quickness and accuracy of the bandwidth calculation is important for sizing the window credit value sent toD24017W001 each server device 110 properly when the client device 102 requests data from the server devices 110 as explained herein.

[0077] In some instances, the electronic processor 310 of the client device 102 implements a round-trip time (RTT) estimator at block 410 (e.g., in addition to the throughput and bandwidth calculator). To estimate the RTT value, the client device 102 may calculate a current RTT value observed and save it to a list in the memory 315. The client device 102 may then uses this list of historical RTT value(s) to estimate the next RTT value. In some instances, the client device 102 calculates the current RTT value by calculating a difference between (i) a time a request was sent to a server device 110 by the client device 102 and (ii) a time-to-first byte that corresponds to a time at which the client device 102 receives a first byte of a response from the server device 110. In some instances, the client device 102 calculates the current RTT value by utilizing the functionality of a Ping Frame in HTTP / 2 and sending a Ping Frame to the server device 110. In such instances, the client device 102 calculates the RTT value as the difference of Ping Ack Receive time minus Ping Frame sent time. When a new RTT value is available, the client device 102 saves the RTT value to the list in the memory 315.

[0078] In some instances, the client device 102 provides estimates of RTT using one or more of multiple approaches including, but not limited to, the following described approaches. One example approach includes the client device 102 estimating the RTT value for a communication path 115 by taking the average value of historical RTT values on the communication path 115. Another example approach includes the client device 102 using the first ever RTT value for the communication path 115 as the estimated RTT value for the communication path 115. Another example approach includes using the most recent RTT value for the communication path 115 as the estimated RTT value for the communication path 115. Yet another example approach includes the client device 102 estimating the RTT value for the communication path by first discarding one or more extreme (e.g., most outlying) RTT value(s) from the list and then averaging the remaining RTT values. This final example approach may be useful for providing a smoothed RTT value. Other (e.g., more advanced) techniques may additionally or alternatively be used to estimate the RTT value for a respective communication path 115.

[0079] At block 415, the client device 102 determines a window credit value for each server device 110 based at least in part on one or a combination of the respective operating characteristic of each communication path 115, a size of the file, an amount of the file that has been received by the client device 102 on all of the plurality of communication paths, a bytes-in- flight (BIF) value corresponding to data that has been requested by the client device 102 but hasD24017W001 not yet been received by the client device 102 (e.g., on each communication path 115), a historic value of window credits sent by the client device 102 to one or more server devices 110, a total window credit given to all server devices 110, and / or the like. The client device 102 is configured to take one or more of the above-noted values into account to determine flow control and load balancing for requesting data of the file (e.g., portions of the file) from the plurality of server devices 110. In some instances, the client device 102 implements a flow controller (e.g., a multipath flow controller) for flow control and load balancing.

[0080] In some instances, the client device 102 has numerous goals when implementing the flow controller. One goal of the flow controller may be to ensure that each server device 110 is busy and not idle. In other words, one goal may be to ensure that data is continuously or almost continuously being transmitted on each communication path 115 from each respective server device 110. This first goal can be achieved by sending a large enough window credit value (e.g., WINDOW_UPDATE frame of the HTTP / 2 specification) to each server device 110 in order to keep each server 110 and its respective communication path 115 busy sending data to the client device 102. A second goal of the flow controller may be to ensure that each communication path 115 is being used to communicate a certain amount of data (e.g., an ideal amount of data) according to its operating characteristic(s) (e.g., bandwidth / throughput, RTT, etc.). A third goal of the flow controller may be to avoid sending too large of a window credit value to any of the server devices 110 because the network conditions (e.g., operating characteristic(s) of each communication path 115) may be highly variable, which means that the speed of downloading data on each communication path 115 may change (e.g., significantly increase or decrease) in the future.

[0081] FIGS. 10A and 10B illustrate example use cases related to the first goal of keeping the server devices 110 busy. In FIG. 10A, the HTTP window size assigned to the server device 110 by the client device 102 is smaller than the TCP window size of the server device 110. Accordingly, the network bandwidth of the communication path 115 between the server device 110 and the client device 102 is not being fully utilized. On the other hand, in FIG. 10B, the HTTP window size assigned to the server device 110 by the client device 102 is larger than the TCP window size of the server device 110. Accordingly, the network bandwidth of the communication channel between the server device 110 and the client device 102 is being fully utilized in the example of FIG. 10B. In summary, as long as the HTTP window size assigned to the server device 110 by the client device 102 is larger than the TCP window size of the server device 110, then the server device 110 will be kept busy and will not be idle.D24017W001

[0082] The first goal to keep the server devices 110 busy and the third goal to avoid sending too large of a window credit value to any of the server devices 110 may conflict with each other in some instances. For example, to ensure that a server device 110 remains busy, the client device 102 must transmit a large enough window credit value (e.g., the larger the better in order to accomplish the first goal). However, when the client device 102 sends too large of a window credit value, it could negatively impact load balancing between server devices 110 and communication paths 115, for example, if the network conditions / operating characteristics are changing rapidly.

[0083] To address this conflict, the client device 102 may take advantage of characteristics of protocols that are used to communicate data from the server device 110 to the client device 102. For example, HTTP / 2 runs on top of the TCP layer, and the maximum amount of data the TCP layer can transmit is determined by the congestion window of the TCP layer. Thus, by ensuring that the HTTP layer has a larger window size than the TCP layer, the first goal may be achieved automatically as explained previously herein with respect to FIGS. 10A and 10B.

[0084] However, the client device 102 may not have direct access to the TCP layer and may not know the congestion window size of the TCP layer. Thus, the client device 102 may be configured to estimate the congestion window size of the TCP layer of each server device 110 / communication path 115. For example, the client device 102 may estimate the congestion window size of the TCP layer using the bandwidth of a communication path 115 multiplied with the RTT value of the communication path 115. The result of this calculation may also be referred to as a bandwidth delay product (BDP) of the communication path 115 (see Equation 7 below). In other words, in some instances, the client device 102 may be configured to determine a bandwidth of each communication path 115 and a RTT of each communication path 115, and determine a BDP of each communication path 115 by determining a product of the bandwidth and the RTT of each communication path 115. In some instances, the product of the bandwidth and the RTT indicates how much data the server device 110 can send to the client device 102 in one RTT on a respective communication path 115 of the server device 110. In some instances, the calculation of the BDP may occur at block 410 instead of at block 415.Equation 7: BDP = bandwidth*RTT

[0085] In some instances, the BDP of each communication path 115 and / or a multiple thereof may be used to establish a minimum window credit value and a maximum window credit value that may be transmitted to each server device 110 by the client device 102 as describedD24017W001 herein. For example, to ensure that the client device 102 does not over-request data on any of the communication paths 115 during the download session, a maximum window credit value for each respective server device 110 / communication path 115 may be set to alpha*bandwidth*RTT (e.g., alpha*BDP), where alpha is a constant and is tunable / user selectable. In some instances, alpha is set to 2.0, 2.25, 1.75, or the like. Continuing this example, to ensure that each server device 110 / communication path 115 is busy and not idle, a minimum window credit value for each respective server device 110 / communication path 115 may be set to the BDP or beta*BDP, where beta is a constant and is tunable / user selectable but that is set to be 1.0 or higher and less than alpha.

[0086] In some instances, the client device 102 is configured to determine the amount of the file that has been received by the client device 102 based on the portions of the file received from each server device 110. In some instances, the client device 102 is configured to determine the BIF value by subtracting the amount of the file that has been received by the client device 102 from a total amount of window credit values that have been transmitted to all of the plurality of server devices 110 (e.g., a total amount of window credit values previously transmitted to the server devices 110 by the client device 102). In some instances, the client device 102 is configured to determine the BIF value for each communication path by subtracting the amount of the file that has been received by the client device on each communication path from a total amount of window credit values that have been transmitted to the respective server device corresponding to each communication path (e.g., BIF value determined for each respective server device 110 / communication path 115).

[0087] In some instances, the client device 102 determines the window credit value for each server device 110 differently depending on how far along in the download process the client device 102 is for a given file. For example, at the begging or in the middle of a download process for a large file, the client device 102 may determine the window credit value for each server device 110 based solely on the BDPs of the communication paths 115 over which the server devices 110 communicate with the client device 102. More specifically, in some instances, the client device 102 determines whether a difference between the size of the file and the total amount of window credit values that have been transmitted to all of the plurality of server devices 110 is greater than a predetermined threshold. For example, the predetermined threshold may be a sum of the maximum window credit values (e.g., maximum window credit value = alpha*BDP) of the plurality of server devices 110 (i.e., for all communication paths 115). Such a determination indicates that there is still a significant amount of data of the file toD24017W001 be requested relative to the BDPs of the communication paths 115. Accordingly, a more precise load balanced downloading approach may not be entered yet since a significant amount of data still must be requested (i.e., given window credit to) in order to complete the download process. Rather, in response to determining that the difference between the size of the file and the total amount of window credit values that have been transmitted to all of the plurality of server devices 110 is greater than the predetermined threshold, the client device 102 may determine the window credit value for each server device 110 based on the BDPs of the communication paths 115 over which the server devices 110 communicate with the client device 102.

[0088] In some instances, the client device 102 is configured to determine the window credit value for each server device 110 based on the BDPs of the communication paths 115 by setting a maximum window credit value (e.g., alpha*BDP) for each server device 110 as a multiple (e.g., a multiple of greater than 1.0 such as 1.75, 2.0, 2.25, or the like) of the BDP of the communication path 115 over which the respective server device 110 communicates with the client device, and by setting a minimum window credit value for each server device 110 to be equal to the BDP of the communication path 115 over which the respective server device 110 communicates with the client device 102. The maximum and minimum window credit values may define a range of potential window credit values from within which the client device 102 can select to be sent to each server device 110 when a significant amount of data of the file still needs to be transmitted to the client device 102. In other words, the client device 102 may be configured to determine the window credit value for each server device 110 such that the window credit value for each server device 110 is in a range between and including its respective minimum window credit value and its respective maximum window credit value.

[0089] In some instances, the client device 102 may determine a first window credit value for a first server device 110 and a second window credit value for a second server device 110, where the first window credit value and the second window credit value are set to different values. For example, because the BDPs of different communication paths 115 associated with respective server devices 110 may be different, the window credit values for different server devices 110 may be different. As a more specific example, if the first server device 110 has a higher BDP at a given time compared to the BDP of the second server device 110, then the client device 102 may transmit a larger window credit value to the first server device 110 than to the second server device 110 (e.g., until the network conditions such as BDPs of communication paths 115 change).D24017W001

[0090] In some instances, the client device 102 is configured to determine, while the difference between the size of the file and the total amount of window credit values that have been transmitted to all of the plurality of server devices 110 is greater than the predetermined threshold, an adjusted window credit value for at least one of the server devices 110 based on a change of the BDP of the communication path 115 over which the at least one of the server devices 110 communicates with the client device 102. In other words, the client device 102 may be configured to dynamically adjust a window credit value that is sent to the server device 110 for the next chunk / portion of data based on the BDP of the communication path 115 of the server device 110 changing since the last time it was evaluated. In some instances, the client device 102 may determine the adjusted window credit value such that the adjusted window credit value is proportionate to the change of the BDP of the at least one of the server devices 110. For example, a 20% decrease in the BDP of the communication path 115 of the server device 110 may result in the client device 102 reducing the next window credit value sent to the server device 110 by 20% compared to the previous window credit value that was sent to the server device 110. In general, while the difference between the size of the file and the total amount of window credit values that have been transmitted to all of the plurality of server devices 110 is greater than the predetermined threshold, the client device 102 is configured to adjust the respective minimum window credit value, the respective maximum window credit value, and / or the respective window credit value of each request to each server device 110 in accordance with changes of the BDP of the communication path 115 corresponding to the respective server device 110. One example algorithm for determining a respective window credit value to transmit to each server device 110 in accordance with the above-noted general principles is shown in pseudocode in FIGS. 14A-14B as Algorithm 1. In some instances, Algorithm 1 is performed by the flow controller implemented by the client device 102 while downloading a first file and / or subsequent files. In some instances, other algorithms may be used.

[0091] With reference to Algorithm 1, for a new connection, in some instances, the initial maximum value of the server- side window size may be defined by Equation 8 below.Equation 8: WNDmax i= 64 X S

[0092] In Equation 8, the default values of S in the first two instances of downloaded data chunks may be (2, 4). These values may be used as an initial maximum window size because, for a new connection between a server device 110 and the client device 102, the TCP will be at a slow start phase. Thus, the TCP window size will grow exponentially as increasedD24017W001 communication between the server device 110 and the client device 102 occurs. To make sure there is sufficient back pressure for the TCP window, the next two RTTs worth of data may be requested by increasing the window size (e.g., using 2, 4 as the default value of S). For example, after receiving an initial 64KB of data on the TCP layer, the TCP congestion window will increase by a factor of 2 (2*64KB) (e.g., an additive increase). After receiving the second batch / chunk of data on the TCP layer, the TCP congestion window will become 4*64KB. Thus, using 2 and 4 as the default values of S during the receipt of the first few data chunks as noted above sets the upper limit of window credit that can be sent to the server device as 2*RTT worth of data on the TCP layer (in the initial phase, where the socket connection is first established and TCP connection still in slow start phase). In some instances, the initial value of the a parameter is 1, the initial value of ft is 2, the initial value of f is 2.25, the initial value of y and 0 is 16, and the initial value of <p is 192. Other initial values may be used in other instances. For example, the default value of the parameters works well for most general network conditions but can also be tuned / adjusted (e.g., by a user) according to different network conditions. During the downloading process, the HTTP Window Size for a given server device 110 may increase exponentially and / or linearly depending on the measured BDP value of its corresponding communication path 115. In some instances, if a future BDP value decreases, a maximum allowed WINDOW UPDATE size may also decrease, but linearly in some instances. This linear decrease aims to ensure that the HTTP layer can fully back-pressure the TCP layer and fully utilize the network bandwidth.

[0093] As indicated previously herein, in some instances, the client device 102 determines the window credit value for each server device 110 differently depending on how far along in the download process the client device 102 is for a given file. For example, near the end of a download process for a file, the client device 102 may implement a load balanced downloading approach, for example, to minimize, reduce, or eliminate excess egress data. For example, to detect that a download process is nearly complete, the client device 102 may determine that the difference between the size of the file and the total amount of window credit values that have been transmitted to all of the plurality of server devices 110 is less than the predetermined threshold (e.g., a sum of the maximum window credit values of the plurality of server devices 110 for all communication paths 115 as explained previously herein). Such a determination indicates that there is only a small amount of data of the file to be requested relative to the BDPs of the communication paths 115. Accordingly, a more precise load balanced downloading approach may be implemented as explained herein. In response to determining that the difference between the size of the file and the total amount of window credit values that haveD24017W001 been transmitted to all of the plurality of server devices 110 is less than the predetermined threshold, the client device 102 may be configured to implement the load balanced downloading approach where the client device 102 determines an adjusted window credit value for at least one of the server devices 110. In some instances, the adjusted window credit value may be smaller than previous window credit values for the server device 110 because only a small portion of the file remains to be requested and received by the client device 102. In some instances, when implementing the load balanced downloading approach, the adjusted window credit value may be less than its respective minimum window credit value that was previously calculated based on the BDP of the communication path 115 of the respective server device 110 (e.g., to sacrifice throughput to minimize excess egress data).

[0094] In some instances, the client device 102 is configured to implement the load balanced downloading approach by determining the adjusted window credit value for the at least one of the server devices 110 based at least in part on the operating characteristic and the difference between the size of the file and the total amount of window credit values that have been transmitted to all of the plurality of server devices 110 (e.g., an amount of data of the file remaining to be requested (i.e., given window credit to)). In some instances, the load balanced downloading approach is implemented with a goal of making the downloading of data / portions of the file on all communication paths 115 finish at the same time while also avoiding or having a limited amount of excess egress data. For example, the client device 102 may divide up the amount of data of the file remaining to be requested proportionately among the server devices 110 / communication paths 115 based on the BDP of the communication paths 115. Thus, the client device 102 may transmit higher window credit values to server devices 110 with higher with communication paths 115 that have higher BDPs and lower window credit values to server devices with communication paths 115 with lower window credit values (e.g., on a proportionate basis. FIG. 11 is a diagram that illustrates an example multipath downloading process of the client device 102 while implementing the load balanced download process. In FIG. 11, the communication paths 115A and 115B have a BDP that is approximately the same and that is twice as much as the BDP of the communication path 115C. Accordingly, the client device 102 may transmit window credit values of 40% of the amount of data of the file remaining to be requested to the communication paths 115A and 115B and a window credit value of 20% of the amount of data of the file remaining to be requested to the communication path 115C.Assuming the BDPs of each communication path 115 remains the same during the final stage of the downloading process, assuming the window credit values are sent to the server devices 110 at the same time, and assuming that all data is successfully downloaded, downloading from eachD24017W001 of the communication paths 115 should finish at the same time without any excess egress data being sent by any of the server devices 110. In some instances of the load balanced downloading approach, the client device 102 may utilize only a subset of the communication paths 115 that were used during the rest of the downloading of the file. As a variation of the example in FIG. 11, depending on the BDPs of each communication path 115, the client device 102 may request 50% of the remaining portion of the file on each of the two fastest communication paths 115 as long as the BDT of each communication path indicates that each requested chunk of data can be sent in one RTT on communication path 115.

[0095] FIG. 12 illustrates another example use case of a load balanced download process with three communication paths 115, where the RTT, BIF, throughput (Thp) is given. The goal of the client device 102 when implementing the load balanced downloading approach is to determine the optimal file size Fteach communication path 115 should download in the final stage of the multipath downloading process so that the client device 102 can receive 100% of the file as quickly as possible while eliminating or reducing excess egress data (e.g., all data from each communication path 115 finishes downloading at the same time). Mathematically, this goal can be formulated as set forth below in Equations 9-11.Equation 9: min=1T2such that:Equation 10:=1Ft= FrandEquation 11:

[0096] The above-noted optimization problem mathematically set forth by equations 9-11 can be solved by known solvers such as CasADI™. However, in some instances, the load balanced downloading approach may be lightweight (e.g., require less processing time and resources). For example, lighter weight optimizers may be implemented by the client device 102 (e.g., a look-up table or a series of “If’ statements) and may be able to solve the above-noted optimization problem more quickly (e.g., by approximately one or two orders of magnitude) compared to using a known optimization solver such as CasADI™. This lighter weight optimizer is well-suited for network-related usage where each millisecond (ms) of delay that is avoided may be beneficial.

[0097] FIG. 12 shows an example use case at a time when the load balanced download process begins. In FIG. 12 different communication paths 115 are in different situations, and theD24017W001 goal of the client device 102 during the load balanced download process is to prevent communication channels 115 from being idle at all or for very long. With respect to the communication path 115A shown in FIG. 12, the RTT of the communication path 115A is smaller than the time needed to download the BIF of the communication path 115A at the time the load balanced download process begins. This means that if the client device 102 sends a window update command (e.g., an additional window credit value) to the server device 110 of the communication path 115A at this moment, the server device 110 will be able to receive the window update command before the server device 110 runs out of window credit to send data to the client device 102. In other words, in this case, the server device 110 associated with the communication path 115A will not become idle, and the bandwidth of the communication path 115A may be fully utilized. The communication path 115C is in a similar situation as the communication path 115A. On the other hand, with respect to the communication path 115B, the RTT of the communication path 115B is larger than the time needed to download the BIF of the communication path 115B at the time the load balanced download process begins. In this situation, there will be an idle time as shown in FIG. 12 where the server device 110 associated with the communication path 115B will not be able to send the client device 102 data because the server device 110 has run out of window credit. The algorithm used during the load balanced download process considers the types of situations shown in FIG. 12 and solves Equation 11 above to determine when to send window credit values to each server device 110 and what value to send as the window credit values to each server device 110 to attempt to reduce or eliminate idle time on any communication path 115 based on current network conditions / characteristics .

[0098] In some instances, the client device 102 may implement the load balanced downloading approach in a different manner such that the client device 102 self-regulates requesting of data from each server device 110 without making the optimization calculations described immediately above. For example, the client device 102 may be configured to implement the load balanced downloading approach by determining a reduced window credit value for each or all of the server devices 110 of the plurality of server devices 110, where the reduced window credit value for each server device 110 may remain constant (for each server device 110 or for all server devices 110) and is included in repeated transmissions from the client device 102 to the respective server device 110 until the total amount of window credit values that have been transmitted to all of the plurality of server devices is greater than or equal to the size of the file. In this implementation of the load balanced downloading approach, the window credit value for each server device 110 may be set to a small amount (e.g., the amountD24017W001 of data of the file remaining to be requested divided by 2*n where n is the number of communication paths). However, in other instances, the constant 2 may be replaced with another constant such as 2.5, 3, 4 or the like. With this small of a window credit value for all server devices 110, the client device 102 is configured to re-send the same small window credit value (or a dynamically adjusted window credit value as explained below) to a server device 110 each time the downloading of a certain amount of data (e.g., one http data frame, several http data frames, etc.) from the corresponding communication path 115 of the server device 110 is completed. In some instances, the client device 102 may be configured to resend the small window update credit to a server device 110 each time the client device received some data (e.g., one or more http data frame) from that server device 110, and the value of the small window credit value is proportional to the size of data the client device 102 just received (e.g., some constant of a dynamically changing value times the size of data it just received). In some instances, the client device 102 may be configured to resend a small window update credit to the server device 110 whose value is proportional to the coded packet size, each time the client device 102 receives one or more coded packets from the server device 110. Accordingly, the server devices 110 corresponding to faster communication paths 115 will naturally receive more requests and additional window credit values compared to the server devices 110 corresponding to slower communication paths 115. Thus, more data will be transmitted on the faster communication paths 115 than on the slower communication paths 115 in a self-regulated manner. As noted above, the client device 102 may continue sending window credit values to server devices 110 upon completion of downloaded data on their corresponding communication paths 115 until the total amount of window credit values that have been transmitted to all of the plurality of server devices 110 is greater than or equal to the size of the file. At this point, the client device 102 ceases sending additional window credit values and downloads the remaining BIF, which should correspond to the size of the file with minimal or no excess egress data from the server devices 110. While the reduced window credit value for each server device 110 may remain constant in the self-regulated load balanced downloading approach, in other instances, the reduced window credit value may be dynamically adjusted for one or more server devices 110 based on the network conditions of one or more of the communication paths 115. For example, the reduced window credit value may be adjusted for each server device 110 based on the network conditions of its respective communication path 115. As another example, the reduced window credit value may be adjusted for all window credit values based on the network conditions of one or more of the communication paths 115.D24017W001

[0099] In some instances, the load balanced downloading approaches (e.g., the selfregulating approach) may be implemented by the client device 102 when the difference between the size of the file and the total amount of window credit values that have been transmitted to all of the plurality of server devices 110 is greater than the predetermined threshold (e.g., a large amount of the file remains to be requested and received). However, because the amount of data that remains to be requested and received is much larger in this situation, the load balanced downloading approaches may be implemented on a larger scale (e.g., with larger window credit values, for example, up to a respective maximum window credit value of each communication path 115 as opposed to the smaller window credit values that are used when the difference between the size of the file and the total amount of window credit values that have been transmitted to all of the plurality of server devices 110 is less than the predetermined threshold (e.g., only a small amount of the file remains to be requested and received)).

[0100] In some instances, despite the efforts of the client device 102 in implementing the load balanced downloading approach, downloading of the data of the file from all communication paths 115 may not finish at the same time (e.g., due to changes in network conditions, corrupted transmissions / data lost in transmission, etc.). Accordingly, one or more of the communication paths 115 may be idle while other communication paths 115 are still providing (or expected to provide) data to the client device 102 based on previous window credit values that were transmitted to server devices 110 by the client device 102. In response to the client device 102 detecting one or more idle communication paths 115, the client device 102 may determine whether to take additional actions depending on its operational mode / settings.

[0101] In some instances, the client device 102 may operate in an economy mode where the client device 102 is configured such that its priority is to reduce or eliminate excess egress data, even at the expense of throughput. Accordingly, when the client device 102 is operating in the economy mode, the client device 102 may not take any action in response to detecting idle communication paths 115 and may instead merely wait for downloading to be completed from active communication paths 115 (unless, as described herein with respect to other modes, the flow controller implemented by the client device 102 deems one or more of the communication paths 115 is unresponsive, then in order to successfully decode the object, the flow controller sends a window credit value with minimally needed values to one or more active communication paths 115 to offload the downloading task from an unresponsive communication path 115). While this non-action will reduce throughput of the downloading process, it will also minimize excess egress data because excess data (e.g., beyond 100% of the size of the file) is notD24017W001 requested from a server device 110 corresponding to an idle communication path 115. In some instances, even when throughput is not prioritized / maximized, the throughput of the downloading process using the method(s) described herein may have an increased throughput compared to a single communication path downloading approach, for example, due to changes in characteristics / capabilities of the communication paths 115 and the ability of the client device 115 to dynamically send window credit values in accordance with such changes as described herein.

[0102] On the other hand, in some instances, the client device 102 may operate in a performance mode for enhanced throughput while still reducing or limiting excess egress data. The performance mode may be utilized in situations where throughput is more important than minimizing excess egress data (e.g., as indicated by user or network manager configuration / preferences of the client device 102). The performance mode is also useful in situations where one of the server devices 110 is or becomes unresponsive because operating in the economy mode in these situations greatly affects throughput as illustrated in FIG. 13 (however, as explained previously herein, even in the economy mode, the client device 102 may have the ability to offload the downloading task from a communication path 115 deemed to be unresponsive). For example, FIG. 13 illustrates a situation where the client device 102 downloads all data that was transmitted by servers 1 and 2 (e.g., server devices 1 and 2) but must then wait to download the rest of the file from server 3 that is slower than servers 1 and 2. This situation results in idle time for servers 1 and 2.

[0103] To address the situation shown in FIG. 13, when operating in the performance mode (or when a communication path 115 is deemed / determined to be unresponsive when operating in the economy mode), the client device 102 may trade off a small amount of excess egress data to increase the throughput by taking advantage of the idle communication paths 115. For example, in situations where throughput is a priority and / or some excess egress data is allowed, it is advantageous for the client device 102 to keep requesting on the idle communication path(s) 115 (e.g., the communication path(s) 115 that already finished downloading the files that were assigned to it) so that 100% of the file can be received as soon as possible. In some instances, an excess egress limit (e.g., 3% of the file size, 5% of the file size, 1% of the file size, a raw amount of data independent of the file size such as 100KB, 200KB, etc., or the like) may be predetermined by the client device 102 (and may be adjustable by a user or network manager) so that more than the excess egress limit of the file size will not be requested on the idle communication path(s) 115 unless the flow controller determines that one or more of theD24017W001 communication paths 115 are broken / unresponsive and that other communication paths 115 should be used to offload the downloading task of the unresponsive communication path 115, which may result in egress data higher than some preset limit. In some instances, when the client device 102 determines that one or communication paths 115 are not responsive / broken, the client device 102 may retry connecting to the server device 102 associated with such a communication path 115 (i) if the client device 102 is about to download a next / new file and / or (ii) after a period of wait time has passed.

[0104] In some instances (e.g., when the client device 102 is operating in performance mode), the client device 102 may implement a refill phase where the client device 102 detects an idle communication path 115 and determines that the entire file (e.g., the full size of the file) has not yet been received. For example, the client device 102 may determine whether any one of the plurality of communication paths 115 has become idle due to the client device 102 downloading all data to which its respective server device 110 transmitted in accordance with the respective adjusted window credit value (that was sent during the load balanced downloading approach). The client device 102 may also determine whether an amount of the file that has been received by the client device 102 on all of the plurality of communication paths 115 is greater than or equal to the size of the file. In response to determining that one of the plurality of communication paths 115 has become idle and that the amount of the file that has been received by the client device 102 on all of the plurality of communication paths 115 is less than the size of the file, the client device 102 may transmit the adjusted window credit value or a further adjusted window credit value to a server device 110 corresponding to the idle communication path 115 in order to allow the server device 110 corresponding to the idle communication path 115 to transmit additional data of the file to the client device 102. In some instances, the further adjusted window credit value corresponds to an amount of the file that has not been received by the client device 102 or corresponds to a maximum excess egress data percentage (e.g., the excess egress limit) of the file that is preprogrammed into the client device 102 (but that is adjustable, for example, by a user or network manager).

[0105] In some instances, the client device 102 is configured to wait for a predetermined period of time after the idle communication path 115 has become idle to determine whether the amount of the file that has been received by the client device 102 on all of the plurality of communication paths 115 is greater than or equal to the size of the file. Such waiting gives additional time for data to be received on the active communication path(s) 115 and prevents the client device 102 from acting too quickly to request additional data on the idle communicationD24017W001 path 115. In some instances, the predetermined period of time may be preset or may be determined based on the RTT of the active communication path(s) 115. For example, the predetermined period of time may be set based on the difference between the RTT for an active communication path 115 and an amount of time that has passed since the request for data (e.g., the most recent window credit value) on the active communication path 115 was transmitted by the client device 102. In some instances, the predetermined time period is determined as a factor of the RTT of the idle communication path 115 or one of the active communication paths 115. In some instances, the client device 102 does not wait the predetermined period of time before taking action.

[0106] In some instances, the client device 102 may implement the refill phase in other manners. For example, the client device 102 may implement a more aggressive refill phase that is theoretically faster than the refill phase implementation described above. In the more aggressive refill phase implementation, the client device 103 may determine an estimated downloading time for the data from each of the server devices 110 based on the operating characteristic of the respective communication path 115 (e.g., the BDP, RTT, and / or the like) and the respective adjusted window credit transmitted to each server device 110. The client device 102 may determine a maximum estimated downloading time as a highest estimated downloading time from among the estimated downloading times of the server devices 110, and may determine a minimum estimated downloading time as a lowest estimated downloading time from among the estimated downloading times of the server devices 110. The client device 102 may also determine whether a difference between the maximum estimated downloading time and the minimum estimated downloading time is greater than a predetermined downloading time difference threshold. In other words, the client device 102 determines whether the downloading of the data on all communication paths 115 is likely to finish at approximately the same time or if the end time of downloading on one communication path 115 (e.g., the communication path 115 estimated to be the fastest) is likely to significantly vary from that of another communication path 115 (e.g., the communication path 115 estimated to be the slowest). In response to determining that the difference between the maximum estimated downloading time and the minimum estimated downloading time is greater than the predetermined downloading time difference threshold, the client device 102 may transmit the adjusted window credit value or the further adjusted window credit value to the server device 110 corresponding to the communication path 115 with the lowest estimated downloading time (e.g., the fast communication path 115) in order to allow the server device 110 corresponding to the communication path 115 with the lowest estimated downloading time to transmit additional dataD24017W001 of the file to the client device 102. In other words, the client device 102 transmits additional window credit to the faster communication path 115 so that the faster communication path 115 can off-load the downloading task given to the slower communication paths 115. As is apparent, the additional window credit to the faster communication path 115 increases throughput to allow the entire size of the file to be received by the client device 102 more quickly, but it also increases excess egress data because the server device 110 corresponding to the slower communication path 115 already received a window credit value for the original data in a previous request. Accordingly, in some instances, the further adjusted window credit value corresponds to a maximum excess egress data percentage (e.g., the excess egress limit) that is preprogrammed into the client device 102 (but that is adjustable, for example, by a user or network manager). In some instances, the further adjusted window credit value corresponds to an amount of the file remaining to be received by the client device 102. In some instances, the refill phase explained immediately above is more aggressive than the previously described refill phase because it may not wait for an idle communication path 115 to be detected (e.g., it may not wait for all requested data to be received on any of the communication paths 115) before transmitting a request for additional data (e.g., transmitting an additional window credit value) to a server device 110 corresponding to a fast communication path 115. Thus, this second refill phase may be particularly useful in situations where throughput is prioritized over minimizing excess egress data.

[0107] As indicated throughout the explanation herein of the functionality of the client device 102, at block 420 of the method 400 of FIG. 4, the client device 102 transmits the respective window credit value to each of the server devices 110. The respective window credit value indicates an amount of the file that each server device 110 is configured to transmit to the client device, for example, until the client device 102 provides another window credit value to one or more server devices 110 to extend the overall window (e.g., overall amount of data) that the respective server device 110 can transmit to the client device 102. As described previously herein, in some instances, the client device 102 communicates with the server devices 110 using HyperText Transfer Protocol-2 (HTTP-2), and the respective window credit value is transmitted by the client device 102 to each of the server devices 110 as a WINDOW_UPDATE frame parameter included as a flow control mechanism that is native to HTTP-2.

[0108] At block 425, the client device 102 receives a portion of the file from each of the server devices 110, where a size of the portion of the file from each server device 110 is limited by the respective window credit value of the respective server device 110. Because eachD24017W001 window credit value is added onto previously sent window credit values, the total size of the portion of the file from each server device 110 is limited by the respective total amount of window credit values transmitted to the server device 110 (e.g., the overall window provided to the respective server device 110 by the client device 102). In some instances, all data transmitted by the server devices 110 is not received by the client device 102 (e.g., due to corrupted transmissions / data lost in transmission, etc.). Nevertheless, the window credit values limit the amount of data that can be transmitted by the server devices 110 to the client device 102.

[0109] In some instances, a sum of the window credit values transmitted to all of the server devices 110 by the client device 102 is equal to the size of the file such that none of the server devices 110 transmit excess egress data to the client device 102 (e.g., when the client device 102 is operating in economy mode and all of the server devices 110 / communication paths 115 are responsive). In some instances, a sum of the window credit values transmitted to all of the server devices 110 by the client device 102 is less than or equal to a an excess egress limit (e.g., a predetermined excess egress percentage of the size of the file) such that the server devices 110 transmit a limited amount of excess egress data to the client device 102. For example, the excess egress data is less than or equal to the predetermined excess egress percentage of the size of the file (e.g., when the client device 102 is operating in a version of the performance mode).

[0110] In some instances, the client device 102 implements a downloading state logger to store details about downloading processes so that historical downloading information / states can be used by the client device 102 during future downloads to determine downloading strategies to reduce or eliminate excess egress data, to increase throughput, or combination of both.

[0111] Numerous use cases / examples in various situations in accordance with the methods described herein are described below.

[0112] In some instances, the flow controller implemented by the client device 102 operates in an initial file downloading stage (e.g., when downloading a first file where communication paths 115 are not yet established and their operating characteristics are not known) or in a subsequent file downloading stage (e.g., when downloading additional files after the first file). In the initial file downloading stage, the client device establishes socket connections with one or more server devices 110 concurrently, sends HTTP “get requests” to the server devices 110, and waits for responses from the server devices 110 as described previously herein. Once any of the server devices 110 respond, the client device 102 is able to determine the size of the file asD24017W001 described previously herein (e.g., by analyzing header information provided by the server device 110). The client device 102 may then determine a relationship between the file size and the amount of file requested from all of the server devices 110 to which requests were sent. For example, a default initial request size sent to each server device 110 may be 64KB. Thus, if the file size is larger than 64KB and the amount of data requested from all server devices 110 (e.g., N*64KB) did not exceed the total file size, then the client device 102 may transmit the HTTP / 2 “get request” on the new socket to the first server device 110 in response to determining that the amount of the file that has not yet been requested is larger than 64KB. On the other hand, the client device 102 may transmit a modified HTTP / 2 “get request” that changes the initial window size of HTTP / 2 connection to the remaining file size in response to determining that the remaining file size is less than 64KB.

[0113] For example, suppose the file size is 150KB, and three socket connections to server A, B, and C are made. If server B's socket connection is first established, then a HTTP / 2 “get request” is directly sent to the server B. At this stage, 64KB of the file in total has been requested (as the initial window size of the server B is 64KB and server B will not send more data unless it receives an additional window update frame(s)). If server A's socket connection is established while still waiting for the HTTP / 2 response from server B, a HTTP / 2 “get request” is also sent to server A. At this stage, 128KB of file has been requested. If both server A and B have not yet sent a response to the client device 102 when server C's socket connection is established, then a HTTP / 2 “get request” is sent to server C as well. However, if at least one of server A and B have responded (and provided the size of the file), the client device 102 is configured to modify the window credit value sent to server C so as to avoid requesting more data than 100% of the size of the file. For example, the window credit value sent to server C may be a difference between the size of the file and a total amount of the file that was previously requested on all communication channels 115 (e.g., 150KB - 128KB = 22KB). This amount is less than default window size of 64KB, which minimizes the excess egress data because a goal of this disclosure is to request just enough of the file to allow the client device 102 to decode the coded file. In some instances, the HTTP / 2 “get request” to any new server device 110 (e.g., server C in the above-noted example) to set up a new communication path 115 may be set to a default value (e.g., 64KB) and may not be adjusted based on the size of the file.

[0114] As a variation of this example, if one of the servers (e.g., server A) does not respond at approximately the same time as the other servers, the effect on the downloading process depends on whether the client device 102 is operating in economy or in performance mode. InD24017W001 economy mode, the client device 102 is configured to wait for server A to respond (unless / until the client device 102 determines that server A and / or its communication path 115 is broken / unresponsive as explained previously herein), which will decrease the overall throughput / speed of the downloading process, because the priority is to reduce or eliminate excess egress data in the economy mode. However, in performance mode, the client device 102 may send requests for additional data on the faster communication path(s) 115 to increase throughput at the expense of some excess egress data as explained previously herein.

[0115] Assuming that the file size is large enough to warrant that additional window credit values be sent to the server devices 110 to allow the server devices 110 to transmit additional data, the client device 102 continues to transmit window update frames with window credit values set at least partially based on the BDP of each communication path 115 as explained previously herein.

[0116] When operating in a performance mode of the initial file downloading stage, the client device 102 may sacrifice some excess egress data to increase throughput as explained previously herein. In doing so, the client device 102 may take different actions (e.g., request a different amount of additional data on a faster communication path 115) depending on a size of the file that is being downloaded. For example, the client device 102 may determine whether the file size is larger than 64KB * N, where N is the number of communication paths 115 available. When the file size is larger than 64KB * N, the client device 102 may request additional data only up to the excess egress limit as explained previously herein. On the other hand, when the file size is smaller than 64KB * N, the file size is small and may only take one RTT to finish downloading. Thus, for small file size situations, the client device 102 may request all of the remaining file size from the server device 110 on the detected idle communication path 115. Two specific examples of implementation of the performance mode are included below.

[0117] In the first example of implementation of the performance mode, suppose the file size is 100 KB, and two communication paths 115 are used to download the file. Additionally, assume only one request on each communication path 115 to download the file: a 64KB request sent to a first communication path 115 and a 36KB request sent to a second communication path 115. Thus, in total, the amount of file size requested by the client device 102 is 64KB + 36KB = 100KB, and the amount of excess egress data assuming that both downloads occur as expected is 0%. Continuing this first example, suppose the response from the first server device 110 is received and 64KB is downloaded from the first server device 110, and the client device 102 is still waiting to receive the 36KB requested from the second server device 110. However, inD24017W001 some instances, especially at the start of a download, a long time may pass until the remaining 36KB of file can be received. In economy mode, the client device 102 is configured to wait, as there may be a strict restriction on the excess egress data (e.g., 0% of the file size). However, even in economy mode, the client device 102 may only wait until the client device 102 determines that second server device 110 and / or its communication path 115 is broken / unresponsive as explained previously herein. On the other hand, in performance mode, a 36KB request on the first communication path 115 may be transmitted as soon as the first communication path 115 becomes idle so that the remaining 36 KB of file is received as quickly as possible to allow decoding of the file (e.g., a coded file) to be completed as quickly as possible. In this example, the amount of excess egress data is 36% of the file size. In summary, when the file size is smaller than 64*N (where N is the number of communication paths 115 being used to download the file), the client device 102 may request, the portion of the file size remaining to be received, from the server device 110 corresponding to the idle communication path 115. This approach is useful to increase the throughput when the (a) file size is small and (b) only needs one or two RTT to complete downloading of the entire file size.

[0118] In the second example of implementation of the performance mode, suppose the file size to download is 1MB, and two communication paths 115 are used to download the file.Additionally, assume one request by the client device 102 is made on each of the communication paths 115 to download the file. Suppose when the first communication path 115 becomes idle, 400KB of the file was requested from the first server device 110, and all 400KB of the file have been received over the first communication path 115. On the second communication path 115, 600KB of the file was requested, and only 500KB of the file was received. In the economy mode, the client device 102 is configured to wait for the remaining 100KB to be received from the second server device 110 via the second communication path 115 (unless / until the client device 102 determines that the second server device 110 and / or its communication path 115 is broken / unresponsive as explained previously herein). On the other hand, in performance mode, the client device 102 may determine an excess egress limit (e.g., as a percentage of the total size of the file). For example, an excess egress limit of 3% means that the amount of excess egress data should be limited to 1MB * 3% = 0.03MB = 30KB. The client device 102 may then compare how much more of the file that the idle communication path 115 (the first communication path 115 in this example) should request by determining the minimum of the total amount of the file remaining to be received and the amount of data corresponding to the excess egress limit (e.g., min(100KB, 30KB) = 30KB). The client device 102 may then request additional data from the first server device 110 on the idle communication path 115 thatD24017W001 corresponds to the lower of the total amount of the file remaining to be received and the amount of data corresponding to the excess egress limit (in this example, 30KB). Thus, throughput may increased (e.g., faster downloading of 100% of the size of the file) while excess egress data can still be limited below a specified limit.

[0119] In some instances, the client device 102 may take additional factors into consideration when implementing the performance mode. For example, with reference to the second example immediately above, at the time when the first communication path 115 went idle, 100KB of data still needed be downloaded on the second communication path 115. To attempt to speed up the downloading process, the client device 102 additionally requested 30 KB on the first communication path 115. However, requesting 30KB of the file might not actually speed up the downloading process at all in some instances. Rather, in some instances, the downloading process is only made faster when the congestion window size (e.g., the TCP congestion window size) of the slower second communication path 115 is less than 100 KB.

[0120] For example, when the TCP congestion window of the second communication path 115 is larger than 100KB, it means that all 100KB of the file will be transmitted by the second server device 110 in one transmission. Thus, even if the client device 102 receives the additionally requested 30KB of file from the first communication path 115, the client device 102 still needs to wait for the second communication path 115 as 70KB more are still needed to finish the downloading process (e.g., download 100% of the size of the file). Thus, when the TCP congestion window is larger than 100KB (e.g., the remaining amount of the file), there is no speed performance gain by additionally requesting 30KB of the file (e.g., a portion of the remaining amount of the file) on the first communication path 115.

[0121] On the other hand, when the TCP congestion window size of the second communication path 115 is smaller than 100KB, multiple RTT transmissions on the second communication path 115 are needed to deliver 100 KB of file. Accordingly, speed performance gain may be possible by downloading an additional portion of the file on the first communication path 115. For example, suppose the second communication path 115 delivered 100KB of the file in three RTT transmissions (e.g., 40KB, 30KB, 30KB), then by requesting 30KB on the first communication path 115, the downloading process may be sped up by an amount of time corresponding to one RTT. However, if the second communication path 115 delivers the 100KB in two RTT (50KB, 50KB), then there still would not be a speed performance gain.D24017W001

[0122] As indicated by the previous two examples, the possibility of speed performance increase in the download process depends on how the slow communication path 115 delivers the remaining portion of the file. In some instances, the client device 102 may take this factor (e.g., the estimated congestion window size of the slow the communication path 115) into account when determining whether to request additional data on an idle communication path 115. For example, to achieve speed improvement, the client device 102 may be configured to request all of the remaining file size (e.g., 100 KB) on the first communication path 115 (e.g., the idle communication path 115). However, doing so may cause the excess egress data to exceed the user defined threshold (e.g., 3% of the total file size). Thus, the client device 102 may be configured to optimize throughput / downloading speed (e.g., in performance mode) or minimize excess egress data (e.g., in economy mode) depending on different situations.

[0123] As indicated previously herein, in some instances, the flow controller implemented by the client device 102 operates in the initial file downloading stage or in the subsequent file downloading stage. In the subsequent file downloading stage, the flow controller may implement one of various file downloading strategies such as (i) CDN Switching (Eager) Approach, and (ii) Load Balanced Downloading Approach. Moreover, variations of each strategy are described which depend on whether the file size of file / object of interest is known.

[0124] In some instances of the subsequent filing downloading stage, the client device 102 implements a CDN switching approach and the file size is not known. For example, the client device 102 may have finished downloading a first object / file and went through the initial file downloading stage. Accordingly, the client device 102 will already have information regarding the bandwidth and RTT of each communication path 115. The product of bandwidth and RTT (e.g., BDP) indicates how much data each server device 110 can send to the client device 102 in one RTT transmission on a respective communication path 115. In this example, suppose there are three communication paths 115, each with known bandwidth and RTT after the initial downloading phase. If the file size of the next file (e.g., a second file) to be downloaded is not known ahead of time, the client device 102 transmits a request on each communication path 115 to respective server devices 110 to get a response from the server devices 110 as quickly as possible. When the client device 102 receives a response from one of the server devices 110, the file size of the second object / file for download is known, and the client device 102 starts to compare the file size and the BDP of each communication path 115. On the communication path 115 that responded first, if the BDP of this communication path 115 is larger than the remaining file size that needs to be downloaded, then the client device 102 sends a windowD24017W001 update frame (including a window credit value) on this communication path 115 and downloads the remaining file size. In this case, the downloads from the other two communication paths 115 will be canceled by the client device 102 once the client device 102 receives respective responses from the other two communication paths 115. Control of the downloading process according to the above example is equivalent to a CDN switching approach, as the client device 102 switches to the communication path 115 that is fastest. If the BDP of the communication path 115 that responded first is not larger than the remaining file size that needs to be downloaded, the client device 102 sends a window update frame with a size of the BDP of the communication path 115 that responded first and uses the same strategy explained with respect to the initial file downloading stage examples explained previously herein to finish the downloading of the file (e.g., using Algorithm 1 of FIGS. 14A-14B and / or implementing economy mode or performance mode and then using the load balanced downloading approach calculated using a solver or using self-regulation as explained previously herein).

[0125] In some instances of the subsequent filing downloading stage, the client device 102 implements a CDN switching approach and the file size is known. If the file size of the object / file for download is known, for example by modifying the manifest file, the client device 102 may identify the communication path 115 that is historically most responsive (e.g., fastest), and determine whether the BDP of the most responsive communication path 115 is larger than the file size. If the BDP of the most responsive communication path 115 is larger than the file size, the client device 102 may send a request to this communication path 115 and complete the downloading of the file. On the other hand, if the BDP of the most responsive communication path 115 is smaller than the file size, the client device 102 may send a request to the most responsive communication path 115 and adjust its window size to the BDP of the most responsive communication path 115. Then, for a second most responsive communication path 115, the client device 102 may determine whether its BDP is larger than the remaining unrequested file size. If the BDP of the second most responsive communication path 115 is larger than the remaining unrequested file size, the client device 102 may send only two requests to finish the downloading of the file (e.g., one request to on each of the most responsive communication path 115 and the second most responsive communication path 115). If the BDP of the second most responsive communication path 115 is smaller than the remaining unrequested file size, the client device 102 may use the same strategy explained with respect to the initial file downloading stage examples explained previously herein to finish the downloading of the file (e.g., using Algorithm 1 of FIGS. 14A-14B and / or implementingD24017W001 economy mode or performance mode and then using the load balanced downloading approach calculated using a solver or using self-regulation as explained previously herein).

[0126] In some instances of the subsequent filing downloading stage, the client device 102 implements a load balanced downloading approach and the file size is not known. If the file size is not known in advance, the client device 102 starts by sending a request to all of the communication paths 115 with an initial window size (e.g., window credit value) set to default value. Then, once the file size is known, the client device 102 use the same strategy explained with respect to the initial file downloading stage examples explained previously herein to finish the downloading of the file (e.g., using Algorithm 1 of FIGS. 14A-14B and / or implementing economy mode or performance mode and then using the load balanced downloading approach calculated using a solver or using self-regulation as explained previously herein).

[0127] In some instances of the subsequent filing downloading stage, the client device 102 implements a load balanced downloading approach and the file size is known. If the file size is already known by the client device 102, the client device 102 may send a request to each communication path 115 and set the initial window size to a respective BDP value of each communication path 115 (assuming that the sum of the BDP values of all the communication paths 115 is smaller than the file size). The client device 102 may then use the same strategy explained with respect to the initial file downloading stage examples explained previously herein to finish the downloading of the file (e.g., using Algorithm 1 of FIGS. 14A-14B and / or implementing economy mode or performance mode and then using the load balanced downloading approach calculated using a solver or using self-regulation as explained previously herein). If the sum of BDP values of all communication paths 115 is larger than the file size, the client device 102 may proceed directly to use the load balanced downloading approach calculated using a solver or using self-regulation, for example, to attempt to have the data from all communication paths 115 finish downloading at the same time.

[0128] The foregoing description, for purpose of explanation, has been described with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The embodiments were chosen and described in order to best explain the principles of the techniques and their practical applications. Others skilled in the art are thereby enabled to best utilize the techniques and various embodiments with various modifications as are suited to the particular use contemplated.D24017W001

[0129] Although the disclosure and examples have been fully described with reference to the accompanying drawings, it is to be noted that various changes and modifications will become apparent to those skilled in the art. Such changes and modifications are to be understood as being included within the scope of the disclosure and examples as defined by the claims.

[0130] Various features and advantages are set forth in the following claims.

Claims

D24017W001CLAIMSWhat is claimed is:

1. A method for reducing excess egress data in a communication session, the method comprising: transmitting, with a client device, a request for a file to a plurality of server devices; determining, with the client device, an operating characteristic of each communication path of a plurality of communication paths, wherein each communication path is between the client device and a respective server device of the plurality of server devices; determining, with the client device, a window credit value for each server device based at least in part on the respective operating characteristic, a size of the file, an amount of the file that has been received by the client device on all of the plurality of communication paths, and a bytes-in-flight (BIF) value corresponding to data that has been requested by the client device but has not yet been received by the client device; transmitting, with the client device, the respective window credit value to each of the server devices, wherein the respective window credit value indicates an amount of the file that each server device is configured to transmit to the client device; and receiving, with the client device, a portion of the file from each of the server devices, wherein a size of the portion of the file from each server device is limited by the respective window credit value of the respective server device.

2. The method of claim 1, wherein determining the operating characteristic of each communication path includes: determining, with the client device, a bandwidth of each communication path and a round-trip time (RTT) of each communication path; and determining, with the client device, a bandwidth delay product (BDP) of each communication path by determining a product of the bandwidth and the RTT of each communication path, and further comprising: determining, with the client device, the amount of the file that has been received by the client device based on the portions of the file received from each server device; determining, with the client device, the BIF value for each communication path by subtracting the amount of the file that has been received by the client device on each communication path from a total amount of window credit values that have been transmitted to the respective server device corresponding to each communication path;D24017W001 determining, with the client device, whether a difference between the size of the file and the total amount of window credit values that have been transmitted to all of the plurality of server devices is greater than a predetermined threshold; and in response to determining that the difference between the size of the file and the total amount of window credit values that have been transmitted to all of the plurality of server devices is greater than the predetermined threshold, determining, with the client device, the window credit value for each server device based on the BDPs of the communication paths over which the server devices communicate with the client device.

3. The method of claim 2, further comprising: setting, with the client device, a maximum window credit value for each server device as a multiple of the BDP of the communication path over which the respective server device communicates with the client device; and setting, with the client device, a minimum window credit value for each server device to be equal to the BDP of the communication path over which the respective server device communicates with the client device; wherein determining the window credit value for each server device based on the BDPs of the communication paths over which the server devices communicate with the client device and in response to determining that the difference between the size of the file and the total amount of window credit values that have been transmitted to all of the plurality of server devices is greater than the predetermined threshold includes determining, with the client device, the window credit value for each server device such that the window credit value for each server device is in a range between and including its respective minimum window credit value and its respective maximum window credit value.

4. The method of claim 2 or claim 3, wherein determining the window credit value for each server device based on the BDPs of the communication paths over which the server devices communicate with the client device includes determining a first window credit value for a first server device and a second window credit value for a second server device, wherein the first window credit value and the second window credit value are set to different values.

5. The method of any one of claims 2-4, further comprising determining, with the client device and while the difference between the size of the file and the total amount of window credit values that have been transmitted to all of the plurality of server devices is greater than theD24017W001 predetermined threshold, an adjusted window credit value for at least one of the server devices based on a change of the BDP of the communication path over which the at least one of the server devices communicates with the client device.

6. The method of claim 5, wherein determining the adjusted window credit value for the at least one of the server devices includes determining, with the client device, the adjusted window credit value for the at least one of the server devices such that the adjusted window credit value is proportionate to the change of the BDP of the at least one of the server devices.

7. The method of any one of claims 2-6, wherein determining the bandwidth of each communication path includes: determining an inter-chunk arrival (ICA) time of each chunk of data received; discarding one or more chunks of data from a bandwidth determination in response to determining that the ICA time for the one or more chunks of data is greater than a predetermined threshold; determining, with the client device, an amount of time elapsed to receive a specified amount of data from the respective server device associated with each communication path; and determining the bandwidth of each communication path by dividing the specified amount of data by the amount of time elapsed to receive the specified amount of data; wherein the specified amount of data excludes the discarded one or more chunks of data, and wherein the time elapsed to receive the specified amount of data excludes time associated with receipt of the discarded one or more chunks of data.

8. The method of any one of claims 2-7, further comprising: determining, with the client device, that the difference between the size of the file and the total amount of window credit values that have been transmitted to all of the plurality of server devices is less than the predetermined threshold; and implementing, by the client device and in response to determining that the difference between the size of the file and the total amount of window credit values that have been transmitted to all of the plurality of server devices is less than the predetermined threshold, a load balanced downloading approach where the client device determines an adjusted window credit value for at least one of the server devices.D24017W0019. The method of claim 8, wherein implementing the load balanced downloading approach includes: determining, with the client device, the adjusted window credit value for the at least one of the server devices based at least in part on the operating characteristic and the difference between the size of the file and the total amount of window credit values that have been transmitted to all of the plurality of server devices; transmitting, with the client device, the adjusted window credit value to the at least one of the server devices; and receiving, with the client device, a second portion of the file from the at least one of the server devices, wherein a size of the second portion of the file from each server device is limited by the adjusted window credit value of the at least one of the server devices.

10. The method of claim 8, wherein implementing the load balanced downloading approach includes determining a reduced window credit value for each of the server devices of the plurality of server devices, wherein the reduced window credit value for each server device is included in repeated transmissions from the client device to the respective server device until the total amount of window credit values that have been transmitted to all of the plurality of server devices is greater than or equal to the size of the file.

11. The method of any one of claims 8-10, further comprising: determining, with the client device, whether any one of the plurality of communication paths has become idle due to the client device downloading all data to which its respective server device transmitted in accordance with the respective adjusted window credit value; determining, with the client device, whether an amount of the file that has been received by the client device on all of the plurality of communication paths is greater than or equal to the size of the file; and in response to determining that one of the plurality of communication paths has become idle and that the amount of the file that has been received by the client device on all of the plurality of communication paths is less than the size of the file, transmitting, with the client device, the adjusted window credit value or a further adjusted window credit value to a server device corresponding to the idle communication path in order to allow the server device corresponding to the idle communication path to transmit additional data of the file to the client device.D24017W00112. The method of claim 11, further comprising waiting for a predetermined period of time after the idle communication path has become idle to determine whether the amount of the file that has been received by the client device on all of the plurality of communication paths is greater than or equal to the size of the file.

13. The method of any one of claims 8-10, further comprising: determining, with the client device, an estimated downloading time for the data from each of the server devices based on the operating characteristic of the respective communication path and the respective adjusted window credit transmitted to each server device; determining, with the client device, a maximum estimated downloading time as a highest estimated downloading time from among the estimated downloading times of the server devices; determining, with the client device, a minimum estimated downloading time as a lowest estimated downloading time from among the estimated downloading times of the server devices; determining, with the client device, whether a difference between the maximum estimated downloading time and the minimum estimated downloading time is greater than a predetermined downloading time difference threshold; and in response to determining that the difference between the maximum estimated downloading time and the minimum estimated downloading time is greater than the predetermined downloading time difference threshold, transmitting, with the client device, the adjusted window credit value or a further adjusted window credit value to a server device corresponding to the communication path with the lowest estimated downloading time in order to allow the server device corresponding to the communication path with the lowest estimated downloading time to transmit additional data of the file to the client device.

14. The method of any one of claims 11-13, wherein the further adjusted window credit value corresponds to an amount of the file remaining to be received by the client device or corresponds to a predetermined maximum amount of excess egress data allowed by the client device.

15. The method of any one of claims 3-14, wherein the predetermined threshold corresponds to a sum of the maximum window credit values of the plurality of server devices.

16. The method of any one of the preceding claims, wherein the client device communicates with the server devices using HyperText Transfer Protocol-2 (HTTP-2), and wherein theD24017W001 respective window credit value is transmitted by the client device to each of the server devices as a WINDOW_UPDATE frame parameter included as a flow control mechanism that is native to HTTP-2.

17. The method of any one of the preceding claims, wherein the client device communicates with the server devices using Quick User Datagram Protocol (UDP) Internet Connections (QUIC), and wherein the respective window credit value is transmitted by the client device to each of the server devices as a MAX_STREAM_DATA or MAX_DATA frame parameter included as a flow control mechanism that is native to QUIC.

18. The method of any one of the preceding claims, wherein a sum of the window credit values transmitted to all of the server devices is equal to the size of the file such that none of the server devices transmit excess egress data to the client device.

19. The method of any one of the preceding claims, wherein a sum of the window credit values transmitted to all of the server devices is less than or equal to a predetermined excess egress percentage of the size of the file such that the server devices transmit a limited amount of excess egress data to the client device, wherein the excess egress data is less than or equal to the predetermined excess egress percentage of the size of the file.

20. A non-transitory computer-readable storage medium storing instructions which, when executed by a computing apparatus, cause the computing apparatus to perform the method of any one of claims 1-19.

21. A client device, wherein the client device includes a computing apparatus, comprising: an electronic processor; and a memory storing instructions, which when executed by the electronic processor, cause the computing apparatus to perform the method of any one of claims 1-19.

Images