REDUCED BANDWIDTH USE VIA GENERATIVE ADVERSARIAL NETWORKS

Abstract

A method implemented on a computer, comprising, on a client data processing unit (102): determining a desired resolution quality of a media file (112); executing, in sequence: determining available resources of the client data processing unit (102); selecting one from a plurality of Generative Adversarial Network (GAN) plug-ins based on the determined available resources of the client data processing unit (102); determining a second resolution quality of the media file (112) corresponding to the selected GAN plug-in (104), the second resolution quality being lower than the desired resolution quality; exchanging data with a server (110) to receive the media file (112) in the determined second resolution quality; receiving the media file (112) in the determined second resolution quality;Converting the received media file (112) from the second resolution quality to the desired resolution quality using the selected GAN plug-in; and outputting the converted media file (112) with the desired resolution quality.

Description

BACKGROUND

[0001] A user often uses a browser or other application running on the user's device to stream media content such as videos. Sometimes, pauses or delays in the streamed media content may occur due to insufficient bandwidth to accommodate the file size of the streamed media content. For example, a user may want to watch a streamed video in high definition (HD) quality, which has a larger file size than standard definition (SD) quality. Depending on the available bandwidth, the HD quality video may experience issues such as buffering latency, resulting in reduced user satisfaction while consuming the streamed video. SUMMARY

[0002] Aspects of the disclosure may include a computer-implemented method, a computer program product, and a system. An example of the method includes determining a desired resolution quality of a media file; determining available resources of a computing device; selecting one of a plurality of Generative Adversarial Network (GAN) plug-ins based on the determined available resources of the computing device; and estimating a second resolution quality of the media file corresponding to the selected GAN plug-in. The second resolution quality is less than the desired resolution quality.The method further comprises exchanging data with a server to receive the media file at the estimated second resolution quality; converting the received media file from the second resolution quality to the desired resolution quality using the selected GAN plug-in; and outputting the converted media file at the desired resolution quality. DRAWINGS

[0003] While the drawings are illustrative only of exemplary embodiments and are therefore not to be considered limiting in scope, the exemplary embodiments will be described with additional specificity and detail through the use of the accompanying drawings, in which: Fig. 1 is a block diagram of one embodiment of an example system. Fig. 2 is a block diagram of one embodiment of an exemplary client device. Fig. 3 is a flowchart illustrating one embodiment of an exemplary method for reducing bandwidth consumption via Generative Adversarial Networks (GANs).

[0004] As is generally customary, the various features described are not drawn to scale, but are drawn to emphasize the particular features relevant to the exemplary embodiments. DETAILED DESCRIPTION

[0005] In the following detailed description, reference is made to the accompanying drawings, which form a part hereof, and in which, for the purpose of illustration, specific illustrative embodiments are shown. It should be understood, however, that other embodiments may be utilized and that logical, mechanical, and electrical changes may be made. Furthermore, the method illustrated in the drawing figures and description should not be construed to limit the order in which the individual steps may be performed. The following detailed description, therefore, is not to be taken in a limiting sense.

[0006] When "a number of" is used here in reference to elements, it means one or more elements. For example, "a number of different types of networks" refers to one or more different types of networks.

[0007] Furthermore, the phrases “at least one,” “one or more,” and “and / or” are open-ended expressions that can have both a conjunctive and a disjunctive function. For example, the phrases “at least one of the following: A, B, and C,” “at least one of the following: A, B, or C,” “one or more of A, B, and C,” “one or more of A, B, or C,” and “A, B, and / or C” mean A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together. In other words, “at least one,” “one or more,” and “and / or” mean any combination of elements, and any number of elements may be used from the list, but not all of the elements in the list are required. The element can be a specific item, thing, or category. Furthermore, the quantity or number of each element in a combination of the listed elements need not be the same.In some illustrative examples, "at least one of the following: A, B, and C" can mean, for example, without limitation, two of element A; one of element B; and ten of element C; or zero of element A, four of element B, and seven of element C; or other suitable combinations.

[0008] The term "any" ("a" or "an") entity refers to one or more of that entity. Therefore, the terms "any," "one or more," and "at least one" may be used interchangeably herein. It should also be noted that the terms "comprising," "comprising," "having," and "possessing" may be used interchangeably.

[0009] Furthermore, the term "automatic" and variants thereof, as used herein, refers to any process or function that can be performed without substantial human input when the process or function is performed. However, a process or function may be automatic even if the performance of the process or function uses substantial or non-substantial human input if the input is received before the process or function is performed. Human input is considered substantial if such input affects how the process or function is performed. Human input that consents to the performance of the process or function is not considered "substantial."

[0010] As discussed above, network bandwidth limitations can cause problems, such as difficulties with latency buffering, when playing streamed media content at a desired media quality (e.g., HD, Full HD, Quad HD, Ultra HD, etc.) when that desired media quality requires more bandwidth than is available. The embodiments described herein address this problem by using selectable GAN (Generative Adversarial Network) plug-ins to reduce the amount of network bandwidth consumed by the streamed media content while still delivering the media content at the user's desired media quality.

[0011] Fig. 1 is a block diagram of one embodiment of an exemplary system 100. The system 100 includes a client device 102 communicatively connected to a server 110 via a network 108. The network 108 may be implemented using any number of suitable physical and / or logical data transmission topologies. The network 108 may include one or more private or public data processing networks. For example, the network 108 may include a private network. Alternatively or additionally, the network 108 may include a public network such as the Internet. Thus, the network 108 may be part of a packet-based network such as a local area network, a wide area network, and / or a worldwide network such as the Internet.The network 108 may include one or more servers, networks, or databases and may use one or more data transmission protocols to transfer data between the server 110 and the client device 102.

[0012] In addition, the network 108 is in Fig. 1 as a single entity, in other examples, it may comprise a plurality of networks, for example, a combination of public and / or private networks. The data transmission network 108 may include a variety of types of physical data transmission channels or links. The connections may be wired, wireless, optical, and / or via any other suitable medium. In addition, the data transmission network 108 may include a variety of network hardware and software for performing routing, switching, or other functions, such as routers, switches, base stations, bridges, or any other equipment that may be useful to enable the transmission of data.

[0013] The server 110 stores a media file 112 that can be provided to the client device 102 via the network 108, for example, in response to a request for the media file 112 from the client device 102. It is understood that for the sake of simplicity of explanation in Fig. 1 depicts a single client device 102, a single server 110, and a single media file 112, in other embodiments, more than one client device 102, more than one server 110, and / or more than one media file 112 may be utilized. For example, in some embodiments, copies of the same media file 112 may be stored on a plurality of servers 110. Additionally, in some embodiments, a single media file 112 may be distributed across a plurality of servers 110. Furthermore, a plurality of different media files may be stored on the same server 110 and / or on a plurality of servers 110. Moreover, in some embodiments, a plurality of client devices 102 may request the same media file 112 or different media files 112 from the same or different servers 110.

[0014] The client unit 102 includes a plurality of GAN plugins 104 and a resource monitor 106. GANs are a type of neural network that can use unsupervised machine learning to convert low-quality images / videos into high-quality images / videos. Specifically, a GAN architecture includes a generator model and a discriminator model. The generator model is used to generate new possible examples for the sample area, e.g., new pixels for an image. The discriminator model is used to determine which pixels are real or original pixels and which are fake or fabricated pixels. The two models can be trained together to reach a point where the discriminator model is unable to distinguish between the fabricated pixels and the real or original pixels.However, it is not necessary for a particular GAN model to reach a point where the discriminator is in any case unable to distinguish the generated pixels from the original pixels.

[0015] The generator model and the discriminator model can be implemented using convolutional neural networks (CNNs). Convolutional neural networks (CNNs) are a class of artificial neural networks known as deep feedforward networks, i.e., networks with many layers between input and output layers, which have been successfully applied to analyzing visual imagery. Convolutional neural networks are constructed from artificial or neuron-like structures that have learnable weights and biases. Each neuron receives some input and computes a dot product. The architecture of a convolutional neural network typically features a stack of layers whose function is to receive an input (e.g., a single vector) and transform it through a series of hidden layers.Each hidden layer is composed of a set of neurons, each with learnable weights and biases. Each neuron can be fully connected to all neurons in the previous layer, and neurons within a single layer can function independently without any shared connections. The final layer is the fully connected output layer, and in classification settings, it corresponds to class scores, which can be arbitrary real-valued numbers or real-valued targets (e.g., in regression). However, it should be understood that the embodiments described herein are not limited to the use of CNNs to implement GANs.

[0016] The accuracy of a GAN network depends on the architecture of the GAN, for example, the network size or the number of layers used in the GAN. A larger GAN requires more processing resources from the client device 102 than a GAN with a smaller network size. The majority of GAN plugins 104 comprise GANs of varying sizes. As a result, each of the GAN plugins 104 has different requirements for processing resources such as memory, CPU, etc. Furthermore, since each GAN plugins 104 has a different network size, each of the GAN plugins 104 is capable of transforming inputs of different sizes.For example, in some embodiments, the largest GAN plug-in, which has the highest resource requirements, is capable of converting a 144-pixel (144p) video to a 1080p video, while the smallest GAN plug-in, which has the lowest resource requirements, is only capable of converting a 720p video to a 1080p video. The size or resolution quality of the required input is thus inversely proportional to the size of the GAN plug-in network (i.e., the smaller the GAN plug-in, the higher the pixel or video quality required to convert to a desired resolution).

[0017] In some embodiments, there are five GAN plug-ins from which the client device 102 selects. In some embodiments, the first or smallest GAN plug-in is configured to convert a 720p video to 1080p. The second or next GAN plug-in is configured to convert a 480p video to 1080p. The third GAN plug-in is configured to convert a 360p video to 1080p. The fourth GAN plug-in is configured to convert a 244p video to 1080p. The fifth or largest GAN plug-in is configured to convert a 144p video to 1080p. It should be understood that the video quality / sizes used above are provided for purposes only, and that other input video sizes, as well as output video sizes, may be used in other embodiments. Furthermore, it is understood that in other embodiments, more than 5 or fewer than 5 GAN plugins of variable sizes may be used.

[0018] The client device 102 further includes a resource monitor 106. The resource monitor 106 is configured to monitor the processing capability of the client device 102. That is, the resource monitor 106 tracks the processes and applications being executed and the amount of resources used (e.g., CPU, memory, bus bandwidth, etc.). Based on the amount of resources used and the total resource capacity of the client device 102, the resource monitor 106 can determine the amount of resources available for use by one of the GAN plug-ins 104. By comparing the amount of resources available for use and the respective amounts of resources required by each of the GAN plug-ins 104, the client device 102 selects one of the GAN plug-ins 104 to process the media content received from the server 110.

[0019] After selecting one of the GAN plug-ins 104, the client device 102 estimates the minimum video resolution required for the selected GAN plug-in 104 to convert the video to the desired resolution quality. As discussed above, each of the GAN plug-ins 104 has a minimum video resolution quality required as input for converting to the desired output resolution quality. Based on the selected GAN plug-in 104 and the desired output quality, the client device 102 can thus estimate the minimum required input resolution quality. The client device 102 exchanges data with the server 110 to instruct the server 110 to send the video file at the minimum input resolution determined by the client device 102.The video file is received at the requested minimum input resolution, and the selected GAN plug-in 104 processes the received video file to convert it to the desired resolution quality for playback / viewing by the user.

[0020] It should be understood that the conversion using the selected GAN plug-in 104 may occur in real time while receiving the video. For example, in some embodiments, the user consumes or watches the video file while the video file is being streamed from the server 110. However, in some other embodiments, the video file may be completely downloaded or received from the server 110 before the user views the video file. For example, the user may specify a time at which the user wishes to view the video file. In both scenarios, the embodiments described herein reduce the amount of bandwidth required to transmit the video file from the server 110 to the client device 102 because the video file may be transmitted at a resolution quality that is lower than the output resolution quality viewed by the user.Because the lower resolution quality results in a smaller file size, the amount of bandwidth required to transfer the video file is reduced.

[0021] Furthermore, in some embodiments, the available resources of client device 102 are not consistently the same or static throughout the process of receiving the video file from server 110. For example, client device 102 may begin to use more resources for other applications or background processes compared to when media file 112 was originally requested from server 110, thereby decreasing the amount of resources available to GAN plug-ins 104. Alternatively, client device 102 may begin to use fewer resources for other applications or processes compared to when media file 112 was originally requested from server 110, thereby increasing the amount of resources available to GAN plug-ins 104.Thus, in some embodiments, resource monitor 106 updates the amount of resources available for use by GAN plug-ins 104 when the amount of resources consumed by other processes / applications changes. Client device 102 is then able to select a different GAN plug-in 104 based on the changed available resources and exchanges data with server 110 to change the appropriate resolution of the media file 112 currently being transmitted to client device 102.

[0022] In some embodiments, resource monitor 106 further maintains a historical log of resource usage data, which may include various factors, such as, but not limited to, which processes / applications are currently running; the average length of time and / or maximum / minimum length of time each process / application is running; the average, maximum, and / or minimum amount of each processing resource consumed by each of the processes / applications; and so on. In this way, client device 102 can predict the amount of available resources for a given period of time. For example, in some implementations, client device 102 does not know how large the media file 112 is or how long the video is.However, the client device 102 may use the historical data to predict the available resources for a specific period of time, for example, but not limited to, a period of 45 seconds. Periods may be measured, for example, in seconds, minutes, etc.

[0023] The total time for receiving the media file 112 can thus be divided into a plurality of consecutive time periods. The time periods into which the total time is divided can be of equal length or they can be periods of different sizes. For each time period, the client device 102 can predict the resources available for that time period, select a GAN plug-in based on the predicted available resources, and exchange data with the server 110 to adjust the resolution quality of the media file 112 for that time period so that the appropriate resolution quality of the media file 112 is transmitted during each time period.In this way, the client device 102 can adjust the input resolution quality of the media file 112 received during the process of receiving the media file 112, as well as adjust the GAN plug-in 104 used to adapt to changes in available resources, while still providing a relatively consistent output resolution quality to the user. For example, a background process continues to collect data about resource usage over time. If more resources are available or predicted to be available after an initial period, the client device 102 can upgrade to a larger GAN plug-in 104 for the next subsequent period. Similarly, if fewer resources are available or predicted to be available after an initial period, the client device 102 can downgrade to a smaller GAN plug-in 104 for the next subsequent period.

[0024] In some embodiments, each of the GAN plug-ins 104 may be pre-trained and, after training, loaded into the client device 102. The GAN plug-ins 104 may further be implemented as plug-ins for an Internet browser application or for other applications or software programs capable of streaming and / or otherwise receiving media files 112 from a server 110. Furthermore, it should be understood that the GAN plug-ins 104 and the resource monitor 106 are referred to as Fig. 1 as separate units. However, it should be understood that the functions described above with respect to the GAN plug-ins 104, the client unit 102 and the resource monitor 106 may be implemented in different ways than those described above in the example of Fig. 1 may be performed by a single unit and / or distributed among multiple units. In some embodiments, the functionality of monitoring and predicting available resources may be performed by a portion of the selected GAN plug-in 104 executing during a particular period of time.

[0025] Fig. 2 is a block diagram of one embodiment of a computing device 200 configured to implement the functionality of the client device 102, including the GAN plug-ins 104 and resource monitoring 106. The Fig. The components of the data processing unit 200 shown in Figure 2 include one or more processors 202, a memory 204, a memory interface 216, an input / output ("I / O") unit interface 212, and a network interface 218, all of which are connected in a data-exchange-capable manner directly or indirectly for exchanging data via a memory bus 206, an I / O bus 208, a bus interface unit (interface, "IF") 209, and an I / O bus interface unit 210.

[0026] In the Fig. 2, the computing unit 200 includes one or more general-purpose programmable central processing units (CPUs) 202A and 202B, generally referred to herein as processor 202. In some embodiments, the computing unit 200 includes multiple processors. However, in other embodiments, the computing unit 200 is a single CPU system. Each processor 202 executes instructions stored in memory 204. While the embodiments are described with reference to central processing unit chips, it should be understood that the embodiments described herein are also applicable to a computer system that utilizes digital signal processors (DSPs) and / or graphics processing unit (GPU) chips in addition to or in place of CPU chips.Therefore, reference to a processor or processing unit herein may refer to CPU chips, GPU chips, and / or a DSP.

[0027] In some embodiments, memory 204 comprises a random-access semiconductor memory, a storage device, or a storage medium (either volatile or non-volatile) for storing or encoding data and programs. For example, memory 204 stores instructions 211, resource data 213, and a plurality of GAN plug-ins 215. When executed by a processor such as processor 202, instructions 211 cause processor 202 to perform the functions and computations discussed herein regarding monitoring, collecting resource data 213, predicting resource usage, selecting one of the GAN plug-ins 215, and executing the selected GAN plug-in 215 to transform a video quality of a received input media file.

[0028] In some embodiments, memory 204 corresponds to the entire virtual memory of computing device 200 and may also include the virtual memory of other computing devices connected to computing device 200 over a network. In some embodiments, memory 204 is a single monolithic device, but in other embodiments, memory 204 includes a hierarchy of caches and other storage devices. For example, memory 204 may exist in multiple levels of caches, and these caches may be further subdivided by function, such that one cache contains instructions while another contains non-instruction data used by the processor.The memory 204 may further be distributed and belong to different processing units or groups of processing units, as is known, for example, in any computer architecture with non-uniform memory access, so-called non-uniform memory access (NUMA) computer architectures. In the embodiment shown in . Fig. 2, for explanatory purposes, the instructions 240, resource data 213, and GAN plug-ins 215 are stored on the same memory 204, but it should be understood that other embodiments may be implemented in different ways. For example, the instructions 240, resource data 213, and / or GAN plug-ins 215 may be distributed across multiple physical media.

[0029] In the Fig. In the embodiment shown in Figure 2, the data processing unit 200 also includes a bus interface unit 209 for handling data transfers between the processor 202, the memory 204, the display system 224, and the I / O bus interface unit 210. The I / O bus interface unit 210 is connected to the I / O bus 208 for transferring data to and from the various I / O units. In particular, the I / O bus interface unit 210 can exchange data with a plurality of I / O interface units 212, 216, and 218, also referred to as I / O processors (IOPs) or I / O adapters (IOAs), via the I / O bus 208. The display system 224 includes a display controller, a display memory, or both. The display controller may provide video, still images, audio, or a combination thereof to a display unit 226. The display memory may be a dedicated memory for buffering video data.The display system 224 is connected to the display unit 226. In some embodiments, the display unit 226 also includes one or more speakers for playing audio. Alternatively, one or more speakers for playing audio may be connected to an I / O interface unit. In alternative embodiments, one or more functions provided by the display system 224 are integrated into an integrated circuit that also includes the processor 202. Furthermore, in some embodiments, one or more functions provided by the bus interface unit 209 are integrated into an integrated circuit that also includes the processor 202.

[0030] The I / O interface units support data transfer with a variety of memory and I / O units. For example, the I / O unit interface unit 212 supports the connection of one or more user I / O units 220, which may include user output units and user input units (such as a keyboard, mouse, keypad, touchpad, trackball, buttons, light pen, or other pointing units). A user may operate the user input units using a user interface to provide input data and commands to the user I / O unit 220. In addition, a user may receive output data via the user output units. A user interface may be provided via the user I / O unit 220, for example, displayed on a display unit or played back through a speaker.

[0031] The storage interface 216 supports the connection of one or more storage devices 228, for example, flash memory. The contents of the memory 204, or any portion thereof, can be stored to and retrieved from the storage device 228 as needed. The network interface 218 provides one or more data transmission paths from the computing device 200 to other digital devices and computing devices. In some embodiments, for example, the computing device 200 can exchange data with a server, such as the server 110, to request and receive a media file via the network interface 218. The computing device 200 can further exchange data with the server to specify a desired resolution quality of the media file via the network interface 218.

[0032] The Fig. While the computing device 200 shown in Figure 2 illustrates a particular bus structure that provides a direct data transmission path between the processors 202, the memory 204, the bus interface unit 209, the display system 224, and the I / O bus interface unit 210, in alternative embodiments, the computing device 200 includes other buses or data transmission paths that may be arranged in any of a variety of forms, such as point-to-point connections in hierarchical, star, or mesh configurations, multiple hierarchical buses, parallel and redundant paths, or any other suitable configuration. While the I / O bus interface unit 210 and the I / O bus 208 are each shown as individual units, the computing device 200 may further include multiple I / O bus interface units 210 and / or multiple I / O buses 208 in other embodiments.While multiple I / O interface units are shown separating the I / O bus 208 from various data transmission paths passing through the various I / O units, in other embodiments, some or all of the I / O units are directly connected to one or more I / O buses of the system.

[0033] As discussed above, in some embodiments, one or more of the Fig. 2 include instructions or statements executed on the processor 202, or instructions or statements interpreted by the instructions or statements executed on the processor 202 to perform the functions described herein. In other embodiments, one or more of the components and data shown in Fig. 2 are implemented in hardware via semiconductor devices, chips, logic gates, circuits, circuit boards, and / or other physical hardware units instead of or in addition to a processor-based system. In other embodiments, furthermore, some of the components shown in Fig. 2 components may be omitted and / or other components may be included.

[0034] Fig.3 is a flowchart illustrating one embodiment of an example method 300 for reducing bandwidth consumption via generative adversarial networks (GANs). The method 300 may be implemented by a computing device, such as the computing device 200 discussed above. It should be understood that the order of steps in the example method 300 is provided for explanatory purposes, and that the method may be performed in a different order in other embodiments. Similarly, it should be understood that some steps may be omitted or additional steps may be included in other embodiments.

[0035] At 302, a plurality of GAN plugins are created and trained. As discussed herein, each of the plurality of GAN plugins may utilize a different network size and have different minimum input resolution requirements to achieve a selected output resolution. Each of the GAN plugins may further have different processing resource requirements such as memory, CPU, etc., from the other GAN plugins. In some embodiments, five GAN plugins are created, each having a different network size. Each of the GAN plugins may be pre-trained before being loaded into the computing device. In some embodiments, the GAN plugins may further be internet browser plugins.In other embodiments, the GAN plug-ins may be standalone applications and / or may be plug-ins that are included in other programs capable of retrieving media data from a server.

[0036] At 304, a desired resolution quality of a media file (e.g., image or video file) is determined. For example, a user may select a media file to be retrieved from a server. Such servers may include, but are not limited to, a video sharing website, a social media platform, a File Transfer Protocol (FTP) server, or another remote location accessed via a local area network or a wide area network such as the Internet. In response to detecting the user's selection, in some embodiments, the user may be prompted with a question to select a desired resolution quality of the media file. In other embodiments, the user may initially enter a desired default resolution quality for media files.In yet other embodiments, the system may learn from the user's responses to automatically determine, based on historical data, what resolution quality should be delivered to the user. The resolution quality may be specified using conventional notation, such as 1080p, 720p, etc.

[0037] At 306, the available resources of the user's computing device are determined. As discussed above, available resources may include, but are not limited to, CPU, memory, bus bandwidth, etc. As used herein, available resources refer to those resources that are not claimed or reserved by another process or application and are thus available for use by a GAN plug-in to process and transform the received media file. In some embodiments, currently available resources are determined. In other embodiments, as discussed above, in addition to currently available resources, future available resources are predicted. For example, as discussed above, resource usage on the user's computing device may be monitored over time to build a record of historical data.Based on the historical data, in some embodiments, future available resources may be determined. In some embodiments, predicting future available resources further comprises predicting future available resources for a first period of time, and, prior to completing the first period of time, predicting second available resources for a second period of time. For example, as discussed above, the file size of the media file is unknown, and thus the time to download or receive the media file is divided into smaller periods of time, e.g., 45 seconds. The future available resources for the first 45 seconds may be predicted, and thereafter, the future available resources for the next 45 seconds may be predicted, and so on, until the media file is fully received.

[0038] At 308, one of a plurality of Generative Adversarial Network (GAN) plug-ins is selected based on the determined available resources of the computing device. As discussed above, each of the GAN plug-ins may have a different size. Each GAN plug-in may have different processing or resource requirements based on the size of the GAN. In some embodiments, the same GAN plug-in is used to process the entire media file. However, in other embodiments, multiple GAN plug-ins may be used. For example, in embodiments where the time to download or receive the media file is divided into smaller consecutive time periods, the computing device may select a GAN plug-in for each consecutive time period based on the currently available resources and / or predicted available resources for the respective time period.In the above example, with respect to the first and second time periods, for example, a first GAN plug-in may be selected for use during the first time period based on the predicted first future resources, and a second GAN plug-in may be selected for use during the second time period based on the predicted second future resources. If the first and second predicted future available resources are similar or the same, the same GAN plug-in may be selected (i.e., the first GAN plug-in and the second GAN plug-in are the same). However, if the first and second predicted available resources are different, different GAN plug-ins may be selected (i.e., the first GAN plug-in is different from the second GAN plug-in).

[0039] At 310, a second resolution quality for the media file is estimated based on the selected GAN plug-in. As discussed above, each GAN plug-in has a minimum resolution quality required to convert a media file to a desired resolution quality. For example, a smaller GAN plug-in requires a larger minimum input resolution quality than a larger GAN plug-in. The second resolution quality is, as discussed above, lower than the desired resolution quality, thus reducing the bandwidth required to receive the media file.

[0040] At 312, the computing device exchanges data with the server to receive the media file at the estimated second resolution quality. For example, the computing device may contact the server to request that the media file be transmitted at the second resolution quality. Furthermore, as discussed above, in some embodiments, different GAN plug-ins with different sizes may be used at different points in receiving the media file. If a new GAN plug-in is selected, the computing device may exchange data with the server to request the media file at a new resolution quality corresponding to the new GAN plug-in (e.g., a third resolution quality).In this way, if more computing unit resources become available, a larger GAN plug-in can be selected, and the media file can be received at a lower resolution quality than the desired or second resolution quality. Conversely, if fewer computing unit resources become available, a smaller GAN plug-in can be selected, which requires comparatively fewer resources than a larger GAN plug-in. The third resolution quality can thus be larger than the second resolution quality but still lower than the desired resolution quality, so that the bandwidth consumption is still reduced compared to receiving the media file at the desired resolution quality.

[0041] Furthermore, it should be understood that in some embodiments, the media file may be requested at a resolution quality that is not the minimum required resolution quality for the selected GAN plug-in. If the minimum resolution quality for a selected GAN plug-in is 240p, in some embodiments, the computing device may estimate a second resolution quality that is higher than the minimum resolution quality of 240p but lower than the desired resolution quality, for example, 360p or 480p.

[0042] At 314, the received media file is converted from the second resolution quality to the desired resolution quality using the selected GAN plug-in. As discussed above, in embodiments where multiple GAN plug-ins of different sizes are sequentially selected based on changes in available resources, the respective portions of the received media file are then converted using the corresponding GAN plug-in. The media file is converted to the desired resolution quality and output at 316 with the desired resolution quality. The converted media file may be displayed, for example, on a display screen of the computing device. Further, it should be understood that in some embodiments, the steps performed at 306 through 314 may be performed substantially concurrently in real time when the media file is output at 316.That is, while one portion of the media file is being output / displayed, another subsequent portion is received and converted, while the computing device monitors resource availability and adjusts the selected GAN as needed in response to detected changes in resource availability. In other embodiments, a user may specify a scheduled time to view the media file or download the media file without simultaneously playing the media file. In such embodiments, the conversion may occur before the media file is output.

[0043] The embodiments described herein thus enable the reduction of bandwidth consumption when retrieving a media file. This reduction in bandwidth consumption can increase user satisfaction when viewing the media file because problems such as buffer latency can be mitigated. The reduction in bandwidth consumption can also have other benefits, for example, in embodiments where bandwidth consumption is measured. However, despite the reduction in bandwidth and the reduced resolution quality of the retrieved media file, the output resolution quality still corresponds to the user's desired resolution quality.

[0044] The present invention may be a system, a method, and / or a computer program product at any possible level of integration of technical details. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to carry out aspects of the present invention.

[0045] The computer-readable storage medium may be any physical device that can retain and store instructions for use by an instruction-executing system. The computer-readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of the computer-readable storage medium includes the following: a removable computer diskette, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM).Flash memory), static random access memory (SRAM), removable compact disk read-only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, mechanically encoded device such as punched cards or raised structures in a groove on which instructions are stored, and any suitable combination thereof. A computer-readable storage medium, as used herein, shall not be construed as carrying transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., pulses of light traveling through an optical fiber cable), or electrical signals carried through a wire.

[0046] Computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to respective computing / processing units or to an external computer or storage unit via a network such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, fiber optic transmission lines, wireless transmission, routers, firewalls, switching units, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing unit receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium within the respective computing / processing unit.

[0047] Computer-readable program instructions for performing operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, integrated circuit configuration data, or either source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, or the like, as well as conventional procedural programming languages ​​such as the C programming language or similar programming languages.The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In the latter case, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, over the Internet using an Internet service provider).In some embodiments, electronic circuits, including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute the computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuits to perform aspects of the present invention.

[0048] Aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowcharts and / or block diagrams, as well as combinations of blocks in the flowcharts and / or block diagrams, may be implemented by computer-readable program instructions.

[0049] These computer-readable program instructions may be provided to a processor of a suitably configured computer, special purpose computer, or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce a means for implementing the functions / steps defined in the flowchart block(s) and / or block diagram(s).These computer-readable program instructions may also be stored on a computer-readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer-readable storage medium having instructions stored thereon comprises an article of manufacture, including instructions that implement aspects of the function / step specified in the block(s) of the flowchart and / or block diagrams.

[0050] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of process steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-executable process such that the instructions executing on the computer, other programmable apparatus, or other device implement the functions / steps defined in the block(s) of flowcharts and / or block diagrams.

[0051] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions comprising one or more executable instructions for performing the specified logical function(s). In some alternative implementations, the functions specified in the block may occur in a different order than shown in the figures. For example, two blocks shown in succession may actually execute substantially concurrently, or the blocks may sometimes execute in reverse order depending on the corresponding functionality.It is further understood that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by special purpose hardware-based systems that perform the specified functions or steps, or by combinations of special purpose hardware and computer instructions.

[0052] While specific embodiments have been illustrated and described herein, it will be apparent to one skilled in the art that any arrangement calculated to achieve the same purpose may be substituted for the specific embodiments shown. Therefore, it is clearly intended that this invention be limited only by the claims and their equivalents.

Claims

[1] A method implemented on a computer, comprising: Determining a desired resolution quality of a media file; Determining available resources of a data processing unit; Selecting one of a plurality of Generative Adversarial Network (GAN) plug-ins based on the determined available resources of the computing device; Estimating a second resolution quality of the media file corresponding to the selected GAN plug-in, wherein the second resolution quality is lower than the desired resolution quality; Exchanging data with a server to receive the media file in the estimated second resolution quality; Converting the received media file from the second resolution quality to the desired resolution quality using the selected GAN plug-in; and Output the converted media file with the desired resolution quality. [2] The method of claim 1, wherein the second resolution quality is a minimum resolution quality associated with the selected GAN plug-in. [3] The method of claim 1, wherein the plurality of GAN plug-ins comprises five GAN plug-ins, each of the plurality of GAN plug-ins having a different size from the other GAN plug-ins in the plurality of GAN plug-ins. [4] The method of claim 1, wherein the plurality of GAN plug-ins are Internet browser plug-ins that are pre-trained before being loaded into the computing device. [5] The method of claim 1, wherein determining available resources of a computing device comprises predicting future available resources of the computing device based on historical data; and wherein selecting one of the plurality of GAN plug-ins comprises selecting one of the plurality of GAN plug-ins based on the predicted future available resources. [6] The method of claim 5, wherein predicting future available resources of the computing unit comprises: Predicting initial future available resources of the data processing unit for a first period; and before completing the first period, predicting second future available resources of the data processing unit for a second period; wherein selecting one of the plurality of GAN plug-ins comprises: Selecting a first GAN plug-in to use during the first period based on the predicted first future available resources; and Selecting a second GAN plug-in to use during the second period based on the predicted second future available resources. [7] The method of claim 6, wherein the first GAN plug-in is different from the second GAN plug-in. [8] The method of claim 1, further comprising: Monitoring available resources of the computing device while receiving the media file from the server to detect a change in the available resources of the computing device; and in response to detecting the change in the available resources of the computing device, selecting a second GAN plug-in from the plurality of GAN plug-ins based on the detected change in the available resources of the computing device; estimating a third resolution quality of the media file corresponding to the selected second GAN plug-in, wherein the third resolution quality is lower than the desired resolution quality and different from the second resolution quality; Exchanging data with the server to receive the media file in the estimated third resolution quality; and Convert the received media file from the third resolution quality to the desired resolution quality using the selected GAN plug-in. [9] Data processing unit comprising: an interface connected to a server in a manner capable of exchanging data; and a processor connected to the interface in a data-exchange-capable manner, the processor being configured to: Determining a desired resolution quality of a media file; Determining available resources of the data processing unit; Selecting one of a plurality of Generative Adversarial Network (GAN) plug-ins based on the determined available resources of the computing device; Estimating a second resolution quality of the media file corresponding to the selected GAN plug-in, wherein the second resolution quality is lower than the desired resolution quality; Exchanging data with the server via the interface to request the media file in the estimated second resolution quality; Converting the media file received from the server in the second resolution quality to the desired resolution quality using the selected GAN plug-in; and Output the converted media file with the desired resolution quality. [10] The data processing unit of claim 9, wherein the second resolution quality is a minimum resolution quality associated with the selected GAN plug-in. [11] The computing device of claim 9, wherein the processor is configured to predict future available resources of the computing device based on historical data; and wherein the processor is configured to select one of the plurality of GAN plug-ins based on the predicted future available resources. [12] The data processing unit of claim 9, wherein the processor is configured to: Predicting initial future available resources of the data processing unit for a first period; and before completing the first period, predicting second future available resources of the data processing unit for a second period; Selecting a first GAN plug-in to use during the first period based on the predicted first future available resources; and Selecting a second GAN plug-in to use during the second period based on the predicted second future available resources. [13] The data processing unit of claim 12, wherein the first GAN plug-in is different from the second GAN plug-in. [14] The data processing unit of claim 9, wherein the processor is further configured to: Monitoring available resources of the computing device while receiving the media file from the server to detect a change in the available resources of the computing device; and in response to detecting the change in the available resources of the computing device, selecting a second GAN plug-in from the plurality of GAN plug-ins based on the detected change in the available resources of the computing device; estimating a third resolution quality of the media file corresponding to the selected second GAN plug-in, wherein the third resolution quality is lower than the desired resolution quality and different from the second resolution quality; Exchanging data with the server to request the media file in the estimated third resolution quality; and Convert the media file received from the server in the third resolution quality to the desired resolution quality using the selected second GAN plug-in. [15] A computer program product comprising a computer-readable storage medium having stored thereon a computer-readable program, the computer-readable program, when executed by a processor, causing the processor to: to determine a desired resolution quality of a media file; to determine available resources of a data processing unit; select one of a plurality of Generative Adversarial Network (GAN) plug-ins based on the determined available resources of the computing device; determine a second resolution quality of the media file corresponding to the selected GAN plug-in, wherein the second resolution quality is lower than the desired resolution quality; Exchange data with a server to request the media file in the estimated second resolution quality; convert the media file received from the server in the second resolution quality to the desired resolution quality by using the selected GAN plug-in; and output the converted media file with the desired resolution quality. [16] The computer program product of claim 15, wherein the second resolution quality is a minimum resolution quality associated with the selected GAN plug-in. [17] The computer program product of claim 15, wherein the computer-readable program is further configured to cause the processor to: predict future available resources of the data processing unit based on historical data; and to select one of the plurality of GAN plugins based on the predicted future available resources. [18] The computer program product of claim 15, wherein the processor is configured to: Predicting initial future available resources of the data processing unit for a first period; and before completing the first period, predicting second future available resources of the data processing unit for a second period; Selecting a first GAN plug-in to use during the first period based on the predicted first future available resources; and Selecting a second GAN plug-in to use during the second period based on the predicted second future available resources. [19] The computer program product of claim 18, wherein the first GAN plug-in is different from the second GAN plug-in. [20] The computer program product of claim 15, wherein the computer-readable program is further configured to cause the processor to: monitor available resources of the computing device while receiving the media file from the server to detect a change in the available resources of the computing device; and in response to detecting the change in the available resources of the computing device, select a second GAN plug-in from the plurality of GAN plug-ins based on the detected change in the available resources of the computing device; estimate a third resolution quality of the media file corresponding to the selected second GAN plug-in, wherein the third resolution quality is lower than the desired resolution quality and different from the second resolution quality; exchange data with the server to receive the media file in the estimated third resolution quality; and convert the received media file from the third resolution quality to the desired resolution quality by using the selected GAN plug-in.