Reinforcement learning in real-time communication

Abstract

The agent interacts with the sending and receiving computing devices to automatically adjust parameters of one-way or two-way real-time audio and real-time video transmissions in response to changing network conditions and / or application demands. The agent incorporates a reinforcement learning model that adjusts transmission parameters to maximize the expected total future reward. The expected total future reward is based on the current state of the sending computing device, the current behavior at the sending computing device (e.g., the current set of transmission parameters), and the reward provided by the receiving computing device. The reward represents the user's perceived quality of experience at the receiving computing device.

Description

[Technical field]

[0001] Reinforcement learning in real-time communication. [Background technology]

[0002]

[0001] Bandwidth estimation, congestion control, and video quality optimization for real-time communications (e.g., audiovisual conferencing) remain challenging problems due to frequently changing network conditions and application requirements. Delivering real-time media with high quality and reliability (e.g., end-user quality of experience) requires continuous updates to react to new application requirements and network behavior. The process of continuous updates can be a slow process that results in a degradation of the end-user experience.

[0003]

[0002] It is with respect to these and other general considerations that various aspects of the present disclosure have been made. Moreover, while relatively specific problems and examples of solving these problems are discussed herein, it should be understood that the examples should not be limited to solving any particular problem identified in the context of this disclosure or elsewhere. Summary of the Invention

[0004]

[0003] The present disclosure is generally directed to systems and methods for implementing reinforcement learning in real-time communications. A particular feature of the present disclosure is directed to reinforcement learning for optimizing user-perceived quality in real-time audio and video communications. An agent interacts with a sending computing device and a receiving computing device to automatically adjust parameters of the real-time audio and video transmission in response to changing network conditions and / or application requirements. The sending computing device sends real-time audio and / or video data. The receiving computing device receives the real-time audio and video transmission from the sending device and determines an actual user-perceived quality of experience (QoE), which is provided to the agent as a reward. The agent incorporates a reinforcement learning model that includes a control policy and a state-action value function. The agent observes the current state of the sending computing device and determines an estimate of the expected total future reward based on the current state, a current action (e.g., a current adjustment or set of adjustments made to transmission parameters at the sending computing device), and the reward provided by the receiving computing device. The agent adjusts a control policy based on the goal of maximizing the expected total future reward. The adjustments in the control policy change the actions applied to the real-time audio and / or video data.

[0005]

[0004] Aspects of the present disclosure are directed to methods, systems, and articles of manufacture for optimizing expected user-perceived QoE in real-time communications. The features include determining a current state of a sending computing device and a current behavior of the sending computing device, where the current behavior includes a plurality of transmission parameters. The features further include sending a real-time communication from the sending computing device to a receiving computing device. The real-time communication includes one or both of a real-time audio communication and a real-time video communication. Further, a reward, e.g., a QoE metric, is determined at the receiving computing device based on one or more parameters of the real-time communication transmitted and received at the receiving computing device. An expected value of a sum of future rewards is determined based on the current state, the current behavior, and the reward, and at least one of the plurality of transmission parameters of the sending computing device is modified to maximize the expected value of the sum of future rewards.

[0006]

[0005] A feature of the present disclosure is directed to a method, system, and article of manufacture for training a reinforcement learning model to optimize expected user-perceived QoE in real-time communications. The feature includes determining a current state of a sender and providing the current state to an agent in communication with the sender. The feature further includes determining a current behavior of the sender, the current behavior being known to the agent and including a plurality of transmission parameters. The feature further includes sending a real-time communication from the sender to a receiver. The real-time communication includes one or both of a real-time audio transmission and a real-time video transmission. The feature further includes receiving, at the agent, a reward from the receiver, the reward being determined at the receiver. The reward is based on one or more parameters associated with the real-time communication received at the receiver. The agent determines an expected value of a sum of future rewards based on the current state, the current behavior, and the reward, and directs a change in at least one of the plurality of transmission parameters to maximize the expected value of the sum of future rewards. The training can be performed in a simulated environment, an emulated environment, or an actual network environment.

[0007]

[0006] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Additional characteristics, features, and / or advantages of the examples will be set forth in part in the following description, and in part will be apparent from the description, or may be learned by the practice of the disclosure.

[0008]

[0007] Non-limiting and non-exhaustive examples are described with reference to the following drawings. [Brief description of the drawings]

[0009] [Figure 1]FIG. 1 illustrates an environment in which reinforcement learning in real-time communication as disclosed herein can take place. [Figure 2A] 2A-2C provide additional details of the environment in which reinforcement learning in real-time communication herein can take place. [Figure 2B] 2A-2C provide additional details of the environment in which reinforcement learning in real-time communication herein can take place. [Figure 2C] 2A-2C provide additional details of the environment in which reinforcement learning in real-time communication herein can take place. [Diagram 3] FIG. 3 illustrates a simulated training environment for reinforcement learning to maximize user-perceived Quality of Experience (QoE) in real-time communications. [Figure 4] FIG. 4 shows an emulated training environment for reinforcement learning to maximize user-perceived QoE in real-time communications. [Diagram 5] Fig. 5 shows a real network training environment for reinforcement learning to maximize user-perceived QoE in real-time communications. [Figure 6] FIG. 6 is a block diagram illustrating example physical components of a computing device that can be used to implement features of the present disclosure. [Figure 7A] 7A and 7B are simplified block diagrams of a mobile computing device that can be used to implement features of the present disclosure. [Figure 7B] 7A and 7B are simplified block diagrams of a mobile computing device that can be used to implement features of the present disclosure. [Figure 8] FIG. 8 is a simplified block diagram of a distributed computing system in which features of the present disclosure can be implemented. [Figure 9] FIG. 9 illustrates a tablet computing device for carrying out one or more features of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010]

[0017] Various features of the present disclosure will now be described more fully with reference to the accompanying drawings, which are a part of the disclosure. Other features of the present disclosure may be implemented in many different forms and should not be construed as being limited to the features set forth herein. Rather, the features, when provided, will be so thorough and complete that they will fully convey the scope of the features to those skilled in the art. The features may be implemented as a method, system, or device. Thus, the features may take the form of a hardware implementation, an all-software implementation, or an implementation that combines software and hardware features. Thus, the following detailed description should not be taken in a limiting sense.

[0011]

[0018] The present disclosure is generally directed to systems and methods for implementing reinforcement learning in real-time communications. A particular feature of the present disclosure is directed to reinforcement learning for optimizing user-perceived quality in real-time audio and video communications. An agent interacts with a sending computing device and a receiving computing device to automatically adjust parameters of the real-time audio and video transmission in response to changing network conditions and / or application requirements. The sending computing device transmits real-time audio and / or video data. The receiving computing device receives the real-time audio and video transmission from the sending device and determines an actual user-perceived quality of experience (QoE), which is provided as a reward to the agent. The agent incorporates a reinforcement learning model that includes a control policy and a state-action value function. The agent observes the current state of the sending computing device and determines an estimate of the expected value of the total future reward based on the current state, a current action (e.g., a current adjustment or set of adjustments made to transmission parameters at the sending computing device), and a reward provided by the receiving computing device. Based on the goal of maximizing the expected value of the sum of future rewards, the agent adjusts the control policy. Adjustments in the control policy change the behavior applied to the real-time audio and / or video data.

[0012]

[0019] Thus, the present disclosure provides multiple technical advantages, including, but not limited to, a continuously updated reinforcement learning model that responds instantaneously to adjust parameters of a sending computing device's real-time audio and video transmission based on the goal of maximizing expected value of a sum of future rewards. The parameters of the real-time audio and video transmission are instantly adjusted to accommodate changing network conditions and / or application demands. Degradation of the transmitted real-time audio and video streams that may occur under a process of hand-coded reactive updates to data transmission parameters previously used to combat degradation is minimized.

[0013]

[0020] 1, an environment 100 for implementing reinforcement learning in real-time communication is shown. The environment 100 includes a network 102 through which multiple computing devices 104 communicate via various communication links 106. The term "real-time" refers to data processing in which received data is processed by the computing devices nearly instantly, e.g., a level of computing device responsiveness that a user perceives as sufficiently immediate or that allows the computing device to perform some external process without delay.

[0014]

[0021] Network 102 is any type of wired and / or wireless network capable of transmitting, receiving, and switching data, voice, and video traffic. Examples of networks include local area networks (LANs) that interconnect endpoints in a domain, wide area networks (WANs) that interconnect multiple LANs, as well as subnetworks, metropolitan area networks, storage area networks, personal area networks (PANs), wireless local area networks (WLANs), campus area networks (CANs), virtual private networks (VPNs), passive optical networks, etc.

[0015]

[0022] Computing device 104 comprises an endpoint of network 102. Computing device 104 may include one or more general purpose or application specific computing devices. Such devices may include, for example, personal computers, server computers, handheld or laptop devices, multiprocessor systems, microcontroller based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, cell phones, personal digital assistants (PDAs), gaming devices, printers, appliances, media centers, computing devices embedded or attached to vehicles, other mobile devices, distributed computing environments including any of the above systems and devices, etc. Further details regarding computing devices are described in Figures 6-9.

[0016]

[0023] Communications between computing devices 104 travel through links 106. Links can include any type of guided or unguided transmission medium capable of transmitting data, voice, and / or video from one computing device 104 to another. Guided media transmit signals along a physical path. Examples of guided media include twisted pair cable, coaxial cable, optical fiber, etc. Unguided media transmit signals without the use of a physical means to define the path taken by the signal. Examples of unguided media include radio waves, microwaves, infrared waves, etc.

[0017]

[0024] For illustrative purposes, FIG. 2A and FIG. 2B show an environment 200 including one sender computing device 204S and one receiver computing device 204R communicating in real time over the network 102 via the link 106. Although the sender computing device 204S is shown as including only transmission capabilities, it should be appreciated that the sender computing device 204S can also operate as a receiver computing device. Similarly, the receiver computing device 204R can also operate as a sender computing device. Thus, bidirectional real-time communication can occur between the sender computing device 204S and the receiver computing device 204R. The environment 200 communicates in real time with an agent 206 to implement reinforcement learning based on real-time communication of data, which may include voice and video data. Reinforcement learning optimizes the expected user-perceived quality of real-time communication by maximizing the expected value of the sum of future rewards. The agent 206 may include code or applications that reside on one or both of the sending computing device 204S and the receiving computing device 204R. The agent 206 may also include code or applications that reside on a computing device other than the sending computing device 204S and the receiving computing device 204R, such as a server computing device, a cloud computing device, etc.

[0018]

[0025] As shown, the sending computing device 204S includes a data capture module 210, a data encoder module 212, and a data transmitter module 214. The data capture module 210 captures state data representing a current observed state of the sending computing device 204S. For real-time audio and video communications, the current observed state may include observed sending parameters that affect the transmission of real-time audio data and real-time video data. The observed sending parameters may include, for example, resolution, bit rate, frame rate, streams-to-send, codec (encode / decode), the user's physical environment (e.g., dark / light levels, background noise, motion, etc.), or any other parameters that may affect real-time data transmission. The data encoder module 212 of the sending computing device 204S converts the state data into a specified format for real-time transmission over the network 102. The data transmitter module 214 transmits the formatted status data in real time over the network 102 .

[0019]

[0026] The receiving computing device 204R includes a data receiver module 220, a data decoder module 222, and a QoE metric module 224. The data receiver module 220 receives formatted state data from the network 102 in real time and outputs network statistics to the agent 206. Examples of network statistics include loss, jitter, round trip time (RTT) also known as network delay, receive rate, packet size, packet type, receive timestamp, transmit timestamp, burst length in packet loss, gap between packet losses, or any other network statistic that can be used to assess the quality of the received audio and video data. The data decoder module 222 does the reverse of the data encoder module 212 and extracts the received state data from the formatted state data in real time.

[0020]

[0027] The QoE metric module 224 determines one or more quality of experience (QoE) metrics based on the extracted state data. The QoE metrics represent a user-perceived quality of the received state data as determined by a QoE machine learning model, such as a deep neural network (DNN) or other suitable model. The QoE machine learning model analyzes various reception parameters, such as the payload of the received audio and video data streams, where the payload is the portion of the received data that is the actual intended message. The analysis of the payload of the audio and video streams can include the use of one or more predefined objective models that approximate the outcome of a subjective quality assessment (e.g., a human observing a rating of quality). In a particular example, the objective model can include one or more of models for assessing real-time audio quality (e.g., a Perception Evaluation of Audio Quality (PEAQ) model, a PEMO-Q model, a Signal-to-Noise Ratio (PSNR) model, or any other objective model capable of assessing a received real-time audio signal). In certain examples, the objective model may include one or more of models for assessing real-time video quality (e.g., a Full Reference (FR) model, a Reduced Reference (RR) model, a No-Reference (NR) model, a Peak Signal-to-Noise Ratio (PSNR) model, a Structural Similarity Index (SSIM) model, or any other objective model capable of assessing a received real-time video signal).

[0021]

[0028] In certain features, the QoE machine learning model may further analyze the statistics of the receiving computing device 204R and network statistics as reception parameters to determine one or more of the QoE metrics described above. As described herein, examples of network statistics include loss, jitter, round trip time (RTT), also known as network delay, reception rate, packet size, packet type, reception timestamp, transmission timestamp, burst length in packet loss, gap between packet losses, or any other network statistic that may be used to assess the quality of the received audio and video data. Examples of statistics of the receiving computing device 204R include display size, display window size, device type, whether a hardware or software encoder / decoder is used, etc. In certain configurations, the QoE machine learning model may further analyze user (e.g., human) feedback as reception parameters to determine one or more of the QoE metrics described above. User feedback may be provided, for example, through a user rating or survey to indicate a personal quality of experience, for example, what the user thinks about the quality of the audio and video received at the receiving computing device 204R. The determined one or more QoE metrics, which are indicative of the user-perceived quality of the audio and / or video, are communicated to the agent 206 .

[0022]

[0029] The agent 206 includes a state module 230 and a reinforcement learning model 232. In a particular aspect, the reinforcement learning model 232 can incorporate any suitable machine learning algorithm (a learning algorithm in which an action is taken, an outcome is observed, and a subsequent action takes into account the outcome of the first action based on a reward signal). The machine learning algorithm can include, for example, actor-critic, q-learning, policy gradient, temporal difference, Monte-Carlo tree search, or any other machine learning algorithm appropriate for the data involved. The reinforcement learning model 232 actively controls data transmission parameters of the sending computing device 204S in real time.

[0023]

[0030] FIG. 2B shows an example of an actor-critic reinforcement learning model 232, which includes a control policy 234 and a state-action value function 236, and FIG. 2C provides an example of an actor-critic architecture. Actor-critic reinforcement learning is a temporal difference learning model, where the control policy 234 is independent of the expected state value function 236, which in this context is the expected value of the sum of future rewards. The control policy 234 includes an actor because it is used to select actions, e.g., data transmission parameters of a sending computing device, and the state value function 236 is a critic because it criticizes the actions taken by the control policy 234. The state value function 236 learns and criticizes the current control policy 234.

[0024]

[0031] The control policy 234 includes a first reinforcement learning model within the agent 206, such as a neural network, that produces one or more output actions in the form of one or more changes to one or more of the data transmission parameters used by the sending computing device 204S. The output actions are designed to optimize the expected user-perceived quality (QoE) of the audio and video data based on maximizing the expected value of future rewards determined by the state-action value function 236. Examples of transmission parameters include transmission rate, resolution, frame rate, object event provided quantization parameter (QP), forward error correction (FEC), or any other controllable parameter usable to change the quality of the transmission of state data from the sending computing device 204S to the receiving computing device 204R.

[0025]

[0032] The state-action value function 236 includes a second machine learning model within the agent 206, such as a neural network, that is trained to predict or estimate an expected value of the sum of future rewards. The expected value of the sum of future rewards is determined based on the current state of the sending computing device, the current behavior (e.g., current transmission parameters used to transmit real-time audio and / or video data), and the reward provided by the receiving computing device. The control policy adjusts the output behavior in response to the determination of the expected value. The control policy 234 can be trained together with the state-action value function 236 or can be derived based on an already trained state-action value function 236.

[0026]

[0033] In certain aspects, during training of the actor-critic reinforcement learning model 232 of FIGS. 2B-2C, the agent 206 need not always follow the actions of the control policy 234. Rather, the agent 206 can explore other actions (e.g., other changes to the data transmission parameters of the sending computing device 204S), which allows the agent 206 to improve the reinforcement learning model 232. The agent 206 can explore other actions through one or more exploration strategies, e.g., epsilon-greedy.

[0027]

[0034] In certain aspects, the control policy 234 of the reinforcement learning model 232 can be decoupled from its learning environment and deployed as a real-time model on a client (e.g., a sending computing device and / or a receiving computing device). The transfer to a real-time model can be accomplished through one or more model transport tools, such as ONNX (Open Neural Network Exchange), tflite (TensorFlow Lite), etc.

[0028]

[0035] 3-5, the agent 206 can be trained using one or more of the simulated environment 300, the emulated environment 400, and the real network environment 500. The environment to be used depends on the requirements of the data collection rate and the type of data. In the simulated environment 300 of FIG. 3, all the processes of the sending computing device 204S (including the processes of the data capture module 210, the data encoder module 212, and the data transmitter module 214), all the processes of the receiving computing device 204R (including the processes of the data receiver module 220, the data decoder module 222, and the QoE metric module 224), and the network 202 are simulated. In the emulated environment 400 of FIG. 4, the sender computing device 204S is replicated in a first emulation including an emulated sender process 404S, the receiver computing device 204R is replicated in a second emulation including an emulated receiver process 404R, and the network 202 is replicated in a third emulation including a network emulation 402. In certain aspects, a physical sender computing device and a physical receiver computing device may be used in conjunction with the emulated network. In the actual network environment of FIG. 5, a physical sender computing device 204S, a physical receiver computing device 204R, and a physical network 202 are used.

[0029]

[0036] The environment used to train the agent 206 depends on the requirements regarding the speed of data collection and the type of data. For example, for fast data collection and training, network simulation tools such as ns-2 and ns-3 (which are discrete event network simulators) can be used in the simulated environment 300. To allow real code to be executed in a controlled environment, network emulation tools such as NetEm (which is an enhancement of the Linux® traffic control facility that allows the addition of delays, packet loss, duplications, and other characteristics of outgoing packets from selected network interfaces) can be used in the emulated environment 400. This controlled environment allows communication applications (e.g., Skype, Microsoft® Teams, WhatsApp, WeChat, etc.) to be tested in an environment with reproducible network conditions. Actual networks (e.g., cellular, Wi-Fi, Ethernet, etc.) using actual Internet Service Providers (ISPs) in the actual network environment 500 provide the most realistic test environment and allow online learning of the conditions experienced by end users. In certain aspects, the same reinforcement learning policies can be used in simulated, emulated, or actual network environments, but each environment will provide different performance. Alternatively or additionally, the agent 206 can be trained using transfer learning, in which previously created hand-coded rules associated with new application requirements and / or network behaviors for streaming real-time audio and video data are used to train the agent 206.

[0030]

[0037] After being trained, the agent 206 is applied in a live network environment for real-time audio and video communication. In the live network, the reinforcement learning model 232 is continuously updated based on the transmission of real-time audio and video data streams from a sending computing device, e.g., device 204S, to a receiving computing device 204R. In a particular aspect, the sending computing device, e.g., device 204S, can include an agent 206 or multiple agents 206 that operate to modify real-time audio and video data transmission parameters, each agent modifying only one data transmission parameter or multiple data transmission parameters. In a particular aspect, the receiving computing device, e.g., device 204R, can determine a QoE or multiple QoEs. The QoE or multiple QoEs can be provided to the agent 206 or multiple agents 206.

[0031]

[0038] Thus, rather than enduring degraded real-time audio and video communications, as may result in environments where only hand coding is used to respond to changing network conditions and / or changing application requirements, the agent 206 and the sender computing device 204S are updated instantly (i.e., in real-time) to continuously optimize the expected user-perceived quality of real-time audio and video communications through maximization of the expected sum of future rewards based on the agent's 206 continuous, live updates.

[0032]

[0039] 6-9 and the associated description provide a discussion of various operating environments in which features of the present disclosure may be implemented. However, the devices and systems shown and discussed with respect to FIGS. 6-9 are intended as examples and illustrations, and are not intended to limit the numerous configurations of computing devices that may be used to implement features of the present disclosure described herein.

[0033]

[0040] FIG. 6 is a block diagram illustrating the physical components (e.g., hardware) of a computing device 600 that may implement features of the present disclosure. The computing device components described below may have computer-executable instructions for implementing reinforcement learning to maximize a user-perceived QoE in real-time communications at a computing device (e.g., a sending computing device 204S and a receiving computing device 204R), including computer-executable instructions for a reinforcement learning application 620 that may be executed to implement the methods described herein. In a basic configuration, the computing device 600 may include at least one processing unit 602 and a system memory 604. Depending on the configuration and type of computing device, the system memory 604 may include, but is not limited to, volatile storage (e.g., random access memory), non-volatile storage (e.g., read-only memory), flash memory, or any combination of such memory. The system memory 604 may include an operating system 605 and one or more program modules 606, such as one or more of the components with respect to FIG. 2, such as a data capture, data encoder, and data transmitter module 611 (e.g., data capture module 210, data encoder module 212, and data transmitter module 214), a data receiver, data decoder, and QoE metric module 613 (e.g., data receiver module 220, data decoder module 222, and QoE metric module 224), and / or an agent module 615 (e.g., agent 206).

[0034]

[0041] The operating system 605 may be, for example, suitable for controlling the operation of the computing device 600. Additionally, embodiments of the present disclosure may be implemented in conjunction with a graphics library, other operating systems, or any other application program, and are not limited to any particular application or system. This basic configuration is illustrated in FIG. 6 by those components within dashed line 608. The computing device 600 may have additional features and functionality. For example, the computing device 600 may also include additional data storage devices (removable and / or non-removable), such as, for example, magnetic disks, optical disks, or tape. Such additional storage is illustrated in FIG. 6 by removable storage 609 and non-removable storage 610. Any number of program modules and data files may be stored in the system memory 604. When executing on the processing unit 602, the program modules 606 (e.g., reinforcement learning application 620) may perform processes including, but not limited to, those features described herein.

[0035]

[0042] Additionally, embodiments of the present disclosure may be implemented in electrical circuits including discrete electronic elements, packaged or integrated electronic chips including logic gates, circuits using a microprocessor, or a single chip including electronic elements or a microprocessor. For example, embodiments of the present disclosure may be implemented via a system-on-chip (SOC) in which each or many of the components shown in FIG. 6 are integrated into a single integrated circuit. Such an SOC device may include one or more processing units, graphics units, communication units, system virtualization units, and various application functions, all of which are integrated (or "embedded") into the chip substrate as a single integrated circuit. When operating via a SOC, the functionality described herein with respect to the client's ability to switch protocols may be operated via application specific logic integrated with other components of the computing device 600 on a single integrated circuit (chip). Embodiments of the present disclosure may also be implemented using other technologies capable of performing logical operations such as, for example, AND, OR, NOT, etc., including, but not limited to, mechanical, optical, fluidic, and quantum technologies. Furthermore, embodiments of the present disclosure may be implemented in a general purpose computer or in any other circuits or systems.

[0036]

[0043] The computing device 600 may also have one or more input device(s) 612, such as a keyboard, mouse, pen, sound or voice input device, touch or swipe input device, etc. Output device(s) 614, such as a display, speakers, printer, etc. may also be included. The above-mentioned devices are examples and others may be used. The computing device 600 may include one or more communication connections 616 that enable communication with other computing devices 650. Examples of suitable communication connections 616 include, but are not limited to, radio frequency (RF) transmitter, receiver, and / or transceiver circuitry, a universal serial bus (USB), a parallel port, and / or a serial port.

[0037]

[0044] The term computer readable medium as used herein may include computer storage media. Computer storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, and the like. System memory 604, removable storage 609, and non-removable storage 610 are all examples of computer storage media (e.g., memory storage). Computer storage media may include RAM, ROM, Electrically Erasable Read Only Memory (EEPROM), Flash memory, or other memory technology, CD-ROM, Digital Versatile Disk (DVD), or other optical storage, magnetic cassettes, magnetic tapes, magnetic disks, or other magnetic storage devices, or any other article of manufacture that can be used to store information and that can be accessed by computing device 600. Any such computer storage media may be part of computing device 600. Computer storage media do not include carrier waves or other propagated or modulated data signals.

[0038]

[0045] Communication media may be embodied by computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" may describe a signal that has one or more characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media may include wired media, such as wired networks and direct-wired connections, and wireless media, such as acoustic, radio frequency (RF), infrared and other wireless media.

[0039]

[0046] 7A and 7B show a mobile computing device 700 capable of implementing embodiments of the present disclosure, examples of which include a mobile phone, a smart phone, a wearable computer (such as a smart watch), a tablet computer, a laptop computer, and the like. In some aspects, the client can be a mobile computing device. Referring to FIG. 7A, one aspect of a mobile computing device 700 for implementing the above features is shown. In a basic configuration, the mobile computing device 700 is a handheld computer having both input and output elements. The mobile computing device 700 typically includes a display 705 and one or more input buttons 710 that allow a user to input information into the mobile computing device 700. The display 705 of the mobile computing device 700 can also function as an input device (e.g., a touch screen display). If an optional side input element 715 is included, further user input is possible. The side input element 715 can be a rotary switch, a button, or any other type of manual input element. In alternative aspects, the mobile computing device 700 can incorporate more or fewer input elements. For example, in some embodiments, the display 705 can be a non-touch screen. In yet another alternative embodiment, the mobile computing device 700 can be a portable phone system, such as a cellular phone. The mobile computing device 700 can also include an optional keypad 735. The optional keypad 735 can be a physical keypad or a "soft" keypad created on a touch screen display.In various embodiments, the output elements include a display 705 for displaying a graphical user interface (GUI), a visual indicator 720 (e.g., a light emitting diode), and / or an audio transducer 725 (e.g., a speaker). In some features, the mobile computing device 700 incorporates a vibration transducer for providing tactile feedback to a user. In yet other features, the mobile computing device 700 incorporates input and / or output ports for transmitting signals to and receiving signals from external devices, such as an audio input (e.g., a microphone jack), an audio output (e.g., a headphone jack), and a video output (e.g., an HDMI port).

[0040]

[0047] 7B is a block diagram illustrating the architecture of one aspect of a mobile computing device. That is, the mobile computing device 700 can incorporate a system (e.g., architecture) 702 for implementing some aspects. In one embodiment, the system 702 is implemented as a "smart phone" capable of running one or more applications (e.g., a browser, email, calendaring, contact management, messaging client, games, and a media client / player). In some aspects, the system 702 is integrated as a computing device, such as a combined personal digital assistant (PDA) and wireless phone.

[0041]

[0048] One or more application programs 766 may be loaded into memory 762 and may execute with or in association with operating system 764. Examples of application programs include a phone dialer program, an e-mail program, a personal information manager (PIM) program, a word processing program, a spreadsheet program, an Internet browser program, a messaging program, and the like. System 702 also includes a non-volatile storage area 768 in memory 762. Non-volatile storage area 768 may be used to store persistent information that should not be lost even if system 702 is powered down. Application programs 766 may use and store information in non-volatile storage area 768, such as e-mails and other messages used by an e-mail application. A synchronization application (not shown) is also present in system 702 and is programmed to interact with a corresponding synchronization application present in the host computer to maintain synchronization between the information stored in non-volatile storage area 768 and corresponding information stored in the host computer. It should be understood that other applications can be loaded into memory 762 and executed on the mobile computing device 700, including instructions for providing the consensus determination applications described herein (e.g., a message parser, a suggestion interpreter, an opinion interpreter, and / or a consensus presenter, etc.).

[0042]

[0049] The system 702 includes a power source 770, which may be implemented as one or more batteries. The power source 770 may also include an external power source, such as an AC adapter or a powered docking cradle, that supplements or recharges the battery.

[0043]

[0050] The system 702 may also include a radio interface layer 772, which performs the functions of transmitting and receiving radio frequency communications. The radio interface layer 772 facilitates wireless connectivity between the system 702 and the "outside world" via a communications carrier or service provider. Transmissions to and from the radio interface layer 772 are under the control of the operating system 764. In other words, communications received by the radio interface layer 772 may be propagated through the operating system 764 to the application programs 766, and vice versa.

[0044]

[0051] The visual indicator 720 can be used to provide a visual notification and / or the audio interface 774 can be used to generate an audible notification via the audio transducer 725 (e.g., the audio transducer 725 shown in FIG. 7A). In an exemplary embodiment, the visual indicator 720 can be a light emitting diode (LED) and the audio transducer 725 can be a speaker. These devices can be directly connected to the power source 770 so that when activated, they remain on for a period of time dictated by the notification mechanism even if the processor 760 and other components are shut down to avoid wasting battery power. The LED can be programmed to remain on indefinitely until the user takes an action to indicate the power on status of the device. The audio interface 774 can be used to provide audible signals to and receive audible signals from the user. For example, in addition to being coupled to the audio transducer 725, the audio interface 774 can also be coupled to a microphone for receiving audible input, such as to facilitate a telephone conversation. According to an embodiment of the present disclosure, the microphone also serves as an audio sensor to facilitate control of notifications as described below. The system 702 may further include a video interface 776, which enables operation of the peripheral device 730 (e.g., an on-board camera) to record still images, video streams, and the like. The audio interface 774, video interface 776, and keyboard 735 may be operated to generate one or more messages as described herein.

[0045]

[0052] A mobile computing device 700 implementing system 702 may have additional features and functionality. For example, mobile computing device 700 may also include additional data storage devices (removable and / or non-removable) such as magnetic disks, optical disks, tape, etc. Such additional data storage is illustrated in FIG. 7B as non-volatile storage area 768.

[0046]

[0053] The data / information generated or captured by the mobile computing device 700 and stored via the system 702 may be stored locally on the mobile computing device 700 as previously described, or the data may be stored on any number of data storage media that may be accessed by the device via the wireless interface layer 772 or via a wired connection between the mobile computing device 700 and a separate computing device associated with the mobile computing device 700, such as a server computer in a distributed computing network such as the Internet. It should be understood that such data / information may be accessed via the mobile computing device 700, via the wireless interface layer 772, or via a distributed computing network. Similarly, such data / information may be readily transferred between computing devices for storage and use in accordance with known means of data / information transfer and storage, including electronic mail and collaborative data / information sharing systems.

[0047]

[0054] It should be understood that Figures 7A and 7B are described for purposes of illustrating the present method and system and are not intended to limit the present disclosure to any particular sequence of steps or combination of hardware or software components.

[0048]

[0055] 8 illustrates one aspect of an architecture of a system for processing data received at a computing system from a remote source, such as a general-purpose computing device 804 (e.g., a personal computer), a tablet computing device 806, or a mobile computing device 808, as described above. Content displayed at the server device 802 may be stored in various communication channels or other storage types. For example, various messages may be received and / or stored using a directory service 822, a web portal 824, a mailbox service 826, an instant messaging storage 828, or a social networking service 830. A reinforcement learning application 821 may be employed by a client communicating with the server device 802 and / or a reinforcement learning application 820 may be employed by the server device 802. The server device 802 can provide data to and from client computing devices, such as a general-purpose computing device 804, a tablet computing device 806, and / or a mobile computing device 808 (e.g., a smart phone) over a network 815. By way of example, the computer system described above can be embodied in a general-purpose computing device 804 (e.g., a personal computer), a tablet computing device 806, and / or a mobile computing device 808 (e.g., a smart phone). Any of these embodiments of a computing device can obtain content from storage device 816 in addition to receiving graphical data available for pre-processing in a graphic-originating system or post-processing in a receiving computing system.

[0049]

[0056] It should be understood that FIG. 8 is described for purposes of illustrating the present methods and systems and is not intended to limit the present disclosure to any particular sequence of steps or combination of hardware or software components.

[0050]

[0057] FIG. 9 illustrates an example tablet computing device 900 capable of implementing one or more features disclosed herein. Additionally, features of the functionality described herein may be operated on a distributed system (e.g., a cloud-based computing system) where application functions, memory, data storage and retrieval, and various processing functions may be operated remotely from one another over a distributed computing network, such as the Internet or an intranet. The user interface and various types of information may be displayed via an on-board computing device display or via a remote display unit associated with one or more computing devices. For example, the user interface and various types of information may be displayed and interacted with on a wall surface on which the user interface and various types of information are projected. Interactions with the numerous computing systems on which embodiments of the present invention may be implemented include keystroke input, touch screen input, voice or other audio input, gesture input, etc., in cases where the associated computing device includes detection (e.g., camera) capabilities to capture and interpret user gestures to control functions of the computing device.

[0051]

[0058] It should be understood that FIG. 9 is described for purposes of illustrating the present methods and systems and is not intended to limit the present disclosure to any particular sequence of steps or combination of hardware or software components.

[0052]

[0059] The description and illustration of one or more features provided in this application are not intended to limit or restrict the scope of the claimed disclosure in any way. The features, examples, and details provided in this application are believed to be sufficient to convey the proprietary and to enable others to make and use the best mode of the claimed disclosure. The claimed disclosure should not be construed as being limited to any features, examples, or details provided in this application. It is intended that various features (both structural and methodological), whether shown and described in combination or individually, may be selectively included or omitted to produce an embodiment with a particular set of features. Those skilled in the art, having been provided with the description and illustrations of this application, may appreciate modifications, changes, and alternative features that are within the spirit of the broad aspects of the general inventive concept embodied in this application without departing from the broad scope of the claimed disclosure.

Claims

1. 1. A method for optimizing an expected user-perceived quality of experience (QoE) in a real-time communication between a sending computing device and a receiving computing device, comprising: an agent of the sending computing device determining a current state of the sending computing device based on current state data of the sending computing device and network statistics received from the receiving computing device, the network statistics indicative of a quality of real-time communications received at the receiving computing device; the agent determining a current behavior of the sending computing device, the current behavior including a plurality of sending parameters; the agent providing the transmission parameters to the sending computing device; transmitting a real-time communication, including one or more of a real-time audio communication and a real-time video communication, from the sending computing device to the receiving computing device in accordance with the transmission parameters; receiving, by the agent, from the receiving computing device, a reward determined by the receiving computing device and second network statistics, the reward being based on one or more reception parameters associated with the real-time communication transmitted and received at the receiving computing device; the agent determining an expected value of a total future reward based on the current state, the current action, the second network statistic, and the reward; modifying at least one of the plurality of transmission parameters of the sending computing device such that the agent maximizes the expected value of the sum of future rewards; The method includes:

2. 2. The method of claim 1, wherein a state-action value function of a reinforcement learning model determines the predicted value of the sum of future rewards.

3. 3. The method of claim 2, further comprising providing an output of the state-action value function to a control policy learning model of the reinforcement learning model, and the control policy learning model modifying the at least one of the plurality of transmission parameters based on the output of the state-action value function.

4. 2. The method of claim 1, wherein the reward comprises a user-perceived quality of experience (QoE) metric based on the one or more reception parameters associated with the real-time communication transmitted and received at the receiving computing device.

5. 5. The method of claim 4, further comprising determining the user's perceived QoE using a QoE machine learning model, the QoE machine learning model evaluating network statistics at the receiving computing device, receiving computing device statistics, and user feedback of the real-time communication transmitted and received at the receiving computing device.

6. 6. The method of claim 4 or 5, further comprising determining the user's perceived QoE using a QoE machine learning model, the QoE machine learning model evaluating a payload of the real-time communication transmitted and received at the receiving computing device.

7. 7. The method of claim 1, wherein the at least one of the plurality of transmission parameters comprises a transmission rate parameter, a resolution parameter, a frame rate parameter, a quantization parameter (QP), or a forward error correction (FEC) parameter.

8. 8. The method according to claim 1, wherein the sending computing device further acts as a receiving computing device and the receiving computing device further acts as a sending computing device for two-way real-time communication.

9. 1. A method for training a reinforcement learning model for optimizing an expected user-perceived quality of experience (QoE) in real-time communications, comprising: an agent of a sending computing device determining a current state of the sending computing device based on current state data of the sending computing device and network statistics received from a receiving computing device, the network statistics indicative of a quality of real-time communications received at the receiving computing device; the agent determining a current behavior of the sending computing device, the current behavior including a plurality of sending parameters; the agent providing the transmission parameters to the sending computing device; transmitting a real-time communication, including one or more of a real-time audio communication and a real-time video communication, from the sending computing device to the receiving computing device in accordance with the transmission parameters; receiving, at the agent, from the receiving computing device, a reward determined by the receiving computing device and second network statistics, the reward being based on one or more reception parameters associated with the real-time communication received at the receiving computing device; the agent determining an expected value of a total future reward based on the current state, the current action, the second network statistic, and the reward; modifying at least one of the plurality of transmission parameters such that the agent maximizes the expected value of the sum of future rewards; The method includes:

10. 10. The method of claim 9, a state-action value function of a reinforcement learning model that determines the predicted value of the sum of future rewards; The method further includes providing an output of the state-action value function to a control policy learning model of the reinforcement learning model; and the control policy learning model altering the at least one of the plurality of transmission parameters based on the output of the state-action value function. method.

11. 10. The method of claim 9, the reward comprises a user-perceived quality of experience (QoE) metric based on the one or more reception parameters associated with the real-time communication transmitted and received at the receiving computing device; The method further includes determining the user's perceived QoE using a QoE machine learning model that evaluates network statistics at the receiving computing device, receiving computing device statistics, and user feedback of the real-time communication transmitted and received at the receiving computing device. method.

12. 12. The method of any one of claims 9 to 11, wherein the sending computing device, the receiving computing device and the network are simulated.

13. 13. The method of claim 12, wherein the sending computing device, the receiving computing device, and the network are simulated using discrete events.

14. 14. A method according to any one of claims 9 to 13, wherein each of the sending and receiving computing devices runs a communication application, and one or more conditions of the network are controlled according to one or more predetermined parameters.

15. 12. The method of any one of claims 9 to 11, wherein the network comprises a live, actual network.

16. 16. The method of claim 15, wherein the sending computing device, the receiving computing device, and the network are in a live environment, and further comprising the step of continuously training the agent based on transmissions of live, real-time communications.

17. 1. A system for optimizing an expected user-perceived quality of experience (QoE) in real-time communications, comprising: A memory having executable instructions stored therein; a processor for executing said executable instructions; the executable instructions, when executed, cause the processor to: an agent of a sending computing device determining a current state of the sending computing device based on current state data of the sending computing device and network statistics received from a receiving computing device, the network statistics indicative of a quality of real-time communications received at the receiving computing device; the agent determining a current behavior of the sending computing device, the current behavior including a plurality of sending parameters; the agent providing the transmission parameters to the sending computing device; transmitting real-time communications, including one or more of real-time audio and video communications, from the sending computing device to a receiving computing device in accordance with the transmission parameters; receiving, by the agent, from the receiving computing device, a reward determined by the receiving computing device and second network statistics, the reward being based on one or more reception parameters associated with the real-time communication transmitted and received at the receiving computing device; determining, by the agent, an expected value of a total future reward based on the current state, the second network statistic, the current action, and the reward; modifying at least one of the plurality of transmission parameters of the sending computing device such that the agent maximizes the expected value of the sum of future rewards; A system that executes the above.

18. 20. The system of claim 17, The executable instructions further cause the processor to determine the predicted value of the sum of future rewards using a state-action value function of a reinforcement learning model; The executable instructions further cause the processor to provide an output of the state-action value function to a control policy learning model of the reinforcement learning model; and the control policy learning model alters the at least one of the plurality of transmission parameters based on the output of the state-action value function. system.

19. 20. The system of claim 17, the reward comprises a user-perceived quality of experience (QoE) metric based on the one or more reception parameters associated with the real-time communication transmitted and received at the receiving computing device; The executable instructions further cause the processor to determine the user's perceived QoE using a QoE machine learning model that evaluates network statistics at the receiving computing device, receiving computing device statistics, and user feedback of the real-time communication transmitted and received at the receiving computing device. system.

20. 20. The system of claim 17, wherein the executable instructions further cause the processor to operate the sending computing device as a receiving computing device for two-way real-time communication.

21. 21. The system of claim 17, wherein the at least one of the plurality of transmission parameters comprises a transmission rate parameter, a resolution parameter, a frame rate parameter, a quantization parameter (QP), or a forward error correction (FEC) parameter.

22. 20. The system of claim 18, wherein the reinforcement learning model comprises an actor-critic model, a q-learning model, a policy gradient model, a temporal difference model, or a Monte-Carlo tree search model.

Images