Reinforcement Learning in Real-Time Communications

Abstract

The agent interfaces with the sending computing device and the receiving computing device to automatically adjust one-way or two-way real-time audio and real-time video transmission parameters in response to changing network conditions and / or application requirements. The agent incorporates a reinforcement learning model that adjusts the transmission parameters to maximize the expected value of a sum of future rewards; the expected value of the sum of future rewards is based on the current state of the sending computing, the current action at the sending computing device (e.g., the current set of transmission parameters), and the reward provided by the receiving computing device. The reward represents the quality of experience perceived by the user at the receiving computing device.

Description

Background Art

[0001] Bandwidth estimation, congestion control, and video quality optimization for real-time communications (e.g., voice and video conferencing) remain a challenge as network conditions and application requirements change frequently. Delivery of real-time media with high quality and reliability (e.g., end-user quality of experience) requires continuous updates to react to new application requirements and network behavior. The process of continuous updates can be a slow process, resulting in a degradation of the end-user experience.

[0002] Various aspects of the present disclosure are made in view of these and other general considerations.In addition, although relatively specific problems and examples that solve these problems may be discussed herein, it should be understood that the examples should not be limited to solving specific problems identified elsewhere in the background or this disclosure. Summary of the invention

[0003] The present disclosure generally relates to systems and methods for implementing reinforcement learning in real-time communications. Certain aspects of the present disclosure relate to reinforcement learning for optimizing user-perceived quality in real-time audio and video communications. An agent interfaces with a sending computing device and a receiving computing device to automatically adjust real-time audio and video transmission parameters in response to changing network conditions and / or application requirements. The sending computing device transmits real-time audio and / or video data. The receiving computing device receives the real-time audio and video transmission from the sending device and determines the actual user-perceived quality of experience (QoE), which is provided to the agent as a reward. The agent incorporates a reinforcement learning model that includes a control policy and a state-action-value function. The agent observes the current state of the sending computing device and determines an estimate of the expected value of the future reward sum based on the current state, the current action (e.g., the current adjustment or adjustment set made to the transmission parameters at the sending computing device) and the reward provided by the receiving computing device. Based on the goal of maximizing the expected value of the future reward sum, the agent adjusts the control policy. The adjustment in the control policy can change the action applied to the real-time audio and / or video data.

[0004] One aspect of the present disclosure relates to methods, systems, and articles of manufacture for optimizing expected user-perceived QoE in real-time communications. The aspect includes determining a current state of a sending computing device and a current action of the sending computing device; the current action includes multiple transmission parameters. The aspect also includes transmitting real-time communications from the sending computing device to a receiving computing device. The real-time communications include one or both of real-time audio communications and real-time video communications. In addition, based on one or more parameters of the transmitted real-time communications received at the receiving computing device, a reward (e.g., a QoE metric) is determined at the receiving computing device. Based on the current state, the current action, and the reward, an expected value of a future sum of rewards is determined, and at least one of the multiple transmission parameters of the sending computing device is changed to maximize the expected value of the future sum of rewards.

[0005] One aspect of the present disclosure relates to methods, systems, and articles of manufacture for a reinforcement learning model that optimizes expected user-perceived QoE in real-time communications. This aspect includes determining a current state of a sender and providing the current state to an agent communicating with the sender. This aspect also includes determining a current action of the sender; the current action is known to the agent and includes multiple transmission parameters. This aspect also includes transmitting real-time communications from the sender to the receiver. The real-time communication includes one or both of real-time audio transmission or real-time video transmission. This aspect also includes receiving a reward determined at the receiver from the agent. The reward is based on one or more parameters associated with the real-time communication received at the receiver. The agent determines an expected value of a future reward sum based on the current state, current action, and reward, and guides changes in at least one of the multiple transmission parameters to maximize the expected value of the future reward sum. Training can be performed in a simulated environment, an emulated environment, or a real network environment.

[0006] This Summary is provided to introduce in a simplified form a selection of concepts that are further described in the Detailed Description below. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Additional aspects, features, and / or advantages of examples will be set forth in part in the description that follows, and in part will become apparent from the description, or may be learned by practice of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Non-limiting and non-exhaustive examples are described with reference to the following figures.

[0008] Figure 1 An environment is shown in which reinforcement learning in real-time communications as disclosed herein can be practiced.

[0009] Figures 2A-2C Additional details of an environment in which reinforcement learning in real-time communications herein may be practiced are shown.

[0010] Figure 3 A simulated training environment for a hardened environment for maximizing user-perceived Quality of Experience (QoE) in real-time communications is shown.

[0011] Figure 4 A simulation training environment for reinforcement learning to maximize user-perceived QoE in real-time communications is shown.

[0012] Figure 5 A real network training environment for reinforcement learning to maximize user-perceived QoE in real-time communications is shown.

[0013] Figure 6 is a block diagram illustrating example physical components of a computing device that can be used to practice aspects of the present disclosure.

[0014] Fig. 7A and 7B is a simplified block diagram of a mobile computing device that can be used to practice aspects of the present disclosure.

[0015] Figure 8 is a simplified block diagram of a distributed computing system in which aspects of the present disclosure may be practiced.

[0016] Fig. 9 A tablet computing device for performing one or more aspects of the present disclosure is shown. DETAILED DESCRIPTION

[0017] Various aspects of the present disclosure are described more fully below with reference to the accompanying drawings that form a part of the present disclosure. The different aspects of the present disclosure can be implemented in many different forms and should not be interpreted as being limited to the aspects set forth herein; rather, these aspects are provided to make the present disclosure thorough and complete, and to fully convey the scope of these aspects to those skilled in the art. Aspects can be practiced as methods, systems, or devices. Therefore, aspects can take the form of hardware implementations, complete software implementations, or implementations combining software and hardware aspects. Therefore, the following detailed description should not be considered as limiting.

[0018] The present disclosure generally relates to systems and methods for implementing reinforcement learning in real-time communications. Certain aspects of the present disclosure relate to reinforcement learning for optimizing user-perceived quality in real-time audio and video communications. An agent interfaces with a sending computing device and a receiving computing device to automatically adjust real-time audio and video transmission parameters in response to changing network conditions and / or application requirements. The sending computing device transmits real-time audio and / or video data. The receiving computing device receives the real-time audio and video transmission from the sending device and determines the actual user-perceived quality of experience (QoE), which is provided to the agent as a reward. The agent incorporates a reinforcement learning model that includes a control policy and a state-action-value function. The agent observes the current state of the sending computing device and determines an estimate of the expected value of the sum of future rewards based on the current state, the current action (e.g., the current adjustment or set of adjustments made to the transmission parameters at the sending computing device) and the reward provided by the receiving computing device. Based on the goal of maximizing the expected value of the sum of future rewards, the agent adjusts the control policy. The adjustment in the control policy can change the action applied to the real-time audio and / or video data.

[0019] Thus, the present disclosure provides a number of technical benefits, including but not limited to a continuously updated reinforcement learning model that immediately responds to adjusting real-time audio and video transmission parameters of a sending computing device based on a goal of maximizing the expected value of a sum of future rewards. Real-time audio and video transmission parameters are immediately adjusted in response to changing network conditions and / or application requirements. Degradation of the transmitted real-time audio and video streams is minimized, which may occur in previously used manually coded responsive update processes for data transmission parameters to combat degradation.

[0020] refer to Figure 1 , an environment 100 for practicing reinforcement learning in real-time communications is shown. The environment 100 includes a network 102 through which multiple computing devices 104 communicate via various communication links 106. The term "real-time" refers to data processing where received data is processed by the computing device almost immediately, e.g., a level of computing device response that is immediate enough for a user to perceive or that enables the computing device to keep up with some external process.

[0021] The network 102 is any type of wired and / or wireless network that can transmit, receive, and exchange data, voice, and video services. Examples of networks include a local area network (LAN) that interconnects endpoints in a single domain and a wide area network (WAN) that interconnects multiple LANs, as well as a subnet, a metropolitan area network, a storage area network, a personal area network (PAN), a wireless local area network (WLAN), a campus local area network (CAN), a virtual private network (VPN), a passive optical network, etc.

[0022] Computing devices 104 comprise endpoints of network 102. Computing devices 104 may include one or more general or special purpose computing devices. Such devices may include, for example, personal computers, server computers, handheld or laptop devices, multiprocessor systems, microcontroller-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, cellular phones, personal digital assistants (PDAs), gaming devices, printers, appliances, media centers, automotive embedded or attached computing devices, other mobile devices, distributed computing environments that include any of the above systems or devices, and the like. Further details regarding computing devices are provided in Figure 6-9 Described in.

[0023] Communications between computing devices 104 proceed over links 106. Links may include any type of guided or unguided transmission medium capable of transmitting data, voice, and / or video from one computing device 104 to another computing device 104. Guided media transmit signals along a physical path. Examples of guided media include twisted pair cables, coaxial cables, optical fibers, and the like. Unguided media transmit signals without using physical means to define the path the signals take. Examples of unguided media include radio waves, microwaves, infrared waves, and the like.

[0024] Figure 2A-2B An environment 200 is shown, which, for illustrative purposes, includes a single sending computing device 204S and a single receiving computing device 204R communicating in real time through a network 202 via a link 206. Although the sending computing device 204S is shown as including only a sending capability, it should be appreciated that the sending computing device 204S can also operate as a receiving computing device. Similarly, the receiving computing device 204R can also operate as a sending computing device. Therefore, two-way real-time communication can occur between the sending computing device 204S and the receiving computing device 204R. The environment 200 communicates in real time with the agent 206 to implement reinforcement learning based on real-time communication of data, which data may include voice data and video data. Reinforcement learning optimizes the expected user-perceived quality in real-time communication by maximizing the expected value of the sum of future rewards. The agent 206 may include coding or applications resident on one or both of the sending computing device 204S and the receiving computing device 204R. The agent 206 may also include code or an application residing on a computing device different from the sending computing device 204S or the receiving computing device 204R, such as a server computing device, a cloud computing device, or the like.

[0025] As shown, the sending computing device 204S includes a data capture module 210, a data encoder module 212 and a data sender module 214. The data capture module 210 captures state data, which represents the currently observed state of the sending computing device 204S. In the context of real-time audio and video communication, the currently observed state may include observed sending parameters that affect the transmission of real-time audio data and real-time video data. The observed transmission parameters may include, for example, resolution, bit rate, frame rate, stream to be sent, codec (encoding / decoding), the user's physical environment (e.g., darkness / light level, background noise, motion, etc.), or any other parameters that may affect the transmission of real-time data. The data encoder module 212 of the sending computing device 204S converts the state data into a specified format for real-time transmission through the network 202. The data sender module 214 sends the formatted state data to the network 202 in real time.

[0026] The receiving computing device 240R includes a data receiver module 220, a data decoder module 222, and a QoE metric module 224. The data receiver module 220 receives formatted status data from the network 202 in real time and outputs network statistics to the agent 206. Examples of network statistics include loss, jitter, round trip time (RTT) (also known as network delay), reception rate, packet size, packet type, reception timestamp, sender timestamp, burst length in packet loss, gap between packet losses, or any other network statistics that can be used to evaluate the quality of the received audio and video data. The data decoder module 222 performs the inverse operation of the data encoder module 212 and extracts the received status data in real time from the formatted status data.

[0027] The QoE metric module 224 determines one or more metrics in the quality of experience (QoE) metric based on the extracted state data. The QoE metric represents the quality of the user's perception of the received state data determined by a QoE machine learning model (such as a deep neural network (DNN) or other appropriate model). The QoE machine learning model analyzes various reception parameters, such as the payload of the received audio and video data streams, where the payload is the portion of the received data that is the actual expected message. The analysis of the payload of the audio and video streams may include the use of one or more predefined objective models that approximate the results of the subjective quality assessment (e.g., the rating of the quality by a human observer). In some examples, the objective model may include one or more models for evaluating the quality of real-time audio (e.g., a perceptual assessment of audio quality (PEAQ) model, a PEMO-Q model, a signal-to-noise ratio (PSNR) model, or any other objective model that can evaluate the received real-time audio signal). In some examples, the objective model may include one or more models for evaluating real-time video quality (e.g., a full reference (FR) model, a reduced reference (RR) model, a no reference (NR) model, a peak signal-to-noise ratio (PSNR) model, a structural similarity index (SSIM) model, or any other model that can evaluate a received real-time video signal).

[0028] In some aspects, the QoE machine learning model can additionally analyze network statistics and statistics of the receiving computing device 204R as reception parameters to determine one or more QoE metrics. As described herein, examples of network statistics include loss, jitter, round trip time (RTT) (also known as network delay), reception rate, packet size, packet type, reception timestamp, sender timestamp, burst length in packet loss, gaps between packet losses, or any other network statistics that can be used to evaluate the quality of the received audio and video data. Examples of statistics of the receiving computing device 204R include display size, display window size, device type, whether hardware or software encoder / decoder is used, etc. In some aspects, the QoE machine learning model can additionally analyze user (e.g., human) feedback as reception parameters to determine one or more QoE metrics. User feedback can be provided, for example, through user ratings or surveys to indicate their personal experience quality, such as the user's opinion on the quality of the audio and video received at the receiving computing device 204R. The determined one or more QoE metrics representing the audio and / or video quality perceived by the user are transmitted to the agent 206.

[0029] The agent 206 includes a state module 230 and a reinforcement learning model 232. In certain aspects, the reinforcement learning model 232 can incorporate any suitable reinforcement learning algorithm (a learning algorithm in which actions occur, results are observed, and the next action takes into account the results of the first action based on a reward signal). The reinforcement learning algorithm can include, for example, actor-critic, q-learning, policy gradient, temporal difference, Monte Carlo tree search, or any other reinforcement learning algorithm suitable for the data involved. The reinforcement learning model 232 actively controls the data transmission parameters of the sending computing device 204S in real time.

[0030] Figure 2B An example of an actor-evaluator reinforcement learning model 232 including a control policy 234 and a state-action-value function 236 is shown. Figure 2C An example of an actor-evaluator architecture is provided. Actor-evaluator reinforcement learning is a temporal difference learning method in which the control policy 234 is independent of the estimated state-value function 236, which, in the current context, is the expected value of the sum of future rewards. The control policy 234 includes an actor because it is used to select actions, such as sending data transmission parameters of a computing device, and the state-value function 236 is an evaluator because it evaluates the actions made by the control policy 234. The state-value function 236 learns and evaluates the current control policy 234.

[0031] The control strategy 234 includes a first machine learning model, such as a neural network, within the agent 206 that produces one or more output actions in the form of one or more changes to one or more of the data transmission parameters used by the sending computing device 204S. The output actions are designed to optimize the expected user-perceived quality (QoE) of the audio and video data based on maximization of the expected value of the sum of future rewards determined by the state-action-value function 236. Examples of data transmission parameters include transmission rate, resolution, frame rate, object events provided to a quantization parameter (QP), forward error correction (FEC), or any other controllable parameter that can be used to modify the quality of transmission of state data from the sending computing device 204S to the receiving computing device 204R.

[0032] The state-action-value function 236 includes a second machine learning model within the agent 206, such as a neural network, whose value function is trained to predict or estimate the expected value of the sum of future rewards. The expected value of the sum of future rewards is determined based on the current state of the sending computing device, the current action (e.g., the current transmission parameters for transmitting real-time audio and / or video data), and the reward provided by the receiving computing device. The control strategy adjusts the output action in response to the expected value determination. The control strategy 234 can be trained with the state-action-value function 236 or can be acquired based on the state-action-value function 236 that has already been trained.

[0033] In some respects, Figure 2B-2C During the training of the actor-critic reinforcement learning model 232, the agent 206 need not always follow the actions of the control policy 234. Instead, the agent 206 can explore other actions (e.g., other modifications to the data transmission parameters of the sending computing device 204S), which allows the agent 206 to improve the reinforcement learning model 232. The agent 206 can explore other actions through one or more exploration strategies (e.g., epsilon greedy).

[0034] In some aspects, the control strategy 234 of the reinforcement learning model 232 can be separated from its learning environment and deployed as a real-time model in the client (e.g., a sending computing device and / or a receiving computing device). The transmission to the real-time model can be achieved through one or more model transmission tools such as ONNX (Open Neural Network Exchange), tflite (TensorFlow Lite), etc.

[0035] refer to Figure 3-Figure 5 , the agent 206 can be trained using one or more of the simulation environment 300, the emulation environment 400, and the real network environment 500. Which environment to use depends on the data collection speed and data diversity requirements. In the simulation environment of Figure 300, all processes of the sending computing device 204S (including the processes of the data capture module 210, the data encoder module 212, and the data sender module 214), all processes of the receiving computing device 204R (including the processes of the data receiver module 220, the data decoder module 222, and the QoE metric module 224), and the network 202 are simulated. Figure 4In the simulation environment 400 of FIG. 500 , the sending computing device 204S is replicated in a first simulation including the simulated sending process 404S, the receiving computing device 204R is replicated in a second simulation including the simulated receiving process 404R, and the network 202 is replicated in a third simulation including the network simulation 402. In some aspects, the physical sending computing device and the physical receiving computing device can be used in conjunction with the simulated network. In the real network environment of FIG. 500 , the physical sending computing device 204S, the physical receiving computing device 204R, and the physical network 202 are used.

[0036] Which environment is used to train the agent 206 depends on the data collection speed and data collection diversity requirements. For example, network simulation tools such as ns-2 or ns-3 (which are discrete event network simulators) can be used in the simulation environment 300 for fast data collection and training. Network simulation tools such as NetEm (which is an enhancement of the Linux business control facility, which allows the addition of delays, packet loss, duplication, and other characteristics of the outgoing transmission packets from the selected network interface) can be used in the simulation environment 400 to allow real code to run in a controlled environment. This controlled environment allows testing communication applications (e.g., Skype, Microsoft Teams, WhatsApp, WeChat, etc.) in an environment with reproducible network conditions. In the real network environment 500, the real network (e.g., cellular, Wi-Fi, Ethernet, etc.) of the real Internet service provider (ISP) is used to provide the most realistic test environment and allow online learning of the conditions experienced by the end-user. In some aspects, the same reinforcement learning strategy can be used in simulation, simulation or real network environments, but each environment will provide different performance. Alternatively or additionally, the agent 206 may be trained using transfer learning, wherein manually encoded rules that were previously created as a reaction to new application requirements and / or network behaviors related to real-time audio and video data streaming are used to train the agent 206.

[0037] Once trained, agent 206 is applied in a live network environment for real-time audio and video communication. In the live network, reinforcement learning model 232 is continuously updated based on the transmission of real-time audio and video data stream from a sending computing device (e.g., device 204S) to a receiving computing device 204R. In some aspects, a sending computing device (e.g., device 204S) may include a single agent 206 or multiple agents 206, and agents 206 operate to modify real-time audio and video data transmission parameters, and each agent only modifies one data transmission parameter or multiple data transmission parameters. In some aspects, a receiving computing device (e.g., device 204R) may determine a QoE or multiple QoEs. One or more QoEs may be provided to a single agent 206 or multiple agents 206.

[0038] Thus, based on the continuous real-time updates of the agent 206, the agent 206 and the sending computing device 204S are immediately (e.g., in real time) updated to continuously optimize the expected user-perceived quality in real-time audio and video communications by maximizing the expected value of the sum of future rewards, rather than suffering from degraded real-time audio and video transmission that would otherwise result in the context of an environment where only manual encoding is used to respond to changing network conditions and / or changing application requirements.

[0039] Figure 6-Figure 9 The and associated descriptions provide a discussion of various operating environments in which aspects of the present disclosure may be practiced. Figure 6-Figure 9 The devices and systems illustrated and discussed are for purposes of example and illustration and are not limiting of the wide variety of computing device configurations that may be used to practice aspects of the present disclosure, as described herein.

[0040] Figure 6 6 is a block diagram illustrating the physical components (e.g., hardware) of a computing device 600 that can be used to practice aspects of the present disclosure. The computing device components described below may have computer-executable instructions for implementing reinforcement learning that maximizes user-perceived QoE in real-time communications on computing devices (e.g., sending computing device 204S and receiving computing device 204R), including computer-executable instructions for a reinforcement learning application 620 that can be executed to implement the methods disclosed herein. In a basic configuration, the computing device 600 may include at least one processing unit 602 and a system memory 604. Depending on the configuration and type of the computing device, the system memory 604 may include, but is not limited to, volatile memory (e.g., random access memory), non-volatile memory (e.g., read-only memory), flash memory, or any combination of such memories. The system memory 604 may include an operating system 605 and one or more program modules 606, such as one or more components with respect to FIG. 2 , particularly data capture, data encoder, and data sender modules 611 (e.g., data capture module 210, data encoder module 212, and data sender module 214), data receiver, data decoder, and QoE metric modules 613 (e.g., data receiver module 220, data decoder module 222, and QoE metric module 224), and / or agent modules 615 (e.g., agent 206).

[0041] For example, operating system 605 may be adapted to control the operation of computing device 600. In addition, embodiments of the present disclosure may be practiced in conjunction with a graphics library, other operating systems, or any other application presentation, and are not limited to any particular application or system. Figure 6608. The computing device 600 may have additional features or functionality. For example, the computing device 600 may also include additional data storage devices (removable and / or non-removable) such as, for example, magnetic disks, optical disks, or tapes. Such additional storage devices may be used to store data. Figure 6 604 by removable storage 609 and non-removable storage 610. Any number of program modules and data files may be stored in system memory 604. When executed on processing unit 602, program modules 606 (e.g., reinforcement learning application 620) may perform processes including, but not limited to, those described herein.

[0042] In addition, embodiments of the present disclosure may be practiced in circuits including discrete electronic components, packaged or integrated electronic chips containing logic gates, circuits utilizing microprocessors, or on a single chip containing electronic components or microprocessors. For example, embodiments of the present disclosure may be practiced via a system on a chip (SOC), wherein Figure 6 Each or many of the components shown can be integrated onto a single integrated circuit. Such a SOC device may include one or more processing units, a graphics unit, a communication unit, a system virtualization unit, and various application functionalities, all of which are integrated (or "burned") onto a chip substrate as a single integrated circuit. When operated via a SOC, the functionality described herein regarding the ability to switch protocols for the client can be operated via dedicated logic integrated with other components of the computing device 600 on a single integrated circuit (chip). Embodiments of the present disclosure may also be practiced using other technologies capable of performing logical operations such as, for example, AND, OR, and NOT, including but not limited to mechanical, optical, fluid, and quantum technologies. In addition, embodiments of the present disclosure may be practiced within a general-purpose computer or in any other circuit or system.

[0043] The computing device 600 may also have one or more input devices 612, such as a keyboard, a mouse, a pen, a sound or voice input device, a touch or sliding input device, etc. Output device(s) 614 (such as a display, a speaker, a printer, etc.) may also be included. The aforementioned devices are examples and other devices may be used. The computing device 600 may include one or more communication connections 616 that allow communication with other computing devices 650. Examples of suitable communication connections 616 include, but are not limited to, radio frequency (RF) transmitters, receivers, and / or transceiver circuitry; universal serial bus (USB), parallel and / or serial ports.

[0044] The term computer-readable media as used herein may include computer storage media. Computer storage media may include volatile and nonvolatile removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, or program modules. System memory 604, removable storage device 609, and non-removable storage device 610 are all examples of computer storage media (e.g., memory storage devices). Computer storage media may include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other article that can be used to store information and can be accessed by computing device 600. Any such computer storage media may be part of computing device 600. Computer storage media do not include carrier waves or other propagated or modulated data signals.

[0045] Communication media may be embodied by computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" may describe a signal that has one or more characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media may include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0046] Fig. 7A and 7B A mobile computing device 700 is shown, such as a mobile phone, a smart phone, a wearable computer (such as a smart watch), a tablet computer, a laptop computer, etc., with which embodiments of the present disclosure may be practiced. In some aspects, the client may be a mobile computing device. Fig. 7A, shows one aspect of a mobile computing device 700 for implementing these aspects. In a basic configuration, the mobile computing device 700 is a handheld computer having both input elements and output elements. The mobile computing device 700 typically includes a display 705 and one or more input buttons 710 that allow a user to enter information into the mobile computing device 700. The display 705 of the mobile computing device 700 can also be used as an input device (e.g., a touch screen display). If included, an optional side input element 715 allows further user input. The side input element 715 can be a rotary switch, a button, or any other type of manual input element. In alternative aspects, the mobile computing device 700 can incorporate more or fewer input elements. For example, in some embodiments, the display 705 may not be a touch screen. In yet another alternative embodiment, the mobile computing device 700 is a portable telephone system, such as a cellular phone. The mobile computing device 700 can also include an optional keypad 735. The optional keypad 735 can be a physical keypad or a "soft" keypad generated on the touch screen display. In various embodiments, the output elements include a display 705 for showing a graphical user interface (GUI), a visual indicator 720 (e.g., a light emitting diode), and / or an audio transducer 725 (e.g., a speaker). In some aspects, the mobile computing device 700 incorporates a vibration transducer for providing tactile feedback to the user. In yet another aspect, the mobile computing device 700 incorporates input and / or output ports, such as an audio input (e.g., a microphone jack), an audio output (e.g., a headphone jack), and a video output (e.g., an HDMI port) for sending signals to or receiving signals from external devices.

[0047] Figure 7B 700 is a block diagram illustrating an architecture of one aspect of a mobile computing device. That is, a mobile computing device 700 may incorporate a system (e.g., architecture) 702 to implement some aspects. In one embodiment, the system 702 is implemented as a "smart phone" capable of running one or more applications (e.g., a browser, email, calendar, contact manager, messaging client, game, and media client / player). In some aspects, the system 702 is integrated as a computing device, such as an integrated personal digital assistant (PDA) and a wireless phone.

[0048] One or more application programs 766 may be loaded into the memory 762 and run on or in association with the operating system 764. Examples of application programs include a phone dialer, an email program, a personal information management (PIM) program, a word processing program, a spreadsheet program, an Internet browser program, a messaging program, and the like. The system 702 also includes a non-volatile storage area 768 within the memory 762. The non-volatile storage area 768 may be used to store persistent information that should not be lost when the system 702 loses power. The application programs 766 may use and store information in the non-volatile storage area 768, such as emails or other messages used by an email application, and the like. A synchronization application (not shown) also resides on the system 702 and is programmed to interact with a corresponding synchronization application resident on the host to keep the information stored in the non-volatile storage area 768 synchronized with the corresponding information stored at the host. It should be understood that other applications may be loaded into memory 762 and executed on mobile computing device 700, including instructions for providing consensus determination applications as described herein (e.g., a message parser, suggestion interpreter, opinion interpreter, and / or consensus demonstrator, etc.).

[0049] System 702 has a power source 770, which can be implemented as one or more batteries. Power source 770 can also include an external power source, such as an AC adapter or a powered docking station cradle to supplement or recharge the batteries.

[0050] The system 702 may also include a radio interface layer 772 that performs the functions of transmitting and receiving radio frequency communications. The radio interface layer 772 facilitates wireless connections between the system 702 and the "outside world" via a communications operator or service provider. Transmissions to and from the radio interface layer 772 are performed under the control of the operating system 764. In other words, communications received by the radio interface layer 772 may be propagated to the application 766 via the operating system 764, and vice versa.

[0051] Visual indicator 720 can be used to provide visual notifications, and / or audio interface 774 can be used to communicate information via audio transducer 725 (e.g., Fig. 7AThe audio transducer 725 shown in the figure) produces an audible notification. In the illustrated embodiment, the visual indicator 720 is a light emitting diode (LED) and the audio transducer 725 can be a speaker. These devices can be directly coupled to the power supply 770 so that when activated, they will remain on for the duration specified by the notification mechanism, even if the processor 760 and other components may be turned off to save battery power. The LED can be programmed to remain lit indefinitely until the user takes action to indicate the power-on state of the device. The audio interface 774 is used to provide audible signals to the user and receive audible signals from the user. For example, in addition to being coupled to the audio transducer 725, the audio interface 774 can also be coupled to a microphone to receive audible input, such as facilitating a telephone conversation. According to an embodiment of the present disclosure, the microphone can also be used as an audio sensor to facilitate the control of the notification, as described below. The system 702 can also include a video interface 776, which enables the peripheral device 730 (e.g., a vehicle-mounted camera) to operate to record still images, video streams, etc. The audio interface 774, the video interface 776, and the keyboard 735 may be operated to generate one or more messages as described herein.

[0052] The mobile computing device 700 implementing the system 702 may have additional features or functionalities. For example, the mobile computing device 700 may also include additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tapes. Such additional storage devices may be used to store data in a manner that is convenient for the user to use. Figure 7B , shown by non-volatile storage area 768.

[0053] Data / information generated or captured by the mobile computing device 700 and stored via the system 702 may be stored locally on the mobile computing device 700, as described above, or the data may be stored on any number of storage media that may be accessed by the device via the radio interface layer 772 or via a wired connection between the mobile computing device 700 and a separate computing device associated with the mobile computing device 700 (e.g., a server computer in a distributed computing network such as the Internet). It should be understood that such data / information may be accessed via the mobile computing device 700 via the radio interface layer 772 or via a distributed computing network. Similarly, such data / information may be readily transferred between computing devices for storage and use according to well-known data / information transmission and storage means, including electronic mail and collaborative data / information sharing systems.

[0054] It should be understood that Fig. 7A and 7B The methods and systems are described for purposes of illustrating the present invention and are not intended to limit the present disclosure to a specific sequence of steps or a specific combination of hardware or software components.

[0055] Figure 8 One aspect of the architecture of a system for processing data received at a computing system from a remote source, such as a general-purpose computing device 804 (e.g., a personal computer), a tablet computing device 806, or a mobile computing device 808, as described above, is shown. The content displayed at the server device 802 may be stored in different communication channels or other storage types. For example, a directory service 822, a web portal 824, a mailbox service 826, an instant messaging repository 828, or a social networking service 830 may be used to receive and / or store various messages. A reinforcement learning application 821 may be employed by a client communicating with the server device 802, and / or a reinforcement learning application 820 may be used by the server device 802. The server device 802 may provide data to and from client computing devices, such as a general-purpose computing device 804, a tablet computing device 806, and / or a mobile computing device 808 (e.g., a smart phone), over a network 815. For example, the computer system described above may be embodied in a general-purpose computing device 804 (e.g., a personal computer), a tablet computing device 806, and / or a mobile computing device 808 (e.g., a smart phone). In addition to receiving graphics data that may be used to be pre-processed at a graphics initiating system or post-processed at a receiving computing system, any of these embodiments of the computing device may retrieve content from repository 816 .

[0056] It should be understood that Figure 8 The methods and systems are described for purposes of illustrating the present invention and are not intended to limit the present disclosure to a specific sequence of steps or a specific combination of hardware or software components.

[0057] Fig. 9 An exemplary tablet computing device 900 that can perform one or more aspects disclosed herein is shown. In addition, the aspects and functions described herein can be operated on a distributed system (e.g., a cloud-based computing system), where application functionality, memory, data storage and retrieval, and various processing functions can be remotely operated from each other through a distributed computing network (such as the Internet or an intranet). Various types of user interfaces and information can be displayed via an onboard computing device display or via a remote display unit associated with one or more computing devices. For example, various types of user interfaces and information can be displayed on a wall on which various types of user interfaces and information are projected and interacted with. Interaction with a variety of computing systems that can be used to practice embodiments of the present invention includes keystroke entry, touch screen entry, voice or other audio entry, gesture entry, where the associated computing device is equipped with detection (e.g., camera) functionality for capturing and interpreting user gestures to control the functionality of the computing device, etc.

[0058] It should be understood that Fig. 9What is described is for the purpose of illustrating the present methods and systems and is not intended to limit the present disclosure to a specific sequence of steps or a specific combination of hardware or software components.

[0059] The description and illustration of one or more aspects provided in this application are not intended to limit or restrict the scope of the disclosure claimed for protection in any way. The aspects, examples and details provided in this application are considered to be sufficient to convey ownership and enable others to make and use the best mode of the disclosure claimed for protection. The disclosure claimed for protection should not be interpreted as being limited to any aspect, example or detail provided in this application. Whether it is shown and described in combination or separately, various features (structural and methodological) are intended to be selectively included or omitted to produce an embodiment with a specific feature group. Description and illustration of the application have been provided, and those skilled in the art can envision the changes, modifications and alternative aspects that fall within the spirit of the broader aspects of the general inventive concept embodied in this application but do not depart from the broader scope of the disclosure claimed for protection.

Claims

1. A method for optimizing an expected user-perceived quality of experience (QoE) in real-time communications between a sending computing device and a receiving computing device, comprising: determining, by an agent of the sending computing device, a current state of the sending computing device based on current state data of the sending computing device and network statistics received from the receiving computing device, the network statistics indicating a quality of real-time communications received at the receiving computing device; determining, by the agent, a current action of the sending computing device, wherein the current action includes a plurality of transmission parameters; providing, by the agent, the transmission parameters to the sending computing device; transmitting a real-time communication from the sending computing device to the receiving computing device according to the transmission parameters, wherein the real-time communication includes one or more of a real-time audio communication and a real-time video communication; receiving, by the agent, from the receiving computing device, a reward determined by the receiving computing device, wherein the reward is based on one or more reception parameters associated with the transmitted real-time communication received at the receiving computing device; Determining, by the agent, an expected value of a sum of future rewards based on the current state, the current action, and the reward; as well as At least one of the plurality of transmission parameters of the sending computing device is varied, by the agent, to maximize the expected value of the sum of future rewards.

2. The method of claim 1 , wherein a state-action-value function of a reinforcement learning model determines the expected value of the sum of future rewards.

3. The method according to claim 2 further includes providing the output of the state-action value function to the control strategy learning model of the reinforcement learning model, and changing at least one of the multiple transmission parameters based on the output of the state-action value function by the control strategy learning model.

4. The method of claim 1, wherein the reward comprises a user-perceived quality of experience (QoE) metric based on the one or more reception parameters associated with the transmitted real-time communication received at the receiving computing device.

5. The method of claim 4, utilizing a QoE machine learning model to determine the user-perceived QoE, wherein the QoE machine learning model evaluates the payload of the transmitted real-time communication received at the receiving computing device.

6. The method of claim 4, wherein the QoE perceived by the user is determined using a QoE machine learning model, wherein the QoE machine learning model evaluates: network statistics at the receiving computing device; receiving computing device statistics; and user feedback of the transmitted real-time communication received at the receiving computing device.

7. The method of claim 1 , wherein the at least one transmission parameter of the plurality of transmission parameters comprises: Transmission rate parameter, resolution parameter, frame rate parameter, quantization parameter QP, or forward error correction FEC parameter.

8. The method of claim 1, wherein the sending computing device additionally operates as a receiving computing device and wherein the receiving computing device additionally operates as a sending computing device for two-way real-time communication.

9. A method for training a reinforcement learning model for optimizing expected user-perceived quality of experience (QoE) in real-time communications, the method comprising: determining, by an agent of the sending computing device, a current state of the sending computing device based on current state data of the sending computing device and network statistics received from a receiving computing device, the network statistics indicating a quality of real-time communications received at the receiving computing device; determining, by the agent, a current action of the sending computing device, wherein the current action includes a plurality of transmission parameters; providing, by the agent, the transmission parameters to the sending computing device; transmitting a real-time communication from the sending computing device to the receiving computing device according to the transmission parameters, wherein the real-time communication includes one or more of a real-time audio communication and a real-time video communication; receiving, at the agent from the receiving computing device, a reward from and determined by the receiving computing device, wherein the reward is based on one or more reception parameters associated with the real-time communication received at the receiving computing device; At the agent, determining an expected value of a sum of future rewards based on the current state, the current action, and the reward; as well as At least one transmission parameter of the plurality of transmission parameters is varied, by the agent, to maximize the expected value of the sum of future rewards.

10. The method of claim 9, wherein the sending computing device, the receiving computing device, and the network are simulated.

11. The method of claim 10, wherein the sending computing device, the receiving computing device, and the network are simulated using discrete events.

12. The method of claim 9, wherein each of the sending computing device and the receiving computing device executes a communication application, and wherein one or more conditions of the network are controlled according to one or more predetermined parameters.

13. The method of claim 9, wherein the network comprises a live, real network.

14. The method of claim 13, wherein the sending computing device, the receiving computing device, and the network are in a live environment, and the method further comprises continuously training the agent based on live real-time communication transmissions.

15. A system for optimizing expected user perceived quality of experience (QoE) in real-time communications, comprising: a memory for storing executable instructions; as well as a processor executing the executable instructions, which when executed cause the processor to: determining, by an agent of the sending computing device, a current state of the sending computing device based on current state data of the sending computing device and network statistics received from a receiving computing device, the network statistics indicating a quality of real-time communications received at the receiving computing device; determining, by the agent, a current action of the sending computing device, wherein the current action includes a plurality of transmission parameters; providing, by the agent, the transmission parameters to the sending computing device; transmitting a real-time communication from the sending computing device to the receiving computing device according to the transmission parameters, wherein the real-time communication includes one or more of a real-time audio communication and a real-time video communication; receiving, by the agent from the receiving computing device, a reward from and determined by the receiving computing device, wherein the reward is based on a parameter associated with the transmitted real-time communication received at the receiving computing device; Determining, by the agent, an expected value of a sum of future rewards based on the current state, the current action, and the reward; as well as At least one of the plurality of transmission parameters of the sending computing device is varied, by the agent, to maximize the expected value of the sum of future rewards.

16. The system of claim 15, further comprising operating the sending computing device as a receiving computing device for two-way real-time communication.

17. The system of claim 15, wherein the at least one transmission parameter of the plurality of transmission parameters comprises: Transmission rate parameter, resolution parameter, frame rate parameter, quantization parameter QP, or forward error correction FEC parameter.

18. The system of claim 15, further comprising utilizing a reinforcement learning model to perform the determination of the expected value of the sum of future rewards.

19. The system of claim 18, wherein the reinforcement learning comprises an actor-critic model, a q-learning model, a policy gradient model, a temporal difference model, or a Monte Carlo tree search model.

Images