Video processing adjustments based on user being watched / not being watched

Abstract

The invention relates to video processing adjustments based on a user watching / not watching. In one aspect, a first device includes a processor component, a display, and a storage device. The storage includes instructions executable by the processor component to: receive a video from a second device different from the first device; determining whether the user is viewing the display; and based on the determination that the user is not viewing the display, processing the video in a first manner for presentation on the display. The instructions may also be executed to process the video in a second manner for presentation on the display based on a determination that the user is viewing the display. The first approach may include using a lower resolution or a slower frame rate than the second approach. The first approach may also include using a CPU core rather than a GPU.

Description

Technical Field

[0001] The following disclosure relates to technically creative and unconventional solutions that are necessarily rooted in computer technology and produce specific technical improvements. In particular, the following disclosure relates to techniques for dynamic video decoding based on whether a user is watching a video. Background Art

[0002] As recognized herein, video decoding is typically performed by the graphics processing unit (GPU) of a computing device. However, this often results in significant battery drain and / or excessive power consumption from a wall socket. In fact, high-quality video streaming applications often require a dedicated GPU for decoding, as this may be the only processing unit capable of maintaining the optimal frame rate and resolution for the video stream. This, however, can further accelerate battery drain / power consumption.

[0003] Additionally, performing video decoding on the GPU over extended periods of time can lead to excessive heat buildup within the device, adversely affecting the overall lifespan of other hardware components that absorb the generated heat. Heat buildup can also adversely affect system performance in the short term, sometimes even causing other components to become inoperable before the heat dissipates.

[0004] There is currently no adequate solution to the aforementioned computer-related technical problems. Summary of the Invention

[0005] Thus, in one aspect, a first device includes: a processor component; a display accessible by the processor component; and a storage device accessible by the processor component. The storage device includes instructions executable by the processor component to: receive a video from a second device different from the first device; determine whether a user is viewing the display; and based on the determination that the user is not viewing the display, process the video in a first manner for presentation on the display. The instructions are also executable to: based on the determination that the user is viewing the display, process the video in a second manner for presentation on the display, wherein the second manner is different from the first manner.

[0006] In some example implementations, the first mode may include processing the video at a first resolution for presentation on a display, and the second mode may include processing the video at a second resolution for presentation on a display, where the second resolution may be higher than the first resolution.

[0007] Additionally or alternatively, the first mode may include processing the video for presentation on a display at a first frame rate, and the second mode may include processing the video for presentation on a display at a second frame rate. The second frame rate may be faster than the first frame rate.

[0008] As another example, the first approach may include processing the video for presentation on a display using a central processing unit (CPU), and the second approach may include processing the video for presentation on a display using a graphics processing unit (GPU). The GPU may be different from the CPU. Additionally, in various specific example implementations, the first approach may include processing the video for presentation on a display using a larger core of two cores on the CPU, while in other example implementations, the first approach may include processing the video for presentation on a display using a smaller core of at least two cores on the CPU.

[0009] As another specific example, the first manner may include each of the following: processing the video at a first resolution for presentation on a display, processing the video at a first frame rate for presentation on a display, and processing the video using a CPU for presentation on a display. In this case, the second manner may include each of the following: processing the video at a second resolution for presentation on a display, processing the video at a second frame rate for presentation on a display, and processing the video using a GPU for presentation on a display.

[0010] Furthermore, in some cases, the instructions may be executed to receive input from a camera. The instructions may then be executed to use the input from the camera to perform eye tracking to determine whether the user is looking at the display. In some cases, the first device may even include a camera.

[0011] Additionally or alternatively, the instructions may be executed to receive input from a human presence detection (HPD) sensor and then use the input from the HPD sensor to determine whether the user is viewing the display. Here, the first device may even include the HPD sensor itself.

[0012] Furthermore, in various example implementations, the instructions may be further executable to: maintain a buffer of a threshold number of past video frames while processing the video in the first manner. In one specific example, the instructions may even be executable to: maintain a buffer of a threshold number of past video frames at multiple resolutions for each corresponding past video frame while processing the video in the first manner.

[0013] In another aspect, a method includes receiving, at a first device, a video from a second device different from the first device. The method also includes determining whether a user is viewing a display on which the video is to be presented. The method includes: based on determining that the user is not viewing the display, processing the video in a first manner at the first device for presentation on the display. Additionally, the method includes: based on determining that the user is viewing the display, processing the video in a second manner at the first device for presentation on the display. The second manner is different from the first manner.

[0014] In a specific example implementation, the first manner may include refusing to process a first non-I frame of the video for presentation on a display, and the second manner may include processing the first non-I frame of the video for presentation on a display. The first non-I frame may include a P frame of the video and / or a B frame of the video.

[0015] In yet another aspect, at least one computer-readable storage medium (CRSM) that is not a transient signal includes instructions executable by a processor component to: receive, at a first device, a video from a second device different from the first device. The instructions are further executable to: determine whether a user is viewing a display, and based on the determination that the user is not viewing the display, process the video in a first manner for presentation on the display. The instructions are executable to: based on the determination that the user is viewing the display, process the video in a second manner for presentation on the display, the second manner being different from the first manner.

[0016] Thus, in one example, a first approach may include refusing to process the first (e.g., any / all) non-I frames of a video for presentation on a display, and a second approach may include processing the first non-I frame of a video for presentation on a display.

[0017] Additionally or alternatively, the first approach may include processing the video for presentation on a display using a central processing unit (CPU), and the second approach may include processing the video for presentation on a display using a graphics processing unit (GPU). The GPU may be different from the CPU.

[0018] The details of the present principles, both as to their structure and operation, may be best understood with reference to the accompanying drawings in which like reference numerals refer to like parts, and in which: BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a block diagram of an example system consistent with the present principles;

[0020] Figure 2 is a block diagram of an example network of devices consistent with the present principles;

[0021] Figure 3 An example notification consistent with present principles that may be presented when a video has been degraded because the user is watching off-screen is shown;

[0022] Figure 4 An example notification consistent with present principles that may be presented when a video is playing at optimal / maximum quality based on the user currently viewing the video is shown;

[0023] Figure 5 shows example logic in example flow chart format that may be executed by a device consistent with present principles; and

[0024] Figure 6 An example settings GUI that may be presented on a display to configure one or more settings of a device is shown, consistent with present principles. DETAILED DESCRIPTION

[0025] To save power, reduce heat buildup, and improve the functionality and lifecycle of computing devices, the present principles recognize that if the target user is not viewing the video stream window during video playback, the device does not need to present the video at the highest quality resolution and fastest frame rate available, because the user will not be viewing the video anyway. However, when the user revisits the video stream again, the video stream can seamlessly switch back to a higher quality resolution and faster frame rate. In addition, when the user is not watching, a processor that consumes less power overall (e.g., a large CPU core, or a small CPU core for greater power savings) can also be used for video decoding. Note that a given processor chip can include multiple large cores and multiple small cores.

[0026] Therefore, a combination of CPU and GPU processing in accordance with the present principles can be used to decode / process incoming video streams, such as online pre-recorded videos and real-time video conferencing streams. In fact, many ARM (Advanced RISC Machine) processors and x86 processors have both large and small cores. The large core can be performance-oriented and can handle heavier tasks, while the small core is even more efficiency-oriented and can run background tasks while consuming much less energy than the larger cores of the CPU. Therefore, the present principles discuss, in particular, dynamically switching between which processor component handles video decoding to save system power and reduce heat generation where possible (e.g., while also reducing power consumption by not requiring a system fan or not requiring too many system fans to cool system components, which in turn can also help reduce the amount of noise generated by the fan).

[0027] Specifically, disclosed below are systems and methods for detecting when a user is not viewing a video streaming window so that the device can adjust playback to conserve the device's battery or overall power consumption (e.g., from an AC wall outlet). The device can make several adjustments to playback, which can be used individually or in combination.

[0028] For example, for streaming video, there are typically multiple resolutions available for reception. This is done for network bandwidth purposes, but devices operating consistent with the present principles can take advantage of this availability to reduce power consumption by processing a lower resolution / higher compression video stream when the user is looking away.

[0029] Additionally, a device operating in accordance with the present principles may decode / process only the periodic I frames of a video when the user is not watching, and skip or not process the P frames and B frames of the video. The implementation of this option may vary per video based on the frame rate of the I frames of the corresponding video. For example, if an I frame is decoded only every 5 seconds, this may be too slow and result in what is essentially a series of still images rather than a video being presented on the display. Therefore, in this case, P frames and / or B frames may also be decoded. As another example, if the intra-frame (I frame) rate is 2 frames per second (fps), a device operating in accordance with the present principles may skip every other intra-frame or every N intra-frames to further reduce power consumption.

[0030] Furthermore, as described above, the processing / decoding of periodic frames can be moved from the GPU to another processor, such as a power-efficient small core of the CPU. Whether to do so, and which CPU core to assign to handle decoding, can be determined on a video-by-video basis based on the resolution and frame rate of any video being received and the capabilities of the corresponding CPU core itself. Thus, if downgraded video decoding is still a relatively high-load task, but some power savings are desired, a larger CPU core can be used, while for relatively lighter tasks and even greater power savings, a smaller CPU core can be used instead. In one specific, but non-limiting, example implementation, a device can start by always moving decoding to a small CPU core to save the most power, then work backward to figure out the resolution, frame rate, etc. that the small core can handle, and can move decoding to a large CPU core where appropriate. Thus, for example, a device may want to complete 640×480 at one frame per second and therefore determine whether the small core can handle this processing. If so, the small core can be used, but if not, decoding can be moved to a large CPU core. So this can vary based on how long the corelet takes to decode the I frame and whether the corelet is doing anything else that might be adversely affecting decoding capabilities.

[0031] Or as another specific non-limiting example, the device may calculate what is required to utilize the large CPU core and the small CPU core for decoding in a given situation, and then determine based on the calculation whether the small core can decode, and if not, use the large core.

[0032] Then, when the user returns or looks back at the screen, the device can switch video decoding back to the GPU for normal / optimal frame rate and resolution processing and presentation. Given this, in some cases, the video can be processed faster than real time from the last intra frame to all predicted frames (P frames and B frames) to reach the current frame and resume normal playback (for example, especially resuming playback from a P frame or B frame). As a result, a buffer of past frames can be maintained at multiple video resolutions as much as possible.

[0033] For example, while the user is looking at the screen, the video can be decoded with GPU hardware assistance at 60fps and ultra-high-definition (HD) resolution (3840×2160 resolution). When the user looks away, the video can be decoded at 1fps and VGA resolution (640×480 resolution) on the CPU core (for example, using a software codec).

[0034] With respect to gaze detection, the device can detect whether the user is not looking at the video stream window by using human presence detection methods (including eye tracking using a camera, among other methods). Note that the delay / latency in gaze tracking can be short enough that the user's experience can be changed in an almost imperceptible manner based on the above-mentioned frame rate and resolution adjustments. Similarly, moving decoding tasks between cores and / or codecs and / or GPUs can also be done imperceptibly and quickly.

[0035] As for the decoding itself, it should be understood consistent with the present principles that video decoding can include converting a compressed digital video file (encoded video) into a decompressed, playable format that can be presented by a device. Thus, during decoding, the compressed video data can be decompressed and converted back to its original format (and therefore larger overall size), allowing the video to be presented in a playable form. Thus, a video codec can be used to decode and decompress received video data.

[0036] Before going into further detail of the present technology, it is noted with respect to any of the computer systems discussed herein that the system may include server and client components connected via a network such that data may be exchanged between the client and server components. The client components may include one or more computing devices including televisions (e.g., smart TVs, Internet-enabled televisions), computers such as desktop computers, laptop computers, and tablet computers, so-called convertible devices (e.g., having a tablet configuration and a laptop configuration), and other mobile devices including smartphones. As non-limiting examples, these client devices may employ an operating system from Apple Inc. of Cupertino, California, Google Inc. of Mountain View, California, or Microsoft Corporation of Redmond, Washington. The operating system may be used or similar An operating system, such as Chrome, Android, Windows, or macOS, can also be used. These operating systems can execute one or more browsers, such as those made by Microsoft, Google, or Mozilla, or another browser program that can access web pages and applications hosted by Internet servers through a network, such as the Internet, a local intranet, or a virtual private network.

[0037] As used herein, instructions refer to computer-implemented steps for processing information in a system. Instructions can be implemented in software, firmware, or hardware, or a combination thereof, and include any type of programming steps undertaken by components of the system; therefore, exemplary components, blocks, modules, circuits, and steps are sometimes described in terms of their functionality.

[0038] The processor can be any single-chip processor or multi-chip processor that can execute logic by means of various lines such as address lines, data lines and control lines, as well as registers and shift registers. In addition, any logic blocks, modules and circuits described herein can be implemented or executed by the following designed to perform the functions described herein: a system processor, a digital signal processor (DSP), a field programmable gate array (FPGA), or other programmable logic devices such as application specific integrated circuits (ASICs), discrete gate or transistor logic, discrete hardware components, or any combination thereof. The processor can also be implemented by a combination of a controller or a state machine or a computing device. Therefore, the method herein can be implemented as software instructions executed by a processor, a suitably configured application specific integrated circuit (ASIC) or a field programmable gate array (FPGA) module, or implemented as any other convenient form that a person skilled in the art will understand. In the case of adoption, the software instructions can also be embodied in a non-transient device being sold and / or provided, and the non-transient device is not a transient propagation signal and / or the signal itself. For example, the non-transient device can be or include a hard drive, a solid-state drive or a CD ROM. A flash drive can also be used to store instructions. Additionally, the software code instructions may also be downloaded via the Internet (e.g., as part of an application ("app") or software file). Thus, it should be understood that while a software application for implementing the present principles may be sold with a device such as the system 100 described below, such an application may also be downloaded to the device from a server via a network such as the Internet. The application may also be run on the server, and the associated presentation may be displayed on a client device in communication with the server via a browser (and / or via a dedicated companion app).

[0039] The software modules and / or applications described herein by flow charts and / or user interfaces may include various subroutines, programs, etc. Without limiting the present disclosure, logic stated as being performed by a particular module may be reallocated to other software modules and / or combined together in a single module and / or made available in a shareable library. Furthermore, the user interfaces (UIs) / graphical UIs described herein may be merged and / or extended, and UI elements may be mixed and matched between UIs.

[0040] When implemented in software, the logic may be written in a suitable language such as, but not limited to, Hypertext Markup Language (HTML)-5, The program may be written in C# or C++ and may be stored on or transferred from a computer-readable storage medium, such as a hard disk drive (HDD) or solid-state drive (SSD), random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), a hard disk drive (HDD) or solid-state drive, a compact disk read-only memory (CD-ROM), or other optical disk storage such as a digital versatile disk (DVD), a magnetic disk storage, or other magnetic storage including a removable thumb drive, etc.

[0041] In an example, the processor can access information from a data storage device, such as a computer-readable storage medium, via its input lines, and / or the processor can wirelessly access information from an Internet server by activating a wireless transceiver to send and receive data. Data is typically converted from analog to digital by circuitry between an antenna and a processor register when being received, and from digital to analog by circuitry between an antenna and a processor register when being transmitted. The processor then processes the data via its shift register to output the calculated data on an output line so that the calculated data can be presented to the device.

[0042] The components included in one embodiment may be used in any appropriate combination in other embodiments. For example, any of the various components described herein and / or depicted in the accompanying drawings may be combined, interchanged, or excluded from other embodiments.

[0043] “A system having at least one of A, B, and C” (similarly, “a system having at least one of A, B, or C” and “a system having at least one of A, B, and C”) includes a system having A alone, a system having B alone, a system having C alone, a system having both A and B, a system having both A and C, a system having both B and C, and / or a system having both A, B, and C, etc.

[0044] The term "circuit" or "circuitry" may be used in the abstract, description, and / or claims. As is well known in the art, the term "circuitry" includes all levels of available integration, for example, from discrete logic circuits to the highest levels of circuit integration such as VLSI, and includes programmable logic components programmed to perform the functions of the embodiments and processors (e.g., special purpose processors) programmed with instructions to perform those functions.

[0045] Now refer to Figure 1, shows an example block diagram of an information handling system and / or computer system 100, which is understood to have housings for the components described below. Note that in some embodiments, system 100 may be a desktop computer system such as a personal computer sold by Lenovo (USA), Inc. of Morrisville, North Carolina. or series, or may be a workstation computer such as the one sold by Lenovo (USA) Inc. of Morrisville, North Carolina However, as is apparent from the description herein, a client device, server, or other machine according to the present principles may include other features or only some of the features of the system 100. Furthermore, the system 100 may be, for example, a , and / or system 100 may include a mobile communication device such as a mobile phone, a laptop computer, and / or other portable computerized device.

[0046] like Figure 1 As shown, system 100 may include a so-called chipset 110. A chipset refers to a group of integrated circuits or chips designed to work together. Chipsets are typically sold on the market as a single product (e.g., consider the and other brands of chipsets).

[0047] exist Figure 1 In the example of , chipset 110 has a specific architecture that can vary to some extent depending on the brand or manufacturer. The architecture of chipset 110 includes cores and memory control group 120 and I / O controller hub 150 that exchange information (e.g., data, signals, commands, etc.) via, for example, direct management interface or direct media interface (DMI) 142 or link controller 144. Figure 1 In the example shown, DMI 142 is a chip-to-chip interface (sometimes referred to as the link between a "north bridge" and a "south bridge").

[0048] The core and memory control group 120 includes a processor component 122 (e.g., one or more single-core or multi-core processors, etc.) that exchanges information via a front-side bus (FSB) 124, and a memory controller hub 126. Thus, a processor component, such as component 122, may include one or more processors that act independently or in conjunction with each other to execute an algorithm, whether the processors are in one device or in more than one device. Additionally, as described herein, various components of the core and memory control group 120 may be integrated onto a single processor die, for example, to create a chip that replaces a "Northbridge" architecture.

[0049] The memory controller hub 126 interfaces with the memory 140. For example, the memory controller hub 126 can provide support for DDR SDRAM memory (e.g., DDR, DDR2, DDR3, etc.). Typically, the memory 140 is a type of random access memory (RAM). The memory 140 is often referred to as "system memory."

[0050] The memory controller hub 126 may also include a low voltage differential signaling interface (LVDS) 132. The LVDS 132 may be a so-called LVDS display interface (LDI) for supporting a display device 192 (e.g., a CRT, a flat panel, a projector, a touch-enabled light-emitting diode (LED) display, or other video display). Block 138 includes some examples of technologies that can be supported via the LVDS interface 132 (e.g., serial digital video, HDMI / DVI, display port). The memory controller hub 126 also includes one or more PCI-express interfaces (PCI-E) 134, for example, for supporting a discrete graphics card 136. Dedicated graphics cards using a PCI-E interface have become an alternative to accelerated graphics ports (AGP). For example, the memory controller hub 126 may include a 16-lane (x16) PCI-E port for an external PCI-E-based graphics card (including, for example, one or more GPUs). Example systems may include AGP or PCI-E for supporting graphics cards.

[0051] In examples using the I / O hub controller 150 , the I / O hub controller 150 may include various interfaces. Figure 1 Examples include a SATA interface 151, one or more PCI-E interfaces 152 (optionally, one or more legacy PCI interfaces), one or more universal serial bus (USB) interfaces 153, a local area network (LAN) interface 154 (more generally a network interface for communicating under the direction of the processor 122 over at least one network, such as the Internet, a WAN, a LAN, a Bluetooth network using Bluetooth 5.0 communication, etc.), a general purpose I / O interface (GPIO) 155, a low pin count (LPC) interface 170, a power management interface 161, a clock generator interface 162, an audio interface 163 (e.g., for a speaker 194 to output audio), a total cost of operation (TCO) interface 164, a system management bus interface (e.g., a multi-host serial computer bus interface) 165, and a plurality of other interfaces. Figure 1Examples of the I / O hub controller 150 include a basic input / output system (BIOS) 168 and a serial peripheral flash memory / controller interface (SPI flash memory) 166 for boot code 190. Regarding network connectivity, the I / O hub controller 150 may include an integrated Gigabit Ethernet controller line multiplexed with the PCI-E interface port. Other network features may operate independently of the PCI-E interface. Example network connections include Wi-Fi and wide area networks (WANs) such as 4G and 5G cellular networks.

[0052] The interfaces of the I / O hub controller 150 can provide communication with various devices, networks, etc. For example, where used, the SATA interface 151 and / or the PCI-E interface 152 provide for reading, writing, or both reading and writing information on one or more drives 180, such as HDDs, SDDs, or a combination thereof, but in any case, the drive 180 is understood to be a tangible computer-readable storage medium, such as a non-transitory propagating signal. The I / O controller hub 150 may also include an Advanced Host Controller Interface (AHCI) to support one or more drives 180. The PCI-E interface 152 allows wireless connections 182 to devices, networks, etc. The USB interface 153 provides input devices 184, such as a keyboard (KB), a mouse, and various other devices (e.g., a camera, a phone, a storage device, a media player, etc.).

[0053] exist Figure 1 In the example of FIG. 1 , LPC interface 170 provides access to one or more ASICs 171, a trusted platform module (TPM) 172, a super I / O 173, a firmware hub 174, BIOS support 175, and various types of memory 176, such as ROM 177, flash memory 178, and non-volatile RAM (NVRAM) 179. With respect to TPM 172, the module may be in the form of a chip that can be used to authenticate software and hardware devices. For example, the TPM may be capable of performing platform authentication and may be used to verify that the system seeking access is the intended system.

[0054] System 100 may be configured to execute boot code 190 for BIOS 168 stored in SPI flash memory 166 upon power-up and thereafter process data under the control of one or more operating systems and application software (e.g., stored in system memory 140). The operating system may be stored in any of a variety of locations and accessed, for example, according to instructions from BIOS 168.

[0055] Additionally, the system 100 may also include one or more cameras 191 that collect one or more images and provide images and related input (e.g., for eye tracking consistent with the present principles) to the processor component 122. The camera may be a thermal imaging camera, an infrared (IR) camera, a digital camera such as a webcam, a three-dimensional (3D) camera, and / or a camera that is otherwise integrated into the system 100 and controllable by the processor component 122 to collect still images and / or video of the user's eyes during video consumption.

[0056] The system 100 may also include one or more human presence detection (HPD) sensors 193, such as, but not limited to, a microphone (for detecting user speaking and thereby detecting user presence), an infrared (IR) sensor (for detecting heat and / or motion and thereby detecting user presence), a keyboard and trackpad and a mouse and touch-enabled display (for detecting user input via use of input devices and thereby detecting user presence), and a Bluetooth transceiver and other wireless transceivers (for detecting wireless signals from another device associated with the user and thereby detecting user presence).

[0057] Additionally, although not shown for the sake of brevity, in some embodiments, system 100 may include a gyroscope that senses and / or measures the orientation of system 100 and provides related input to processor assembly 122, an accelerometer that senses acceleration and / or motion of system 100 and provides related input to processor assembly 122, and / or a magnetometer that senses and / or measures directional motion of system 100 and provides related input to processor assembly 122. Still further, system 100 may include an audio receiver / microphone that provides input from a microphone to processor assembly 122 based on audio detected, for example, via a user providing audible input to the microphone.

[0058] Additionally, the system 100 may include a Global Positioning System (GPS) transceiver configured to communicate with a satellite to receive / identify geographic location information and provide the geographic location information to the processor component 122. However, it should be understood that another suitable location receiver other than a GPS receiver may be used to determine the location of the system 100 in accordance with present principles.

[0059] It should be understood that an example client device or other machine / computer may include, for example, Figure 1 The system 100 may have fewer or more features than those shown. In any case, it should be understood, at least based on the foregoing, that the system 100 is configured to undertake the present principles.

[0060] Now go to Figure 2, shows example devices communicating over a network 200 such as the Internet (e.g., for video streaming from one device to another) consistent with the present principles. It should be understood that with reference to Figure 2 Each of the described devices may include at least some of the features, components, and / or elements described above for system 100. In fact, any device disclosed herein may include at least some of the features, components, and / or elements described above for system 100.

[0061] Figure 2 Shown are a laptop and / or convertible computer 202, a desktop computer 204, a wearable device 206 such as a smartwatch, a smart television (TV) 208, a smart phone 210, a tablet 212, and a server 214 such as an Internet server that can provide cloud storage accessible by the devices 202 to 212. It should be understood that the devices 202 to 214 can be configured to communicate with each other over the network 200 to undertake the present principles.

[0062] Now refer to Figure 3 , assume that the end user is streaming a video to his or her device, such as a smartphone, laptop, wearable device, computerized smart glasses, or even an augmented reality (AR) or virtual reality (VR) head-mounted device. For example, the video may come from a video sharing website or a social media platform. The video can also be streamed as part of a real-time video conference in which the end user participates, and thus the video can be a real-time video showing the faces of other conference participants. Other types of videos can also be received and processed consistent with the present principles.

[0063] in any case, Figure 3 A graphical user interface (GUI) 300 is shown that may be presented on a display of the device. The GUI 300 itself may present the received and decoded video in real time as the video is received and processed (in Figure 3 310 in the figure). Note, however, that in this example, the video presentation has been downgraded to a lower resolution and slower frame rate because the device determined that the user is not generally viewing the device's display or is not even viewing the particular display area where the video is actually being presented (e.g., where the video is presented in a non-maximized or non-full screen window). Additionally or alternatively, when the user is not viewing, the device can offload video decoding from the GPU to a larger or smaller core of the CPU, also resulting in a downgraded video presentation but one that consumes less power to present.

[0064] In addition to performing the above actions in response to determining that the user is not viewing the display / video window, the device may present a graphical notification 320 over the video 310. The notification 320 may include a textual indication 330 that the video has been degraded. The notification 320 may also include a selector 340 that may be selected to instruct the device to stop degrading the video presentation and use the device's GPU to present the video 310 at the best / maximum available resolution and frame rate. Thus, if the device's eye tracking results in a false positive—the user did not actually look away but the device has determined so—the user may decide to select the selector 340. Alternatively, the user may decide to select the selector 340 in other situations, such as when more than one person is viewing the video 320 on the device's display and the device may only be tracking the gaze of one of the two users who happens to be viewing off-screen (while the other user continues viewing the video and therefore does not want the video to be degraded).

[0065] Go to Figure 4 , GUI 300 is again presented on the display, which again includes presenting video 310 itself as part of GUI 300. However, notification 320 may be provided by Figure 4 300 in the GUI 300. Notice 400 can be presented in response to determining that the user is looking back at the display of the device again. Therefore, notice 400 can include the text indication 410 that video has been upgraded and / or presented with the best or optimum available video quality again. Notice 400 also includes a selector 420 that meets the present principle, and this selector 420 can be selected to command equipment to degrade video presentation (for example, using slower frame rate and lower resolution to process video 310 on CPU core rather than GPU), even if the user may still be determined to be watching display / video window. For example, if the user does not pay special close attention to video 310 and hopes to save power, if the user is always scanning back and forth between display and other things and therefore does not expect optimal video quality, or if the user only wants to reduce power consumption and heat generation during video playback, then the user can decide to select selector 420.

[0066] Considering the above, now refer to Figure 5 The diagram illustrates a method that may be implemented by a first device, such as system 100, processor component 122, and / or Figure 3 and Figure 4 Example logic executed by the client device of . Note that although Figure 5 The logic of is shown in a flow chart format, but other suitable logic may be used. Also note that the first device may be a client device, such as a smartphone, laptop, wearable device, head mounted device, etc.

[0067] Beginning at block 500, a first device can receive a video from a second device that is different from the first device. For example, the second device can be an Internet-based server or a cloud platform that hosts pre-recorded videos uploaded by other end users, such as a YouTube server. Or the second device can be a server for a video streaming service, such as Netflix or Amazon Prime. The second device can also be a video conferencing server that routes audio and video streams between participating client devices, or even another one of the conference client devices itself. The second device can also be another client device outside the video conferencing environment, such as another client device from which a pre-recorded video or a live, real-time video stream is received. Other implementations of the second device are also encompassed by the present principles.

[0068] The logic may then proceed from block 500 to block 510. At block 510, the first device may receive input from one or more sensors, such as a camera (e.g., camera 191) or other HPD sensors (including the other HPD sensors 193 mentioned above). The logic may then proceed to block 520, where, if camera input is received, an eye tracking algorithm may be executed using the camera input to determine whether the user is viewing the display of the first device (at decision diamond 530).

[0069] Additionally or alternatively, at block 520, the first device may process other types of HPD sensor input that may be received at block 510 to determine (at diamond 530) whether the user is viewing the display of the first device. For example, if the HPD sensor is a microphone, the sound input to the microphone may be processed to identify that a person is speaking, in which case it can be assumed that the person is viewing the display. Alternatively, the microphone input may be processed using natural language understanding (NLU) to determine that the person speaking is specifically talking about content associated with the video itself (e.g., as identified from video metadata or NLU performed on the video's associated audio content), in which case it can be assumed that the person is viewing the display. Additionally or alternatively, speech recognition may be performed using the microphone input to identify the user speaking as a specific user associated with the first device, in which case it can be assumed that the person is viewing the display because the person is already a registered / identified user.

[0070] As another example, input from an IR sensor can be used to detect heat and / or motion via the IR sensor input, in which case it can be assumed that the user who is the source of the heat or motion is viewing the display. As yet another example, input received at a connected input device such as a keyboard, a touch-enabled laptop trackpad, a mouse, and / or a touch-enabled display can be used to detect user presence due to a person first initiating the input, in which case it can be assumed that the person is viewing the display.

[0071] As another non-limiting example, the present principles also encompass other HPD technologies that may also use input from a Bluetooth transceiver and other wireless transceivers (e.g., ultra-wideband transceivers, Wi-Fi transceivers, and / or near-field communication transceivers) to determine that another client device is present due to a wireless signal being received from the other client device, in which case it may be assumed that a person nearby is viewing the display due to the presence of the other client device. In some specific examples, the wireless signal received from the other client device may include identification information of the other client device itself, which may then be associated with a particular user registered with the first device or otherwise identifiable as a past user of the first device (and the first device may therefore assume that the identified / registered user is viewing the display). Additionally or alternatively, a unique user identifier (UUID) by which a user may be similarly identified may be provided via the wireless signal, and this unique user identifier may then be used to assume that a user is viewing the display based on the presence of the UUID in the signal.

[0072] Further, in some examples, a received signal strength indicator (RSSI) algorithm can be performed using the received wireless signal to not only identify a given user that may be registered or otherwise associated with the first device as being present somewhere around the first device, but also to determine that the other client device itself is within a threshold non-zero distance of the first device. The threshold distance can be less than the maximum range over which the respective wireless signals can be transmitted, and can be an amount sufficient to infer not only the presence of a user in the general area around the first device, but also the actual user proximity to the first device, assuming that the user is close enough to potentially be viewing the display of the first device. Thus, when the first device is a laptop, smartphone, wearable device, or head-mounted device, the threshold distance can be ten feet or less. If the first device is instead a television, the threshold distance may be more based on the recognition that people typically sit farther from a television than they might from a laptop or smartphone display, and thus the threshold distance for a television implementation may be twenty feet.

[0073] Still refer to Figure 3, and thus any of the eye tracking / HPD techniques mentioned above can be used to determine whether the user is actually looking at the display. In response to the first device determining that the user is not looking at the display, one or more of blocks 550 to 556 can then be executed to process the video in a first manner for presentation on the display. However, in response to the first device determining that the user is actually looking at the display, one or more of blocks 560 to 564 can alternatively be executed to process the video in a second manner for presentation on the display.

[0074] Starting from blocks 550 and 560, at block 550, the first device may process the video at a first resolution for presentation on the display, and at block 560, the first device may alternatively process the video at a second resolution higher than the first resolution for presentation on the display. For example, while the user is viewing the display, at block 560, the video may be decoded at an ultra-high-definition (UHD) resolution (3840×2160 resolution), while when the user looks away from the display (or not at all), at block 550, the video may be decoded at a VGA resolution (640×480 resolution). Thus, for example, in streaming video, there are typically multiple resolutions available for streaming / receiving. Therefore, the first device can take advantage of this availability to reduce power consumption by requesting and then processing an available lower resolution / higher compression video stream when the user looks away from the display, and then switching to requesting and processing the available higher resolution / lower compression video stream when the user looks back at the display.

[0075] Turning to blocks 552 and 562, at block 552, the first device may process the video at a first frame rate for presentation on the display, and at block 562, the first device may process the video at a second frame rate that is faster than the first frame rate for presentation on the display. For example, when the user is viewing the display, the video may be decoded at 60 fps, and when the user looks away from the display, at block 562, the video may be decoded at 1 fps.

[0076] Still with respect to frame rate and to provide another example, at block 552, the device may refuse to process the first non-I frame of the video for presentation on the display (e.g., refuse to process P frames and B frames) because the user is not viewing the display at this time. However, at block 562, based on determining that the user is viewing the display, the first device may instead process the first non-I frame of the video for presentation on the display. The first non-I frame may include all available / received non-I frames.

[0077] Note that I-frames are the least compressible of the three, but typically do not require other video frames to decode these I-frames (e.g., these I-frames can be fully rendered on the display by themselves without using data from previous or subsequent frames to render the complete image). In contrast, P-frames can use data from previous frames for decompression and rendering of the P-frame (and are more compressible than I-frames). B-frames can use both previous and subsequent frames for data references to achieve higher data compression for subsequent decompression and rendering by the first device.

[0078] As for blocks 554 and 564, the first device can use a larger core or a smaller core of the CPU to process the video for presentation on the display, and at block 564, the first device can alternatively use a relatively high-performance GPU to process the video for presentation on the display. For example, when the user is watching the display, the GPU can be used to decode the video, and when the user looks away from the display, the larger "big" core of the CPU can be used to decode the video to reduce power consumption and heat generation while still maintaining a relatively high performance level. Or for even greater power savings and heat reduction, the smaller "little" core of the CPU can be used instead of the GPU. For example, in either case, the CPU can still use a software codec for decoding.

[0079] Note that the logic can proceed directly from block 564 to block 570 , as will be briefly described. However, with respect to block 554 , note that the logic can proceed from block 554 to block 556 before ultimately proceeding to block 570 .

[0080] At block 556 and still based on determining that the user is not viewing the display, the first device may maintain a buffer of a threshold amount of past video frames while processing the video in the first manner. The buffer may be maintained in random access memory (RAM) of the first device or in another storage location, such as a persistent solid state drive (SDD) or hard disk drive (HDD) storage location. In any case, buffering of past images may be performed to process images faster than real time from the last / most recent past I-frame to all subsequent predicted frames (P-frames and B-frames) that may also be past frames, to reach rendering of the current frame and then resume normal playback, even without any intervening additional I-frames being received and / or processed (e.g., resuming playback from the current P-frame or B-frame). Thus, the threshold amount of past video frames maintained in the buffer may include all P-frames and B-frames following the most recently received I-frame (the most recently received I-frame is also maintained in the buffer until the next I-frame in the sequence of I-frames is received). Thus, reference to the (most recent) I frame and intermediate P and B frames can be accumulated in a buffer and then removed / deleted from the buffer in response to another I frame being received (because the past P and B frames that referenced the past I frame will no longer be needed for image rendering now that the subsequent I frame has been received).

[0081] Additionally, in one specific example implementation, the first device may maintain a buffer of a threshold amount of past video frames at not only one resolution but also multiple resolutions for each corresponding past video frame at block 556. This may be done so that despite any possible network bandwidth issues or other constraints, the highest available resolution of the frame may be dynamically selected and used on the fly, where each corresponding I-frame, B-frame, and / or P-frame buffered at the multiple resolutions has been cached in local storage for real-time playback without delay.

[0082] The aforementioned block 570 is now described, noting that it can be reached via block 556 or block 564. At block 570, the first device can return to decision diamond 530 to again determine whether the user is viewing the display during playback of a later portion of the same video. Figure 5 The process may be repeated continuously during video playback to adjust playback back and forth between different resolutions, frame rates, and decoding processors, thereby enabling the first device to be more energy-efficient in operation where possible.

[0083] Therefore, in accordance with Figure 5In one specific example implementation of the logic, when the first device detects that the user is not viewing the display, the first device can move video decoding from the high-power GPU to the lower-power small CPU core. Recognizing that the small CPU core may not have the same performance as the GPU, in this case, when moving video decoding to the small CPU core, the first device can reduce the resolution, reduce the frame rate, and skip B frames, P frames, and even I frames to meet the performance factors of the small CPU core.

[0084] about Figure 5 It should also be noted that if the user follows the logic above, for example Figure 3 and Figure 4 If the user input described is used to override the current processing of video decoding by the GPU or the CPU big core or small core, the user's command will override the above logic, and the device will operate according to the user's command.

[0085] Also, note that in the case where decoding is offloaded to the CPU big or small cores, the GPU can be shut down or otherwise powered down during CPU decoding to save additional power. Then, when the user is determined to be looking back at the display again, the GPU can be powered back on and used for video decoding.

[0086] Now go to Figure 6 , shows another example GUI 600 that may be presented on a display of a client device configured to implement the present principles. The GUI 600 may be presented to set up or enable a first device (e.g., its guest operating system) or a particular video streaming app stored at the first device (which itself may execute Figure 5 Thus, for example, the GUI 600 can be navigated through a device or app menu. Also note that each of the example options discussed below can be selected by directing a touch or cursor input to an associated checkbox adjacent to the corresponding option.

[0087] like Figure 6 As shown, the GUI 600 may include an option 610 that may be selected to set or configure the device / app to assume the present principles. Thus, a single selection of option 610 may be made to set or enable the device / app to perform in multiple future instances of video streaming (e.g., for different discrete videos / video files). Figure 5 The logic and presentation Figure 3 and Figure 4 Thus, option 610 may be selected to enable dynamic video decoding / processing using user gaze to save power and reduce heat generation that may otherwise shorten the overall life of the various components within the first device.

[0088] GUI 600 may also include option 620. Option 620 may be selected to set the device to, or enable the device to, specifically not perform dynamic video processing based on user gaze when two or more people are determined to be at least present (if not looking directly at the display of the first device). This may be accomplished using camera input and computer vision and / or facial recognition to identify the number of people shown in the camera input. As another example, microphone input may be processed to identify that two different voices are detected, thereby inferring the presence of two or more people. IR sensors and other types of HPD sensors may also be used to determine this situation. Thus, in a specific example, both options 610 and 620 may be selected simultaneously so that when a single user present is determined to have moved their gaze away from the display, the video presentation may be dynamically degraded, and so that when more than one user is present, the video presentation is not degraded, so that even if one of the users happens to move their gaze away and the device detects this situation, the best video quality may be used.

[0089] from Figure 6 Continuing, as described above, in some examples, a device operating consistent with the present principles may determine more than just whether a user is generally viewing a display. Instead, the device may specifically determine whether the user is viewing a particular display area or window on which the video content itself is being presented, where that area or window may be smaller than the entire area of the display on which images may be presented. Thus, if a user is viewing a video in a relatively small window on a display while also processing a word processing document, replying to an email, or viewing other content that is simultaneously presented on the display, the device may determine that the user is viewing the other content based on the window. Figure 5 The first way to present the video, and then when the user switches back to watching the video itself according to Figure 5 The second way to present the video.

[0090] As another example, if the user maximizes another window in the foreground as the active window, and minimizes the video's window or at least minimizes it in the background as an inactive window (e.g., even if still partially or fully presented on some portion of the display), the device may maximize the video's window in the foreground as the active window. Figure 5 The first way to present the video, and then when the user switches back to the video window as the active window in the foreground Figure 5 The second way to present the video.

[0091] It can now be appreciated that the present principles provide an improved computer-based user interface that increases the functionality and overall lifespan of the devices disclosed herein while also making the devices more energy efficient.The disclosed concepts are rooted in computer technology that enables computers to perform their functions.

[0092] The components included in one embodiment may be used in any appropriate combination in other embodiments. For example, any of the various components described herein and / or depicted in the accompanying drawings may be combined, interchanged, or excluded from other embodiments.

[0093] It should be understood that although the present principles have been described with reference to certain example embodiments, these example embodiments are not intended to be limiting, and that various alternative arrangements may be used to implement the subject matter claimed herein. Thus, although specific techniques and devices have been shown and described in detail herein, it should be understood that the subject matter encompassed by this application is limited only by the claims.

Claims

1. A first device, comprising: Processor components; a display accessible by the processor component; as well as a storage device accessible by the processor component and comprising instructions executable by the processor component to: receiving a video from a second device different from the first device; determining whether a user is viewing the display; processing the video in a first manner for presentation on the display based on a determination that the user is not viewing the display; as well as Based on a determination that the user is viewing the display, the video is processed for presentation on the display in a second manner that is different from the first manner.

2. The first device according to claim 1, wherein The first manner comprises processing the video at a first resolution for presentation on the display, and wherein the second manner comprises processing the video at a second resolution for presentation on the display, the second resolution being higher than the first resolution.

3. The first device according to claim 1, wherein The first mode comprises processing the video for presentation on the display at a first frame rate, and wherein the second mode comprises processing the video for presentation on the display at a second frame rate, the second frame rate being faster than the first frame rate.

4. The first device according to claim 1, wherein The first manner includes processing the video for presentation on the display using a central processing unit (CPU), and wherein the second manner includes processing the video for presentation on the display using a graphics processing unit (GPU), the GPU being different from the CPU.

5. The first device according to claim 4, wherein The first manner includes using a larger of two cores on the CPU to process the video for presentation on the display.

6. The first device according to claim 4, wherein The first manner includes using a smaller core of at least two cores on the CPU to process the video for presentation on the display.

7. The first device according to claim 1, wherein: The first method includes any of the following: processing the video at a first resolution for presentation on the display, processing the video at a first frame rate for presentation on the display, and processing the video using a central processing unit (CPU) for presentation on the display; and The second method includes any of the following: processing the video at a second resolution for presentation on the display, processing the video at a second frame rate for presentation on the display, and processing the video using a graphics processing unit (GPU) for presentation on the display; The second resolution is higher than the first resolution, the second frame rate is faster than the first frame rate, and the GPU is different from the CPU.

8. The first device according to claim 1, wherein The instructions can be executed to perform the following operations: receiving input from a camera device; and Eye tracking is performed using input from the camera to determine whether the user is looking at the display.

9. The first device according to claim 8, comprising the camera arrangement.

10. The first device according to claim 1, wherein The instructions can be executed to perform the following operations: receiving input from a human presence detection (HPD) sensor; and Input from the HPD sensor is used to determine whether the user is viewing the display. The first device of claim 10 , comprising the HPD sensor.

12. The first device according to claim 1, wherein The instructions can be executed to perform the following operations: While processing the video in the first manner, a buffer of a threshold amount of past video frames is maintained.

13. The first device according to claim 1, wherein The instructions can be executed to perform the following operations: While processing the video in the first manner, a buffer of a threshold amount of past video frames is maintained at multiple resolutions for each corresponding past video frame.

14. A method comprising: receiving, at a first device, a video from a second device different from the first device; determining whether a user is viewing a display on which the video is to be presented; processing the video in a first manner at the first device for presentation on the display based on determining that the user is not viewing the display; as well as Based on determining that the user is viewing the display, the video is processed at the first device in a second manner for presentation on the display, the second manner being different from the first manner.

15. The method according to claim 14, wherein The first manner comprises refusing to process a first non-I frame of the video for presentation on the display, and wherein the second manner comprises processing the first non-I frame of the video for presentation on the display.

16. The method according to claim 15, wherein The first non-I frame includes a P frame of the video.

17. The method according to claim 15, wherein: The first non-I frame comprises a B frame of the video.

18. At least one computer-readable storage medium (CRSM), the at least one CRSM not being a transient signal, the at least one CRSM comprising instructions executable by a processor component to: receiving, at a first device, a video from a second device different from the first device; determining whether a user is viewing the display; processing the video in a first manner for presentation on the display based on a determination that the user is not viewing the display; as well as Based on a determination that the user is viewing the display, the video is processed for presentation on the display in a second manner that is different from the first manner.

19. The at least one CRSM of claim 18, wherein: The first manner comprises refusing to process a first non-I frame of the video for presentation on the display, and wherein the second manner comprises processing the first non-I frame of the video for presentation on the display.

20. The at least one CRSM of claim 18, wherein: The first manner includes processing the video for presentation on the display using a central processing unit (CPU), and wherein the second manner includes processing the video for presentation on the display using a graphics processing unit (GPU), the GPU being different from the CPU.

Images