Gaze as a substitute for attention in video streaming services
By using user gaze tracking and machine learning to dynamically adjust stream quality and resolution, multi-view streaming applications provide a more natural and seamless user experience by adapting to user attention shifts.
Patent Information
- Application Number
- JP2024531050
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-11-24
- Filing Date
- 2022-11-22
- Publication Date
- 2024-11-19
AI Technical Summary
Multi-view streaming applications face challenges in supporting rich user experiences within constraints of technology resources and user attention, limiting the number of simultaneous high-quality views due to user focus shifts that current methods fail to capture naturally.
Utilizing user gaze as a substitute for attention, leveraging inexpensive cameras and software for eye tracking, combined with machine learning to dynamically adjust stream quality and resolution based on user focus, creating backup streams for seamless transitions.
Enables a more natural and seamless user experience in multi-view streaming by adaptively adjusting stream quality and resolution according to user attention, supporting multiple views with high quality and minimizing performance degradation.
Smart Images

Figure 2024543133000001_ABST
Abstract
Description
[Technical field]
[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Patent Application No. 63 / 282,954, filed November 24, 2021, entitled EYE GAZE AS A PROXY OF ATTENTION FOR VIDEO STREAMING SERVICES, which is incorporated by reference in its entirety herein.
[0002] Various embodiments disclosed herein relate to gaze as a substitute for attention in video streaming services. [Background technology]
[0003] Internet streaming is the most dominant application generating traffic on the Internet. In a typical streaming service, video and / or audio data is streamed as a sequence of packets from a collection of servers to multiple users for playback. Unlike video-on-demand, Internet streaming content is recorded and broadcast as it occurs to one or more users. Internet streaming can be live or real-time.
[0004] For the purposes of this discussion, live media streaming refers to streaming content that is transmitted over the Internet without first being recorded and stored. Live video streaming services have a high tolerance for stream latency, the amount of time between when a streamed event or specific action within an event occurs and when it is delivered to subscribers. Streaming services exploit this latency by using large buffers where packets can be collected before being played out. For live video streaming provided by solutions such as Apple's HLS, stream latency of tens of seconds is not uncommon and is generally considered to be acceptable.
[0005] For ease of explanation, streaming real-time media refers to online streaming of media that is recorded and broadcast simultaneously as it occurs to one or more users, with latency low enough to allow natural interaction between participants. These streaming services have low stream latency, allowing participants to respond to and / or interact with the content and other participants in the same stream. A scenario diagram of streaming real-time media depicts performers responding to requests from online viewers or from players placing bets in a real-time stream game of cards while cards are being turned. It should be noted that this definition is stated not in terms of an inherent latency threshold (e.g., less than 0.5 seconds), but more generally in terms of supporting natural user interactivity.
[0006] Both live and real-time streaming require the use of codecs to compress, localize, transport, and use media, network, and application protocols. Commonly used video compression standards include AVC H.264, H.265, VP8, and VP9.
[0007] A media stream consists of a sequence of keyframes, sometimes called intraframes in VP8 and keyframes in MPEG, and predicted frames, called interframes in VP8 and P-frames in MPEG terminology. Keyframes are decoded into a sequence without reference to other frames; that is, the decoder reconstructs such frames from their default state (see Figure 1). Keyframes provide random access or seek points in the video stream. Predicted frames are encoded with reference to previous frames, in particular all previous frames including the most recent keyframe. Correct decoding of a predicted frame depends on the correct decoding of the most recent keyframe and all predicted frames following it. Thus, the decoding algorithm does not tolerate missing keyframes. In an environment where frames may be missing or corrupted, correct decoding is not possible until a keyframe is correctly received.
[0008] State-of-the-art media streaming systems rely on multi-bitrate (MBR) encoding, where the source video content is encoded with alternative bitstreams whose coding rates vary. In this case, the content is streamed in segments or chunks at various quality levels corresponding to different coding rates based on changing network conditions (usually using bitstream switching between segments).
[0009] A common approach for real-time streaming is to use adaptive bitrate streaming, where an encoder encodes a single source video at multiple bitrates. The player client switches between streaming different encodings depending on available resources. Specifically, the source content is encoded at multiple bitrates and may be divided into small parts of a few seconds. The streaming client knows, via a manifest file, which streams are available at different bitrates and, if necessary, segments of the stream. At the start, the client can subscribe or request a segment from a stream with a moderately low bitrate. If the client determines that the download speed allows for a higher bitrate, the client can request the next higher bitrate (segment).
[0010] On the other hand, if the client determines that the download speed is lower than the bitrate of the stream, the client can subscribe or request a lower bitrate segment.
[0011] Today, ultra-high-definition video is not uncommon at events with dozens or hundreds of users, many of whom have large-screen devices. These users' increasingly capable devices are being leveraged by multiparty applications (e.g., online education or entertainment) that most commonly approximate a shared-room experience by instantly showing a replica of the in-person experience to many or most of the remaining participants. A new set of services is radically redefining the user experience by leveraging pervasive cameras as alternative and simultaneous perspectives on the event. Imagine attending a concert and being able to select a different seat in the theater or hop on the stage with a few movements. Or consider watching your favorite sports team score the winning goal from multiple cameras in a stadium, all at the same time and in sync. Referring to Figure 2, this shows the interface of a multiview streaming application with several windows corresponding to streams of different sizes and different resolutions. It shows that different streams can be associated either to different one or more participants of a multiparty application, or to different views of a multiperspective application.
[0012] The number of participants in a multi-party application, or the number of viewpoints in a multi-view application, can be hundreds, thousands, or even more, but there are two fundamental constraints: technology resources and user attention. Application and user demands continue to grow with higher resolution, more participants, more viewpoints, and lower user tolerance for performance degradation. Meanwhile, in video streaming technology, near-optimal adaptive bit rate encoding algorithms and video codec improvements are diminishing. Thus, there is a limit to the number of viewpoints or participants that can be displayed simultaneously with sufficiently high quality. Beyond technical constraints, even if the size and resolution of high-end displays continue to improve and networks can support the growing demand for streaming high-resolution media, there is a limit to the number of simultaneous elements (e.g., views that a user can track) (Non-Patent Document 1). [Prior art documents] [Non-patent literature]
[0013] [Non-Patent Document 1] George A Alvarez, Steven L. Franconeri; "How many objects can you track?: Evidence for a resource-limited attentive tracking mechanism." Journal of Vision 2007;7(13): 14 [Non-Patent Document 2] Bolt, RA; "Gaze-orchestrated Dynamic windows." Computer Graphics 15, 3 (Aug. 1981), 109-119. https: / / dLacm.org / doi / pdf / 10.1145 / 965161.806796 [Non-Patent Document 3] Zhai, S., Morimoto, C., and Ihde, S. "Manual and Gaze Input Cascaded (MAGIC) Pointing." Proc. ACM CHl'99 (1999), pp. 246-253. https: / / dLacm.org / doi / pdf / 10.1145 / 302979.303053 [Non-Patent Document 4] Papoutsaki, A, Sangkloy, P., Laskey, J., Daskalova, N. Huang, J. and Hays, J.: "Scalable webcam eye tracking using user interactions." In Proc. IJCAI, pp. 3839-3845, 2016 Summary of the Invention [Problem to be solved by the invention]
[0014] A key challenge for multi-party and multi-view applications (collectively known as multi-view streaming applications) is to support a rich experience that meets user preferences within these constraints.
[0015] Current multi-view applications address this challenge by limiting the number of views in the display, either of participants or viewpoints, using a fixed number of applications (two or three are common), such as one main view and a limited number of secondary views (e.g., zoom "speaker" mode), and by relying on simple interfaces where the user explicitly enters the desired mode (e.g., clicking an icon for the desired modality). This approach does not capture the natural shifts in a user's focus during a face-to-face event, when a user's attention may shift seemingly randomly from a speaker to other participants or audience members, or from one side of a stadium to another, or to a bird's-eye view.
[0016] (summary) Various disclosed embodiments relate to streaming of live and real-time media over packet-switched networks such as the Internet. For purposes of explanation, live media streaming refers to streaming content transmitted over the Internet without first being recorded and stored, and real-time media streaming refers to online streaming of media that is recorded and broadcast simultaneously as it occurs to one or more users, where the delay is low enough to allow natural interaction between participants.
[0017] Motivated by the above-mentioned need for multi-view applications, embodiments of the present invention build on advances in eye-tracking hardware and software. Research into using gaze as a source of computer input, alone or in combination with other inputs, has been ongoing for over 40 years. Early seminal work in 1981 by R. Bolt in non-patent literature 2 suggests that computers need to capture and understand natural human modes of expression. His "Worm of Windows" shows a wall of windows selectable by gaze (non-patent literature 2). Later, the use of gaze in combination with other modes such as mouse clicks was also proposed. This helps to disambiguate user input and enrich the output (non-patent literature 3). It is only recently that advances in eye-tracking hardware and software have led to the proposal of the possibility of a general-purpose eye-tracking interface built on an inexpensive webcam. As an example, Papoutsaki et al. in Non-Patent Document 4 presented a real-time online eye tracking system that uses only a common webcam and shows predictions that are comparable to those made by a commercially available eye tracker, the Tobii EyeX, with a mean error of 128.9 pixels and a mean visual angle of 4.17 or 1.6 inches (Non-Patent Document 4).
[0018] In view of the above considerations, an embodiment of the present invention provides a method to leverage a user's gaze as a proxy for the user's attention to a video streaming service, thereby providing a more natural experience in a multi-view video streaming service. The method takes advantage of new powerful and inexpensive cameras (e.g., webcams) and new software that uses these cameras to perform eye tracking of products. An embodiment of the present invention also takes advantage of data collected about the user's interaction with machine learning techniques to customize responses for unique users. A system for implementing the method described above in a streaming architecture is also defined.
[0019] An embodiment of the present invention receives information about a window focus of a user's attention based on the user's gaze gaze during the last observation period; receives information about available bandwidth based on passive observation of network conditions; and selects the window focus value, screen percentage, and percentage of total screen size for each view as a function of the resolution associated with the main window and the maximum allocatable bandwidth available and the selected resolution.
[0020] In a further embodiment of the present invention, the determination of the user's attentional focus is based on a machine learning model of the user that is trained using past information regarding the user's changes in focus.
[0021] In yet another embodiment of the present invention, a more seamless focus change is achieved by creating backup streams of the required resolution for the views surrounding the current focus view.
[0022] In yet another embodiment of the present invention, a more seamless focus change is achieved by leveraging a user machine learning model to identify views that the user typically selects and creating backup streams of those views at the required resolution. [Brief description of the drawings]
[0023] [Figure 1] Shows keyframes and predicted frames for a chunk (top) and a stream (bottom). [Diagram 2] 1 illustrates an interface for a multi-view streaming application with different streams of different sizes and resolutions, and shows that different streams can be associated with different participants in a multi-party application or different views in a multi-perspective application. [Diagram 3] 1 illustrates a user and a device equipped with a camera supporting one embodiment of the present invention. [Figure 4]4 is a flowchart illustrating a process used to determine a particular window or icon's percentage of total screen size and selected resolution in accordance with one embodiment of the present invention. [Diagram 5] FIG. 1 is a flowchart illustrating a process for determining a particular window or icon percentage of total screen size and selected resolution using a model generated for a given user based on recorded data of the user's interactions, in accordance with one embodiment of the present invention. [Figure 6] FIG. 1 illustrates an approach for generating alternative quality versions of a stream that facilitates switching to higher resolution as a user's attention shifts focus, according to one embodiment of the present invention. [Figure 7] FIG. 1 illustrates a second approach to generating alternative quality versions of a stream based on a user's past interactions that facilitates switching to higher resolutions as the user's attention shifts focus, according to an embodiment of the present invention. [Figure 8] An exemplary form of a computer system illustrates a machine capable of executing a set of instructions for causing the machine to perform one or more of the methodologies discussed herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0024] The following detailed description describes embodiments of the present invention that comprise a method for capturing and using a user's focus of attention to provide a more natural experience in multi-view video streaming services.
[0025] Embodiments of the present invention utilize a user's gaze as a proxy for user attention, thereby providing product eye tracking using new powerful and inexpensive cameras, and new software that uses these cameras.
[0026] An embodiment of the invention comprises a user device, either in the form of a desktop or laptop computer, a pad, or a smartphone, connected wirelessly or by wire to at least one camera facing the user, and including a display, computational resources, and persistent and random access memory.
[0027] FIG. 3 shows a user and a device with a camera supporting one embodiment of the present invention in which a user 30 operating a personal device 32 gazes at a focus window 34 and the user's gaze is tracked by an eye gaze tracking camera 36.
[0028] A multi-view application displays multiple windows or icons on a screen to a user that represent multiple participants or viewpoints. In one embodiment, the multiple windows or icons represent all or some of the participants in a multi-party call or multiple angles of view of a sporting event. FIG. 2 illustrates an interface of a multi-view streaming application having different streams of different sizes and resolutions as shown in windows 1-13, where each window comprises one of a number of different streams of different sizes and resolutions. FIG. 2 illustrates that the different streams can be associated with either different participants in a multi-party application or different views in a multi-view application.
[0029] An embodiment of the present invention is based on gaze gaze, which identifies the unique window or windows on which the user's attention is focused. Every window or icon has a focus value ranging from 0 to 100.
number
[0030] The sum of the focus values of all windows is
number
number
number
[0031] At any given time, a particular window or icon's percentage of the total screen size and the resolution selected for that window or icon are a function of that window's focus value, the percentage of the screen resolution associated with the main window, and the maximum allocatable bandwidth available. There is a known bandwidth budget for every stream quality level; the available levels of stream quality form a distinct set (e.g., low resolution, standard resolution, high resolution).
[0032] The user device's camera and associated software track the user's gaze with sufficient accuracy to select an available view in a multi-view application. The focus value associated with a window is determined by this device's input. For example, if the main window occupies 40% of the screen and its maximum resolution requires 30% of the maximum allocable bandwidth, then the total screen allocation for other windows cannot exceed 60%. Also, the sum of the bandwidth requests cannot exceed 70% of the maximum allocated bandwidth.
[0033] (Changes in focus of attention) During a session, the user's attention shifts between windows, possibly focusing on different participants or the entire audience. An embodiment of the present invention dynamically adjusts the budget allocated to a particular window and its associated stream quality level as a function of user attention. This is done by dividing the session into observation periods (e.g., of a few seconds) and tracking the user's focus on different views of the multi-view application interface during each period. The window's focus value determined during observation period t is used to allocate its proportion of the screen size and thus the optimal resolution for the associated stream during observation period t+1.
[0034] FIG. 4 shows a process 42 in which at the start 40 of the process, the screen size percentage and resolution are determined.
[0035] The method disclosed herein of using gaze gaze to identify the focus of a user's attention and adjust views in a multi-view application accordingly complements other interaction modes available to the user, such as mouse clicks, and helps to disambiguate user input, which is labeled "user override" in Figures 4 and 5.
[0036] The method identifies available bandwidth and provides updated screen size and resolution information. If the display corresponds to the focus value, i.e., the current display is true 46, the process ends by the next observation period 48. During the next observation period, the method determines a focus value for this period that replaces the focus value of the previous observation period 44, and the process is repeated as described above.
[0037] (User-specific attention estimation) Different users may operate a multi-view interface in different ways, constantly shifting their gaze between multiple windows or focusing on a particular window. An embodiment of the present invention includes data collected during previous user interactions by gaze gaze and by alternative inputs such as mouse clicks ("user overrides"). These data are compiled and processed with machine learning techniques to customize the response provided to a unique user, taking the device's gaze information as input and generating a focus value.
[0038] Figure 5 shows a process 52 in which at the start of the process 50 a screen size percentage and resolution determination is made. The method identifies available bandwidth and provides updated screen size and resolution information. The method also uses the focus value determination for the period as input 55 and updates the user model of focus 57 based on data collected during previous user interactions with gaze gaze and alternative inputs stored in the user log 56. If the display corresponds to the focus value, i.e., the current display is true 54, the process ends 58 and the process is repeated as described above until the next observation period.
[0039] (Improved seamless focus transitions) As the user's focus shifts from one window to another, the new dominant window occupies a proportionally larger portion of the interface and is streamed at a higher quality level, while the previously dominant window occupies a proportionally smaller portion of the interface and is streamed at a lower quality level.
[0040] The different streams that make up a multi-view application are streamed using an adaptive bitrate scheme. This scheme allows seamless transitions between different levels of quality and resolution, to higher or lower quality for new or old focus windows, respectively. For real-time streaming, this seamless transition may require dynamically or proactively generating alternative versions of the same stream at different quality levels. Since doing this for potentially tens or hundreds of windows in a multi-view application may not be scalable, an embodiment of the invention distributes the bandwidth budget allocated to the backup stream to windows surrounding the dominant window (see FIG. 6). This approach assumes that the user's focus shifts when he or she notices other, possibly nearby, viewpoints or participants, and the bandwidth budget is allocated to the backup stream accordingly (see views 1-8 in FIG. 6). Additional backup streams are used to support seamless transitions of attention focus back to and from the dominant window.
[0041] (Note about dominant windows) Over time, a user's attention tends to prioritize a particular window (eg, a speaker or a friend) over other windows, so that the user keeps returning to that window during the session (see FIG. 7).
[0042] An embodiment of the present invention uses machine learning techniques to analyze data collected from previous user interactions to identify user-specific attention dominant windows and bandwidth budget allocations for backup streams associated with these windows (see views 13 and 17 in FIG. 7). As in the previous case, the additional backup streams support seamless transitions of attention focus back to and from the dominant windows.
[0043] The embodiments also detect the attention or inattention of the user during a video conference call or while viewing content such as advertisements, performances, etc. Such attention / inattention metrics can be obtained and used to generate various reports. Such information can also be used in real time to inform, for example, a broadcast service or a promoter of a sporting event that most of their viewers prefer one viewing over another. This information can be used to make global broadcast decisions in real time. Alternatively, viewer gaze information can be used to automatically modify the broadcast stream (e.g., select a dominant camera for a stream source).
[0044] (Computer implementation) 8 is a block diagram of a computer system that can be used to implement certain features of some embodiments, which can be a server computer, a client computer, a personal computer (PC), a user device, a tablet PC, a laptop computer, a personal digital assistant (PDA), a mobile phone, an iPhone, an iPad, a Blackberry, a processor, a telephone, a web appliance, a web application, a network router, a switch or bridge, a console, a handheld console, a handheld gaming device, a music player, a portable, a mobile, a handheld device, a wearable device, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be performed by the machine.
[0045] Computing system 80 may include one or more central processing units ("processors") 81, memory 82, input / output devices 85 (e.g., keyboard and pointing devices, touch devices, display devices), storage devices 84 (e.g., disk drives), and network adapters 86 (e.g., network interfaces connected to interconnect 83). Interconnect 83 is an abstract representation of any one or more separate physical buses, point-to-point connections, or both connected by appropriate bridges, adapters, or controllers. Thus, interconnect 83 may include, for example, a system bus, a Peripheral Component Interconnect (PCI) bus or PCI-Express bus, a HyperTransport or Industry Standard Architecture (ISA) bus, a Small Computer System Interface (SCSI) bus, a Universal Serial Bus (USB), an IIC (I2C) bus, or an Institute of Electrical and Electronics Engineers (IEEE) standard 1394 bus (also known as Firewire).
[0046] Memory 82 and storage device 84 are computer-readable storage media that can store instructions that implement at least a portion of the various embodiments. Additionally, data structures and message structures (e.g., signals on a communications link) can be stored or transmitted over a data transmission medium. Various communications links (e.g., the Internet, a local area network, a wide area network, or a point-to-point dial-up connection) may be used. Thus, computer-readable media can include computer-readable storage media (e.g., non-transitory media and computer-readable transmission media).
[0047] The instructions stored in memory 82 may be implemented as software and / or firmware that programs processor 81 to perform the operations described above. In some embodiments, such software or firmware may be provided to computing system 80 by initially downloading it from a remote system via computer system 80 (e.g., via network adapter 86).
[0048] Various embodiments described herein may be implemented, for example, using programmable circuitry (e.g., one or more microprocessors programmed by software and / or firmware, or completely dedicated hard-wired (non-programmable) circuitry, or a combination of such forms. The dedicated hard-wired circuitry may take the form of, for example, one or more ASICs, PLDs, FPGAs, and the like.
[0049] The language used herein has been selected primarily for ease of reading and explanation. It has not been selected to delineate the subject matter or to limit the scope. Accordingly, it is intended that the scope of the technology be limited not by this detailed description, but by the claims that will be issued on an application based on this specification. Accordingly, the disclosure of various embodiments is intended to illustrate, but not limit, the scope of the claimed technology.
Claims
1. 1. A method for streaming live and real-time media, comprising: A multi-view application presents a user with multiple windows or icons on a screen of a user device representing multiple participants or viewpoints; capturing a user's gaze with at least one camera facing the user when the user gazes at a focus window while operating the user device; using the user's gaze gaze as a focus of the user's attention to identify a unique window or windows on which the user's attention is focused; and determining, at any given time, the percentage of total screen size and selected resolution of a particular window or icon as a function of that window's focus value, the percentage of screen resolution associated with the main window and the maximum allocable bandwidth available; method.
2. The method of claim 1 , wherein the user device comprises a display, computational resources, and persistent and random access memory.
3. The method of claim 2 , wherein the user device is in communication with the at least one camera facing the user.
4. The method of claim 1 , wherein the user device comprises one of a desktop, a laptop computer, a pad, or a smartphone.
5. The method of claim 1 , wherein the multiple windows or icons represent all or a subset of participants at multiple view angles of a multiparty call and / or event.
6. 2. The method of claim 1, wherein the multi-view application comprises a multi-view streaming application interface having different streams of different sizes and resolutions represented by windows, each window comprising one of a plurality of different streams of different sizes and resolutions.
7. The method of claim 1 , wherein different streams are associated with different participants in a multi-party application and / or different views in a multi-perspective application.
8. Every window or icon has an associated focus value proportional to the percentage of time that the user's gaze is focused on that particular window or icon over the observation period. [0050] 2. The method of claim 1, comprising:
9. The sum of the focus values of all windows is [006] The method of claim 8, wherein the .times. ...
10. Additionally, it provides a known bandwidth budget for every stream quality level, The method of claim 1 , wherein the available levels of stream quality form a distinct set.
11. a main window comprises x% of the screen and the maximum resolution of the main window requires y% of the maximum allocable bandwidth; the sum of the screen allocations for all other windows on said screen does not exceed 100-x%, and the bandwidth requirements for all other windows on said screen exceed 100-y% of the maximum allocated bandwidth; The method of claim 1.
12. and dynamically adjusting the budget allocated to a particular window and its associated stream quality level as a function of the user's attention as the user's attention shifts between windows during the session by dividing the session into observation periods and tracking the user's focus on the different views of the multi-view application interface during each period. The method of claim 1.
13. The focus value of a particular window determined during observation period t is used to allocate a percentage of the screen size of the particular window and to allocate an optimal resolution to the associated stream of the particular window during the next observation period t+1. The method of claim 12.
14. and adjusting views of the multi-view application to complement other interaction modes available to the user to clarify the user's input. The method of claim 12.
15. Additionally, the present invention detects the user's attention or inattention during a video conference or while viewing content, obtaining said attention / inattention metrics; and generating a report using said metrics; The method of claim 1.
16. Additionally, real-time viewer information regarding the gaze gaze of multiple users is used to inform the broadcaster that most viewers prefer one viewing over another; and using said audience information to determine a worldwide broadcast in real time; Using that viewer information to automatically modify the broadcast stream; and Using the viewer information to select a dominant camera for a broadcast stream source Do one of the following: The method of claim 7.
17. 1. A method for streaming live and real-time media, comprising: A multi-view application presents a user with multiple windows or icons representing multiple participants or viewpoints on a screen of the user device; capturing a gaze gaze of the user with at least one camera facing the user while the user is gazing at a focus window while operating the user device; selecting from among available views of a multi-view application using the user's gaze gaze as a focus of the user's attention; identifying a unique window or windows on which the user's attention is focused; Determine the screen size percentage and resolution, providing updated screen size percentage and resolution information when the unique window or windows on which the user's attention is focused during an initial observation period correspond to an initial focus value; During a next observation period, determining a next focus value for the next observation period; replacing the initial focus value for the initial observation period with the next focus value for the next observation period; and Update the screen based on the next focus value for the next observation period. method.
18. Further, collecting data during user interaction through gaze gaze and alternative inputs when different users interact with the multi-view application in different ways by constantly shifting their gaze among multiple windows or by focusing on a particular window; and Using the gaze information of the device as input, compile and process the data using machine learning techniques to generate a focus value; and using the focus value to customize the response provided to the unique user; 20. The method of claim 17.
19. Identifying available bandwidth and providing updated screen size and resolution information, and updating a user model of focus using as input the focus value determination for the current time period and based on data collected during previous user interactions via gaze gaze and alternative inputs stored in a user log; 20. The method of claim 17.
20. 1. A method for streaming live and real-time media, comprising: A multi-view application presents a user with multiple windows or icons on a screen of a user device representing multiple participants or viewpoints; capturing a gaze gaze of the user with at least one camera facing the user when the user gazes at a focus window while operating the user device; selecting from among available views of a multi-view application using the user's gaze gaze as a focus of the user's attention; identifying a unique window or windows on which the user's attention is focused; providing a new dominant window in a proportionally larger portion of the screen as the user's focus shifts from one window to another; and The new dominant window is streamed at a higher quality level, while the previous dominant window occupies a smaller portion of the screen and is streamed at a lower quality level. method.
21. Further, streaming the different streams constituting said multi-view application using an adaptive bitrate scheme to enable seamless transitions between different levels of quality and resolution, to higher or lower quality for each of said new or old focus windows.
21. The method of claim 20.
22. Additionally, dynamic or proactive generation of alternative versions of the same stream at different quality levels for seamless transitions during real-time streaming; 22. The method of claim 21.
23. Further, as the user's focus shifts, supporting seamless transition of attentional focus back to or away from the dominant window by distributing the allocated bandwidth budget to additional backup streams around the dominant window.
23. The method of claim 22.
24. further allocating a bandwidth budget to the backup stream if the user's attention is prioritized to a particular window over other windows and the user continues to return to the prioritized window throughout the session; 24. The method of claim 23.
25. using machine learning techniques to analyze collected data of previous user interactions to identify one or more user-specific attention dominant windows and the allocation of bandwidth budgets to backup streams associated with the one or more user-specific attention dominant windows to support seamless transitions of attention focus back to and away from the one or more user-specific attention dominant windows; 24. The method of claim 23.
26. 1. An apparatus for streaming live and real-time media, comprising: a user device having a display, computational resources, and persistent random access memory; At least one camera facing the user when the user gazes at a focus window for capturing the user's gaze while operating the user device, the user device communicating with the at least one camera facing the user; a multi-view application configured to present a user with a plurality of windows or icons on a screen of said user device representing a plurality of participants or viewpoints; a processor configured to use the user's gaze gaze as a focus of the user's attention to identify a unique window or windows on which the user's attention is focused; and the processor is configured to determine, at any given time, a percentage of total screen size and a selected resolution for a particular window or icon as a function of that window's focus value, a percentage of the screen resolution associated with the main window, and a maximum allocable bandwidth available; Device.
Citation Information
Patent Citations
Information processing device and video conference system and program
JP2014050018A
Methods and apparatus for delivering content and / or playing back content
US20150346832A1
Data stream bandwidth assignments
WO2019152028A1
