System and method for predictive mixed-scale encoding of a media stream

Abstract

Predictive mixed-scale encoding of a media stream is described. A composite gaze path is determined based on multiple gaze paths. Each of the gaze paths includes gaze direction(s) obtained when a user views the media stream with a user device. A foveation weight map associated with a scene of the media stream is generated. The generation includes: determining, based on the plurality of gaze paths, a gaze heatmap that includes for each direction within the scene a score based on a number of users that had that direction as a gaze direction when viewing the scene; determining, based on the gaze heatmap, a region of interest that includes region(s) of high gaze concentration; and determining, based on the region of interest, the foveation weight map that includes foveation weights within a foveation area surrounding the region of interest and centered at a composite gaze direction from the composite gaze path.

Description

SYSTEM AND METHOD FOR PREDICTIVE MIXED-SCALE ENCODING OF A MEDIASTREAMTECHNICAL FIELD

[0001] Embodiments of the invention relate to the field of communication networks; and more specifically, to a system and method for predictive mixed scale encoding of a media stream.BACKGROUND ART

[0002] “Foveation” is a phenomenon of human vision, where the highest level of detail and acuity is concentrated in an area surrounding the center of the eye’ s pupil. Beyond this area, the perceived resolution of human vision declines precipitously, turning instead toward greater perception of motion or contrast rather than acuity. Foveated rendering techniques aim to take advantage of this drop in quality by concentrating rendering resources on the area of the display where the user looks. For example, foveation rendering techniques are used in extended reality environments. In these environments, some techniques rely on the orientation of headsets to estimate the direction of a user’ s gaze and concentrate the rendering resources on a foveation area centered at the estimated direction.

[0003] Video encoding is the backbone of ultra-fast, reliable video streaming technologies. Video encoding codecs allow the efficient, standardized transport of video data by encoding video frame feature information into tiles that are then iteratively compressed, transmitted, and decompressed for display on a user device. Mixed scale encoding refers to encoding techniques that allow for the encoding of tiles at differing pixel density and resolutions according to a set of one or more rules.

[0004] Mixed scale encoding reduces the bandwidth requirement of video transmission by taking advantage of foveation and reducing the resolution of tiles unlikely to be within the foveation area of a user’s gaze.SUMMARY

[0005] In some aspects, the techniques described herein relate to a method of encoding a media stream. The method includes: determining, based on a plurality of gaze paths, a composite gaze path, where each one of the plurality of gaze paths includes one or more gaze directions obtained when a user from one or more users views the media stream with a user device from one or more user devices; and generating a foveation weight map associated with a scene of the media stream. The generating includes: determining, based on the plurality of gaze paths, a gaze heatmap that includes for each direction within the scene a score based on a number of usersfrom the one or more users that had that direction as a gaze direction when viewing the scene, determining, based on the gaze heatmap, a region of interest that includes one or more regions of high gaze concentration, and determining, based on the region of interest, the foveation weight map that includes a set of foveation weights within a foveation area surrounding the region of interest and centered at a composite gaze direction from the composite gaze path, where a foveation weight from the set of foveation weights is indicative of a resolution at which to encode a tile from one or more tiles forming the scene.

[0006] In some aspects, the techniques described herein relate to a machine-readable medium including computer program code which when executed by a computer carries out operations including: determining, based on a plurality of gaze paths, a composite gaze path, where each one of the plurality of gaze paths includes one or more gaze directions obtained when a user from one or more users views the media stream with a user device from one or more user devices; and generating a foveation weight map associated with a scene of the media stream. The generating includes: determining, based on the plurality of gaze paths, a gaze heatmap that includes for each direction within the scene a score based on a number of users from the one or more users that had that direction as a gaze direction when viewing the scene, determining, based on the gaze heatmap, a region of interest that includes one or more regions of high gaze concentration, and determining, based on the region of interest, the foveation weight map that includes a set of foveation weights within a foveation area surrounding the region of interest and centered at a composite gaze direction from the composite gaze path, where a foveation weight from the set of foveation weights is indicative of a resolution at which to encode a tile from one or more tiles forming the scene.

[0007] In some aspects, the techniques described herein relate to a computing device for encoding a media stream, the computing device including: one or more processors; and a machine-readable storage medium that stores instructions, which when executed by the one or more processors, causes the computing device to perform operations. The operations includes determining, based on a plurality of gaze paths, a composite gaze path, where each one of the plurality of gaze paths includes one or more gaze directions obtained when a user from one or more users views the media stream with a user device from one or more user devices; and generating a foveation weight map associated with a scene of the media stream. The generating includes: determining, based on the plurality of gaze paths, a gaze heatmap that includes for each direction within the scene a score based on a number of users from the one or more users that had that direction as a gaze direction when viewing the scene, determining, based on the gaze heatmap, a region of interest that includes one or more regions of high gaze concentration, and determining, based on the region of interest, the foveation weight map that includes a set offoveation weights within a foveation area surrounding the region of interest and centered at a composite gaze direction from the composite gaze path, where a foveation weight from the set of foveation weights is indicative of a resolution at which to encode a tile from one or more tiles forming the scene.

[0008] In some aspects, the techniques described herein relate to a method in a user device including: determining a gaze direction for a user viewing a scene from a media stream with the user device; determining whether the gaze direction is within a foveation area that is centered at a composite gaze direction and determined based on a gaze heatmap for the scene that includes for each direction within the scene a score based on a number of users from one or more users that had that position as a gaze direction when viewing the scene; and responsive to determining that the gaze direction is not within the foveation area, transmitting to a remote computing device an indication that the gaze direction is not within the foveation area, and receiving from the remote computing device one or more updated encoded scenes that corresponds to the gaze direction, where the updated encoded scenes were encoded from scenes of the media stream based on an updated foveation area that is centered at the gaze direction.

[0009] In some aspects, the techniques described herein relate to a machine-readable medium including computer program code which when executed by a computer carries out operations including: determining a gaze direction for a user viewing a scene from a media stream with the user device; determining whether the gaze direction is within a foveation area that is centered at a composite gaze direction and determined based on a gaze heatmap for the scene that includes for each direction within the scene a score based on a number of users from one or more users that had that position as a gaze direction when viewing the scene; and responsive to determining that the gaze direction is not within the foveation area, transmitting to a remote computing device an indication that the gaze direction is not within the foveation area, and receiving from the remote computing device one or more updated encoded scenes that corresponds to the gaze direction, where the updated encoded scenes were encoded from scenes of the media stream based on an updated foveation area that is centered at the gaze direction.

[0010] In some aspects, the techniques described herein relate to a user device including: one or more processors; and a machine-readable storage medium that stores instructions, which when executed by the one or more processors, causes the user device to perform operations that includes determining a gaze direction for a user viewing a scene from a media stream with the user device; determining whether the gaze direction is within a foveation area that is centered at a composite gaze direction and determined based on a gaze heatmap for the scene that includes for each direction within the scene a score based on a number of users from one or more users that had that position as a gaze direction when viewing the scene; and responsive to determiningthat the gaze direction is not within the foveation area, transmitting to a remote computing device an indication that the gaze direction is not within the foveation area, and receiving from the remote computing device one or more updated encoded scenes that corresponds to the gaze direction, where the updated encoded scenes were encoded from scenes of the media stream based on an updated foveation area that is centered at the gaze direction.BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The invention may best be understood by referring to the following description and accompanying drawings that are used to illustrate embodiments of the invention. In the drawings:

[0012] Figure 1 illustrates an exemplary system for mixed scale encoding of a media stream, in accordance with some embodiments.

[0013] Figure 2 illustrates a flow diagram of exemplary operations for predictive mixed scale encoding of a media stream, in accordance with some embodiments.

[0014] Figure 3 illustrates a flow diagram of exemplary operations for generating a foveation weight map for a scene of the media stream, in accordance with some embodiments.

[0015] Figure 4 illustrates a flow diagram of exemplary operations for generating a foveation weight map based on a region of interest, in accordance with some embodiments.

[0016] Figure 5 illustrates a flow diagram of exemplary operations for displaying an encoded media stream in a user device, in accordance with some embodiments.

[0017] Figure 6 is a flow diagram of exemplary operations for predictive mixed scale encoding of a media stream, in accordance with some embodiments.

[0018] Figure 7 illustrates a flow diagram of exemplary operations for displaying an encoded media stream in a user device, in accordance with some embodiments.

[0019] Figure 8 illustrates a block diagram for an ND that can be used for implementing a computing device described herein, in accordance with some embodiments.

[0020] Figure 9 illustrates an exemplary embodiment in which a node is implemented on multiple network devices.DETAILED DESCRIPTION

[0021] The following description describes methods and apparatus for method for predictive mixed scale encoding of a media stream. In the following description, numerous specific details such as logic implementations, opcodes, means to specify operands, resource partitioning / sharing / duplication implementations, types and interrelationships of system components, and logic partitioning / integration choices are set forth in order to provide a morethorough understanding of the present invention. It will be appreciated, however, by one skilled in the art that the invention may be practiced without such specific details. In other instances, control structures, gate level circuits and full software instruction sequences have not been shown in detail in order not to obscure the invention. Those of ordinary skill in the art, with the included descriptions, will be able to implement appropriate functionality without undue experimentation.

[0022] References in the specification to “one embodiment,” “an embodiment,” “an example embodiment,” etc., indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases do not necessarily refer to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to affect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.

[0023] Bracketed text and blocks with dashed borders (e.g., large dashes, small dashes, dotdash, and dots) may be used herein to illustrate optional operations that add additional features to embodiments of the invention. However, such notation should not be taken to mean that these are the only options or optional operations, and / or that blocks with solid borders are not optional in certain embodiments of the invention.

[0024] In the following description and claims, the terms “coupled” and “connected,” along with their derivatives, may be used. It should be understood that these terms are not intended as synonyms for each other. “Coupled” is used to indicate that two or more elements, which may or may not be in direct physical or electrical contact with each other, co-operate or interact with each other. “Connected” is used to indicate the establishment of communication between two or more elements that are coupled with each other.Overview

[0025] Foveated encoding techniques take advantage of foveation to perform mixed-scale tilebased encoding. One such technique estimates the user’ s gaze direction when viewing a media stream and encodes the media stream based on a foveation area of constant radius at the center of the estimated direction. In addition to relying on an estimated user’s gaze direction, some prior art techniques are unable to adapt to dynamic changes in the user’s gaze direction during the viewing experience when the actual user’s gaze direction differs from the estimated gaze direction. Further, existing foveated encoding techniques implemented based on prediction model fail to provide a good viewer experience when the predictive model fails to produce anestimate of the user’s gaze direction or when a user’s current gaze is outside of the foveation area.Predictive mixed scale encoding of a media stream

[0026] The embodiments herein describe methods and systems for predictive mixed scale encoding of a media stream. A composite gaze path is determined based on multiple gaze paths. The gaze paths include one or more gaze directions obtained when a user from one or more users views the media stream with a user device from one or more user devices. A foveation weight map associated with a scene of the media stream is generated. The generation of the foveation weight map includes determining a gaze heatmap based on the gaze paths. The gaze heatmap includes, for a direction within the scene, a score that is based on a number of users from the users that had that direction as a gaze direction when viewing the scene. The generation of the foveation weight map further includes determining, based on the gaze heatmap, a region of interest that includes one or more regions of high gaze concentration, and determining, based on the region of interest, a foveation weight map. The foveation weight map includes a set of foveation weights within a foveation area surrounding the region of interest and centered at a composite gaze direction from the composite gaze path. A foveation weight is indicative of a resolution at which to encode a tile from a scene.

[0027] In some embodiments, an encoded scene is generated from the media stream and based on the foveation weight map. The encoded scene includes encoded tiles determined based on the set of foveation weights. The encoded scene is transmitted to be displayed on a user device. In some embodiments, the set of foveation weights is a first set of foveation weights and the foveation weight map includes a second set of foveation weights. In these embodiments, the encoded scene further includes second encoded tiles determined based on the second set of foveation weights. The second encoded tiles are of lower resolution than the first encoded tiles.

[0028] The embodiments herein present several advantages with respect to existing foveated encoding techniques. The generation of the foveation weight map enables an efficient use of compute resources when encoding the media stream. For example, the encoding of the media stream based on the foveation weight map reduces overhead on the computing device by avoiding the encoding of new tiles based on small or insignificant changes in the user’s gaze direction. Additionally, the embodiments described herein allow a flexible allocation of resources as the computing device can pre-render more of the media stream at off-peak use hours. In addition to providing advantages on the computing device’s side, the embodiments presented herein allow a user device to receive, decode, and cache more of the encoded media stream up-front while allowing for a fast correction of the encoding when a user’ s head or gaze movement leaves the predicted foveation area. In some embodiments the user device transmits,to the computing device, an indication that the user’s gaze is no longer within the predicted foveation area consequently allowing the remote computing device to transmit updated encoded media data to the user device to account, in real time, for difference between an estimated gaze direction and the actual gaze direction of the user when viewing the media stream.

[0029] Figure 1 illustrates an exemplary system 100 for mixed scale encoding of a media stream, in accordance with some embodiments. System 100 includes one or more user devices 104A-N, gaze aware media encoder 110, and network 105. System 100 is operative to record, generate, encode, decode, and / or display media streams for 2D and / or 3D (or 360°) videos viewing in a user device. By way of illustration, system 100 includes user devices 104A- N. A user device, e.g., user device 104A, can include a computer 103A such as a laptop or a desktop and / or a tablet or smartphone 103B associated with head-mounted displays (HMDs) or headsets 103C. In some embodiments, user device 104A can be a standalone HMD 103C that is operative to connect to a remote computing device implementing gaze aware media encoder 110 through network 105 without an intermediary electronic device. One or more of user devices 104A-N are operative to receive encoded media streams, decode and display the media streams, and forward gaze information to the remote computing device through network 105. User devices 104A-N are operative to decode and render several types of 360° video content that may be encoded and bandwidth-optimized according to the embodiments described in additional detail below.

[0030] System 100 further includes computing device 102. Computing device 102 is an electronic device that is remote from a user device from user devices 104A-N (e.g., connected to the user device through a wide area network (WAN)). Alternatively or additionally, computing device 102 is a local electronic device connected to the user device through a local area network. Computing device 102 includes gaze aware media encoder 110. Gaze aware media encoder 110 includes one or more media streams 112, codec 114, multi -resolution encoded media streams 116, composite gaze path determiner 117, foveation weight map determiner 118, tile selector 119, and transmitter 121.

[0031] In some embodiments media streams 112 can be obtained from one or more high- definition cameras (e.g., 4K, SK, etc.), including omnidirectional or panoramic cameras, etc. In some embodiments, the media stream is compatible with one or more interfaces, High- Definition Multimedia Interface (HDMI), Serial Digital Interface (SDI), High-Definition SDI (HD-SDI), or other formats, which may comprise unstitched or stitched streams, with or without projection-mapping, and with or without source video encoding. In some embodiments, the media stream is a projection-mapped stream generated from stitched streams using a suitablemap projection scheme, e.g., a spherical image projection including, without limitation, equirectangular projection, Cube Map projection, Equi-Angular Cube map (EAC) projection, Pyramid projection, Fish-Eye projection, etc.

[0032] Gaze aware media encoder 110 further includes codec 114. Codec 114 is operative to encode or compress the media stream according to one or more video encoding formats, e.g., H.264 or Advanced Video Coding (MPEG-4 AVC), High Efficiency Video Coding (HEVC) or H.265 (MPEG-H Part 2), H.262 (MPEG-2), H.264 (MPEG-4, Part 2), Alliance for Open Media (AOMedia) Video 1 (AVI), H.266, Versatile Video Coding (VVC), Future Video Coding (FVC), etc. In some embodiments, codec 114 is operative to perform tile encoding. In some embodiments, codec 114 is operative to generate encoded media streams 116 of multiple bitrate representations of an input video stream corresponding to a 360° immersive video asset or program. Each bitrate representation has a certain video quality level and may be encoded to contain frames with appropriately modified tile, frame and / or slice data that optimizes bandwidth, video quality, and / or latency of the media stream’s distribution.

[0033] Gaze aware media encoder 110 further includes composite gaze path determiner 117, tile selector 119, foveation weight map determiner 118, and transmitter 121. Composite gaze path determiner 117 is operative to receive gaze information from one or more user devices 104A-N and determine a composite gaze path from the gaze information. The gaze information includes gaze direction recorded by a user device when the user using the device was viewing a media stream. Foveation weight map determiner 118 is operative to determine, from gaze information and / or the composite gaze path, one or more foveation weight maps for one or more scenes of the media stream. Tile selector 119 is operative to select appropriate tiles responsive to control inputs, based on the foveation weight map(s), and generate a multiplexed encoded stream that may be delivered by transmitter 121 to network 105 to be decoded and displayed on user device 104A. In an example implementation, delivery of the multiplexed video streams to end users may be performed based on a number of protocols, e.g., HTTP / S, chunked HTTP / S, RTP / RTCP, etc., over a variety of network infrastructures.

[0034] Exemplary system 100 may be implemented in a hierarchical network architecture, with various aspects of media capture and preparation, including, e.g., source stream stitching, projection mapping, source media compression, tiled / ABR encoding / transcoding, packaging, etc., as well as distributing / uploading and edge node processes taking place in different network portions disposed at different hierarchical levels, involving one or more operators, content delivery networks (CDNs), edge networks, and the like. Further, in some implementations, at least some of the foregoing apparatuses and processes may be cloudbased. A non-limiting example of system 100 may be configured to accept media contentfrom live sources and / or static file sources, e.g., online content providers such as Hulu®, Netflix®, YouTube®, or Amazon® Prime, as well as VOD catalog or content providers or studios such as, e.g., Disney, Warner, Sony, etc. Media content from live sources may comprise live programming captured relative to any type of event, e.g., sporting / entertainment / gaming events, concerts, live TV shows, live news broadcasting sources, such as, for instance, national broadcasters (e.g., NBC, ABC, etc.) as well as cable broadcaster channels like Time Warner channels of CNN, ESPN, CNBC, etc., and local broadcasters, etc., including any secondary media insertions such as advertisement media channels. In another non- limiting example of system 100 is configured to enable cloud gaming, where graphics rendering for video games, is offloaded to Graphics Processing Unit (GPU)- powered cloud servers.

[0035] System 100 is operative to enable predictive mixed scale encoding of media streams. Composite gaze path determiner 117 is operative to determine a composite gaze path based on multiple gaze paths. The gaze paths include one or more gaze directions obtained when a user views the media stream with a user device from the user devices 104A-N. Foveation weight map determiner 118 is operative to generate a foveation weight map associated with a scene of the media stream. The generation of the foveation weight map includes determining a gaze heatmap based on the gaze paths; determining, based on the gaze heatmap, a region of interest that includes region(s) of high gaze concentration; and determining, based on the region of interest, a foveation weight map. The foveation weight map includes a set of foveation weights within a foveation area surrounding the region of interest and centered at a composite gaze direction from the composite gaze path. A foveation weight is indicative of a resolution at which to encode a tile from a scene.

[0036] In some embodiments, an encoded scene is generated from the media stream and based on the foveation weight map. The encoded scene includes encoded tiles determined based on the set of foveation weights. The encoded scene is transmitted to be displayed on a user device. In some embodiments, the set of foveation weights is a first set of foveation weights and the foveation weight map includes a second set of foveation weights. In these embodiments, the encoded scene further includes second encoded tiles determined based on the second set of foveation weights. The second encoded tiles are of lower resolution than the first encoded tiles.

[0037] The operations in the flow diagrams will be described with reference to the exemplary embodiments of the other figures. However, the operations of the flow diagrams can be performed by embodiments of the invention other than those discussed with reference to the other figures, and the embodiments of the invention discussed with reference to these otherfigures can perform operations different than those discussed with reference to the flow diagrams.

[0038] Figure 2 illustrates a flow diagram of exemplary operations 200 for predictive mixed scale encoding of a media stream, in accordance with some embodiments. At operation 210, gaze aware media encoder 110 encodes the media stream into multiple output media streams. Each of the output media streams has a distinct encoding resolution. Each output media stream has a separate video quality that is related to a corresponding Quantization Parameter (QP) value used by codec 114 for encoding the media stream. Each frame of the media stream comprises an array of tiles. In some embodiments, the array of tiles is organized into at least one slice per frame, with multiple frames forming a GOP structure of the media stream. The flow of operations moves to operation 220. Operation 210 is optional. While in some embodiments, operation 210 is performed prior to the selection of the tiles based on the foveation weight map, in other embodiments, gaze aware media encoder 110 is operative to encode on the fly the media stream based on the foveation weight map into a multi -resolution encoded media stream without requiring a pre-encoding of the stream into multiple encoded streams of varying resolutions.

[0039] At operation 220, gaze aware media encoder 110 obtains one or more gaze directions from the user devices 104A-N, when the users view the media stream. A gaze direction is a representation of where a user is looking when viewing the media stream. In one embodiment, the gaze direction is obtained by tracking an orientation of the user’s headset associated with the user device 104A-N when the user is viewing the media stream. In another embodiment, the gaze direction is obtained by tracking a movement of the user’ s eyeballs with respect to different portions of the display environment when the media is displayed on a display of the user device 104A-N. In a further embodiment, a combination of ocular and head movements associated with the user may be used for determining the gaze direction. In some embodiments, the gaze direction is represented by a unit vector that has two degrees of freedom (e.g., pitch and yaw). In other embodiments, the gaze direction can be represented by a unit vector that has three degrees of freedom (e.g., pitch, yaw, and roll). In some embodiments, a gaze direction obtained from a user device is associated with a time indication (e.g., timestamp) indicating the time at which the user viewed the media stream in that direction.

[0040] In some embodiments, the gaze directions can be obtained from a single user device from devices 104A-N that is used by one or more users for viewing the media stream multiple times. Alternatively or additionally, the gaze directions can be obtained from multiple user devices 104A-N used by one or more users for viewing the same media stream. While the embodiments herein are described with respect to gaze directions obtained from viewing a single media stream, the embodiments herein are not so limited, and the gaze directions can beobtained from one or more user devices when one or more users watch multiple media streams, where at least two of the media streams are different from one another. While in some embodiments, the gaze direction is obtained from the user device 104A-N, alternatively or additionally, the gaze direction can be determined at gaze aware media encoder 110 from data obtained from the user devices 104A-N.

[0041] In some embodiments, in addition to obtaining one or more gaze directions, gaze aware media encoder 110 obtains additional metadata related to the gaze direction. The metadata can include a user device identifier that identifies the type of user device from which the gaze direction is obtained, network measurements (e.g., latency, bandwidth, etc.) indicative of the performance of the network (e.g., network 105) in displaying the media stream on a display of the user device, the media stream encoding bitrate (e.g., when the encoder is a variable bitrate encoder). In some embodiments, when gaze aware media encoder 110 is operative to receive gaze directions for multiple media streams, gaze aware media encoder 110 can further obtain an identification of the media stream. In some embodiments, operation 210 is a data collection phase that is performed separately from the remaining operations of Figure 2. Alternatively or additionally, operation 210 can be performed continuously, where gaze information (including gaze direction and / or additional metadata) is obtained simultaneously to the other operations of Figure 2.

[0042] At operation 230, gaze aware media encoder 110 determines one or more gaze paths from the gaze directions. A gaze path includes one or more gaze directions obtained when a user views the media stream with a user device from the user devices 104A-N. In some embodiments, a gaze path is represented as a set of points (gaze directions), areas comprised of gaze directions across a number of frames or scenes, and / or vectors corresponding to the point of gaze and the direction of gaze flow from one frame or scene to the next frame or scene.

[0043] At operation 230, gaze aware media encoder 110 determines, based on the gaze paths, a composite gaze path. The composite gaze path includes one or more composite gaze directions associated with one or more confidence intervals. A confidence interval is a range of estimates for the gaze direction. The confidence interval includes a composite gaze direction and is computed at a designated confidence level. In some embodiments, the confidence level can be any of 90%, 95%, 99%, or any other confidence level.

[0044] In some embodiments, determining the composite gaze path includes generating a prediction model from the gaze directions that outputs a composite gaze path (e.g., a predicted gaze path) for the media stream and takes one or more of the following as inputs: movement measurements, streaming bitrate, and network condition measurements (e.g., latency and / or bandwidth) associated with the one or more user devices. In a non-limiting example,determining the composite gaze path is performed according to a clustering algorithm (e.g., K- means, a density-based clustering algorithm such as DBSCAN) or machine learning techniques (e.g., Gaussian processes, recurrent neural networks, etc.). Alternatively, the composite gaze path can be determined according to a classification technique based on features related to the client use. For example, features corresponding to the user device type, demographic of the user, client use case, etc., can be used. Based on these features, the classification technique outputs the composite gaze path with a confidence interval for each composite gaze direction in the composite gaze path. The flow of operations moves to operation 240.

[0045] At operation 250, gaze aware media encoder 110 generates a foveation weight map associated with a scene of the media stream. In some embodiments, the generation of the foveation weight map is performed as described with reference to Figure 3 in further details below. Upon determination of the foveation weight map, the flow of operations moves to operation 260.

[0046] Figure 3 illustrates a flow diagram of exemplary operations 250 for generating a foveation weight map for a scene of the media stream, in accordance with some embodiments. At operation 310, foveation weight map determiner 118 determines one or more scenes of the media stream. In some embodiments, determining the scene(s) includes dividing the media stream into multiple scenes, where a scene includes a portion of a frame, a frame, or one or more frames with similar visual content. In some embodiments, e.g., when the media stream is a 360 video, a scene is a portion of visual content that corresponds to a continuous view from a single camera. In some embodiments, determining the scenes can be performed manually by a user selecting and delimiting each scene of the media stream. Additionally or alternatively, determining the scene can be performed according to a scene detection algorithm that automatically identifies visual features, changes, and / or movements in the media stream. While embodiments of the invention have been described in relation to foveation weight map determiner 118 performing operation 310, alternative embodiments could be implemented such that a separate element performs operation 310. For example, such an embodiment could be implemented by having a scene detector that determines the scenes of a media stream prior to the scene being encoded into a plurality of mixed resolution encoded streams and separate from the operations of gaze weight map determination. The flow of operations moves to operation 320.

[0047] At operations 320, foveation weight map determiner 118 determines a gaze heatmap based on the gaze paths (e.g., gaze paths obtained at operation 230 of Figure 2). The gaze heatmap includes a score associated with each direction within the scene. The score is based on a number of users from the users of user devices 104A-N that had that direction as a gazedirection when viewing the scene. In some embodiments, foveation weight map determiner 118 applies a 2D Gaussian filter over the gaze paths obtained from the user devices 104A-N to obtain the gaze heatmap. Applying the filter enables gaze aware media encoder 110 to account for small variations in gaze paths across users. The flow of operations moves to operation 330.

[0048] At operation 330, foveation weight map determiner 118 determines a region of interest based on the gaze heatmap. The region of interest includes one or more regions of high gaze concentration. In some embodiments, foveation weight map determiner 118 identifies regions including directions in the gaze heatmap that have scores exceeding a predetermined threshold. In a non- limiting example, the threshold value is determined based on the confidence intervals of composite directions from the composite gaze path. Alternatively, the threshold value is determined by analyzing a distribution of scores within the gaze heatmap. Upon identification of the regions of high gaze concentration, foveation weight map determiner 118 creates the region of interest by merging continuous regions in the heatmap that exceed the threshold. In some embodiments, the creation of the region of interest can be performed according to image processing techniques such as edge or contour detection, connected component labeling, and / or blob detection. In some embodiments, the region of interest is refined by removing small or less significant regions based on a minimum size criterion for the regions and / or by merging nearby regions that do not connect if their distance is less than a second predefined threshold.

[0049] At operation 340, foveation weight map determiner 118 generates a foveation weight map based on the region of interest. The foveation weight map includes a set of foveation weights within a foveation area surrounding the region of interest and centered at a composite gaze direction from the composite gaze path. A foveation weight from the set of foveation weights is indicative of a resolution at which to encode a tile from one or more tiles forming the scene. In some embodiments, generating the foveation weight map is performed as described below with respect to Figure 4.

[0050] Figure 4 illustrates a flow diagram of exemplary operations 340 for generating a foveation weight map based on a region of interest, in accordance with some embodiments. At operation 410, foveation weight map determiner 118 initializes a foveation weight map for a scene. In some embodiments, the foveation weight map’s dimension is the dimension of a portion of a frame when the scene includes a portion of a frame. Alternatively, the foveation weight map’s dimension is the dimension of a frame when the scene includes one or more frames. In some embodiments, the foveation weight map is initialized with weight-map values that indicate that the corresponding tile is to be encoded at the lowest resolution. The flow of operations moves to operation 420.

[0051] At operation 420, foveation weight map determiner 118 creates a foveation area that is a circular weight map and surrounds the region of interest. The foveation area has the smallest radius that includes the region of interest. The foveation area provides a smooth transition from the center, which includes weight values associated with high resolution encoding, to the edges, which includes weight values associated with lower resolution encoding. In some embodiments, the circular weight map is created by applying a radial gradient that gradually decreases in intensity from the center of the foveation area to the edges. The flow of operations moves to operation 430.

[0052] At operation 430, foveation weight map determiner 118 blends the foveation area with the initialized foveation weight map according to a blending function to obtain a foveation weight map. In some embodiments, the blending function can include alpha blending or max blending. Alpha blending includes determining a smooth transition between the initialized weight map and the foveation area. Max blending retains the highest resolution level wherever there is an overlap between the initialized foveation weight map and foveation area. Other blending mechanisms can be used without departing from the scope of the present embodiments. The flow of operations moves to operation 440.

[0053] At operation 440, foveation weight map determiner 118 normalize the foveation weight map so that a weight associated to the highest resolution (i.e., greater than all other resolution levels) corresponds to the region at the center of the foveation area and the weight associated with the lowest resolution (i.e., smaller than all resolutions) corresponds to the background. The normalization operation enables an efficient use of the foveation weight map during the tile selection operation.

[0054] Returning to Figure 2, when the foveation weigh map is generated at operation 250, the flow of operations moves to operation 260. At operation 260, gaze aware media encoder 110 generates an encoded scene from the scene and based on the foveation weight map. The encoded scene includes encoded tiles determined based on the foveation weights of the foveation weight map. In some embodiments, generating the encoded scene includes selecting for each tile of the scene an encoded tile from the encoded media stream of corresponding resolution from the encoded media streams 116. In some embodiments, in addition to determining the tile of appropriate resolution based on the weight, tile selector 119 may consider additional parameters. For example, tile selector 119 may use network measurements, such as bandwidth and / or latency, to select the tile of appropriate resolution. In a non-limiting example, when bandwidth is scarce tile selector 119 selects tiles of medium resolution (i.e., of greater resolution that tiles outside the foveation area and smaller than tiles of the highest resolution) instead of selecting tiles of the highest resolution, even for the tiles located within the foveation area. Alternatively,when there is sufficient and / or generous bandwidth availability, tile selector 119 may select tiles of high resolution in an area that can be wider area than the foveation area.

[0055] In another non-limiting example, when the media stream encoding bit rate decreases below a threshold encoding bit rate that is required to effectively render the media stream on user device 104A, the radius of the foveation area is reduced to a minimum radius that concentrates the high-resolution tiles around the composite gaze direction for a scene.

[0056] In another non-limiting example, when a user device 104A is operative to cache multiple encoded scenes, gaze aware media encoder 110 can adjust the number of encoded scenes transmitted to user device 104A based on network conditions. For example, when network conditions are unreliable with respect to latency (e.g., high latency), gaze aware media encoder 110 may request a larger buffer to transmit a higher number of encoded scenes to the client to accommodate for higher latency. Alternatively, when the network conditions become more reliable, gaze aware media encoder 110 may transmit a smaller number of encoded scenes (i.e., reduce the buffer size) to minimize transport latency.

[0057] At operation 260, transmitter 121 transmits the encoded scene to be displayed on a user device. In some embodiments, transmitter 121 packages the encoded media scene(s) in a transmission container (e.g., Moving Picture Experts Group (MPEG) transport stream) and transmits the encoded scene to the user device 104A through network 105. The transmission container enables delivery of an encoded media stream with varying levels of quality that depend on the user’s network conditions, a composite gaze path determined from multiple gaze directions, and a foveation weight map. Transmitter 121 is operative to switch between the varying levels of quality seamlessly. Thus, in some embodiments, transmitter 121 transmits, for the same scene, first tiles that are of a first quality (e.g., associated with a high resolution when displayed) and second tiles that are of a second quality (e.g., lower resolution than the resolution of the first tiles). In some embodiments, in addition to transmitting the encoded scene, transmitter 121 transmits metadata associated with each scene. The metadata can include one or more of foveation weight maps, the composite gaze path, and foveation radii of foveation areas. In some embodiments, transmitter 121 transmits multiple encoded scenes generated based on the foveation weights. The metadata can be used by the user device to adaptively adjust the foveation during playback of the encoded scene(s) based on real-time user gaze information and network conditions. In some embodiments, each encoded scene is associated with a foveation area and transmitter 121 transmits at least two scenes associated with two foveation areas where a first of the two foveation areas is different from the second of the two foveation areas. In some embodiments, transmitter 121 transmits multiple scenes of a media streams to be cached and decoded according to the foveation weight map at the user device 104A.

[0058] In some embodiments, gaze aware media encoder 110 receives from user device 104 A, an indication that the gaze direction is not within the foveation area following the display of an encoded scene. Upon receipt of such indication, gaze aware media encoder 110 transmits one or more updated encoded scenes that corresponds to the gaze direction. For example, instead of using a composite gaze path and / or direction for determining a foveation weight map, gaze aware media encoder 110 creates an updated weight map based on the current gaze direction obtained from user device 104A to create an updated weight map centered at the current gaze direction. Gaze aware media encoder 110 generates an updated encoded scene based on the updated weight map. When user device 104A caches several encoded scenes, determined based on the composite gaze paths and gaze directions, the updated encoded (associated with the current gaze direction received from user device 104A) is used at the user device to bypass the cached scenes and is displayed to be viewed by the user of user device 104A consequently resulting in the display of a better quality media stream.

[0059] Figure 5 illustrates a flow diagram of exemplary operations 500 for displaying an encoded media stream in a user device, in accordance with some embodiments. While the embodiments below are described with respect to user device 104A, one of ordinary skill in the art would recognize that these embodiments are not so limited, and the operations below can be performed by one or more of user devices 104B-N. At operation 10, user device 104A receives one or more encoded scenes of a media stream. The encoded scene includes encoded tiles determined based on the foveation weights of the foveation weight map. In some embodiments, at least one of the encoded scenes is a multi-resolution scene. Tiles of a multi-resolution scene are encoded according to varying weights from the foveation weight map. Tiles associated with different weights result in portions of a scene of different resolutions, where the resolution of a portion of the scene located at the center of a foveation area for the scene is greater or equal to the resolution of a portion of the scene located outside of the foveation area.

[0060] hi some embodiments, in addition to receiving the encoded scenes, user device 104A receives metadata. The metadata can include one or more of foveation weight maps, the composite gaze path, and foveation radii of foveation areas. The metadata can be used by the user device to adaptively adjust the foveation during playback of the encoded scene(s) based on real-time user gaze information and network conditions as will be described in further details below. In some embodiments, each encoded scene is associated with a foveation area and user device 104A receives at least two scenes associated with two foveation areas where a first of the two foveation areas is different from the second of the two foveation areas. In some embodiments, user device 104A stores the multiple scenes of a media streams in a buffer fromwhich scenes are successively retrieved to be decoded according to their associated foveation weight map. The flow of operations moves to operation 520.

[0061] At operation 520, user device 104A decodes a scene from the encoded scenes. The decoding of the scene is performed according to the associated foveation weight map of a foveation area associated with the scene. The foveation area is centered at a composite gaze direction and determined based on a gaze heatmap for the scene that includes for each direction within the scene a score based on a number of users from one or more users that had that position as a gaze direction when viewing the scene. Tiles of the scene located within the foveation area are decoded to obtain high resolution tiles and tiles of the scene located outside the foveation area are decided to obtain low resolution tiles. The flow moves to operation 530.

[0062] At operation 530, user device 104A displays the decoded scene on a display. In some embodiments, the display is the display of an HMD. In other embodiments, the display is the display of a laptop, tablet, smartphone and / or a monitor connected with a computing device. The flow of operations moves to operation 540.

[0063] At operation 540, user device 104A determines a current gaze direction for a user viewing the decoded scene. In one embodiment, the gaze direction is determined by tracking an orientation of the user’s headset associated with the user device 104 A when the user is viewing the media stream. In another embodiment, the gaze direction is determined by tracking a movement of the user’ s eyeballs with respect to different portions of the display environment when the media is displayed on a display (e.g., HMD or other types of displays) of the user device 104A. In a further embodiment, a combination of ocular and head movements associated with the user may be used for determining the gaze direction. In some embodiments, the gaze direction is represented by a unit vector that has two degrees of freedom (e.g., pitch and yaw). In other embodiments, the gaze direction can be represented by a unit vector that has three degrees of freedom (e.g., pitch, yaw, and roll). In some embodiments, the gaze direction is associated with a time indication (e.g., timestamp) indicating the time at which the user viewed the media stream in that direction.

[0064] In some embodiments, user device 104A transmits the gaze direction to computing device 102. In some embodiments, in addition to transmitting gaze directions, user device 104A transmits additional metadata related to the gaze direction. The metadata can include a user device identifier that identifies the type of user device from which the gaze direction is obtained, network measurements (e.g., latency, bandwidth, etc.) indicative of the performance of the network (e.g., network 105) in displaying the media stream on a display of the user device, the media stream encoding bitrate (e.g., when the encoder is a variable bitrate encoder). In some embodiments, when gaze aware media encoder 110 is operative to receive gaze directions formultiple media streams, user device 104A can further transmit an identification of the media stream. In some embodiments, operation 540 is a data collection phase that is performed separately from the remaining operations of Figure 5. Alternatively or additionally, operation 510 can be performed continuously, where gaze information (including gaze direction and / or additional metadata) is transmitted simultaneously to the other operations of Figure 5 and Figure 2. The flow of operations moves to operation 550.

[0065] At operation 550, user device 104A determines whether the current gaze direction is within the foveation area. In other words, user device 104A determines whether the user’s gaze significantly deviated from a predicted area for the user’ s gaze direction. In response to determining that the gaze direction is not within the foveation area, the flow of operation moves to operation 560. Alternatively, in response to determining that the gaze direction is within the foveation area, the flow of operations moves to operation 580, at which user device 104A retrieves the next encoded scene from the cache and proceeds to decoding the scene (operation 550) to be displayed and viewed by the user.

[0066] At operation 560, user device 104A transmits to computing device 102 an indication that the gaze direction is not within the foveation area. Following the transmission of the indication, user device 104A receives from computing device 102 an updated encoded scene and / or an updated portion of the encoded scene that corresponds to the current gaze direction. The updated encoded scene or portion of a scene was encoded based on an updated foveation area that is centered at the current gaze direction instead of the composite gaze direction. In some embodiments, user device 104A receives from gaze aware media encoder 110 only tiles that were previously associated with a low resolution and are associated with high resolution tiles after the update (i.e., tiles that were outside of the foveation area and became included within the foveation area when the user moved its gaze). In some embodiments, updating the scene can include generating, at the computing device 102, a new buffer including the updated scene or portion of the scene and transmitting the new buffer to be rendered, at the user device 104A, around this new area. In some embodiments, updating the scene can further include upgrading the resolution of areas surrounding the current gaze direction in accordance with an updated region of interest. These operations can be repeated while the current direction is determined to be outside of a foveation area of the cached scenes.

[0067] In some embodiments, the transmission of the current gaze direction and the metadata associated with the gaze direction allow gaze aware media encoder 110 to adapt the encoding of the media stream based on this feedback. Gaze aware media encoder 110 can adapt the encoding by modifying a foveation area based on the current gaze direction and / or network conditions.

[0068] Figure 6 is a flow diagram of exemplary operations 600 for predictive mixed scale encoding of a media stream, in accordance with some embodiments. At operation 610, gaze aware media encoder 110 determines, based on multiple gaze paths, a composite gaze path. A gaze path from the multiple gaze paths includes one or more gaze directions obtained when a user from one or more users views the media stream with a user device from one or more user devices.

[0069] At operation 620, gaze aware media encoder 110 generates a foveation weight map associated with a scene of the media stream. The generation of the foveation weight map includes determining a gaze heatmap, determining a region of interest based on the gaze heatmap, and determining the foveation weight map based on the region of interest. The gaze heatmap is determined based on the multiple gaze paths. The gaze heatmap includes for each direction within the scene a score based on a number of users from the one or more users that had that direction as a gaze direction when viewing the scene. The region of interest is determined based on the gaze heatmap. The region of interest includes one or more regions of high gaze concentration. The foveation weight map is determined, based on the region of interest. The foveation weight map includes a set of foveation weights within a foveation area surrounding the region of interest and centered at a composite gaze direction from the composite gaze path. A foveation weight from the set of foveation weights is indicative of a resolution at which to encode a tile from one or more tiles forming the scene.

[0070] In some embodiments, the operations of Figure 6 further include optional operations 660 and 670. In these embodiments, the set of foveation weights is a first set of foveation weights and the foveation weight map includes a second set of foveation weights. At operation 660, gaze aware media encoder 110 generates an encoded scene from the scene and based on the foveation weight map. The encoded scene includes first encoded tiles determined based on the first set of foveation weights and second encoded tiles determined based on the second set of foveation weights, and the second encoded tiles are of lower resolution that the first encoded tiles. At operation 670, gaze aware media encoder 110 transmits the encoded scene, based the foveation weight map, to be displayed on a user device. Figure 7 illustrates a flow diagram of exemplary operations 700 for displaying an encoded media stream in a user device, in accordance with some embodiments. While the embodiments below are described with respect to user device 104A, the operations herein can be performed by any one of the user devices 104A-N without departing from the scope of the present invention.

[0071] At operation 710, user device 104A determines a gaze direction for a user viewing a scene from a media stream with the user device. The flow of operation moves to operation 720. At operation 720, user device 104A determines whether the gaze direction is within a foveationarea that is centered at a composite gaze direction and determined based on a gaze heatmap for the scene that includes for each direction within the scene a score based on a number of users from one or more users that had that position as a gaze direction when viewing the scene. In response to determining that the gaze direction is not within the foveation area, the flow of operation moves to operation 730. At operation 730, user device 104A transmits to a remote computing device, e.g., computing device 102, an indication that the gaze direction is not within the foveation area. In response to determining that the gaze direction is not within the foveation area, user device 104 A further receives from the remote computing device 102 one or more updated encoded scenes that corresponds to the gaze direction. The updated encoded scenes were encoded from scenes of the media stream based on an updated foveation area that is centered at the gaze direction.Architecture

[0072] An electronic device stores and transmits (internally and / or with other electronic devices over a network) code (which is composed of software instructions and which is sometimes referred to as computer program code or a computer program) and / or data using machine-readable media (also called computer-readable media), such as machine-readable storage media (e.g., magnetic disks, optical disks, solid state drives, read only memory (ROM), flash memory devices, phase change memory) and machine-readable transmission media (also called a carrier) (e.g., electrical, optical, radio, acoustical or other form of propagated signals - such as carrier waves, infrared signals). Thus, an electronic device (e.g., a computer) includes hardware and software, such as a set of one or more processors (e.g., wherein a processor is a microprocessor, controller, microcontroller, central processing unit, digital signal processor, application specific integrated circuit, field programmable gate array, other electronic circuitry, a combination of one or more of the preceding) coupled to one or more machine-readable storage media to store code for execution on the set of processors and / or to store data. For instance, an electronic device may include non-volatile memory containing the code since the non-volatile memory can persist code / data even when the electronic device is turned off (when power is removed), and while the electronic device is turned on that part of the code that is to be executed by the processor(s) of that electronic device is typically copied from the slower nonvolatile memory into volatile memory (e.g., dynamic random access memory (DRAM), static random access memory (SRAM)) of that electronic device. Typical electronic devices also include a set or one or more physical network interface(s) (NI(s)) to establish network connections (to transmit and / or receive code and / or data using propagating signals) with other electronic devices. For example, the set of physical NIs (or the set of physical NI(s) in combination with the set of processors executing code) may perform any formatting, coding, ortranslating to allow the electronic device to send and receive data whether over a wired and / or a wireless connection. In some embodiments, a physical NI may comprise radio circuitry capable of receiving data from other electronic devices over a wireless connection and / or sending data out to other devices via a wireless connection. This radio circuitry may include transmitter(s), receiver(s), and / or transceiver(s) suitable for radiofrequency communication. The radio circuitry may convert digital data into a radio signal having the appropriate parameters (e.g., frequency, timing, channel, bandwidth, etc.). The radio signal may then be transmitted via antennas to the appropriate recipient(s). In some embodiments, the set of physical NI(s) may comprise network interface controller(s) (NICs), also known as a network interface card, network adapter, or local area network (LAN) adapter. The NIC(s) may facilitate in connecting the electronic device to other electronic devices allowing them to communicate via wire through plugging in a cable to a physical port connected to a NIC. One or more parts of an embodiment of the disclosure may be implemented using different combinations of software, firmware, and / or hardware.

[0073] A network device (ND) is an electronic device that communicatively interconnects other electronic devices on the network (e.g., other network devices, end-user devices). Some network devices are “multiple services network devices” that provide support for multiple networking functions (e.g., routing, bridging, switching, Layer 2 aggregation, session border control, Quality of Service, and / or subscriber management), and / or provide support for multiple application services (e.g., data, voice, and video, etc.). In the embodiments described above the components of the system 100 can be implemented on one or more network devices coupled through a physical network. For example, user devices 104A-N and computing device 102 can be implemented on one ND or distributed over multiple NDs.

[0074] Figure 8 illustrates a block diagram for an ND that can be used for implementing a computing device described herein, in accordance with some embodiments. In some embodiments, the node can be a computing device or a client device. According to one embodiment, the ND is an electronic device which includes hardware 805. Hardware 805 includes one or more processors 814, network communication interfaces 860 coupled with a computer readable storage medium 812. The computer readable storage medium 812 may include a computer program 811.

[0075] While one embodiment does not implement virtualization, alternative embodiments may use different forms of virtualization - represented by a virtualization layer 820. In these embodiments, the instance 840 and the hardware that executes it form a virtual server which is a software instance of the modules stored on the computer readable storage medium 812.

[0076] The computer program 811 includes instructions which when executed by the hardware 805 causes the instance 840 to perform the operations described with reference toFigs. 1-7. In this embodiment, computing device 102 is implemented on a single network device.

[0077] Figure 9 illustrates an exemplary embodiment in which a node is implemented on multiple network devices. In some embodiments, the node can be a computing device. In the illustrated example, the node is distributed over multiple network devices 930A-930K, where each network device has a similar architecture as network device 930. The multiple network devices 930A-930K are coupled through one or more links and can be located in a same geographical location or remote from one another. The operations described with reference to the embodiments of Figs. 1-6, and Figs. 6 can be distributed over the multiple network devices, such as each network device is operative to perform a subset of the operations described herein.

[0078] While the flow diagrams in the figures show a particular order of operations performed by certain embodiments of the present disclosure, it should be understood that such order is exemplary (e.g., alternative embodiments may perform the operations in a different order, combine certain operations, overlap certain operations, etc.).

[0079] While the present disclosure has been described in terms of several embodiments, those skilled in the art will recognize that the present disclosure is not limited to the embodiments described, can be practiced with modification and alteration within the spirit and scope of the appended claims. The description is thus to be regarded as illustrative instead of limiting.

Claims

CLAIMSWhat is claimed is:

1. A method of encoding a media stream, the method comprising: determining (610), based on a plurality of gaze paths, a composite gaze path, wherein each one of the plurality of gaze paths includes one or more gaze directions obtained when a user from one or more users views the media stream with a user device from one or more user devices; and generating (620) a foveation weight map associated with a scene of the media stream, wherein the generating includes: determining (630), based on the plurality of gaze paths, a gaze heatmap that includes for each direction within the scene a score based on a number of users from the one or more users that had that direction as a gaze direction when viewing the scene, determining (640), based on the gaze heatmap, a region of interest that includes one or more regions of high gaze concentration, and determining (650), based on the region of interest, the foveation weight map that includes a set of foveation weights within a foveation area surrounding the region of interest and centered at a composite gaze direction from the composite gaze path, wherein a foveation weight from the set of foveation weights is indicative of a resolution at which to encode a tile from one or more tiles forming the scene.

2. The method of claim 1, wherein the set of foveation weights is a first set of foveation weights and the foveation weight map includes a second set of foveation weights, and wherein the method further comprises: generating (660) an encoded scene from the scene and based on the foveation weight map, wherein the encoded scene includes first encoded tiles determined based on the first set of foveation weights and second encoded tiles determined based on the second set of foveation weights, and the second encoded tiles are of lower resolution that the first encoded tiles; and transmitting (670) the encoded scene to be displayed on a user device.

3. The method of any of claims 1-2, wherein the determining (640), based on the gaze heatmap, the region of interest is further based on a confidence interval from one or more confidence intervals for the composite gaze direction.

4. The method of any of claims 1-3, wherein the determining (640), based on the gaze heatmap, the region of interest includes: analyzing a distribution of scores within the gaze heatmap.

5. The method of any of claims 1-4, wherein the determining (640), based on the gaze heatmap, a region of interest includes: determining the one or more regions of high gaze concentration based on a threshold.

6. The method of any one of claims 2-5 further comprising: encoding the media stream into a plurality of output media streams, wherein each one of the plurality of output media streams has a distinct encoding resolution; and wherein the generating (660) the encoded scene includes selecting the plurality of tiles from the plurality of output media streams based at least in part on the foveation weight map.

7. The method of any of claims 1-6 further comprising: receiving movement measurements, streaming bitrate, and network condition measurements associated with the one or more user devices; and wherein the determining the composite gaze path is further based on the movement measurements, the streaming bitrate, and the network condition measurements.

8. The method of any of claims 1-7, wherein the determining (610), based on a plurality of gaze paths, the composite gaze path includes: generating a prediction model based on the plurality of gaze paths; and using the prediction model to obtain the composite gaze path.

9. The method of any of claims 2-8, wherein the transmitting (670) the encoded scene includes transmitting metadata including one or more of foveation weight maps, the composite gaze path, and foveation radii of foveation areas.

10. A machine-readable medium (812) comprising computer program code which when executed by a computer carries out the method steps of any of claims 1-9.

11. A computing device (102) for encoding a media stream, the computing device comprising: one or more processors (814); anda machine-readable storage medium (812) that stores instructions, which when executed by the one or more processors, causes the computing device to perform the method steps of any one of claims 1-9.

12. A method in a user device (104 A) comprising: determining (710) a gaze direction for a user viewing a scene from a media stream with the user device; determining (720) whether the gaze direction is within a foveation area that is centered at a composite gaze direction and determined based on a gaze heatmap for the scene that includes for each direction within the scene a score based on a number of users from one or more users that had that position as a gaze direction when viewing the scene; and responsive (730) to determining that the gaze direction is not within the foveation area, transmitting to a remote computing device an indication that the gaze direction is not within the foveation area, and receiving from the remote computing device one or more updated encoded scenes that corresponds to the gaze direction, wherein the updated encoded scenes were encoded from scenes of the media stream based on an updated foveation area that is centered at the gaze direction.

13. The method of claim 12 further comprising: receiving (510) a plurality of encoded scenes of the media stream generated based on a composite gaze path and one or more gaze heatmaps, wherein the number of the encoded scenes depends on network measurements indicative of network conditions between the user device and the remote computing device for displaying the media stream.

14. The method of claim 13, wherein the network measurements are indicative of network latency and / or network bandwidth.

15. A machine-readable medium (812) comprising computer program code which when executed by a computer carries out the method steps of any of claims 12-14.16 A user device (104A) comprising: one or more processors (814); anda machine-readable storage medium (812) that stores instructions, which when executed by the one or more processors, causes the user device to perform the method steps of any one of claims 12-14.

Images