Methods, systems, and media for rendering immersive video content with grid of gaze points

Abstract

The invention relates to methods, systems, and media for rendering immersive video content with a grid of gaze points. In some embodiments, the method comprises: receiving a video content item; determining, using the hardware processor, whether the video content item satisfies at least one criterion; in response to determining that the video content item satisfies at least one criterion, a point-of-gaze grid on which frames of the video content item are projected is generated using the hardware processor according to a point-of-gaze ratio parameter, where the point-of-gaze grid has a non-uniform position mapping that is greater than a peripheral portion of each frame of the video content item than a peripheral portion of each frame of the video content item. The non-uniform position mapping increases the pixel density in the central portion of each frame of the video content item; and storing the video content item in a file format comprising the generated gaze point grid, wherein the immersive video content is rendered by applying the video content item as a texture to the generated gaze point grid.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 201980098968.3, filed on December 10, 2019.

[0003] CROSS-REFERENCE TO RELATED APPLICATIONS

[0004] This application claims priority to U.S. Provisional Patent Application No. 62 / 879,529, filed on July 28, 2019, the entirety of which is incorporated herein by reference. Technical Field

[0005] The disclosed subject matter relates to methods, systems, and media for rendering immersive video content using a foveated grid. More specifically, the disclosed subject matter can apply video content as a texture to a foveated grid that increases pixel density in a central portion of a video by sacrificing pixel density in peripheral portions of the video. Background Art

[0006] Many users like to watch video content in an immersive environment, such as virtual reality content, augmented reality content, three-dimensional content, 180-degree content, or 360-degree content, which can provide an immersive experience for viewers. For example, a virtual reality system can generate an immersive virtual reality environment for a user, in which the user can interact with one or more virtual objects. In a more specific example, a device such as a virtual reality head-mounted device or a head-mounted display device can be used to provide an immersive virtual reality environment. In another example, an augmented reality system can generate an immersive augmented reality environment for a user, in which computer-generated content (e.g., one or more images) can be superimposed on the user's current view (e.g., using a camera of a mobile device).

[0007] Immersive video content is often rendered by applying the video content to a mesh, such as a fisheye mesh or a three-dimensional equirectangular mesh, which causes the video content to be rendered as if it were part of a filled sphere rather than looking flat. However, rendering immersive video content while optimizing the video quality of such content is a difficult and resource-intensive task.

[0008] Therefore, it is desirable to provide new methods, systems, and media for rendering immersive video content utilizing a foveation grid. Summary of the invention

[0009] Methods, systems, and media for rendering immersive video content using a foveation grid are provided.

[0010] According to some embodiments of the disclosed subject matter, a method for generating immersive video content is provided, the method comprising: receiving a video content item; determining, using a hardware processor, whether the video content item satisfies at least one criterion; in response to determining that the video content item satisfies at least one criterion, generating, using the hardware processor, a foveation grid on which frames of the video content item are projected based on a foveation ratio parameter, wherein the foveation grid has a non-uniform position mapping that increases pixel density in a central portion of each frame of the video content item compared to a peripheral portion of each frame of the video content item; and storing the video content item in a file format that includes the generated foveation grid, wherein the immersive video content is rendered by applying the video content item as a texture to the generated foveation grid.

[0011] In some embodiments, the at least one criterion comprises determining whether the received video content item is associated with a particular file format. In some embodiments, the particular file format comprises a VR180 file format using directional stereoscopic projection with a 180 degree field of view.

[0012] In some embodiments, the at least one criterion comprises determining whether the received video content item comprises stereoscopic content or monoscopic content, and the method further comprises generating the foveation grid in response to determining that the received video content item comprises stereoscopic content.

[0013] In some embodiments, at least one criterion includes determining whether the received video content item is a static video or a dynamic video, static video including video content in which the camera does not substantially move when the video content is captured, and dynamic video including a camera motion metadata track, and the method further includes generating a gaze point grid in response to determining that the received video content item is a static video.

[0014] In some embodiments, the at least one criterion includes determining whether the received video content item is associated with a particular type of input grid, and the method further includes generating a foveation grid in response to determining that the received video content item is associated with a cropped equirectangular grid. In some embodiments, the foveation grid is generated by deforming a two-dimensional UV map of the cropped equirectangular grid using a polynomial fit function.

[0015] In some embodiments, the method further comprises generating a first fixation point grid based on the first fixation point ratio parameter, and generating a second fixation point grid based on the second fixation point ratio parameter.

[0016] In some embodiments, in response to determining that the received video content item is downsampled from a first pixel resolution to a second pixel resolution at a downsampling ratio, and an improvement ratio at a central portion of each frame of the video content item corresponds to the downsampling ratio, a gaze point grid is generated.

[0017] In some embodiments, the method further comprises, in response to receiving a request for the video content item from the user device, sending the video content and the generated gaze point grid to the user device.

[0018] According to some embodiments of the disclosed subject matter, a system for generating immersive video content is provided, the system comprising a memory and a hardware processor, which, when executing computer executable instructions stored in the memory, is configured to: receive a video content item; determine whether the video content item satisfies at least one criterion; generate a foveation grid onto which frames of the video content item are projected based on a foveation ratio parameter in response to determining that the video content item satisfies the at least one criterion, wherein the foveation grid has a non-uniform position mapping, which increases the pixel density in a central portion of each frame of the video content item compared to a peripheral portion of each frame of the video content item; and store the video content item in a file format including the generated foveation grid, wherein the immersive video content is rendered by applying the video content item as a texture to the generated foveation grid.

[0019] According to some embodiments of the disclosed subject matter, a non-transitory computer-readable medium is provided, the non-transitory computer-readable medium containing computer-executable instructions, which, when executed by a processor, cause the processor to implement a method for interacting with immersive video content, the method comprising: receiving a video content item; determining whether the video content item satisfies at least one criterion; in response to determining that the video content item satisfies the at least one criterion, generating a foveation grid onto which frames of the video content item are projected based on a foveation ratio parameter, wherein the foveation grid has a non-uniform position mapping, which increases the pixel density in a central portion of each frame of the video content item compared to a peripheral portion of each frame of the video content item; and storing the video content item in a file format including the generated foveation grid, wherein the immersive video content is rendered by applying the video content item as a texture to the generated foveation grid.

[0020] According to some embodiments of the disclosed subject matter, a system for generating immersive video content is provided, the system comprising: a device for receiving a video content item; a device for determining whether the video content item satisfies at least one criterion; a device for generating a gaze point grid on which frames of the video content item are projected based on a gaze point ratio parameter in response to determining that the video content item satisfies the at least one criterion, wherein the gaze point grid has a non-uniform position mapping that increases the pixel density in a central portion of each frame of the video content item compared to a peripheral portion of each frame of the video content item; and a device for storing the video content item in a file format including the generated gaze point grid, wherein the immersive video content is rendered by applying the video content item as a texture to the generated gaze point grid. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Various objects, features, and advantages of the disclosed subject matter may be more fully understood by reference to the following detailed description of the disclosed subject matter when considered in conjunction with the following drawings, in which like reference numerals identify like elements.

[0022] Figure 1 An illustrative example of a process for determining whether to generate a foveation grid and generating a foveation grid according to some embodiments of the disclosed subject matter is shown.

[0023] Figure 2 An illustrative example of a process for rendering immersive video content using a foveation grid in accordance with some embodiments of the disclosed subject matter is shown.

[0024] Figure 3 A schematic diagram of an illustrative system suitable for implementing the mechanisms described herein for determining whether to generate a foveation grid and generating a foveation grid in accordance with some embodiments of the disclosed subject matter is shown.

[0025] Figure 4 The invention shows that some embodiments of the disclosed subject matter can be Figure 3 Detailed examples of the hardware used in the server and / or user devices. DETAILED DESCRIPTION

[0026] According to various embodiments, mechanisms (which may include methods, systems, and media) are provided for rendering immersive video content using a foveation grid.

[0027] In some embodiments, the mechanisms described herein can determine whether video content satisfies one or more criteria, and in response to the video content satisfying the one or more criteria, a gaze point grid can be generated, which can be used to render the video content as immersive video content, wherein the gaze point grid can increase the pixel density or resolution in the central portion of the video content.

[0028] As used herein, a foveation grid is a grid structure with a non-uniform UV or three-dimensional position mapping that increases pixel density in the central portion of the video by sacrificing pixel density in the peripheral portions of the video.

[0029] In some embodiments, the foveation grid can be a modified version of a cropped equirectangular grid. For example, the UV vertices in the cropped equirectangular grid are evenly distributed, while the UV vertices in the foveation grid are not evenly distributed. In a more specific example, a 180 degree cropped equirectangular grid can have a substantially uniform UV mapping. Continuing with this example, the foveation grid can be generated by deforming the two-dimensional UV mapping of the cropped equirectangular grid using a polynomial fit function so that the pixel density in the central portion of the video content item is increased compared to the pixel density in the peripheral portion of the video content item.

[0030] It should be noted that in some embodiments, a similar deformation may be applied to the three-dimensional position grid such that the pixel density in the central portion of the video content item is increased compared to the pixel density in the peripheral portion of the video content item where the UV mapping remains unchanged.

[0031] It should also be noted that in some embodiments, multiple gaze grids can be generated. For example, the mechanisms can generate a first gaze grid according to a first gaze strategy (e.g., based on a function of a first gaze ratio parameter), and generate a second gaze grid according to a second gaze strategy (e.g., based on a function of a second gaze ratio parameter). In a more specific example, different gaze strategies can be implemented based on any suitable conditions (e.g., video type, resolution, computing resources, etc.). In another example, the mechanisms can select a gaze strategy from a plurality of gaze strategies based on one or more conditions, and can generate a gaze grid according to the selected gaze strategy.

[0032] When rendering video content on a suitable device (e.g., a virtual reality head-mounted device, a head-mounted display device, a game console, a mobile phone, a tablet computer, a television, and / or any other suitable type of user device), the device may receive the video content in a format that includes a foveation grid, wherein each frame of the video content is projected onto the foveation grid. For example, the mechanisms described herein may generate a foveation grid that may be used to render the video content as 180-degree video content, which may allow a viewer of the video content to feel immersed in the video content while viewing the video content. In another example, in some embodiments, the mechanisms described herein may generate a foveation grid that may be used to render the video content as three-dimensional video content having any suitable spatial extent.

[0033] In some embodiments, the mechanisms described herein may generate a gaze grid for a received video based on any suitable criteria.

[0034] For example, in some embodiments, the mechanisms may determine whether the received video is associated with a particular file format. Continuing with this example, in response to determining that the received video has a VR180 file format that uses directional stereo projection with a 180 degree field of view and / or satisfies any other suitable criteria, the mechanisms may generate a foveation grid for rendering immersive video content.

[0035] In another example, in some embodiments, the mechanisms may determine whether the received video includes stereoscopic content (e.g., left-eye / right-eye views or images, three-dimensional images, etc.) or includes monoscopic content (e.g., a set of two-dimensional images). Continuing with this example, the monoscopic content may be rendered as two-dimensional content in a web interface, mobile interface, or other non-head-mounted device interface, while the stereoscopic content may be rendered for presentation on a virtual reality head-mounted device, a head-mounted display device, etc. In response to determining that the received video includes stereoscopic content and / or satisfies any other suitable criteria, the mechanisms may generate a foveation grid for rendering the immersive video content.

[0036] In yet another example, in some embodiments, the mechanisms can determine whether the received video is a static video or a dynamic video, where a static video generally refers to a video in which a camera or camera assembly does not substantially move when the received video is captured. In response to determining that the received video is a static video and / or satisfies any other suitable criteria, the mechanisms can generate a foveation grid for rendering immersive video content.

[0037] In another example, in some embodiments, the mechanisms can determine a type of input grid associated with the received video. Examples of input grids associated with the received video can include a fisheye grid and a cropped equirectangular grid. In response to determining that the received video is associated with a cropped equirectangular grid and / or satisfies any other suitable criteria, the mechanisms can generate a foveation grid for rendering immersive video content.

[0038] It should be noted that in some embodiments, these mechanisms can generate a foveation grid for rendering the received video as immersive video content when downsampling the video content. It should also be noted that the improvement ratio at the central portion of the video can be limited by the downsampling ratio. For example, when downsampling video content from a 4K pixel resolution (e.g., 3849×2160 pixels or 4096×2160 pixels) to a 1080p resolution (e.g., 1920×1080 pixels), the highest improvement ratio at the central portion of the video is 4. Note that in this implementation, the improvement ratio can be measured as the number of pixels in the video content rendered using the foveation grid divided by the number of pixels in the video content rendered using the original grid.

[0039] In a more specific example, in response to determining that the received video is a still video including stereoscopic content in a VR180 file format and associated with a cropped equirectangular input grid, the mechanism can generate a foveation grid for rendering the immersive video content. It should be noted that any suitable criteria and any suitable combination of criteria can be used to determine whether to generate a foveation grid.

[0040] In some embodiments, the video content and the generated foveation grid may be sent to a user device for rendering by the user device (e.g., a virtual reality headset, a head mounted display device, etc.). For example, in some embodiments, the generated foveation grid may be inserted into a video file including the video content, and the video file may be sent to the user device. Additionally or alternatively, in some embodiments, the generated foveation grid may be used as an intermediate step before creating a projection of the video content and / or a three-dimensional equirectangular projection of the video content.

[0041] It should be noted that in some embodiments, video textures included in the video content may also be modified by applying similar foveation strategies. For example, the foveation strategy applied in generating the foveation grid may be used to modify the video textures included in the video content so that when the immersive video content is rendered at the client device, the immersive video content appears to be free of distortion while increasing the quality around the center of the video content at the expense of quality at the edges or peripheral portions of the video content (e.g., non-uniform pixel density or resolution).

[0042] It should also be noted that the foveation grid can be used to render immersive video content, thereby achieving an improvement in the overall perceived video resolution or quality of the immersive video content, where the pixel density or resolution in a central region of interest in each frame of the immersive video content for a given static fixed point is greater than the pixel density or resolution in a peripheral region in each frame of the immersive video content.

[0043] Combine the following Figure 1-4 These and other features of rendering immersive video content using a foveated grid are described.

[0044] Steering Figure 1 , shows an illustrative example 100 of a process for determining whether to generate a gaze point grid and generating a gaze point grid according to some embodiments of the disclosed subject matter. In some embodiments, the blocks of process 100 can be executed by any suitable device, such as a content server that stores video content and sends the stored video content to a user device (e.g., as described above in conjunction with Figure 3 The content server 302 shown and described executes.

[0045] The process 100 may begin by receiving a video content item at 110. For example, the process 100 may access an uploaded video content item stored on a content server. In another example, the process 100 may select a specific video content item that meets certain criteria (e.g., upload date, view count, etc.). In a more specific example, the process 100 may select a specific video content item stored on a content server that meets certain criteria as described below (e.g., a stereoscopic still video with a cropped equirectangular grid). In yet another example, the process 100 may receive a video content item that has been uploaded by a user with a user account.

[0046] It should be noted that the video content items may be received from any suitable source. For example, in some embodiments, stereoscopic video content may be captured using a physical camera device (e.g., a 360-degree stereo camera rig for capturing 360-degree stereo video, a VR180 camera device for capturing 180-degree stereo video, a set of wide-angle cameras, etc.). Continuing with this example, a computing device, such as a virtual reality head-mounted device or a head-mounted display device, may generate an immersive environment, such as an AR environment or a VR environment, based on the stereoscopic content.

[0047] The process 100 may continue at 120 by determining whether to generate a foveation grid for associating with the video content item based on any suitable criteria. For example, the process 100 may determine whether the received video content item satisfies one or more criteria, and in response to the video content item satisfying the one or more criteria, a foveation grid may be generated, the foveation grid being usable for rendering the video content as immersive video content, wherein the foveation grid may increase the pixel density or resolution in the central portion of the video content. In a more specific example, the process 100 may determine to generate a foveation grid to stereoscopically transcode a static video associated with a cropped equirectangular input grid.

[0048] In some embodiments, the criteria for generating the gaze point grid may include determining at 130 whether the received video is associated with a specific file format. Continuing with this example, the process 100 may determine that the received video was captured using a VR180 camera and has a VR180 file format using a directional stereo projection with a 180 degree field of view. In a more specific example, the specific file format may include metadata, for example, the metadata defines a projection from a video frame to a partial viewport within a spherical coordinate system (e.g., spherical video metadata according to the Spherical Video V2 metadata standard), and the process 100 may determine whether the received video is associated with a specific file format by determining the presence of such metadata. In another more specific example, when uploading a video content item to a content server, the uploading user may indicate that the video content item is associated with a specific file format (e.g., a VR180 file format), or may indicate that the video content item contains a specific type of content (e.g., 180 degree stereo video content, 360 degree stereo video content, etc.). Therefore, according to the Spherical Video V2 metadata standard, a VR180 video may be identified for processing or playback by the presence and content of the spherical video metadata.

[0049] It should be noted that in response to determining that the received video content item is associated with a specific file format (e.g., VR180 file format), process 100 may continue to determine whether the received video content item is a candidate for generating a gaze point grid. Alternatively, in response to determining that the received video content item is not associated with a specific file format, process 100 may determine not to generate a gaze point grid for the received video content item.

[0050] In some embodiments, the criteria for generating the gaze grid may include determining at 140 whether the received video content item includes stereoscopic content (e.g., left / right eye views or images, three-dimensional images, etc.) or monoscopic content (e.g., a set of two-dimensional images). Continuing with this example, the monoscopic content may be presented as two-dimensional content in a web interface, mobile interface, or other non-head-mounted device interface, while the stereoscopic content may be rendered for presentation on a virtual reality head-mounted device, a head-mounted display device, etc. In a more specific example, as described above, the received video content item may be associated with spherical video metadata that, for example, defines a projection from a video frame to a partial viewport within a spherical coordinate system, and the process 100 may determine whether the received video includes stereoscopic content or monoscopic content by determining the presence of a stereo-related field in such metadata. In another more specific example, when uploading a video content item to a content server, the uploading user may indicate that the video content item contains a specific type of content (e.g., 180-degree stereoscopic video content, 360-degree stereoscopic video content, etc.).

[0051] It should be noted that in response to determining that the received video content item includes stereoscopic content, such as 180-degree stereoscopic video content, process 100 may continue to determine whether the received video content item is a candidate for generating a foveation grid. Alternatively, in response to determining that the received video content item includes monoscopic content, process 100 may determine not to generate a foveation grid for the received video content item.

[0052] In some embodiments, the criteria for generating the gaze grid may include determining whether the received video is a static video or a dynamic video at 150. For example, a static video generally refers to a video in which a camera or camera assembly, such as a VR180 camera or a stereo camera rig, does not substantially move while the received video is being captured. In contrast, a dynamic video generally refers to a video in which a camera or camera assembly moves during video capture.

[0053] For example, as mentioned above, a video content item may be associated with metadata that defines a projection from a video frame to a partial viewport within a spherical coordinate system. For dynamic video, such metadata associated with the video content item may include a camera motion metadata track that stores different types of camera motion metadata, such as camera orientation information, gyroscope information, accelerometer information, and the like. For example, a camera motion metadata track may provide rotation-based motion stabilization to the video content based on the movement of the camera so that each video frame of the video content item is aligned with a fixed world orientation. Continuing with this example, determining whether a received video is a static video or a dynamic video may include detecting the presence of specific metadata, such as a camera motion metadata track, wherein a video content item containing a camera motion metadata track may be designated as a dynamic video.

[0054] It should be noted that in response to determining that the received video content item is a static video, the process 100 may continue to determine whether the received video content item is a candidate for generating a foveation grid. Alternatively, in response to determining that the received video content item is a dynamic video, the process 100 may determine not to generate a foveation grid for the received video content item. For example, in response to determining that the spherical video metadata of the received video content item includes a camera motion metadata track or other motion stabilization information, the process 100 may determine not to generate a foveation grid for the received video content item.

[0055] In some embodiments, the criteria for generating the foveation grid may include determining the type or grid format of an input grid associated with the received video content item at 160. Illustrative examples of input grids associated with the received video content item may include a fisheye grid and a cropped equirectangular grid. For example, process 100 may determine whether the received video content item is associated with a cropped equirectangular input grid by accessing content of spherical video metadata according to the Spherical Video V2 metadata standard and determining whether the value of the ProjectionType element in the spherical video metadata is set to a cropped equirectangular grid.

[0056] It should be noted that in response to determining that the received video content item is associated with a cropped equirectangular input grid, process 100 may continue to determine whether the received video content item is a candidate for generating a foveation grid. Alternatively, in response to determining that the received video content item is associated with another grid type (e.g., a fisheye grid), process 100 may determine not to generate a foveation grid for the received video content item.

[0057] In some embodiments, the criteria for generating the foveation grid may include determining whether the received video content item is being downsampled at 170. It should be noted that the foveation grid may provide a perception that the pixel density in the central portion of each frame of the video content item is improved by sacrificing or otherwise reducing the pixel density in the peripheral portion of each frame of the video content item, wherein such improvement is seen when the video content item is downsampled.

[0058] It should also be noted that the improvement ratio at the central portion of each frame of the video content item may be limited by the downsampling ratio. For example, when downsampling the video content item from a 4K pixel resolution (e.g., 3849×2160 pixels or 4096×2160 pixels) to a 1080P resolution (e.g., 1920×1080 pixels), the highest improvement ratio at the central portion of the video is approximately 4. It should be noted that in this implementation, the improvement ratio may be measured as the number of pixels in the video content rendered using the foveation grid divided by the number of pixels in the video content rendered using the original grid.

[0059] In response to determining that the received video content item is being downsampled, process 100 may continue to determine whether the received video content item is a candidate for generating a gaze point grid. Alternatively, in response to determining that the received video content item is not downsampled, or in response to determining that the improvement ratio is not greater than a certain threshold (e.g., an improvement ratio of 4), process 100 may determine not to generate a gaze point grid for the received video content item.

[0060] Thus, in response to determining whether the received video content item satisfies one or more criteria in 130 to 170, in 180, process 100 may generate a foveation grid that may be used to render the video content item as immersive video content. For example, in response to determining that the received video is a static video that includes stereoscopic content in a VR180 file format and is associated with a cropped equirectangular input grid, a foveation grid may be generated to render the immersive video content. It should be noted that although the embodiments described herein generally relate to generating a foveation grid for a static video that includes stereoscopic content associated with a cropped equirectangular input grid, any suitable criteria and any suitable combination of criteria may be used to determine whether to generate a foveation grid.

[0061] As described herein, a foveation grid is a grid structure with a non-uniform UV or three-dimensional position mapping that increases pixel density in the central portion of a video by sacrificing pixel density in peripheral portions of the video.

[0062] In some embodiments, a transcoding system or transcoding pipeline for stereoscopic transcoding of a video may be modified to generate a foveation grid or otherwise implement a foveation strategy. The transcoding system may include a transcoder control component that sets video parameters and transcoder metadata, which may include creating a target grid required to reproject a video from one grid format to another grid format or otherwise convert a video from one grid format to another grid format.

[0063] It should be noted that in some embodiments, the transcoder control component can be modified to create a parameterized gaze point grid by passing a gaze point ratio parameter. In some embodiments, the gaze point grid can be a modified version of a cropped equirectangular grid. For example, the UV vertices in the cropped equirectangular grid are evenly distributed, while the UV vertices in the gaze point grid are not evenly distributed. In a more specific example, a 180-degree cropped equirectangular grid can have a substantially uniform UV mapping. Continuing with this example, a gaze point grid can be generated by deforming the two-dimensional UV mapping of the cropped equirectangular grid using a polynomial fit function so that the pixel density in the central portion of the video content item is increased compared to the pixel density in the peripheral portion of the video content item.

[0064] It should be noted that in some embodiments, a similar deformation based on the gaze point ratio parameter may be applied to the three-dimensional position grid such that the pixel density in the central portion of the video content item is increased compared to the pixel density in the peripheral portion of the video content item where the UV mapping remains unchanged.

[0065] In a more specific example, the process 100 can focus the normalized coordinates normalized_coord based on the focus ratio parameter foveation_ratio. It should be noted that in some embodiments, this can include clamping the input coordinates or otherwise limiting the position to the range [0.0, 1.0]. This grid focus point can be represented as follows:

[0066] clamped_coord = clamp(0.0, 1.0, normalized_coord)

[0067] foveated_coord = 0.5 + atan(sqrt(foveation_ratio) * tan((clamped_coord – 0.5) * π)) / π

[0068] Continuing with this example, process 100 may de-normalize the normalized coordinate fixation by performing the inverse process, which may be expressed as follows:

[0069] clamped_coord = clamp(0.0, 1.0, normalized_coord)

[0070] unfoveated_coord = atan(tan((clamped_coord – 0.5) * π)) / sqrt(foveation_ratio)) / π + 0.5

[0071] It should be noted that in some embodiments, multiple gaze point grids can be generated. For example, a first gaze point grid can be generated according to a first gaze point strategy (e.g., based on a polynomial fit function of a first gaze point ratio parameter), and a second gaze point grid can be generated according to a second gaze point strategy (e.g., based on a polynomial fit function of a second gaze point ratio parameter). In a more specific example, different gaze point strategies can be implemented based on any suitable conditions (e.g., video type, resolution, computing resources, etc.). In another example, a gaze point strategy can be selected from a plurality of gaze point strategies based on one or more conditions, and a gaze point grid can be generated according to the selected gaze point strategy.

[0072] Back to Figure 1 In some embodiments, the process 100 may store the video content item and the foveation grid at 190. For example, the process 100 may insert the video content item in a specific file format (eg, VR180 file format) into the foveation grid.

[0073] When rendering video content on a suitable device (e.g., a virtual reality head-mounted device, a head-mounted display device, a game console, a mobile phone, a tablet computer, a television, and / or any other suitable type of user device), the device may receive the video content in a format that includes a foveation grid, wherein each frame of the video content is projected onto the foveation grid. For example, the mechanisms described herein may generate a foveation grid that may be used to render the video content as 180-degree video content, which may allow a viewer of the video content to feel immersed in the video content while viewing the video content. In another example, in some embodiments, the mechanisms described herein may generate a foveation grid that may be used to render the video content as three-dimensional video content having any suitable spatial extent.

[0074] It should be noted that the foveation grid can be used to render immersive video content, thereby achieving an improvement in the overall perceived video resolution or quality of the immersive video content, wherein the pixel density or resolution in a central region of interest in each frame of the immersive video content for a given static fixed point is greater than the pixel density or resolution in a peripheral region in each frame of the immersive video content.

[0075] In some embodiments, the video content and the generated foveation grid may be sent to a user device for rendering by the user device (e.g., a virtual reality headset, a head-mounted display device, etc.).

[0076] Steering Figure 2 , shows an illustrative example 200 of a process for rendering immersive video content according to some embodiments of the disclosed subject matter. In some embodiments, the blocks of process 200 can be performed by any suitable device, such as a virtual reality head-mounted device, a game console, a mobile phone, a tablet computer, a television, and / or any other suitable type of user device.

[0077] Process 200 may begin at 210 by requesting a video content item for presentation on a user device. In some embodiments, the video content item may be any suitable type of video content item, such as a video to be streamed from a content server to a user device, and / or any other suitable type of video content item. In some embodiments, the video content item may be requested on the user device in any suitable manner. For example, in some embodiments, the video content item may be requested in response to a user of the user device selecting a video content item from a page containing available video content, and / or in any other suitable manner. The request for the video content item may be sent from the user device to a content server (e.g., as described above in conjunction with Figure 3 content server 302 as shown and described herein).

[0078] In response to the request sent at 210, process 200 may receive a video content item and a corresponding foveation grid to be used to render the video content item at 220. For example, in some embodiments, the foveation grid may be retrieved from a storage device such as that described above in conjunction with Figure 1 The content server of the video content and / or the generated gaze grid described above (e.g., as described above in conjunction with Figure 3 In some embodiments, the video content item and the corresponding fixation grid may be received by the content server 302 as shown and described above. In some embodiments, the video content item and the corresponding fixation grid may be received in any suitable manner. For example, in some embodiments, the fixation grid may be inserted into a video file including the video content item, and the process 200 may extract the fixation grid from the video file using any suitable technique or combination of techniques. It should be noted that any suitable scheme may be used to inject the corresponding fixation grid into the video content item.

[0079] At 230, process 200 may render the video content item on the user device as immersive content using any suitable technique or combination of techniques. For example, in some embodiments, process 200 may apply or project the video content item onto a foveation grid to render the video content item as immersive content. As a more specific example, in some embodiments, process 200 may use any suitable technique (e.g., UV mapping, and / or any other suitable technique) to map content from the video content item as a texture to the foveation grid. As another example, in some embodiments, process 200 may render the video content item as stereoscopic content by presenting a first portion of the video content item to the left eye of a viewer of the user device, and presenting a second portion of the video content item to the right eye of a viewer of the user device. In some such embodiments, the first portion of the video content item and the second portion of the video content item may be applied or projected onto a foveation grid, respectively, before rendering on the user device.

[0080] In some embodiments, process 200 may render the video content item as immersive content that can be manipulated or interacted with by a viewer of the video content item. For example, in some embodiments, process 200 may render the video content item in a manner that allows the viewer to change the viewpoint of the video content item. As a more specific example, in some embodiments, where the viewer is using a virtual reality head-mounted device or other wearable computing device, process 200 may change the presentation viewpoint of the video content item in response to determining that the viewer has changed the orientation of the viewer's head. As another more specific example, in some embodiments, process 200 may change the presentation viewpoint of the video content item in response to determining that the viewer has input a gesture on the user device to manipulate the viewpoint, such as by selecting and dragging a user interface on which the video content item is presented to change the presentation viewpoint of the video content item.

[0081] Steering Figure 3 , an illustrative example 300 of hardware for determining whether to generate a foveation grid, generating a foveation grid, and / or rendering immersive video content using a foveation grid that can be used according to some embodiments of the disclosed subject matter is shown. As shown, the hardware 300 can include a content server 302, a communication network 304, and / or one or more user devices 306, such as user devices 308 and 310.

[0082] The content server 302 may be any suitable server for storing media content and / or providing media content to the user device 306. For example, in some embodiments, the content server 302 may store media content, such as videos, television programs, movies, live broadcast media content, audio content, animations, video game content, graphics, and / or any other suitable media content. In some embodiments, the content server 302 may transmit the media content to the user device 306, for example, via the communication network 304. In some embodiments, the content server 302 may store video content (e.g., live video content, computer-generated video content, and / or any other suitable type of video content) in association with any suitable information used by a client device (e.g., the user device 306) to render the video content as immersive content, as described above in conjunction with Figure 2 As shown and described.

[0083] In some embodiments, the communication network 304 can be any suitable combination of one or more wired and / or wireless networks. For example, the communication network 304 can include any one or more of the Internet, an intranet, a wide area network (WAN), a local area network (LAN), a wireless network, a digital subscriber line (DSL) network, a frame relay network, an asynchronous transfer mode (ATM) network, a virtual private network (VPN), and / or any other suitable communication network. The user device 306 can be connected to the communication network 304 via one or more communication links (e.g., communication link 312), and the communication network 304 can be linked to the content server 302 via one or more communication links (e.g., communication link 314). The communication link can be any communication link suitable for transmitting data between the user device 306 and the content server 302, such as a network link, a dial-up link, a wireless link, a hard-wired link, any other suitable communication link, or any suitable combination of these links.

[0084] User device 306 may include any one or more user devices suitable for requesting video content, rendering the requested video content as immersive video content (e.g., as virtual reality content, as three-dimensional content, as 360-degree video content, as 180-degree video content, and / or in any other suitable manner), and / or for performing any other suitable function. For example, in some embodiments, user device 306 may include a mobile device, such as a mobile phone, a tablet computer, a wearable computer, a laptop computer, a virtual reality head-mounted device, a vehicle (e.g., a car, a boat, an airplane, or any other suitable vehicle) information or entertainment system, and / or any other suitable mobile device and / or any other suitable non-mobile device (e.g., a desktop computer, a game console, and / or any other suitable non-mobile device). As another example, in some embodiments, user device 306 may include a media playback device, such as a television, a projector device, a game console, a desktop computer, and / or any other suitable non-mobile device.

[0085] In a more specific example where user device 306 is a head mounted display device worn by a user, user device 306 may include a head mounted display device connected to a portable handheld electronic device. The portable handheld electronic device may be, for example, a controller, a smart phone, a joystick, or another portable handheld electronic device that may be paired with and communicate with the head mounted display device to interact in an immersive environment generated by the head mounted display device and displayed to the user, for example, on a display of the head mounted display device.

[0086] It should be noted that the portable handheld electronic device can be operably coupled or paired with the head-mounted display device via, for example, a wired connection or a wireless connection such as a WiFi or Bluetooth connection. Such pairing or operably coupling of the portable handheld electronic device and the head-mounted display device can provide communication between the portable handheld electronic device and the head-mounted display device and data exchange between the portable handheld electronic device and the head-mounted display device. This can allow, for example, the portable handheld electronic device to act as a controller in communication with the head-mounted display device for interacting in an immersive virtual environment generated by the head-mounted display device. For example, manipulation of the portable handheld electronic device, and / or input received on the touch surface of the portable handheld electronic device, and / or movement of the portable handheld electronic device can be converted into corresponding selections, or movements or other types of interactions in the virtual environment generated and displayed by the head-mounted display device.

[0087] It should also be noted that in some embodiments, the portable handheld electronic device may include a housing in which the internal components of the device are housed. A user interface accessible to a user may be provided on the housing. The user interface may include, for example, a touch-sensitive surface configured to receive user touch input, touch and drag input, etc. The user interface may also include a user-operated device, such as an actuation trigger, a button, a knob, a toggle switch, a joystick, etc.

[0088] It should also be noted that in some embodiments, the head-mounted display device may include a housing coupled to a frame, wherein the audio output device includes, for example, a speaker mounted in a headphone and also coupled to the frame. For example, the front of the housing may be rotated away from the base of the housing so that some of the components housed in the housing are visible. The display may be mounted on an inwardly facing side of the front of the housing. In some embodiments, when the front is in a closed position against the base of the housing, the lens may be mounted in the housing at a position between the user's eyes and the display. The head-mounted display device may include a sensing system (including various sensors) and a control system (including a processor and various control system devices) to facilitate operation of the head-mounted display device.

[0089] For example, in some embodiments, the sensing system may include an inertial measurement unit that includes a variety of different types of sensors, such as accelerometers, gyroscopes, magnetometers, and other such sensors. The position and orientation of the head-mounted display device can be detected and tracked based on data provided by the sensors included in the inertial measurement unit. The detected position and orientation of the head-mounted display device can allow the system to detect and track the user's head gaze direction, head gaze movement, and other information related to the position and orientation of the head-mounted display device in turn.

[0090] In some implementations, the head-mounted display device may include a gaze tracking device that includes, for example, one or more sensors to detect and track eye gaze direction and movement. Images captured by the (multiple) sensors may be processed to detect and track the direction and movement of the user's eye gaze. The detected and tracked eye gaze may be processed as user input to be transformed into corresponding interactions in an immersive virtual experience. The camera may capture still and / or moving images that may be used to help track the physical location of the user and / or other external devices that are in communication / operably coupled to the head-mounted display device. The captured images may also be displayed to the user on a display in a pass-through mode.

[0091] Although content server 302 is illustrated as one device, in some embodiments, the functions performed by content server 302 may be performed using any suitable number of devices. For example, in some embodiments, multiple devices may be used to implement the functions performed by content server 302. In a more specific example, in some embodiments, a first content server may store media content items and respond to requests for media content, and a second content server may generate thumbnail representations of virtual objects corresponding to the requested media content items.

[0092] Despite Figure 3 Two user devices 308 and 310 are shown in FIG. 3 to avoid overcomplicating the diagram, but any suitable number of user devices and / or any suitable type of user devices may be used in some embodiments.

[0093] In some embodiments, content server 302 and user device 306 may be implemented using any suitable hardware. For example, in some embodiments, devices 302 and 306 may be implemented using any suitable general purpose computer or special purpose computer. For example, a mobile phone may be implemented using a special purpose computer. Any such general purpose computer or special purpose computer may include any suitable hardware. For example, Figure 4 As shown in example hardware 400 , such hardware may include a hardware processor 402 , a memory and / or storage device 404 , an input device controller 406 , an input device 408 , a display / audio driver 410 , a display and audio output circuitry 412 , communication interface(s) 414 , an antenna 416 , and a bus 418 .

[0094] In some embodiments, the hardware processor 402 may include any suitable hardware processor, such as a microprocessor, a microcontroller, a (multiple) digital signal processor, dedicated logic, and / or any other suitable circuit system for controlling the functions of a general purpose computer or a special purpose computer. In some embodiments, the hardware processor 402 may be controlled by a server program stored in a memory and / or storage device 404 of a server (e.g., such as the content server 402). For example, in some embodiments, the server program may cause the hardware processor 402 to send a media content item to the user device 206, send instructions for rendering a video content item as immersive video content, and / or perform any other suitable action. In some embodiments, the hardware processor 402 may be controlled by a computer program stored in a memory and / or storage device 404 of the user device 406. For example, the computer program may cause the hardware processor 402 to render a video content item as immersive video content, and / or perform any other suitable action.

[0095] In some embodiments, the memory and / or storage device 404 may be any suitable memory and / or storage device for storing programs, data, media content, and / or any other suitable information. For example, the memory and / or storage device 404 may include random access memory, read-only memory, flash memory, hard disk storage, optical media, and / or any other suitable memory.

[0096] In some embodiments, input device controller 406 may be any suitable circuitry for controlling and receiving input from one or more input devices 408. For example, input device controller 406 may be circuitry for receiving input from a touch screen, a keyboard, a mouse, one or more buttons, voice recognition circuitry, a microphone, a camera, an optical sensor, an accelerometer, a temperature sensor, a near-field sensor, and / or any other type of input device.

[0097] In some embodiments, display / audio driver 410 may be any suitable circuitry for controlling and driving outputs to one or more display / audio output devices 412. For example, display / audio driver 410 may be circuitry for driving a touch screen, a flat panel display, a cathode ray tube display, a projector, one or more speakers, and / or any other suitable display and / or presentation device.

[0098] Communication interface(s) 414 may be used to communicate with one or more communication networks (e.g., Figure 3 For example, interface(s) 414 may include network interface card circuitry, wireless communication circuitry, and / or any other suitable type of communications network circuitry.

[0099] In some embodiments, antenna 416 may be any suitable antenna or antennas for wirelessly communicating with a communication network, such as communication network 304. In some embodiments, antenna 416 may be omitted.

[0100] In some embodiments, bus 418 may be any suitable mechanism for communicating between two or more components 402 , 404 , 406 , 410 , and 414 .

[0101] According to some embodiments, any other suitable components may be included in hardware 400 .

[0102] In some embodiments, Figure 1 and 2 At least some of the above blocks of the process of can be performed or implemented in any order or sequence not limited to the order and sequence shown in and described in conjunction with the accompanying drawings. In addition, Figure 1 and Figure 2Some of the above blocks may be executed substantially simultaneously or in parallel where appropriate to reduce waiting time and processing time. Additionally or alternatively, Figure 1 and Figure 2 Some of the above boxes of the process.

[0103] In some embodiments, any suitable computer-readable medium may be used to store instructions for performing the functions and / or processes herein. For example, in some embodiments, the computer-readable medium may be temporary or non-temporary. For example, non-temporary computer-readable media may include, for example, magnetic media in a non-temporary form (e.g., a hard disk, a floppy disk, and / or any other suitable magnetic media), optical media in a non-temporary form (e.g., a compact disk, a digital video disk, a Blu-ray disk, and / or any other suitable optical media), semiconductor media in a non-temporary form (e.g., flash memory, an electrically programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), and / or any other suitable semiconductor media), any suitable medium that is not transient during transmission or is not without any permanent appearance, and / or any suitable tangible medium. As another example, a temporary computer-readable medium may include a signal on a network, a wire, a conductor, an optical fiber, a signal in a circuit, any suitable medium that is transient during transmission and without any permanent appearance, and / or any suitable intangible medium.

[0104] In cases where the systems described herein collect personal information about a user or utilize personal information, the user may be provided with an opportunity to control whether a program or feature collects information about the user (e.g., information about the user's social network, social actions or activities, occupation, the user's preferences, or the user's current location). In addition, certain data may be processed in one or more ways before being stored or used so as to remove personal information. For example, the user's identity may be processed so that personally identifiable information cannot be determined for the user, or where location information is obtained, the user's geographic location may be summarized (e.g., to a city, zip code, or state level) so that the user's specific location cannot be determined. Thus, the user may control how information about the user is collected and used by the content server.

[0105] Thus, methods, systems, and media for rendering immersive video content utilizing a foveation grid are provided.

[0106] Although the present invention has been described and illustrated in the foregoing illustrative embodiments, it should be understood that the present disclosure is made by way of example only and that many changes in the details of the implementation of the present invention may be made without departing from the spirit and scope of the present invention, which is limited only by the following claims. The features of the disclosed embodiments may be combined and rearranged in various ways.

Claims

1. A method for generating immersive video content, the method comprising: generating, using a hardware processor, a foveation grid onto which frames of a video content item are to be projected, wherein the foveation grid transcodes content of a video content item currently associated with an equirectangular grid, and wherein the foveation grid increases pixel density in a central portion of each frame of the video content item as compared to peripheral portions of each frame of the video content item; determining whether an improvement ratio satisfies a predetermined criterion, wherein the improvement ratio reflects a ratio of a number of pixels in a video content item rendered using the foveation grid to a number of pixels in a video content item rendered using the equirectangular grid; and In response to determining that the improvement ratio satisfies the predetermined criterion, storing, using the hardware processor, the video content item in a file format including the generated foveation grid, wherein the immersive video content is rendered by applying the video content item as a texture to the generated foveation grid.

2. The method according to claim 1, wherein: The foveation grid is generated by deforming the two-dimensional UV map of a cropped equirectangular grid using a polynomial fitting function.

3. The method according to claim 1, wherein: The fixation point grid is generated by generating a first fixation point grid based on a first fixation point ratio parameter and generating a second fixation point grid based on a second fixation point ratio parameter.

4. The method according to claim 1, further comprising: Prior to generating the gaze point grid, it is determined that the video content item is downsampled from a first pixel resolution to a second pixel resolution at a downsampling ratio, and wherein an improvement ratio at a central portion of each frame of the video content item corresponds to the downsampling ratio.

5. The method according to claim 1, further comprising: A determination is made as to whether the received video content item is associated with a particular file format.

6. The method according to claim 5, wherein: The specific file format includes a VR180 file format that uses directional stereoscopic projection with a 180 degree field of view.

7. The method according to claim 1, further comprising: Prior to generating the gaze point grid, determining whether the received video content item is a static video or a dynamic video, wherein the static video comprises video content in which the camera does not substantially move when the video content is captured, and wherein the dynamic video comprises a camera motion metadata track.

8. The method of claim 1, further comprising sending the video content item and the generated gaze point grid to a user device in response to receiving a request for the video content item from the user device.

9. A system for generating immersive video content, the system comprising: Memory; as well as a hardware processor that, when executing computer-executable instructions stored in the memory, is configured to: generating a foveation grid onto which frames of a video content item are to be projected, wherein the foveation grid transcodes content of a video content item currently associated with an equirectangular grid, and wherein the foveation grid increases pixel density in a central portion of each frame of the video content item as compared to peripheral portions of each frame of the video content item; determining whether an improvement ratio satisfies a predetermined criterion, wherein the improvement ratio reflects a ratio of a number of pixels in a video content item rendered using the foveation grid to a number of pixels in a video content item rendered using the equirectangular grid; and In response to determining that the improvement ratio satisfies the predetermined criterion, the video content item is stored in a file format including the generated foveation grid, wherein the immersive video content is rendered by applying the video content item as a texture to the generated foveation grid.

10. The system according to claim 9, wherein: The foveation grid is generated by deforming the two-dimensional UV map of a cropped equirectangular grid using a polynomial fitting function.

11. The system according to claim 9, wherein: The processor is further configured to, prior to generating the gaze point grid, determine that the video content item is downsampled from a first pixel resolution to a second pixel resolution at a downsampling ratio, and wherein an improvement ratio at a central portion of each frame of the video content item corresponds to the downsampling ratio.

12. The system according to claim 9, wherein: The processor is further configured to determine whether the received video content item is associated with a particular file format.

13. The system according to claim 9, wherein: The processor is further configured to, prior to generating the gaze point grid, determine whether the received video content item is a static video or a dynamic video, wherein the static video comprises video content in which the camera does not substantially move when the video content is captured, and wherein the dynamic video comprises a camera motion metadata track.

14. The system according to claim 9, wherein: The processor is further configured to, in response to receiving a request for the video content item from a user device, send the video content item and the generated gaze point grid to the user device.

15. A system for generating immersive video content, the system comprising: Memory; as well as a hardware processor that, when executing computer-executable instructions stored in the memory, is configured to: receiving a video content item; Determining whether the video content includes stereoscopic content; In response to determining that the video content item includes stereoscopic content, determining that a foveation grid is to be generated for transcoding the stereoscopic content of the video content item currently associated with an equirectangular grid; generating a fixation grid for association with the video content item; determining whether an improvement ratio satisfies a predetermined criterion, wherein the improvement ratio reflects a ratio of a number of pixels in the video content item rendered using the foveation grid to a number of pixels in the video content item rendered using the equirectangular grid; as well as In response to determining that the improvement ratio satisfies the predetermined criterion, the immersive video content is rendered by applying the video content item as a texture to the generated foveation mesh.

16. The system of claim 15, wherein: The hardware processor is further configured to determine whether the received video content item is associated with a particular file format.

17. The system of claim 16, wherein: The specific file format includes a VR180 file format that uses directional stereoscopic projection with a 180 degree field of view.

18. The system of claim 15, wherein: The hardware processor is further configured to determine whether the received video content item is a still video or a dynamic video, wherein the still video includes video content in which the camera does not substantially move when the video content is captured, wherein the dynamic video includes a camera motion metadata track, and wherein the gaze point grid is generated upon determining that the received video content item is the still video.

19. The system of claim 15, wherein: The hardware processor is further configured to, in response to receiving a request for the video content item from a user device, send the video content item and the generated gaze point grid to the user device.

20. The system of claim 15, wherein: The foveation grid is generated by deforming the two-dimensional UV map of a cropped equirectangular grid using a polynomial fitting function.

Images