Method and related apparatus for displaying video content

Abstract

A method, a computer system, and a computer readable medium for displaying video content are provided. A presentation engine interface is created for implementing real-time rendering of a plurality of objects, wherein the plurality of objects are logically organized together through a plurality of scene graphs, and wherein the plurality of scene graphs are organized into a media exchange format. Geometric information of the plurality of scene graphs is updated through a presentation engine interface. The plurality of objects are streamed based on multi-resolution processing or multi-tessellation processing of heterogeneous visual geometric objects in each of the plurality of scene graphs using the media exchange format.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. Patent Application No. 17 / 231,513, filed on April 15, 2021, “DATA MODELFOR REPRESENTATION AND STREAMING OF HETEROGENEOUS IMMERSIVE MEDIA,” which claims priority based on U.S. Provisional Application No. 63 / 038,591, filed on June 12, 2020, “DATA MODELFOR REPRESENTATION AND STREAMING OF HETEROGENEOUS IMMERSIVE MEDIA,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure generally relates to immersive media processing technology, and in particular to methods and related devices for displaying video content. Background Art

[0004] "Immersive media" generally refers to media that stimulates any or all of the human sensory systems (vision, hearing, body sensation, smell, and possibly taste) to produce or enhance the user's physical experience in a media environment, which goes beyond the timed two-dimensional video and audio distributed on existing commercial networks. Immersive media is either timed or non-timed. Timed media refers to media that is constructed and presented in terms of time, such as movie features, news reports, and plot content, which are all organized according to time periods. Traditional video and audio are generally considered to be timed media. Non-timed media is not constructed in time, but is constructed through logical, spatial, and / or temporal relationships, such as video games that users can control their experience. Non-timed media can incorporate timed media, for example, in a continuously looping audio clip or video clip of a scene in a video game. A device with immersive media capabilities refers to a device that has the ability to access, interpret, and present immersive media information from immersive media to a user. Such media and devices are heterogeneous in the amount and format of data required to represent immersive information, and the amount and type of network resources required to distribute such media information on a large scale (ie, to achieve distribution equivalent to traditional video and audio media).

[0005] Over the past decade, many hardware devices with immersive media capabilities have been introduced to the consumer market, including head-mounted displays, augmented reality glasses, handheld controllers, haptic gloves, and game consoles. Similarly, holographic displays and other forms of stereoscopic displays will also appear in the next decade. Despite the immediate or expected availability of these devices, a coherent end-to-end ecosystem for the distribution of immersive media has not been realized for several reasons. One of these reasons is the lack of a standard representation for immersive media that enables heterogeneous media formats to be streamed to heterogeneous endpoint clients over a commercial network. Therefore, a standard representation for streaming immersive media over a commercial network is needed to drive heterogeneous endpoints (e.g., devices with immersive media capabilities that have appeared or are expected to appear). Summary of the invention

[0006] The techniques described in this article enable the creation of a unified data model for representing and streaming heterogeneous immersive media over commercial networks to serve heterogeneous endpoints, including holographic displays and XR headsets. The ability to stream media capturing live events (e.g., sports, concerts, news, user-generated content, and distributed gaming) is important for enabling real-time distribution of such media for subsequent presentation on a variety of heterogeneous endpoints.

[0007] According to one aspect, a method for displaying video content is provided. The method may include: creating a presentation engine interface to enable real-time rendering of multiple objects, wherein the multiple objects are logically organized together through multiple scene graphs, and wherein the multiple scene graphs are organized into a media exchange format. Updating geometric information of the multiple scene graphs through a representation engine interface. Streaming the multiple objects based on multi-resolution processing or multi-tesselation processing of heterogeneous visual geometric objects in each of the multiple scene graphs using a media exchange format.

[0008] According to another aspect, a device for displaying video content is provided. The device may include: a creation module configured to create a presentation engine interface to implement real-time rendering of multiple objects, wherein the multiple objects are logically organized together through multiple scene graphs, and wherein the multiple scene graphs are organized into a media exchange format; an update module configured to update geometric information of the multiple scene graphs through the presentation engine interface; and a transmission module configured to stream the multiple objects based on multi-resolution processing or multi-subdivision processing of heterogeneous visual geometric objects in each of the multiple scene graphs using the media exchange format.

[0009] According to another aspect, a computer system for displaying video content is provided. The computer system may include one or more processors and one or more computer-readable tangible storage devices. Program instructions are stored on the computer-readable tangible storage device, and the program instructions are used to be executed by one or more processors, so that the computer system can perform the above method.

[0010] According to another aspect, a computer-readable medium for displaying video content is provided. The computer-readable medium stores program instructions, and the program instructions can be executed by a processor to perform the above method.

[0011] According to the above method, multiple objects are logically organized together through multiple scene graphs, and the multiple scene graphs are organized into a media exchange format, and the multiple objects are streamed based on multi-resolution processing or multi-subdivision processing of heterogeneous visual geometric objects in each scene graph among the multiple scene graphs using the media exchange format, so that the streamed media is not limited to traditional visual and audio media, but may include any type of media information that can generate signals that interact with a machine to stimulate human vision, hearing, taste, touch and smell, and a hierarchical representation is implemented so that heterogeneous clients can first generate a simplified representation of the scene, and then receive additional information for refining the simplified representation according to their needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] These and other objects, features and advantages will become apparent from the illustrative embodiments which are to be read in conjunction with the accompanying drawings. The various features of the drawings are not drawn to scale. In the drawings:

[0013] Figure 1 is a schematic diagram of the end-to-end processing of sequential traditional media distribution.

[0014] Figure 2 is a diagram of a standard media format for streaming time-sequential traditional media.

[0015] Figure 3 is a schematic diagram of a data model for representation and streaming of time-sequential immersive media, according to some implementations.

[0016] Figure 4 is a schematic diagram of a data model for representation and streaming of non-timed immersive media, according to some implementations.

[0017] Figure 5 is a system diagram of a data model for representation and streaming of non-temporal immersive media, according to some implementations. DETAILED DESCRIPTION

[0018] Specific embodiments of the claimed structures and methods are disclosed herein; however, it is understood that the disclosed embodiments are merely illustrative of the claimed structures and methods. However, these structures and methods may be implemented in many different forms and should not be construed as limited to the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided so that the disclosure is thorough and complete and fully conveys the scope to those skilled in the art. In the specification, details of well-known features and techniques are omitted to avoid unnecessarily obscuring the embodiments presented.

[0019] Immersive media can be considered to be one or more types of media that, when presented to humans by a device with immersive media capabilities, stimulates any of the five senses of sight, sound, taste, touch, and hearing in a way that is more realistic and consistent with human experience of the natural world, that is, beyond the way that traditional media presented by traditional devices has created. In this context, the term "traditional media" refers to two-dimensional (2D) visual media, still picture frames or moving picture frames, and / or corresponding audio whose user's interactive capabilities are limited to pause, play, fast forward, or rewind; "traditional devices" refer to televisions, laptops, monitors, and mobile devices whose capabilities are limited to the presentation of traditional media. In consumer-oriented application scenarios, a presentation device for immersive media (i.e., a device with immersive media capabilities) is a consumer-oriented hardware device that has the ability to create a presentation that is closer to human understanding of and interaction with the physical world using specific information embodied by immersive media, that is, beyond the ability of traditional devices to do so.

[0020] Over the past decade, many hardware devices with immersive media capabilities have been introduced to the consumer market, including head-mounted displays, augmented reality glasses, handheld controllers, haptic gloves, and game consoles. Similarly, holographic displays and other forms of stereoscopic displays will also appear in the next decade. Despite the immediate or expected availability of these devices, a coherent end-to-end ecosystem for the distribution of immersive media has not been realized for several reasons. One of these reasons is the lack of a standard representation for immersive media that enables heterogeneous media formats to be streamed to heterogeneous endpoint clients over a commercial network. Therefore, a standard representation for streaming immersive media over a commercial network is needed to drive heterogeneous endpoints (e.g., devices with immersive media capabilities that have appeared or are expected to appear).

[0021] As is well known, traditional devices supported by traditional media have gained widespread consumer adoption because they are equally supported by an ecosystem of traditional media content providers and commercial network service providers, who produce standards-based representations of traditional media and commercial network service providers who provide network infrastructure to connect traditional devices to sources of standard traditional content. In addition to their role in distributing traditional media over the network, commercial network service providers also facilitate the pairing of traditional client devices with access to traditional content on a content distribution network (CDN). Once paired with access to content in an appropriate form, a traditional client device can request or "pull" traditional content from a content server to the device for presentation to an end user.

[0022] Various aspects are described herein with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer-readable media according to various embodiments. It should be understood that each block in the flowchart illustration and / or block diagram and the combination of blocks in the flowchart illustration and / or block diagram can be implemented by computer-readable program instructions.

[0023] Figure 1 is an example illustration of the end-to-end processing of time-sequential traditional media distribution. Figure 1In , time-sequential audio-visual content is captured by a camera or microphone in 101A, or generated by a computer in 101B, creating a sequence 102 of 2D images and associated audio that is input to a preparation module 103. The output of the preparation module 103 is edited content in a so-called Master Format (e.g., for post-production including language translation, subtitles, other editing functions), which can be converted by a converter module 104 into a standard Mezzanine Format (e.g., for on-demand media) or as a standard Contribution Format (e.g., for live events). The media is "ingested" by a commercial network service provider, and an adaptation module 105 packages the media into various bit rates, temporal resolutions (frame rates), or spatial resolutions (frame sizes), which are packaged into a standard Distribution Format. The resulting adaptations are stored on the content distribution network 106, and each client 108A-108C makes a pull request for one or more media adaptations 107A-107C from the content distribution network 106 to obtain the media and present it to the end user. It is important to note that the master format may include a mix of media from 101A or 101B, and that format 101A may be obtained in real time, such as media to be obtained from a live sporting event. In addition, the client 108A-108C is responsible for selecting a specific media adaptation 107A-107C that is most suitable for the client's configuration and / or current network conditions.

[0024] Figure 2 is an example of a standard media format for distributing traditional timed media such as video, audio, and supporting metadata (including timed text for subtitles, for example). Figure 1As shown in item 106 in, media is stored on CDN 201A to 201C in a standard-based distribution format. The standard-based format is shown as MPD 202, which includes multiple timing periods (e.g., periods 203A, 203B), and the timing period has a start time and an end time corresponding to a clock. Each period among the periods 203A and 203B quotes one or more adaptation sets 204A to 204C. Each of the adaptation sets 204A to 204C is generally used for a single type of media, such as video, audio, or timed text. For any given period in a given period 203A and 203B, multiple adaptation sets 204A to 204C can be provided, such as one for video and multiple for audio, such as for translation into various languages. Each adaptation set in the adaptation set 204A to 204C quotes one or more representations 205 for providing information about the frame resolution (for video), frame rate, and bit rate of the media. Multiple representations 205 can be used to provide access to various media. For example, each representation 205 is used for ultra-high definition video, high definition video or standard definition video. Each representation 205 references one or more segment files 206, and the media is actually stored in the segment files 206 for clients (such as Figure 1 108 in the figure) is extracted.

[0025] Figure 3 is an example representation of a streamable format for time-sequential, heterogeneous immersive media. Figure 4 is an example representation of a streamable format for non-temporal, heterogeneous immersive media. The two figures each refer to a scene; Figure 3 Scene 301 involving timed media, and Figure 4 Scenes involving non-timed media 401. In both cases, the scenes can be represented by scene graphs.

[0026] Several techniques have been proposed to directly or indirectly support scene graphs as a format suitable for various types of immersive media.

[0027] 1. OTOY

[0028] OTOY's ORBX is one of several scene graph technologies that can support any type of temporal or non-temporal visual media, including ray-traced visual formats, traditional (frame-based) visual formats, volumetric visual formats, and other types of synthetic or vector-based visual formats. ORBX is different from other scene graphs because ORBX provides native support for freely available and / or open source formats for meshes, point clouds, and textures. ORBX's scene graph is designed to facilitate interchange between multiple scene graph operator technologies. In addition, ORBX provides a rich material system, support for an open shader language, a robust camera system, and support for Lua scripting. ORBX is also the basis for the immersive technology media format released by the Alliance for Immersive Digital Experiences under a royalty-free license.

[0029] 2. Pixar’s Generic Scene Description

[0030] Pixar's Universal Scene Description (USD) is another well-known and mature scene graph popular in the VFX and professional content production community. USD is integrated into Nvidia's Omniverse platform, a set of tools for developers to create and render 3D models using Nvidia's GPUs. A subset of USD was released by Apple and Pixar as USDZ. USDZ is supported by Apple's ARKit.

[0031] 3. Khronos’ glTF 2.0

[0032] glTF 2.0 is the latest version of the "Graphics Language Transmission Format" specification written by the Khronos 3D Group. The format supports a simple scene graph format that is typically capable of supporting static (non-temporal) objects in a scene, including "png" and "jpeg" image formats. glTF 2.0 supports simple animations, including support for translation, rotation, and scaling of basic shapes (i.e., geometric objects) described using glTF primitives. glTF 2.0 does not support temporal media, and therefore supports neither video nor audio.

[0033] 4. Unity Engine and Unreal Engine

[0034] Both Unity and Unreal Engine support rich scene graphs including ORBX and USD for rendering, and can also support game applications. Unity and Unreal are examples of game engines. Both can also perform path tracing of ray-traceable media, for example, they can model the physically correct behavior of light.

[0035] 5. Advanced networking technology

[0036] Network Functions Virtualisation and MEC are standards developed by ETSI, while Software Defined Networks (SDN) is a technology area standardized by IEEE. Network-Based Media Processing is also the name of a recently completed standard developed by MPEG. Each of these standards provides features that together enable distributed, flexible and dynamic configuration and adaptation of commercial network resources, in particular for the distribution and processing of large-scale media across commercial networks.

[0037] 6. CableLabs

[0038] The CableLabs proposal describes "Systems and Methods for Network-Based Media Processing." The proposal describes a "unified" media exchange format that is "independent" of the type of endpoint client in the network. The network-based media processing described therein is different from the network-based media processing specified by MPEG. In the CableLabs proposal, the network employs separate control planes and data planes that enable media processing and conversion in an exchange format into a specific format that ultimately "binds" the media to a specific type of endpoint client device. In the MPEG standard, there is no distinction between the control plane and the data plane. In the CableLabs proposal, media in the media exchange format is referred to as "display agnostic," while the resulting converted media (i.e., from CableLabs' network-based media processing) is referred to as "display specific." However, the CableLabs specification does not suggest a mechanism for how to stream media, nor does it provide any information about the organization of media formats, compression of underlying geometry formats, representation of the functional capabilities of endpoint clients, representation of application requirements for endpoint clients, or representation of features corresponding to service level agreements between clients and network service providers. Furthermore, the CableLabs specification does not consider immersive media to include media other than visual or audio formats.

[0039] 7. Moving Picture Experts Group

[0040] The ISO / IEC JTC1 / SC29 / WG11 Moving Picture Experts Group (MPEG) is developing an extension to glTF2.0 to enable support for timed MPEG media in glTF. Since glTF is currently used for rendering of static assets, the MPEG work on extending glTF will naturally not fit in with existing applications or ecosystems that widely deploy glTF. Similarly, the current ecosystem for timed media does not adopt scene graphs to distribute timed media. The fact that MPEG is considering using scene graphs to support timed immersive media shows that MPEG has recognized the importance of scene graphs for the organization of independent volumetric objects, however in its current design, MPEG plans to package the scene graph and the individual objects referenced by the graph into the ISO Base Media File Format file. This is particularly restrictive because in terms of data hierarchy, the scene graph container should be a container wrapper for both MPEG media and non-MPEG media, and therefore should not be at the same level (in terms of logical data hierarchy) as MPEG media.

[0041] MPEG-I is a set of standards currently being developed by MPEG. Starting with MPEG 130, the set of standards consists of 14 separate specifications called "Parts". Part 14 of the set of standards is MPEG's first attempt to create a data structure that can support both MPEG media and non-MPEG media. The latest draft of Part 14 is available from MPEG 130's N19070, entitled "Scene Description for MPEG Media". See N18339 for the requirements for scene descriptions.

[0042] In the proposal submitted to MPEG 130 in M53364, Qualcomm proposed a design to update the scene graph using the JSON patch protocol. Since MPEG media is applied to the scene graph, the proposal does not support updates to the scene graph for non-MPEG media. That is, the proposal in M53364 only works for updates related to MPEG media, and does not support updates to the scene graph for both sequential media (e.g., based on MPEG standards) or non-sequential media (not based on MPEG standards).

[0043] Return to reference Figure 1 ,although Figure 1Each of the clients 108A to 108C in the embodiment is embodied in a different form (e.g., a fixed display, a mobile display, a laptop display, a head-mounted display), but these different forms all apply a homogeneous form of media. That is, each media adaptation result in the media adaptation result 107 basically includes a 2D video and a corresponding audio signal, which is compressed and formatted according to a traditional standard compression scheme for distributing time-sequential 2D video and corresponding audio. Commonly used compression schemes for video include official standards (de-jure standards) established by the ISO / IEC Moving Picture Experts Group (MPEG), the Alliance for Open Media (Alliance for Open Media, AOM) or the Society of Motion Pictures and Television Engineers (Society of Motion Pictures and Television Engineers, SMPTE). Commonly used compression schemes for audio include official standards established by MPEG, SMPTE, ETSI, and de-facto standards established by Dolby and DTS (now wholly owned by Xperi). Among the widely used compression formats for video (including MPEG, SMPTE, and AOM), the compressed video bitstream is also homogeneous, using a common architecture and a relatively common tool set to eliminate spatial redundancy within each picture frame and temporal redundancy between each picture frame. Similarly, there is a common architecture and a common tool set between audio formats.

[0044] The consequence of this homogeneity of audiovisual compression formats and homogeneity of client endpoints is that a significant barrier to entry is created for commercial distribution, as heterogeneous immersive media formats do not necessarily exploit the same visual and audio architectures in their representations. This problem is particularly evident for visual information, the transmission of which currently consumes a large portion of the resources and capacity of the existing Internet.

[0045] However, as advanced network technologies such as 5G for mobile networks and fiber optic cables for fixed networks are deployed, the capacity and capabilities of commercial networks are also improved because such advanced network infrastructure can support the transmission and delivery of increasingly large amounts of visual information. In addition, network infrastructure management technologies such as Multi-access Edge Computing (MEC), Software Defined Networks (SDN), and Network Functions Virtualization (NFV) enable commercial network service providers to flexibly configure their network infrastructure to adapt to changes in demand for certain network resources, for example, in response to dynamic increases or decreases in demand for network throughput, network speed, round-trip latency, and computing resources. In addition, this inherent ability to adapt to dynamic network requirements also facilitates the network's ability to support a variety of immersive media applications with potentially heterogeneous visual media formats for heterogeneous client endpoints.

[0046] Immersive media applications themselves may also have different requirements for network resources, including gaming applications that require significantly lower network latency to respond to real-time updates in the state of the game, telepresence applications that have symmetric throughput requirements for both the uplink and downlink portions of the network, and passive viewing applications that may have increased demands on downlink resources depending on the type of client endpoint display that is consuming the data. In general, any consumer-oriented application may be supported by a variety of client endpoints with various onboard client capabilities for storage, computation, and power, and with various requirements for specific media presentations.

[0047] For example, a fixed holographic display used as a client endpoint will need access to visual information represented in a holographic format, while a virtual reality head-mounted display used as a client endpoint will need access to visual information represented in a stereoscopic omnidirectional format. Similarly, an augmented reality display will need access to visual information represented as a single volumetric object.

[0048] Currently, large-scale streaming and distribution of media over commercial networks is limited to media compressed and represented in traditional 2D media formats. This limitation constrains the deployment of immersive consumer endpoint devices and similar immersive applications to those that can utilize traditional 2D visual media formats to create immersive experiences that are subject to the technical constraints of the 2D formats. Endpoint clients that need access to visual information representing quantities of light, such as light field displays (also known as holographic displays), cannot be supported. Likewise, any endpoint clients that need access to ray-traced visual information or visual information with photo-realism cannot be supported.

[0049] Despite the emergence of advanced network technologies that dynamically support and adapt to various network conditions and application requirements; and despite the availability of consumer-oriented immersive media-capable devices, a coherent ecosystem for the widespread distribution of heterogeneous immersive media to a heterogeneous set of immersive media-capable devices does not exist. The basic components for such a coherent ecosystem remain to be defined. These elements include a unified architecture coupled with a unified media format that will facilitate the development of a coherent ecosystem similar to that currently exists for traditional media. Such a unified architecture and data model that proves that it can be streamed across commercial networks to support interactive applications and heterogeneous endpoint clients that need to access heterogeneous media formats, including: traditional 2D information, single volumetric object information, volumetric scene information, ray-traced visual information, timed media or non-timed media, or some combination of these visual formats, is necessary to break through the current barriers to entry for immersive media.

[0050] Embodiments of the present application describe a heterogeneous media framework and corresponding media representations for streaming heterogeneous media to heterogeneous endpoint client devices to support passive media experiences and interactive media experiences suitable for deployment on commercial networks. The underlying network infrastructure for deploying the present invention can be a MEC-based network or a CDN-based network.

[0051] The content expands upon previous immersive media technologies with the following features:

[0052] 1. Media streamed according to an encompassing media format is not limited to traditional visual and audio media, but may include any type of media information capable of generating signals that interact with a machine to stimulate human vision, hearing, taste, touch, and smell.

[0053] 2. The surround media format is streamable by implementing a manifest and layered representation that enables the client to receive only a minimal amount of information (e.g., in a base layer representation for an object) so that the client can first generate a simplified representation of the scene and then receive additional information for refining the simplified representation (e.g., in an enhancement layer for an object).

[0054] 3. The media streamed according to the surround media format may be sequential media or non-sequential media, or a mixture of the two.

[0055] 4. In addition, the surround media format is further made streamable by using a base layer and enhancement layer architecture to achieve a layered representation for media objects. In one example, the separated base layer and enhancement layer are calculated by applying multi-resolution techniques or multi-segmentation analysis techniques to the media objects in each scene. This is similar to the progressively rendered image formats specified in ISO / IEC10918-1 (JPEG) and ISO / IEC 15444-1 (JPEG2000), but is not limited to raster-based visual formats. In an example embodiment, the progressive representation of the geometric object can be a multi-resolution representation of the object calculated using wavelet analysis.

[0056] In another example of a layered representation of a media format, an enhancement layer applies different properties to a base layer, such as refining the material properties of a surface of a visual object represented by the base layer. In yet another example, these properties may refine the texture of a surface of a base layer object, such as changing a surface from a smooth texture to a porous texture, or from a matte surface to a shiny surface.

[0057] In yet another example of a layered representation, the surface of one or more visual objects in a scene may be changed from a Lambertian surface to a ray-traced surface.

[0058] In yet another example of a layered representation, the network will distribute a base layer representation to a client so that the client can create a nominal presentation of the scene while the client awaits the transmission of additional enhancement layers to refine the resolution or other characteristics of the base representation.

[0059] 5. The resolution of the attributes or refinement information in the enhancement layer is not explicitly coupled to the resolution of the objects in the base layer, as is the case in today's existing MPEG video standards and JPEG image standards.

[0060] 6. Surround media formats support any type of information media that can be presented or driven by a presentation device or machine, thereby enabling support of heterogeneous media formats for heterogeneous client endpoints. In one embodiment of a network that distributes media formats, the network will first query the client endpoint to determine the capabilities of the client, and if the client cannot meaningfully ingest the media representation, the network will remove layers of attributes that are not supported by the client, or transcode the media from its current format to a format suitable for the client endpoint. In one example of such transcoding, the network will convert a volumetric visual media asset into a 2D representation of the same visual asset by using a network-based media processing protocol.

[0061] 7. The manifest for a complete or partially complete immersive experience (live streaming event, game, or playback of on-demand assets) is organized by scene, which is the minimum amount of information that the rendering and game engines can currently ingest in order to create a presentation. The manifest includes a list of individual scenes to be rendered for the entire immersive experience requested by the client. Associated with each scene is one or more representations of geometric objects within the scene corresponding to a streamable version of the scene geometry. A scene representation can reference a low-resolution version of the scene's geometric objects. The same scene can also reference an enhancement layer of the low-resolution representation of the scene to add additional detail or increase subdivision to the geometric objects of the same scene. As described above, each scene can have more than one enhancement layer to increase the detail of the scene's geometric objects in a progressive manner.

[0062] 8. Each layer of media objects referenced within the scene is associated with a token (e.g., URI) that points to an address where a resource can be accessed within the network. These resources are similar to CDN resources, whose content can be obtained by the client.

[0063] 9. The token used for the representation of the geometric object can point to a location within the network or to a location within the client. That is, the client can signal to the network that its resources are available to the network for network-based media processing.

[0064] Figure 3 An implementation of a surround media format for timed media is described as follows. A timed scene manifest includes a list of information corresponding to one or more scenes 301. The scenes 301 reference a list of components 302, which respectively describe processing information and types of media assets comprising the scenes 301. The components 302 reference assets 303, such as visual, audio, tactile, and olfactory assets, which further reference base layers 304 and attribute enhancement layers 305, such as resolution enhancement layers, temporal enhancement layers, and material enhancement layers.

[0065] Figure 4 An implementation of a surround media format for non-timed media is described as follows. Scene information 401 is independent of the start and end times of a clock. Scene information 401 references a list of components 402, which respectively describe the processing information and type of the media assets comprising scene 401. Component 402 references asset 403, which further references base layer 404 and attribute enhancement layer 405. In addition, scene 401 references other scenes for non-timed media. Scene 401 also references a timed media scene.

[0066] The above-described techniques for representing and streaming heterogeneous immersive media may be implemented as computer software including computer-readable instructions physically stored in one or more computer-readable media. Figure 5A computer system 500 suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0067] Computer software may be encoded using any suitable machine code or computer language, which may be subjected to mechanisms such as assembly, compilation, and linking to create code comprising instructions that may be executed directly by a computer central processing unit (CPU), graphics processing unit (GPU), etc., or through interpretation, microcode execution, etc.

[0068] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, etc.

[0069] Figure 5 The components for computer system 500 shown in the example are exemplary and do not limit the scope of use or functionality of computer software implementing embodiments of the present disclosure. The configuration of components should also not be interpreted as having any dependency or requirement related to any one or combination of components shown in the exemplary embodiment of computer system 500.

[0070] Computer system 500 may include certain human-machine interface input devices. Such human-machine interface input devices may be responsive to inputs implemented by one or more human users through, for example, tactile inputs (e.g., keystrokes, swipes, data glove movements), audio inputs (e.g., voice, tapping), visual inputs (e.g., gestures), olfactory inputs (not depicted). Human-machine interface devices may also be used to capture certain media that are not necessarily directly related to human conscious inputs, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), and videos (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0071] The input human interface device may include one or more of the following (only one of each is depicted): keyboard 501 , mouse 502 , trackpad 503 , touch screen 510 , data gloves (not depicted), joystick 505 , microphone 506 , scanner 507 , camera 508 .

[0072] The computer system 500 may also include certain human-machine interface output devices. Such human-machine interface output devices may stimulate one or more senses of a human user through, for example, tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include: tactile output devices (e.g., tactile feedback through touch screen 510, data gloves (not depicted), or joystick 505, but there may also be tactile feedback devices that are not used as input devices); audio output devices (e.g., speakers 509, headphones (not depicted)); visual output devices (e.g., screen 510, including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capabilities, each with or without tactile feedback capabilities - some of which may be able to output two-dimensional visual output or more than three-dimensional output by means such as stereoscopic image output; virtual reality glasses (not depicted); holographic displays and smoke tanks (not depicted)); and printers (not depicted).

[0073] The computer system 500 may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW 520 with CD / DVD etc. media 521, a thumb drive 522, a removable hard drive or solid state drive 523, traditional magnetic media such as magnetic tapes and floppy disks (not depicted), dedicated ROM / ASIC / PLD based devices such as security dongles (not depicted), and the like.

[0074] Those skilled in the art should also understand that the term "computer-readable media" used in conjunction with the presently disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0075] The computer system 500 may also include an interface to one or more communication networks. The network may be, for example, wireless, wired, optical. The network may also be local, wide, metropolitan, vehicle-mounted and industrial, real-time, delay-tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless LAN, cellular networks (including GSM, 3G, 4G, 5G, LTE, etc.), television wired or wireless wide area digital networks (including cable television, satellite television and terrestrial broadcast television), vehicle-mounted and industrial networks (including CANBus), etc. Some networks typically require an external network interface adapter (e.g., such as a USB port of the computer system 500) attached to some universal data port or peripheral bus (549); other networks are typically integrated into the core of the computer system 500 by attaching to the system bus as described below (e.g., to an Ethernet interface in a PC computer system or to a cellular network interface in a smart phone computer system). Using any of these networks, the computer system 500 can communicate with other entities. Such communications may be one-way, receive-only (e.g., broadcast TV), one-way send-only (e.g., CANbus to certain CANbus devices), or two-way (e.g., to other computer systems using a local area digital network or a wide area digital network). Specific protocols and protocol stacks may be used on each of these networks and network interfaces as described above.

[0076] The above-mentioned human-machine interface device, human-accessible storage device, and network interface may be attached to the core 540 of the computer system 500 .

[0077] The core 540 may include one or more central processing units (CPUs) 541, graphics processing units (GPUs) 542, dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) 543, hardware accelerators 544 for certain tasks, etc. These devices, along with read-only memory (ROM) 545, random access memory 546, internal mass storage devices 547 (e.g., internal non-user accessible hard drives, SSDs, etc.), may be connected via a system bus 548. In some computer systems, the system bus 548 may be accessed in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, etc. Peripheral devices may be attached to the system bus 548 of the core directly or via a peripheral bus 549. The architecture of the peripheral bus includes PCI, USB, etc.

[0078] The CPU 541, GPU 542, FPGA 543, and accelerator 544 may execute certain instructions, which in combination may constitute the computer code mentioned above. The computer code may be stored in ROM 545 or RAM 546. Transient data may also be stored in RAM 546, while permanent data may be stored in, for example, an internal mass storage device 547. Fast storage and retrieval of any of the memory devices may be achieved by using a cache memory, which may be closely associated with one or more CPUs 541, GPUs 542, mass storage devices 547, ROM 545, RAM 546, and the like.

[0079] The computer readable medium may have computer code thereon for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of a type well known and available to those skilled in the art of computer software.

[0080] As an example and not limitation, a computer system 500 having an architecture, in particular a core 540, can provide functionality due to a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software implemented in one or more tangible computer-readable media. Such a computer-readable medium can be a medium associated with a user-accessible mass storage device as described above, as well as certain storage devices of the core 540 having non-transitory properties, such as a core internal mass storage device 547 or a ROM 545. Software implementing various embodiments of the present disclosure can be stored in such a device and executed by the core 540. Depending on specific needs, the computer-readable medium may include one or more memory devices or chips. The software can enable the core 540 and in particular the processor therein (including a CPU, GPU, FPGA, etc.) to perform a specific process or a specific part of a specific process described herein, including defining a data structure stored in the RAM 546 and modifying such a data structure according to a process defined by the software. Additionally or as an alternative, the computer system may provide functionality due to logic hard-wired or otherwise implemented in a circuit (e.g., accelerator 544) that may replace software or operate with software to perform a specific process or a specific portion of a specific process described herein. Where appropriate, references to software may include logic, and vice versa. Where appropriate, references to computer-readable media may include circuits (e.g., integrated circuits (ICs)) storing software for execution, circuits implementing logic for execution, or both. The present disclosure includes any suitable combination of hardware and software.

[0081] definition

[0082] Scene graph: A general-purpose data structure commonly used by vector-based graphics editing applications and modern computer games that arranges the logical representation of a graphical scene and usually (but not necessarily) also the spatial representation; a collection of nodes and vertices in a graph structure.

[0083] Node: The basic element of the scene graph, including information related to the logical representation, spatial representation, or temporal representation of vision, audio, touch, smell, or taste, or related processing information; each node should have at most one output edge, zero or more input edges, and at least one edge (input or output) connected to the node.

[0084] Base Tier: A nominal representation of an asset, typically formulated to minimize the computing resources or time required to render the asset or to transfer the asset over a network.

[0085] Enhancement Layer: A set of information that, when applied to a base layer representation of an asset, enhances the base layer to include features or capabilities not supported in the base layer.

[0086] Attribute: Metadata associated with a node that describes a particular property or characteristic of that node in a canonical or more complex form (e.g., in terms of another node).

[0087] Container: A serialization format for storing and exchanging information to represent a fully natural scene, a fully synthetic scene, or a mix of synthetic and natural scenes, including the scene graph and all the media assets needed to render the scene.

[0088] Serialization: The process of converting a data structure or object state into a format that can be stored (e.g., in a file or storage buffer) or transmitted (e.g., over a network connection link) and subsequently reconstructed (possibly in a different computer environment). When the resulting sequence of bits is reread according to the serialization format, the serialization format can be used to create a semantically identical clone of the original object.

[0089] Renderer: An application or process (usually software-based) based on a selective mixture of disciplines related to acoustic physics, light physics, visual perception, audio perception, mathematics, and software development that, given an input scene graph and an asset container, typically emits visual and / or audio signals suitable for presentation on a target device or conforming to desired properties specified by the attributes of a render target node in the scene graph. For visual-based media assets, a renderer may emit visual signals suitable for a target display or for storage as an intermediate asset (e.g., for repackaging into another container, i.e., for use in a series of rendering processes in a graphics pipeline); for audio-based media assets, a renderer may emit audio signals for presentation in multi-channel speakers and / or binauralized headphones, or for repackaging into another (output) container. Popular examples of renderers include: Unity, Unreal.

[0090] Evaluation: Produce abstract output into concrete results (e.g., similar to the evaluation of the Document Object Model of a web page).

[0091] Scripting Language: An interpreted programming language that can be executed by the renderer at runtime to handle dynamic input and mutable state changes made to scene graph nodes that affect the rendering and evaluation of spatial and temporal object topology (including physical forces, constraints, IK, deformations, collisions), and energy propagation and transmission (light, sound).

[0092] Shader: A type of computer program that was originally used for shading (producing appropriate levels of light, darkness, and color within an image), but which now performs a variety of specialized functions in various areas of computer graphics special effects, or to perform video post-processing unrelated to shading or even functions completely unrelated to graphics.

[0093] Path tracing: A computer graphics method for rendering a three-dimensional scene so that the scene's lighting is realistic.

[0094] Timed Media: Media that is ordered by time; for example, has a start time and an end time according to a specific clock.

[0095] Non-temporal media: Media organized in spatial, logical, or temporal relationships; for example, as in an interactive experience that is based on actions taken by the user.

[0096] Some embodiments may involve systems, methods, and / or computer-readable media at any possible level of integrated technical detail. The computer-readable medium may include a computer-readable non-transitory storage medium (or multiple media) having computer-readable program instructions thereon for causing a processor to perform operations.

[0097] A computer-readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following items: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device such as a punch card or a convex structure in a groove on which instructions are recorded, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be interpreted as being a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted by a wire.

[0098] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network, and forwards the computer-readable program instructions to be stored in a computer-readable storage medium in the corresponding computing / processing device.

[0099] The computer readable program code / instruction for performing the operation can be an assembly instruction, an instruction set architecture (ISA) instruction, a machine instruction, a machine-related instruction, a microcode, a firmware instruction, a state setting data, a configuration data for an integrated circuit system, or a source code or an object code written in any combination of one or more programming languages, wherein the one or more programming languages ​​include object-oriented programming languages ​​such as Smalltalk, C++, etc. and process programming languages ​​such as "C" programming language or similar programming languages. The computer readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of executing entirely on a remote computer or server, the remote computer can be connected to the user's computer through any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., the Internet through the use of an Internet service provider). In some embodiments, an electronic circuit system including, for example, a programmable logic circuit system, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer-readable program instructions to personalize the electronic circuit system by utilizing state information of the computer-readable program instructions to perform various aspects or operations.

[0100] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can guide the computer, programmable data processing device, and / or other device to function in a specific manner, so that the computer-readable storage medium having the instructions stored therein includes an article of manufacture, which includes instructions for implementing various aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0101] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing device, or other apparatus, so that a series of operational steps to be performed on the computer, other programmable device, or other apparatus produce a computer-implemented process, so that the instructions executed on the computer, other programmable device, or other apparatus implement the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0102] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functions and operations of possible implementations of systems, methods and computer-readable media according to various embodiments. In this regard, each block in the flowchart or block diagram may represent a module, segment or partial instruction, and the module, segment or partial instruction includes one or more executable instructions for implementing a specific logical function. Methods, computer systems and computer-readable media may include additional blocks, fewer blocks, different blocks or blocks of different arrangements compared to the blocks depicted in the accompanying drawings. In some alternative implementations, the functions noted in the blocks may not occur in the order noted in the accompanying drawings. For example, the two blocks shown in succession can actually be performed simultaneously or substantially simultaneously, or the blocks can sometimes be performed in reverse order according to the functions involved. It should also be noted that each block in the block diagram and / or flowchart illustration and the combination of blocks in the block diagram and / or flowchart illustration can be implemented by a hardware-based dedicated system that performs a specific function or action or realizes a combination of dedicated hardware and computer instructions.

[0103] It will be apparent that the systems and / or methods described herein can be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the implementation. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to specific software codes, and it should be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0104] Unless explicitly described to this, the elements, actions or instructions used in this article should not be interpreted as critical or necessary. In addition, as used herein, the articles "a kind of" and "an" are intended to include one or more items, and can be used interchangeably with "one or more". In addition, as used herein, the term "group" is intended to include one or more items (e.g., related items, unrelated items, combinations of related items and unrelated items, etc.), and can be used interchangeably with "one or more". In the case of meaning only one item, the term "one" or similar language is used. In addition, as used herein, the terms "have", "have", "contain" etc. are intended to be open terms. In addition, unless otherwise explicitly stated, the phrase "based on" is intended to mean "based at least in part on".

[0105] Descriptions of various aspects and embodiments have been presented for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Although combinations of features are described in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features can be combined in ways that are not specifically described in the claims and / or disclosed in the specification. Although each dependent claim listed can directly reference only one claim, the disclosure of possible implementations includes each dependent claim combined with each other claim in the claim group. Without departing from the scope of the described embodiments, many modifications and changes will be obvious to those of ordinary skill in the art. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or technical improvements relative to existing technologies on the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for displaying video content, characterized in that: The method comprises: Creating a rendering engine interface to implement real-time rendering of a plurality of objects, wherein the plurality of objects are logically organized together through a plurality of scene graphs, wherein the plurality of scene graphs are organized into a media exchange format, wherein the media exchange format further comprises scene graph information for each of a plurality of auditory objects, tactile objects, olfactory objects, and taste objects, wherein the scene graph information establishes one or more of a logical relationship, a spatial relationship, and a temporal relationship with heterogeneous visual geometric objects; Updating geometric information of the plurality of scene graphs through a presentation engine interface; and Based on a media interchange format in which scene graphs are organized and their geometric information is updated, multiple objects based on multi-resolution processing or multi-subdivision processing of heterogeneous visual geometric objects in each scene graph are streamed through a base layer representation obtained at least based on the scene graph information and one or more enhancement layers that improve the base layer representation, wherein associated with each scene are one or more representations of geometric objects within the scene corresponding to a streamable version of the scene geometry, the scene representation refers to a low-resolution version of the geometric objects of the scene, and the same scene refers to an enhancement layer of the low-resolution representation of the scene for adding additional details or increasing subdivision to the geometric objects of the same scene, and each scene has more than one enhancement layer to increase the details of the geometric objects of the scene in a progressive manner.

2. The method according to claim 1, characterized in that: The method further comprises: Updates the geometry of a specific scene graph, The updating comprises increasing the resolution of the plurality of objects in the multi-resolution representation of the particular scene graph.

3. The method according to claim 1, further comprising: Interacting with the plurality of objects in the scene graph is performed through the presentation engine interface.

4. The method according to claim 1, characterized in that The method further comprises: The camera position is updated through the presentation engine interface.

5. The method according to claim 1, characterized in that The method further comprises: A representation for immersive media is created in a scene graph, wherein the scene graph creates logical, spatial, and temporal relationships between individual and heterogeneous forms of immersive media, and wherein the representation includes media capable of interacting with or stimulating one or more of human sensory systems, including vision, hearing, touch, taste, and smell.

6. The method according to claim 1, characterized in that The method further comprises: Each geometric object referenced within the scene in the scene graph is associated with a token that points to an address within the network where the resource can be accessed.

7. The method according to claim 6, wherein: The token includes a Universal Resource Identifier (URI).

8. The method according to claim 7, wherein: The content of the resource is obtained by the client by accessing the URI.

9. The method according to claim 6, wherein: The token points to (1) a location within the network or (2) a location within the client.

10. The method according to claim 9, wherein: The client signals to the network in response that its resources are available to the network for network-based media processing.

11. The method according to claim 1, characterized in that: The method further comprises: A network-served representation is created for a streamable geometric object present on a network or within a client, wherein the network-served representation is simultaneously maintained in multiple scenes.

12. The method according to claim 11, wherein: If the streamable geometric object exists on a client managed by the network, the network reserves the streamable geometric object to make it available.

13. The method according to claim 1, characterized in that The method further comprises: Encapsulate the OpenXR API using a set of network APIs and client APIs; capturing interactions from users; and Respond to the interaction using the encapsulated OpenXR API.

14. The method according to claim 13, wherein: The interaction includes one or more of: movement of the user's head, and tactile interaction with a glove.

15. The method according to any one of claims 1 to 14, characterized in that The method further comprises: Prior to receiving the scene graph at a client, availability of specific representations of geometric objects in the scene graph at specific locations within a network is maintained by a network media resource manager, wherein the scene graph references the specific representations of the geometric objects.

16. The method according to claim 15, characterized in that The method further comprises: One or more references to geometric object representations used by clients connected to the network are maintained by the network media resource manager.

17. The method according to claim 16, characterized in that The method further comprises: determining that all references in the plurality of references associated with the particular media resource have been deleted; and In response to the determination, cached resources are purged by the network media resource manager.

18. A device for displaying video content, characterized in that: The device comprises: a creation module configured to create a rendering engine interface to implement real-time rendering of a plurality of objects, wherein the plurality of objects are logically organized together through a plurality of scene graphs, wherein the plurality of scene graphs are organized into a media exchange format, wherein the media exchange format further comprises scene graph information for each of a plurality of auditory objects, tactile objects, olfactory objects, and taste objects, wherein the scene graph information establishes one or more of a logical relationship, a spatial relationship, and a temporal relationship with heterogeneous visual geometric objects; An update module configured to update the geometric information of the plurality of scene graphs through a presentation engine interface; and A transmission module is configured to stream a plurality of objects based on multi-resolution processing or multi-subdivision processing of heterogeneous visual geometric objects in each scene graph based on a media exchange format in which a scene graph is organized and its geometric information is updated, through a base layer representation obtained at least based on the scene graph information and one or more enhancement layers that improve the base layer representation, wherein associated with each scene are one or more representations of geometric objects within the scene corresponding to a streamable version of the scene geometry, the scene representation refers to a low-resolution version of the geometric objects of the scene, the same scene refers to an enhancement layer of the low-resolution representation of the scene, which is used to add additional details or increase subdivision to the geometric objects of the same scene, and each scene has more than one enhancement layer to increase the details of the geometric objects of the scene in a progressive manner.

19. A computer system for displaying video content, the computer system comprising: one or more computer-readable non-transitory storage media configured to store computer program code; as well as One or more computer processors configured to access and execute the computer program code to perform the method for displaying video content according to any one of claims 1 to 17.

20. A non-transitory computer readable medium having stored thereon a computer program for displaying video content, the computer program being configured to cause one or more computer processors to perform the method for displaying video content according to any one of claims 1 to 17.

Images