Real node extension in scene description

Abstract

Methods, devices, and data streams are provided for generating, transmitting, and decoding scene descriptions of an augmented reality scene. According to the present principles, a scene graph links nodes and includes information indicating which nodes correspond to objects of a first list of 3D models corresponding to objects of a real environment of the scene and / or which nodes correspond to objects of a second list of 3D models corresponding to virtual objects of the scene.

Description

1. Technical Field

[0001] The present principles generally relate to the field of extended reality scene description and extended reality scene rendering. In particular, the present principles relate to the description of the management of objects of the real environment in the description of a 3D scene. This document is also understood in the context of formatting and playback of extended reality applications when rendered on an end-user device (such as a mobile device or a head-mounted display (HMD), such as perspective glasses). 2. Background technology

[0002] This section is intended to introduce the reader to various aspects of the art, which may be related to various aspects of the present principles described and / or claimed below. This discussion is considered to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present principles. Therefore, it should be understood that these statements are to be read in this light, and not as admissions of prior art.

[0003] Extended Reality (XR) is a technology that enables interactive experiences in which real-world environments and / or video content are augmented by virtual content, which can be defined across multiple sensory modalities (including vision, hearing, touch, etc.). During the runtime of an application, virtual content (e.g., 3D content or audio / video files) is rendered in real time in a manner consistent with the user context (environment, viewpoint, device, etc.). Scene graphs (such as, for example, those proposed by Khronos / glTF and its extensions defined in the MPEG scene description format or Apple / USDZ) are possible ways to represent the content to be rendered. They combine, on the one hand, a declarative description of a scene structure linking real environment objects and virtual objects, and on the other hand, a binary representation of virtual content. Binary representations of real objects may also be included in the nodes of the scene description. Such representations of real objects may have been obtained, for example, by scanning the real environment. The scene description framework ensures that timed media and corresponding related virtual content are available at any time during the rendering of an application. The scene description may also carry scene-level data that describes how scene objects behave and interact at runtime to obtain an immersive XR experience. The management of representations of real objects (real and / or virtual, the user being the real object) is a technical challenge, as they may be used for different processing when rendering. There is a lack of a solution for scene description that indicates which objects of the scene include representations of virtual objects and which objects include representations of objects of the real environment. 3. Summary of the invention

[0004] A simplified summary of the present principles is presented below to provide a basic understanding of some aspects of the present principles. This summary is not an extensive review of the present principles. It is not intended to identify the key or decisive elements of the present principles. The following summary merely presents some aspects of the present principles in a simplified form as a preface to the more detailed description provided below.

[0005] The present principles relate to a method for generating an extended reality scene description. The method includes obtaining a first list of 3D models of objects of a real environment corresponding to the scene and a second list of 3D models of virtual objects corresponding to the scene. Then, a scene graph is generated, which links nodes and includes information indicating which nodes correspond to objects of the first list and which nodes correspond to objects of the second list. The extended reality scene description is encoded in a data stream, which has node-level information and scene-level information.

[0006] In an embodiment, the information is an array of indices of nodes corresponding to objects of the first list stored at the scene level. In another embodiment, the information is an array of indices of nodes corresponding to objects of the second list stored at the scene level. In another embodiment, the nodes of the scene graph include information indicating whether the node corresponds to an object of the first list or an object of the second list. All of these embodiments can be combined to replicate or verify the information.

[0007] The present principles also relate to a device for implementing the above method. The present principles also relate to a method and device for decoding and processing a scene description generated according to the above method. The present principles also relate to a data stream carrying data representing a scene description generated according to the above method. 4. Description of the drawings

[0008] The present disclosure will be better understood, and other particular features and advantages will emerge, upon reading the following description, which refers to the accompanying drawings, in which:

[0009] - Figure 1 An example diagram of an extended reality scene description according to the present principles is shown;

[0010] - Figure 2 An example architecture of an XR processing engine that may be configured to implement the methods described with respect to the present principles is shown;

[0011] - Figure 3 An example of an embodiment of the syntax of a data stream encoding an extended reality scene description according to the present principles is shown;

[0012] - Figure 4 An example of a rendering of an extended reality scene is shown. 5. Specific implementation methods

[0013] The present principle will be described more fully below with reference to the accompanying drawings, in which examples of the present principle are shown. However, the present principle can be implemented in a variety of alternative forms and should not be interpreted as being limited to the examples set forth herein. Therefore, although the present principle is susceptible to various modifications and alternative forms, its specific examples are shown by the examples in the accompanying drawings and will be described in detail herein. However, it should be understood that there is no intention to limit the present principle to the specific form disclosed, but on the contrary, the present disclosure will cover all modifications, equivalents and alternatives that fall within the spirit and scope of the present principle as defined by the claims.

[0014] The terms used herein are only used for the purpose of describing specific examples and are not intended to be a limitation of this principle. As used herein, the singular forms "one", "an" and "the" are also intended to include plural forms, unless the context clearly indicates otherwise. It will be further understood that when the terms "include", "include", "contain" and / or "contain" are used in this specification, the existence of the features, integers, steps, operations, elements and / or components is specified, but the existence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups is not excluded. In addition, when an element is referred to as "in response to" or "connected to" another element, it can directly respond to or be connected to another element, or there can be an intermediate element. In contrast, when an element is referred to as "directly responding to" or "directly connected to" another element, there is no intermediate element. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items, and can be abbreviated as " / ".

[0015] It will be understood that although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, without departing from the teaching of the present principles, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element.

[0016] Although some of the figures include arrows on communication paths to illustrate a primary direction of communication, it is to be understood that communication can occur in the opposite direction to the depicted arrows.

[0017] Some examples are described with respect to block diagrams and operational flow charts, wherein each block represents a portion of a circuit element, module, or code, which includes one or more executable instructions for implementing a specified (one or more) logical function. It should also be noted that in other implementations, the (one or more) functions indicated in the blocks may not appear in the order indicated. For example, two blocks shown in succession may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order, depending on the functions involved.

[0018] References herein to "according to an example" or "in an example" mean that a particular feature, structure, or characteristic described in conjunction with the example can be included in at least one implementation of the present principles. The phrases "according to an example" or "in an example" appearing in various places in the specification are not necessarily all referring to the same example, nor are they necessarily separate or alternative examples that are mutually exclusive of other examples.

[0019] Reference signs appearing in the claims are by way of illustration only and shall have no limiting effect on the scope of the claims.The present examples and variations may be used in any combination or sub-combination although not explicitly described.

[0020] Figure 1 An example of an extended reality scene description is shown in FIG10. In this example, a scene graph may include descriptions of real objects, such as a "flat horizontal surface" (which may be a table or a road) and descriptions of virtual objects 12, such as an animation of a car. The scene description is organized as an array of nodes 10. Nodes may be linked to child nodes to form a scene structure 11. Nodes may carry descriptions of real objects (e.g., semantic descriptions) or descriptions of virtual objects. Figure 1 In the example of , node 101 describes a virtual camera located in the 3D volume of the XR application. Node 102 describes a virtual car and includes an index to a representation of the car, such as an index in an array of a 3D grid. According to the present principles, a node may also include an index to a representation of the real object. The representation may, for example, have been obtained by scanning a real environment, or may, for example, be a model generated by a content creator. The representation must correspond to the real object described in the node and is therefore located in the 3D scene. In such a scene description, there is no longer a means to distinguish between real nodes and virtual nodes, since both include 3D representations. According to the present principles, information about the properties of the nodes in the scene description is provided to the renderer.

[0021] The scene description may include many arrays containing descriptions of various aspects of the scene, such as an array containing instructions for several animations, an array containing meshes of several virtual or real objects, or an array containing material descriptions. Node 103 is a child of node 102 and includes a description of a wheel of a car. Similarly, it includes an index to the 3D mesh of the wheel. Since the scale, position and orientation of the object are described in the scene node, the same 3D mesh can be used for several objects in the 3D scene. The scene graph 10 also includes nodes for descriptions of the spatial relationships between real objects and virtual objects.

[0022] XR applications are diverse and can be applied to different contexts and real or virtual environments. For example, in an industrial XR application, a virtual 3D content item (e.g., part A of an engine) is displayed when a reference object (part B of an engine) is detected in a real environment by a camera mounted on a head-mounted display device. The 3D content item is positioned in the real world at a position and scale defined relative to the detected reference object. In the same way, when a collision between real or virtual objects is detected (i.e., a collision between two real objects, two virtual objects, or a real object and a virtual object), an action can be performed on the virtual object.

[0023] For example, in an XR application for interior design, the color of a part of a displayed virtual furniture is changed when a user touches a virtual control object or when a user touches a real table. In another application, an audio file may start playing when two moving displayed virtual objects collide. In another example, an advertising song file may play when a user grabs a given can of soda in a real environment. However, detecting collisions (or contacts) between real or virtual objects and rendering the realistic reactions of virtual objects requires the use of a physics engine that is scalable in terms of memory and processing resources. Therefore, it is important to have a scene description that accurately describes different kinds of behaviors associated with collisions in order to optimize the use of the physics engine. According to the present principles, such an XR scene description format is provided herein.

[0024] XR applications can also augment video content instead of the real environment. When a timed event is detected in the video, the video is displayed on the rendering device and the virtual objects described in the node tree are superimposed. In such a context, the node tree only includes virtual object descriptions.

[0025] Real assets, which are representations of real objects, are very useful for improving the XR experience. For example, a scan of a room allows managing collisions between virtual and real objects. This can also be used for rendering purposes, e.g., not rendering real objects in the case of a see-through XR device, or e.g., ensuring consistent lighting between real and virtual objects, including shadow management. This principle provides a solution to indicate the properties of any node of a scene description.

[0026] Figure 4 An example of rendering of an extended reality scene is shown. In the example of FIG. 5 , object 51 is a real table, object 52 is a real speaker, and object 53 is a virtual teddy bear. The scene description generated according to the present principles allows the rendering system to estimate the shadows on the real objects so as to generate consistent shadows for the teddy bear. In such an example, it is key for the rendering system to distinguish between the real and virtual objects of the extended reality scene.

[0027] According to the present principles, a scene description includes information about nodes including representations of real objects in the scene to obtain a rendering close to reality. In a first embodiment, the scene description includes an array of real nodes at the scene level. For example, within the scope of the MPEG-I Scene Description Framework using the Khronos glTF extension mechanism to support additional scene description features, an "MPEG_real_nodes" extension is defined at the scene level. The semantics of the MPEG_real_nodes at the scene level is defined by the following table:

[0028]

[0029] AR scenes with scans of the real environment allow managing collisions between the real and virtual worlds. In AR experiences, meshes and textures of real objects are invisible (the real world is already visible through the AR device), and only mesh colliders are used. In other kinds of applications, such as remote XR experiences, representations of real objects (scanned points of the cloud or 3D models) can be rendered and displayed jointly with virtual objects.

[0030] According to the present principle, the "real node" attribute provides information to the renderer to manage the visibility of real objects. In AR scenes, lighting can be very important. It allows for consistent management of shadows of virtual objects. The output of the lighting shader must be calculated considering different situations. The output of the lighting shader provides the color of the pixel of the final composite image, and its calculation is different depending on whether the pixel belongs to a real object without shadow, a real object in a shadow cast by a real surface, a real object in a shadow cast by a virtual surface, a virtual object without shadow, or a virtual object with shadow.

[0031] To distinguish the cases, a binary flag is provided to the shader, which is set to 0 for vertices of the 3D model corresponding to real objects and to 1 for vertices corresponding to attached virtual objects. Two depth maps are also generated from each light estimation position: one for real objects (i.e., texture meshes) and another for virtual objects. The "real node" attribute provides the renderer with the necessary information so that shadows are not generated for real objects because they already exist.

[0032] Since the concept of "real node" refers to nodes in the scene graph, the concept also refers to light sources. In the AR framework, real light is implemented by virtual light to illuminate virtual objects in order to achieve consistent light rendering. Each light is associated with at least one node. Light inherits the "real node" attribute from the node. This "real node" attribute can be used with respect to possible controls of the light.

[0033] At the rendering side, during runtime, the application iterates over each defined node at the scene level. The information contained in the scene description (collisions, virtual lights) and the information obtained from the application are used in different ways by the renderer. For descriptive purposes, the present invention is described in detail in the context of the MPEG-I Scene Description Framework, which uses the Khronos glTF extension mechanism to support additional scene description features. However, the present principles can be applied to other existing or upcoming descriptions of XR scenes. Figure 1 As illustrated in , each glTF node may include an array called children, which includes the indices of its child nodes. Therefore, each node is an element of a hierarchy of nodes, and together they define the structure of the scene as a scene graph. An example of a possible syntax for such a real node description in an MPEG-I scene description is provided below:

[0034]

[0035]

[0036]

[0037]

[0038] In a second embodiment, the scene description includes a list of virtual objects at the scene level. Thus, by parsing the list, the renderer is informed that the nodes listed with pointers to the 3D model are virtual objects, while the nodes not listed with pointers to the 3D model are real objects. In this second embodiment, the array attribute is called "VirtualNodes". The second embodiment can be combined with the first embodiment. In practice, since the scene graph may include nodes that are not directly related to the objects of the 3D model, it may be valuable for the renderer to have both a list of real objects and a list of virtual objects, allowing the renderer to distinguish object nodes from other types of nodes.

[0039] In a third embodiment, information about the nature of a node is indicated in a node-level attribute. For example, a node-level attribute "isRealObject" may be set to 1 when the associated 3D model corresponds to an object of the real environment of the application, and may be set to 0 when it corresponds to a virtual object. In a variant, a node-level "isVirtual" attribute is set to 1 when the object is virtual, and is set to 0 when the object is real.

[0040] The third embodiment can be combined with the first or second embodiment. In fact, information about the real or virtual nature of a node can be indicated at the scene level and the node level. The renderer can use this duplication information for post-processing operations.

[0041] Figure 2 An example architecture of an XR processing engine 30 that may be configured to implement the present principles is shown. Figure 2 The devices of the architecture are linked to other devices via their buses 31 and / or via I / O interfaces 36 .

[0042] The device 30 comprises the following elements linked together by a data and address bus 31:

[0043] - a microprocessor 32 (or CPU), which is for example a DSP (or digital signal processor);

[0044] - ROM (or read-only memory) 33;

[0045] - RAM (or random access memory) 34;

[0046] - storage interface 35;

[0047] - an I / O interface 36 for receiving data to be transmitted from an application; and

[0048] -power supply( Figure 2 not shown), such as batteries.

[0049] According to the example, the power supply is external to the device. In each of the memories mentioned, the word "register" used in the specification may correspond to an area of ​​small capacity (a few bits) or a very large area (such as an entire program or a large amount of received or decoded data). ROM 33 includes at least programs and parameters. ROM 33 can store algorithms and instructions to perform technology according to the present principles. When turned on, CPU 32 uploads the program to RAM and executes the corresponding instructions.

[0050] RAM 34 includes in registers a program executed by CPU 32 and uploaded after device 30 is turned on, input data in registers, intermediate data in different states of a method in registers, and other variables in registers used to execute the method.

[0051] The implementation described herein can be implemented in, for example, a method or process, a device, a computer program product, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (e.g., discussed only as a method or device), the implementation of the discussed features can also be implemented in other forms (e.g., programs). The device can be implemented in, for example, appropriate hardware, software, and firmware. The method can be implemented in, for example, a device (such as, for example, a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device). The processor also includes a communication device, such as, for example, a computer, a cellular phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate the communication of information between end users.

[0052] The device 30 is linked, for example, via a bus 31, to a set of sensors 37 and a set of rendering devices 38. The sensors 37 may be, for example, cameras, microphones, temperature sensors, inertial measurement units, GPS, humidity sensors, IR or UV light sensors, or wind sensors. The rendering devices 38 may be, for example, displays, speakers, vibrators, heaters, fans, etc.

[0053] According to an example, the device 30 is configured to implement a method according to the present principles and belongs to the set comprising:

[0054] -mobile device;

[0055] -communication equipment;

[0056] -Gaming equipment;

[0057] - Tablet (or tablet computer);

[0058] - Laptop;

[0059] -Still picture camera;

[0060] -Camcorder.

[0061] Figure 3 An example of an embodiment of a syntax for a data stream encoding an extended reality scene description in accordance with the present principles is shown. Figure 3An example structure 4 of an XR scene description is shown. The structure is included in a container that organizes the stream into independent grammatical elements. The structure may include a header portion 41, which is a set of data common to each grammatical element of the stream. For example, the header portion includes some of the metadata about the grammatical elements that describes the properties and role of each of the grammatical elements. The structure also includes a payload that includes grammatical elements 42 and grammatical elements 43. Syntax elements 42 include data representing media content items described in nodes of a scene graph associated with a virtual element. Images, meshes, and other raw data may have been compressed according to a compression method. Syntax elements 43 are part of the payload of the data stream and include data for encoding the scene description as described in accordance with the present principles.

[0062] The implementation described herein can be implemented in, for example, a method or process, a device, a computer program product, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (e.g., discussed only as a method or device), the implementation of the discussed features can also be implemented in other forms (e.g., programs). The device can be implemented in, for example, appropriate hardware, software, and firmware. The method can be implemented in, for example, a device (such as, for example, a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device). The processor also includes a communication device, such as, for example, a smart phone, a tablet, a computer, a mobile phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate the communication of information between end users.

[0063] The implementation of the various processes and features described herein can be implemented in a variety of different equipment or applications (particularly, for example, equipment or applications associated with data encoding, data decoding, view generation, texture processing, and other processing of images and related texture information and / or depth information). Examples of such equipment include encoders, decoders, post-processors that process outputs from decoders, pre-processors that provide inputs to encoders, video encoders, video decoders, video codecs, web servers, set-top boxes, laptop computers, personal computers, cellular phones, PDAs, and other communication devices. As should be clear, the equipment can be mobile and even installed in a mobile vehicle.

[0064] In addition, the method can be implemented by instructions executed by a processor, and such instructions (and / or data values ​​produced by the implementation) can be stored on a processor-readable medium, such as, for example, an integrated circuit, a software carrier, or other storage device, such as, for example, a hard disk, a compact disk ("CD"), an optical disk (such as, for example, a DVD, commonly referred to as a digital versatile disk or digital video disk), a random access memory ("RAM"), or a read-only memory ("ROM"). The instructions can form an application program that is tangibly implemented on a processor-readable medium. The instructions can be, for example, in hardware, firmware, software, or a combination. The instructions can be found in, for example, an operating system, a separate application, or a combination of the two. Therefore, a processor can be characterized as both a device configured to perform a process and a device that includes a processor-readable medium (such as a storage device) having instructions for performing the process. In addition, in addition to or in lieu of instructions, the processor-readable medium can store data values ​​produced by the implementation.

[0065] As will be apparent to one skilled in the art, implementations may generate a variety of signals formatted to carry information that may be stored or transmitted, for example. The information may include, for example, instructions for performing a method, or data generated by one of the described implementations. For example, a signal may be formatted to carry as data the rules for writing or reading the grammar of the described embodiments, or to carry as data the actual grammar values ​​written by the described embodiments. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, the signal may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor-readable medium.

[0066] Many implementations have been described. However, it will be appreciated that various modifications may be made. For example, elements of different implementations may be combined, supplemented, modified, or removed to produce other implementations. In addition, it will be appreciated by those of ordinary skill in the art that other structures and processes may replace those disclosed, and that the resulting implementations will perform at least substantially the same (one or more) functions in at least substantially the same (one or more) manners to achieve at least substantially the same (one or more) results as the disclosed implementations. Therefore, the present application contemplates these and other implementations.

Claims

1. A method for generating a scene description of an extended reality scene, the method comprising: - obtaining a first list of 3D models of objects of the real environment corresponding to the augmented reality scene and a second list of 3D models of virtual objects corresponding to the augmented reality scene; - generating a scene description comprising a scene graph linking nodes and comprising information indicating which nodes correspond to objects of the first list and which nodes correspond to objects of the second list; as well as -Encode the extended reality scene description in the data stream.

2. The method according to claim 1, wherein: The information is an array of indices of nodes corresponding to the objects of the first list stored at the scene level.

3. The method according to claim 1 or 2, wherein: The information is an array of indices of nodes corresponding to objects of the second list stored at the scene level.

4. The method according to any one of claims 1 to 3, wherein: Each node of the scene graph includes information indicating whether the node corresponds to an object of the first list or an object of the second list.

5. A device comprising a memory associated with a processor, the processor being configured to: - obtaining a first list of 3D models of objects of the real environment corresponding to the augmented reality scene and a second list of 3D models of virtual objects corresponding to the augmented reality scene; - generating a scene description comprising a scene graph linking nodes and comprising information indicating which nodes correspond to objects of the first list and which nodes correspond to objects of the second list; as well as -Encode the extended reality scene description in the data stream.

6. The device according to claim 5, wherein: The information is an array of indices of nodes corresponding to the objects of the first list stored at the scene level.

7. The device according to claim 5 or 6, wherein: The information is an array of indices of nodes corresponding to objects of the second list stored at the scene level.

8. The device according to any one of claims 5 to 7, wherein: Each node of the scene graph includes information indicating whether the node corresponds to an object of the first list or an object of the second list.

9. A method for rendering an extended reality scene, the method comprising: -obtaining a scene description of an extended reality scene from the data stream from the data stream; - decoding a scene graph from the scene description, the scene graph linking nodes and comprising information indicating which nodes correspond to objects of the real environment of the augmented reality scene and which nodes correspond to virtual objects of the augmented reality scene; as well as - Processing the nodes of the scene graph according to said information.

10. The method according to claim 9, wherein: The information is an array of indices of nodes corresponding to the objects of the first list stored at the scene level.

11. The method according to claim 9 or 10, wherein: The information is an array of indices of nodes corresponding to objects of the second list stored at the scene level.

12. The method according to any one of claims 9 to 11, wherein: Each node of the scene graph includes information indicating whether the node corresponds to an object of the first list or an object of the second list.

13. An apparatus comprising a memory associated with a processor, the processor being configured to: - Obtain a scene description of the extended reality scene from the data stream; - decoding a scene graph from the scene description, the scene graph linking nodes and comprising information indicating which nodes correspond to objects of the real environment of the augmented reality scene and which nodes correspond to virtual objects of the augmented reality scene; and - Processing the nodes of the scene graph according to said information.

14. The device according to claim 13, wherein: The information is an array of indices of nodes corresponding to the objects of the first list stored at the scene level.

15. The apparatus according to claim 13 or 14, wherein: The information is an array of indices of nodes corresponding to objects of the second list stored at the scene level.

16. The device according to any one of claims 13 to 15, wherein: Each node of the scene graph includes information indicating whether the node corresponds to an object of the first list or an object of the second list.

17. A data stream comprising a scene description of an extended reality scene, the scene description comprising a scene graph, the scene graph linking nodes and including information indicating which nodes correspond to objects of a real environment of the extended reality scene and which nodes correspond to virtual objects of the extended reality scene.

18. The data stream of claim 17, wherein: The information is an array of indices of nodes corresponding to the objects of the first list stored at the scene level.

19. A data stream according to claim 17 or 18, wherein: The information is an array of indices of nodes corresponding to objects of the second list stored at the scene level.

20. The data stream according to any one of claims 17 to 19, wherein: Each node of the scene graph includes information indicating whether the node corresponds to an object of the first list or an object of the second list.

Images