Triggering mechanism in time-evolving scene description

Abstract

Some embodiments of the method may include obtaining scene description data for a 3D scene. The scene description data may include scene element information describing each of a plurality of scene elements in the scene, trigger information describing at least one trigger condition, action information describing an action to perform on one or more scene elements associated with the action, and behavior information. The behavior information associates the at least one trigger condition with the at least one action. Some embodiments of the method may include performing the at least one action on at least a first node associated with the at least one action in response to determining that the activation information indicates that the at least one trigger condition has produced a result that fires the at least one trigger.

Description

[Technical Field]

[0001] Cross-reference to other applications This application claims priority to European Patent Application No. EP22306405.6, filed September 23, 2022, which is incorporated herein by reference in its entirety.

[0002] This application incorporates by reference the following applications in their entireties: European Patent Application No. EP22305024.6, filed January 12, 2022, entitled "METHODS AND DEVICES FOR INTERACTIVE RENDERING OF A TIME-EVOLVING EXTENDED REALITY SCENE" (the "024 Application"); International Application No. PCT / EP2023 / 065281, filed June 7, 2023, entitled "SYSTEMS AND METHODS FOR PROVIDING INTERACTIVITY WITH LIGHT SOURCES IN A SCENE DESCRIPTION" (the "281 Application"); and European Patent Application No. EP22305880.1, filed June 16, 2022, entitled "SYSTEMS AND METHODS FOR PROVIDING INTERACTIVITY WITH LIGHT SOURCES IN A SCENE DESCRIPTION" (the "880 Application"). [Background technology]

[0003] This section is intended to introduce the reader to various aspects of art that may be related to various aspects of the present principles, which are described and / or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of various aspects of the present principles. As such, it should be understood that these statements are to be read in this light, and not as admissions of prior art.

[0004] Augmented reality (XR) is a technology that enables interactive experiences in which real-world environments and / or video content are augmented by virtual content, and can be defined across multiple sensory modalities, including vision, hearing, and touch. During application runtime, virtual content (e.g., 3D content or audio / video files) is rendered in real time to match the user context (environment, viewpoint, device, etc.). Scene graphs (e.g., those proposed by Khronos / glTF and the MPEG Scene Description Format or its extensions defined by Apple / USDZ) are a possible way to represent the content to be rendered. They combine a declarative description of the scene structure linking real-world and virtual objects on the one hand with a binary representation of the virtual content on the other. While such scene description frameworks ensure that timed media and corresponding related virtual content are always available during application rendering, they do not describe how users can interact with scene objects at runtime for immersive XR experiences.

[0005] No XR system is capable of capturing an XR scene description, which contains metadata that describes how a user can interact with scene objects at runtime and how these interactions can be updated during the runtime of an XR application. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] "Information Technology-Coded Representation of Immersive Media-Part14: Scene Description for MPEG Media", ISO / IEC DIS23090-14:2021(E) Summary of the Invention

[0007] The embodiments described herein include methods used for video encoding and decoding (collectively "coding").

[0008] A first exemplary method according to some embodiments includes: obtaining scene description data of a 3D scene, wherein the scene description data may include: scene element information describing each of a plurality of scene elements in the scene; trigger information describing at least one trigger condition; action information describing at least one action to perform on one or more scene elements associated with at least one action; and behavior information, wherein the behavior information includes: a trigger list of at least one trigger; an action list of at least one action; trigger combination information indicating a combination of a first trigger condition corresponding to a first trigger in the trigger list and other trigger conditions corresponding to other triggers in the trigger list; and activation information. and activation information indicating when to perform the at least one action depending on a result of the trigger combination information; and performing the at least one action on one or more scene elements associated with the at least one action in response to a logical determination that (i) a combination of a first trigger condition corresponding to a first trigger in the trigger list and other trigger conditions corresponding to other triggers in the trigger list produces a result, and (ii) the activation information indicates that the result fires the first and other triggers.

[0009] In some embodiments of the first exemplary method, the trigger list may include one trigger, and there are no other triggers in the trigger list, the trigger combination information may include a unary operator that operates on the one trigger, and the activation information may indicate when to execute the unary operator on the one trigger.

[0010] In some embodiments of the first exemplary method, the trigger list may include at least two triggers, the trigger combination information may indicate how the at least two trigger conditions should be combined, and the activation information may indicate when to perform at least one action depending on the result of the trigger combination information.

[0011] In some embodiments of the first exemplary method, performing the first action may include performing the first action one or more times based on the activation information.

[0012] In some embodiments of the first exemplary method, determining a combination of the trigger conditions of at least one trigger in the trigger list with the trigger conditions of other triggers in the trigger list may include performing a logical operation on at least one of the trigger conditions.

[0013] In some embodiments of the first exemplary method, determining a combination of the trigger conditions of at least one trigger in the trigger list with the trigger conditions of other triggers in the trigger list may include performing a logical OR operation of at least two trigger conditions.

[0014] In some embodiments of the first exemplary method, at least a first of the trigger conditions is a visibility condition that is satisfied when a specified scene element is visible to a specified camera node.

[0015] In some embodiments of the first exemplary method, at least a first of the trigger conditions is a proximity condition that is met when the distance from the user camera to a specified scene element is within a specified range.

[0016] In some embodiments of the first exemplary method, at least a first of the trigger conditions is a user input condition that is satisfied when a specified user interaction is detected.

[0017] In some embodiments of the first exemplary method, at least a first of the trigger conditions is a timed condition that is satisfied during a specified period of time.

[0018] In some embodiments of the first exemplary method, at least a first of the trigger conditions is a collider condition that is satisfied in response to detecting a collision between designated scene elements.

[0019] In some embodiments of the first exemplary method, the action information may describe at least two actions, and the behavior information may include, for at least one behavior, information indicating an order in which at least two of the at least two actions should be performed.

[0020] In some embodiments of the first exemplary method, the action information may describe at least two actions, and the behavior information may include information indicating that at least two of the at least two actions are to be performed simultaneously.

[0021] In some embodiments of the first exemplary method, at least one of the scene elements in the scene is a virtual object.

[0022] Some embodiments of the first exemplary method may further include rendering the 3D scene according to the scene description data computed by the first action.

[0023] In some embodiments of the first exemplary method, the trigger information may include an array of two or more triggers in the 3D scene.

[0024] In some embodiments of the first exemplary method, the action information may include an array of two or more actions in the 3D scene.

[0025] In some embodiments of the first exemplary method, the behavior information may include an array of two or more behaviors in the 3D scene.

[0026] In some embodiments of the first exemplary method, the scene description data may be provided in JSON format.

[0027] In some embodiments of the first exemplary method, the scene description data may be provided in GLTF format.

[0028] A first exemplary method / apparatus according to some embodiments may include a processor and a non-transitory computer-readable medium storing instructions that, when executed by the processor, function to cause the apparatus to perform any one of the methods set forth above.

[0029] A second exemplary method / apparatus according to some embodiments may include: acquiring scene description data of a 3D scene, where the scene description data may include: scene element information describing each of a plurality of scene elements in the scene, trigger information describing at least one trigger condition, action information describing at least one action to perform on one or more scene elements associated with the at least one action, and behavioral information associating at least one of the trigger conditions with at least one of the actions; and performing a first action on at least a first node associated with the first action in response to a logical determination that (i) at least one combination of the trigger conditions is satisfied, and (ii) a first action of the at least one action is associated with the combination of trigger conditions by the behavioral information.

[0030] In some embodiments of the second exemplary method, the logical determining may further include determining that a trigger associated with the at least one trigger condition has been activated.

[0031] In some embodiments of the second exemplary method, the logical determination may further include determining an activation state of the combination of the at least one trigger condition, and performing the first action may include performing the first action one or more times based on the activation state.

[0032] In some embodiments of the second exemplary method, determining the combination of the at least one trigger condition may include performing a logical operation on at least one of the trigger conditions.

[0033] In some embodiments of the second exemplary method, determining the combination of the at least one trigger condition may include performing a logical OR operation on the at least two trigger conditions.

[0034] In some embodiments of the second exemplary method, at least a first of the at least one trigger condition is a visibility condition that is satisfied when a specified scene element is visible to a specified camera node.

[0035] In some embodiments of the second exemplary method, at least a first of the at least one trigger condition is a proximity condition that is satisfied when the distance from the user camera to a specified scene element is within a specified range.

[0036] In some embodiments of the second exemplary method, at least a first of the at least one trigger condition is a user input condition that is satisfied when a specified user interaction is detected.

[0037] In some embodiments of the second exemplary method, at least a first of the at least one trigger condition is a timed condition that is satisfied during a specified period of time.

[0038] In some embodiments of the second exemplary method, at least a first of the at least one trigger condition is a collider condition that is satisfied in response to detecting a collision between designated scene elements.

[0039] In some embodiments of the second exemplary method, the behavioral information may identify at least one behavior, and the behavioral information for each of the at least one behavior identifies a trigger associated with one of the at least one trigger condition and one of the at least one action.

[0040] In some embodiments of the second exemplary method, the action information may describe at least two actions, and the behavior information may include, for at least one behavior, information indicating an order in which at least two of the at least two actions should be performed.

[0041] In some embodiments of the second exemplary method, the action information may describe at least two actions, and the behavior information may include information indicating that at least two of the at least two actions are to be performed simultaneously.

[0042] In some embodiments of the second exemplary method, at least one of the scene elements in the scene is a virtual object.

[0043] Some embodiments of the second exemplary method may further include rendering the 3D scene according to the scene description data computed by the first action.

[0044] In some embodiments of the second exemplary method, the trigger information may include an array of two or more triggers in the 3D scene.

[0045] In some embodiments of the second exemplary method, the action information may include an array of two or more actions in the 3D scene.

[0046] In some embodiments of the second exemplary method, the behavior information may include an array of two or more behaviors in the 3D scene.

[0047] In some embodiments of the second exemplary method, the scene description data may be provided in JSON format.

[0048] In some embodiments of the second exemplary method, the scene description data may be provided in GLTF format.

[0049] A second exemplary method / apparatus according to some embodiments may include a processor and a non-transitory computer-readable medium storing instructions that, when executed by the processor, function to cause the apparatus to perform any one of the methods listed above.

[0050] A third exemplary method for rendering an augmented reality scene for a user in a timed environment according to some embodiments may include steps of obtaining a description of the augmented reality scene, the description including: a scene tree connecting nodes describing timed objects, virtual objects, or relationships between objects; and behavior data items, the behavior data items including: at least trigger control parameters, where the trigger control parameters are descriptions of conditions associated with one or more triggers; an activate condition associated with the trigger control parameters; and at least actions, where the action is a description of processing to be performed by the augmented reality engine on the objects described by the nodes of the scene tree; and steps of applying the actions of the behavior data items to the associated objects, conditional on at least one logical combination of the triggers of the behavior data items being triggered and the activate condition associated with the trigger control parameters of the behavior data items being satisfied.

[0051] In some embodiments of the third exemplary method, the logical combination of the at least one trigger may include a logical operation on at least one of the conditions associated with the one or more triggers.

[0052] In some embodiments of the third exemplary method, the logical combination of the at least one trigger may include a logical OR operation of at least two conditions associated with the one or more triggers.

[0053] A third example method / apparatus according to some embodiments may include a processor and a non-transitory computer-readable medium storing instructions that, when executed by the processor, function to cause the apparatus to perform any one of the methods listed above.

[0054] A fourth exemplary method, according to some embodiments, for updating a first description of an augmented reality scene including behavior data items at runtime with a second description of the augmented reality scene may include, for each ongoing behavior data item of the first description, if the ongoing behavior data item is not applicable to the second description: processing an interrupt action if one exists for the ongoing application of the first description; stopping the ongoing behavior; and applying the second description.

[0055] A fifth exemplary apparatus, according to some embodiments, that is a device for rendering an augmented reality scene for a user in a timed environment, may include a processor configured to obtain a description of the augmented reality scene, the description including: a scene tree connecting nodes that describe timed objects, virtual objects, or relationships between objects; and behavioral data items, which may include: at least triggers that are descriptions of conditions; triggers that are activated when the conditions are detected in the timed environment; and at least actions that are descriptions of processing to be performed by the augmented reality engine on the objects described by the nodes of the scene tree; and further include memory associated with the processor configured to apply the behavioral actions to the associated objects, conditional on the trigger of the behavioral data item being activated.

[0056] In some embodiments of the fifth exemplary apparatus, the processor is further configured: to attribute an activation state set to False to at least one description trigger when a description of the augmented reality scene is obtained; to set the activation state of the trigger to True when a condition of the at least one trigger is met for the first time; and to activate the trigger when a condition of the at least one trigger is met.

[0057] In some embodiments of the fifth exemplary apparatus, the processor is further configured to, when a condition of at least one trigger is satisfied, if an activation state of the trigger is set to true, activate the trigger only if a description of the trigger allows a second activation.

[0058] A sixth exemplary apparatus, according to some embodiments, that is a device for runtime updating a first description of an augmented reality scene including behavioral data items with a second description of the augmented reality scene may include memory associated with a processor configured to, for each ongoing behavioral data item of the first description, if the ongoing behavioral data item is not applicable to the second description: process an interrupt action if present for the ongoing application of the first description; stop the ongoing behavior; and apply the second description.

[0059] A seventh exemplary apparatus according to some embodiments may include one or more processors configured to perform any one of the methods listed above.

[0060] An eighth exemplary apparatus according to some embodiments may include a computer-readable medium including instructions for causing one or more processors to perform any one of the methods listed above.

[0061] In some embodiments of the eighth exemplary apparatus, the computer-readable medium is a non-transitory storage medium.

[0062] A tenth exemplary apparatus according to some embodiments may include a computer program product including instructions that, when executed by one or more processors, cause the one or more processors to perform any one of the methods listed above.

[0063] An eleventh exemplary apparatus according to some embodiments may include a signal including scene description data of a 3D scene, wherein the scene description data may include: scene element information describing each of a plurality of scene elements in the scene; trigger information describing at least one trigger condition; action information describing at least one action to perform on one or more scene elements associated with the action; and behavior information, wherein the behavior information associates at least one of the trigger conditions with at least one of the actions.

[0064] A twelfth exemplary method / apparatus according to some embodiments may include a computer-readable medium including scene description data for a 3D scene, wherein the scene description data may include: scene element information describing each of a plurality of scene elements in the scene; trigger information describing at least one trigger condition; action information describing at least one action to perform on one or more scene elements associated with the action; and behavior information, wherein the behavior information associates at least one of the trigger conditions with at least one of the actions.

[0065] In further embodiments, encoder and decoder devices are provided for performing the methods described herein. The encoder or decoder devices may include a processor configured to perform the methods described herein. The devices may include a computer-readable medium (e.g., non-transitory medium) storing instructions for performing the methods described herein. In some embodiments, the computer-readable medium (e.g., non-transitory medium) stores video encoded using any of the methods described herein.

[0066] One or more of these embodiments also provide a computer-readable storage medium having stored thereon instructions for performing bidirectional optical flow and encoding or decoding video data according to any of the methods described above. The embodiments also provide a computer-readable storage medium having stored thereon a bitstream generated by the methods described above. The embodiments also provide a method and apparatus for transmitting a bitstream generated by the methods described above. The embodiments also provide a computer program product including instructions for performing any of the described methods. [Brief explanation of the drawings]

[0067] [Figure 1A] 1 is a schematic side view illustrating an exemplary waveguide display that may be used with extended reality (XR) applications, according to some embodiments. [Figure 1B] 1A-1C are schematic side views illustrating exemplary alternative display types that may be used with augmented reality applications, according to some embodiments. [Figure 1C] 1A-1C are schematic side views illustrating exemplary alternative display types that may be used with augmented reality applications, according to some embodiments. [Figure 1D] FIG. 1 is a system diagram illustrating an example set of interfaces for a system, according to some embodiments. [Figure 1E] FIG. 1 is a system diagram illustrating an exemplary set of interfaces for scene description, according to some embodiments. [Figure 2] FIG. 2 is a system diagram illustrating an exemplary set of interfaces in an MPEG-I node hierarchy that support elements of scene interactivity, according to some embodiments. [Figure 3] FIG. 1 is a block diagram illustrating an example of logical relationships between trigger information (describing triggers 1 through n), action information (describing actions 1 through m), and behavior information (describing the relationship between triggers and actions), according to some embodiments, where triggers and actions may reference one or more nodes in a scene description, such as a hierarchical scene graph. [Figure 4] 1 is a schematic plan view illustrating exemplary relationships of augmented reality scene description objects, according to some embodiments. [Figure 5A] FIG. 1 is a diagram of a data structure illustrating an example set of triggers for an augmented reality scene description, according to some embodiments. [Figure 5B] FIG. 1 is a diagram of a data structure illustrating an example set of actions for an augmented reality scene description, according to some embodiments. [Figure 5C] FIG. 1 is a diagram of a data structure illustrating an example set of behaviors for an augmented reality scene description, according to some embodiments. [Figure 5D] FIG. 1 is a diagram of a data structure illustrating an exemplary set of complementary information for an augmented reality scene description, according to some embodiments. [Figure 6] FIG. 1 is a data syntax diagram illustrating an example syntax of a data stream encoding an extended reality (XR) scene description, according to some embodiments. [Figure 7] 1 is a flowchart illustrating an exemplary process, according to some embodiments. [Figure 8] 10 is a flowchart illustrating an example process for activating a trigger or combination of triggers, according to some embodiments. [Figure 9] 10 is a flowchart illustrating an exemplary process for rendering an augmented reality scene of an updated trigger mechanism, according to some embodiments. [Figure 10] 1 is a flowchart illustrating an exemplary process for trigger activation in a time-evolving scene description, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0068] The entities, connections, configurations, etc. depicted in and described in connection with the various figures are presented by way of example and not limitation. As such, any and all statements or other indications regarding what a particular figure "depicts," or what a particular element or entity of a particular figure "is" or "has," and any and all similar statements (which may be read in isolation and out of context as absolute and therefore limiting) may properly be read only when structurally preceded by a phrase such as "in at least one embodiment...." For brevity and clarity of presentation, this implied precedent phrase will not be repeated at length in the Detailed Description.

[0069] 1A is a schematic side view illustrating an exemplary waveguide display that may be used with extended reality (XR) applications, according to some embodiments. An image is projected by an image generator 102. The image generator 102 may use one or more of a variety of techniques for projecting images. For example, the image generator 102 may be a laser beam scanning (LBS) projector, a liquid crystal display (LCD), a light emitting diode (LED) display (including an organic LED (OLED) or micro LED (μLED) display), a digital light processor (DLP), a liquid crystal on silicon (LCoS) display, or other type of image generator or light engine.

[0070] Light representing an image 112 generated by the image generator 102 is coupled into the waveguide 104 by a diffractive incoupler 106. The incoupler 106 diffracts the light representing the image 112 into one or more diffraction orders. For example, one of the rays representing the lower portion of the image, ray 108, is diffracted by the incoupler 106, and one of the diffraction orders 110 (e.g., the second order) is at an angle that allows it to propagate through the waveguide 104 by total internal reflection. The image generator 102 displays the image when instructed by a control module 124, which operates to render image data, video data, point cloud data, or other displayable data.

[0071] At least a portion of the light 110 coupled into the waveguide 104 by the diffractive in-coupler 106 is coupled out of the waveguide by the diffractive out-coupler 114. At least a portion of the light coupled out of the waveguide 104 replicates the angle of incidence of the light that was coupled into the waveguide. For example, in the illustration, out-coupled light rays 116a, 116b, and 116c replicate the angle of in-coupled light ray 108. Because the light exiting the out-coupler replicates the direction of the light that entered the in-coupler, the waveguide substantially replicates the original image 112. A user's eye 118 can focus on the replicated image.

[0072] In the example of FIG. 1A, outcoupler 114 couples out only a portion of the light with each reflection, allowing a single input beam (such as beam 108) to generate multiple parallel output beams (such as beams 116a, 116b, and 116c). In this way, at least some of the light from each portion of the image can reach the user's eye, even if the eye is not perfectly aligned with the center of the outcoupler. For example, if eye 118 moves downward, beam 116c can enter the eye even though beams 116a and 116b do not, allowing the user to perceive the bottom of image 112 regardless of the change in position. Thus, outcoupler 114 acts, in part, as a vertical exit pupil expander. The waveguide may also include one or more additional exit pupil expanders (not shown in FIG. 1A) that expand the exit pupil horizontally.

[0073] In some embodiments, the waveguide 104 is at least partially transparent to light originating outside the waveguide display. For example, at least a portion of the light 120 from a real-world object (such as object 122) traverses the waveguide 104, allowing a user to view the real-world object while using the waveguide display. Because the light 120 from the real-world object also passes through the diffraction grating 114, there are multiple diffraction orders and therefore multiple images. To minimize the visibility of multiple images, order zero (no deflection by the diffraction grating 114) is desirable to obtain a high diffraction efficiency and zero order of the light 120, while higher diffraction orders have lower energy. Therefore, in addition to magnifying and out-coupling the virtual image, the out-coupler 114 is preferably configured to pass the zero-order real image. In such an embodiment, the image displayed by the waveguide display may appear superimposed on the real world.

[0074] 1B is a schematic side view illustrating an exemplary alternative display type that may be used with an augmented reality application, according to some embodiments. In an XR head-mounted display device 130, a control module 132 controls a display 134, which may be an LCD, to display images. The head-mounted display includes a partially reflective surface 136 that reflects (and, in some embodiments, both reflects and focuses) the image displayed on the LCD so that it is visible to the user. The partially reflective surface 136 also allows at least some external light to pass through, allowing the user to see their surroundings.

[0075] 1C is a schematic side view illustrating an exemplary alternative display type that may be used with an augmented reality application, according to some embodiments. In an XR head-mounted display device 140, a control module 142 controls a display 144, which may be an LCD, to display an image. The image is focused by one or more lenses 146 of the display optics so that the image is visible to the user. In the example of FIG. 1C, external light does not directly reach the user's eyes. However, in some such embodiments, an external camera 148 may be used to capture images of the external environment and display such images on the display 144 along with any virtual content that may also be displayed.

[0076] The embodiments described herein are not limited to any particular type or configuration of XR display device.

[0077] FIG. 1D is a system diagram illustrating an example set of system interfaces, according to some embodiments. An augmented reality display device, together with its control electronics, may be implemented using a system such as the system of FIG. 1D. System 150 may be embodied as a device including various components described below and configured to perform one or more of the aspects described herein. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 150, singly or in combination, may be embodied as a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of system 150 are distributed across multiple ICs and / or discrete components. In various embodiments, system 150 is communicatively coupled to one or more other systems or other electronic devices, for example, via a communication bus or through dedicated input and / or output ports. In various embodiments, the system 150 is configured to implement one or more of the aspects described herein.

[0078] System 150 includes at least one processor 152 configured to execute loaded instructions to implement various aspects described herein, for example. Processor 152 may include embedded memory, input / output interfaces, and various other circuitry known in the art. System 150 includes at least one memory 154 (e.g., a volatile memory device and / or a non-volatile memory device). System 150 may include a storage device 158, which may include non-volatile and / or volatile memory, including, but not limited to, electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash, magnetic disk drives, and / or optical disk drives. Storage device 158 may include, by way of non-limiting example, an internal storage device, an attached storage device (including removable and non-removable storage devices), and / or a network-accessible storage device.

[0079] System 150 includes encoder / decoder module 156, which may include its own processor and memory, configured to process data to provide, for example, encoded or decoded video. Encoder / decoder module 156 represents a module that may be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of an encoding module and a decoding module. Additionally, encoder / decoder module 156 may be implemented as a separate element of system 150 or may be incorporated within processor 152 as a combination of hardware and software, as is known to those skilled in the art.

[0080] Program code to be loaded into processor 152 or encoder / decoder 156 to perform various aspects described herein may be stored in storage device 158 and then loaded into memory 154 for execution by processor 152. According to various embodiments, one or more of processor 152, memory 154, storage device 158, and encoder / decoder module 156 may store one or more of various items during performance of the processes described herein. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from processing of equations, formulas, operations, and arithmetic logic.

[0081] In some embodiments, memory internal to the processor 152 and / or the encoder / decoder module 156 is used to store instructions and to provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device can be either the processor 152 or the encoder / decoder module 156) is used for one or more of these functions. The external memory can be the memory 154 and / or the storage device 158 (e.g., volatile dynamic memory and / or non-volatile flash memory). In some embodiments, the external non-volatile flash memory is used to store, for example, the television's operating system. In at least one embodiment, a high-speed external volatile dynamic memory such as RAM is used as working memory for video coding and decoding operations, such as for MPEG-2 (MPEG refers to the Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard being developed by JVET (Joint Video Experts Team)).

[0082] Inputs to the elements of system 150 can be provided via various input devices, shown in block 172. Such input devices include, but are not limited to, (i) a radio frequency (RF) section for receiving RF signals transmitted over the air, for example by a broadcast station, (ii) a component (COMP) input terminal (or set of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high-definition multimedia interface (HDMI®) input terminal. Other examples not shown in FIG. 1C include composite video.

[0083] In various embodiments, the input devices of block 172 have associated input processing elements known in the art. For example, the RF section may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal or bandlimiting a signal to a frequency band), (ii) downconverting the selected signal, (iii) bandlimiting again to a narrower frequency band to select a signal frequency band (e.g., which in some embodiments may be referred to as a channel), (iv) demodulating the downconverted and bandlimited signal, (v) performing error correction, and (vi) demultiplexing to select a desired stream of data packets. The RF section of various embodiments includes one or more elements for performing these functions, such as a frequency selector, a signal selector, a bandlimiter, a channel selector, a filter, a downconverter, a demodulator, an error corrector, and a demultiplexer. The RF section may include, for example, a tuner for performing various of these functions, including downconverting a received signal to a lower frequency (e.g., an intermediate frequency or a frequency near baseband) or to baseband. In one set-top box embodiment, the RF section and its associated input processing elements receive RF signals transmitted over a wired (e.g., cable) medium and perform frequency selection by filtering, downconverting, and filtering again to a desired frequency band. Various embodiments rearrange the order of the above (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, such as inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0084] Additionally, the USB and / or HDMI terminals may include respective interface processors for connecting system 150 to other electronic devices over the USB and / or HDMI connections. It should be understood that various aspects of the input processing, e.g., Reed-Solomon error correction, may be implemented, for example, in a separate input processing IC or within processor 152, as desired. Similarly, aspects of the USB or HDMI interface processing may be implemented, for example, in a separate interface IC or within processor 152, as desired. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 152 and encoder / decoder 156, which operate in combination with memory and storage elements to process the data stream as desired for presentation to an output device.

[0085] The various elements of system 150 may be provided within a unified housing in which the various elements are interconnected and data may be transmitted therebetween using suitable connection structures 174 (e.g., internal buses known to those skilled in the art, including Inter-IC (I2C) buses, wiring, and printed circuit boards).

[0086] System 150 includes a communication interface 160 that enables communication with other devices over a communication channel 162. Communication interface 160 may include, but is not limited to, a transceiver configured to transmit and receive data over communication channel 162. Communication interface 160 may include, but is not limited to, a modem or a network card, and communication channel 162 may be implemented in wired and / or wireless media, for example.

[0087] In various embodiments, data is streamed or otherwise provided to system 150 using a wireless network such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signal in these embodiments is received via communication channel 162 and communication interface 160 adapted for Wi-Fi communication. Communication channel 162 in these embodiments is typically connected to an access point or router, which provides access to external networks, including the Internet, to enable streaming applications and other over-the-top communications. In other embodiments, a set-top box delivering data via an HDMI connection in input block 172 is used to provide streamed data to system 150. In yet other embodiments, an RF connection in input block 172 is used to provide streamed data to system 150. As noted above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, such as a cellular network or a Bluetooth network.

[0088] System 150 can provide output signals to various output devices, including a display 176, speakers 178, and other peripheral devices 180. Display 176 in various embodiments includes, for example, one or more of a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. Display 176 may be for a television, a tablet, a laptop, a cell phone (mobile phone), or other device. Display 176 can also be integrated with other components (e.g., as in a smartphone) or separate (e.g., an external monitor for a laptop). Other peripheral devices 180, in various example embodiments, include one or more of a standalone digital video disc (or digital versatile disc) (DVD, either term), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 180 that function based on the output of system 150. For example, a disc player performs the function of playing the output of system 150.

[0089] In various embodiments, control signals are communicated between system 150 and display 176, speaker 178, or other peripheral device 180 using signaling such as AV.Link, Consumer Electronic Control (CEC), or other communication protocols that enable inter-device control with or without user intervention. Output devices can be communicatively coupled to system 150 via dedicated connections through respective interfaces 164, 166, 168. Alternatively, output devices can be connected to system 150 using communication channel 162 via communication interface 160. Display 176 and speaker 178 can be integrated into a single unit with other components of system 150, for example, within an electronic device such as a television. In various embodiments, display interface 164 includes a display driver, for example, a timing controller (T Con) chip.

[0090] Display 176 and speakers 178 may alternatively be separate from one or more of the other components, for example, if the RF portion of input 172 is part of a separate set-top box. In various embodiments in which display 176 and speakers 178 are external components, the output signals may be provided via dedicated output connections including, for example, an HDMI port, a USB port, or a COMP output.

[0091] System 150 may include one or more sensor devices 170. Examples of sensor devices that may be used include one or more GPS sensors, gyro sensors, accelerometers, light sensors, cameras, depth cameras, microphones, and / or magnetometers. Such sensors may be used to determine information such as a user's position and orientation. When system 150 is used as a control module for an augmented reality display (e.g., control modules 124, 132), the user's position and orientation may be used to determine how to render image data so that the user perceives the correct portion of a virtual object or scene from the correct viewpoint. In the case of a head-mounted display device, the position and orientation of the device itself may be used to determine the user's position and orientation for purposes of rendering virtual content. In the case of other display devices, such as a phone, tablet, computer monitor, or television, other inputs may be used to determine the user's position and orientation for purposes of rendering content. For example, a user may select and / or adjust a desired viewpoint and / or gaze direction using a touchscreen, keypad or keyboard, trackball, joystick, or other input. If the display device has sensors such as an accelerometer and / or a gyroscope, the viewpoint and orientation used to render the content may be selected and / or adjusted based on movement of the display device.

[0092] The embodiments may be performed by computer software implemented by the processor 152, by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The memory 154 may be of any type appropriate to the technology environment and may be implemented using any suitable data storage technology, such as, by way of non-limiting examples, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. The processor 152 may be of any type appropriate to the technology environment and may include, by way of non-limiting examples, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0093] Scene description framework for XR The present principles generally relate to the domain of rendering of augmented reality scene descriptions and augmented reality rendering, and this specification is also understood in the context of formatting and playback of augmented reality applications when rendered on an end-user device, such as a mobile device or a head-mounted display (HMD).

[0094] In XR applications, scene descriptions are used to combine an explicit, easy-to-parse description of the scene structure with some binary representation of the media content.

[0095] In time-based media streaming, the scene description itself can be time-evolving to provide relevant virtual content for each sequence of the media stream, for example, for advertising purposes a virtual bottle can be displayed during a video sequence in which people are drinking.

[0096] This kind of behavior can be achieved by relying on the framework defined in the MPEG media document scene description, Information technology - Coding representation of immersive media - Part 14: Scene Description for MPEG media, ISO / IEC DIS23090-14:2021(E). A scene update mechanism based on the JSON Patch protocol defined in IETF RFC6902 can be used to synchronize the virtual content with the MPEG media stream.

[0097] FIG. 1E is a system diagram illustrating an example set of interfaces for a scene description, according to some embodiments. In FIG. 1E, the scene description is stored as an item in glTF.json 194. The example scene description has three video tracks 182, 183, and 184, an audio track 185, and a JSON patch update track 186 in an ISOBMFF file structure 181, according to some embodiments. The example in FIG. 1E shows video samples 187, 188, 189, and 190 in the video tracks 182, 183, and 184 of the example structure. The example JSON patch update track 186 has multiple sample update patches 191, 192, and 193. The example ISOBMFF file structure 181 also shows a gltf buffers.bin 195 connected to the item gltf.json 194.

[0098] While the MPEG-I scene description framework ensures that timed media and corresponding associated virtual content are always available, it does not describe how users can interact with scene objects at runtime for immersive XR experiences, and therefore lacks support for user-specific XR experiences for consuming immersive media.

[0099] The example embodiments described herein can be used to provide a scene description that includes virtual objects or light sources, but does not necessarily display or render the virtual objects or light sources even if they are available. In some embodiments, one or more of the following aspects may be considered when determining whether to display a virtual object or light source:

[0100] Spatial aspects may be taken into account when determining whether to display a virtual object or light source. For example, a virtual object or light source may not be displayed if the user environment is not suitable (e.g., the user is too far away from the location of the rendered timed media), or if the user is not looking in the correct direction, or if the virtual object should be displayed in a user-specific area (e.g., above the user's left hand, which has not yet been detected).

[0101] Temporal aspects may be taken into account when deciding whether to display a virtual object or light source: for example, if the user is not yet ready or wants to trigger the display of the object themselves (e.g., with a particular gesture), the virtual object or light source may not be displayed until an appropriate trigger is detected.

[0102] In some embodiments, the scene description specifies which objects or light sources the user is allowed to manipulate or interact with through possible haptic feedback.

[0103] Runtime Interactivity Figure 2 is a system diagram illustrating an exemplary set of interfaces in an MPEG-I node hierarchy that supports elements of scene interactivity, according to some embodiments. Figure 2 shows an exemplary MPEG-I node hierarchy 200. In accordance with the present principles, in addition to the node tree described in connection with Figure 3, behavioral metadata items (referred to herein as "behaviors") are added to the scene description. In an exemplary embodiment, the time-evolving scene description is augmented by adding information that specifies behaviors. These behaviors may be associated with pre-defined virtual objects that allow runtime interactivity for a user-specific XR experience.

[0104] In some embodiments, these behaviors are time-evolving. In such embodiments, the behaviors can be updated through existing scene description update mechanisms.

[0105] In an exemplary embodiment, the behavior is characterized by one or more of the following properties: One or more triggers that define the conditions that must be met for the trigger to fire Trigger control parameters that define the logical operations between defined triggers The action to be performed in response to the firing of the trigger Action control parameters that define the execution order of defined actions A priority number that allows you to select the behavior with the highest priority when several behaviors for the same virtual object occur simultaneously. Optional interrupt actions to specify how to terminate this behavior when it is no longer defined in a newly received scene update, for example, if the associated object has been removed or if the behavior is no longer relevant to this current media (e.g., audio or video) sequence.

[0106] By adding these behaviors, time-dependent user interactivity with immersive content for XR experiences can be defined.

[0107] When the second scene description is received, some of the behaviors in the first scene description may be "in progress." That is, they have been triggered and their actions are being executed. The second scene description may be provided as update metadata. That is, the metadata describes the differences between the first and second scene descriptions. The second scene description includes a node tree that describes objects that may be common or different to objects in the first scene description. Objects in the node tree of the first scene description may no longer exist in the second description. If objects related to the execution actions of ongoing behaviors no longer exist in the second scene description, these ongoing behaviors no longer apply. Similarly, if an ongoing behavior is not defined in the second description, the ongoing behavior no longer applies. The interrupt action field describes how to gracefully interrupt an ongoing action for an ongoing behavior.

[0108] FIG. 3 is a block diagram illustrating an example of logical relationships between trigger information (describing triggers 1 to n), action information (describing actions 1 to m), and behavior information (describing the relationship between triggers and actions), according to some embodiments, where triggers and actions may reference one or more nodes in a scene description, such as a hierarchical scene graph.

[0109] In XR applications, scene descriptions are used to combine an explicit, easy-to-parse description of the scene structure with some binary representation of the media content. The above section describes an action mechanism for the scene description. These behaviors are associated with pre-defined virtual objects that allow runtime interactivity into a user-specific XR experience. Figure 3 shows an example behavior mechanism structure 300. The example structure 300 in Figure 3 shows an example trigger structure 304 and an example action structure 306 within an example behavior structure 302. Figure 3 also shows an example node 308 with connections to specific example triggers and actions.

[0110] 4 is a schematic plan view illustrating exemplary relationships of augmented reality scene description objects, according to some embodiments. In this example, scene graph 400 includes descriptions of real objects 412, e.g., a "flat horizontal surface" (which could be a table or floor or dish), and virtual objects 414, e.g., a description of an animation of a walking character. Scene graph nodes 414 are associated with media content items 416, which are encodings of data used to render and display the walking character (e.g., as a textured animated 3D mesh). Scene graph 400 also includes nodes 410, which are descriptions of spatial relationships between the real objects described by nodes 412 and the virtual objects described by nodes 414. In this example, nodes 410 describe the spatial relationship for having the character walk across the plane. When an XR application is started, media content item 416 is loaded, rendered, and buffered to be displayed when triggered. When a plane is detected in the real environment by a sensor (or, in some embodiments, a camera), the application displays the buffered media content item as described in node 410. The timing is managed by the application according to the features detected in the real world and the timing of the animation. A node in the scene graph may also contain no description and only act as a parent to child nodes.

[0111] XR applications are diverse and can be applied to different contexts and real or virtual environments. For example, in an industrial XR application, a virtual 3D content item (e.g., piece A of an engine) is displayed when a reference object (piece B of the engine) is detected in the real world by a camera attached to a head-mounted display device. The 3D content item is placed in the real world at a position and scale defined relative to the detected reference object.

[0112] For example, in an XR application for interior design, when a given image from a catalog is detected in the input camera view, a 3D model of furniture is displayed. The 3D content is placed in the real world at a defined position and scale relative to the detected reference image. In another application, an audio file may begin playing when a user enters an area near a church (real or virtually rendered in an augmented real environment). In another example, an advertising jingle file may play when a user sees a certain soda can in the real environment. In an outdoor gaming application, various virtual characters may appear depending on the semantics of the scene observed by the user. For example, a bird character may be matched to a tree, so if the XR device's sensors detect a real object described with the semantic label "tree," a bird flying around the tree can be added. In a companion application implemented by smart glasses, when a car is detected in the user's camera's field of view, a car noise may be emitted to the user's headset to alert the user to potential danger. Furthermore, the sound may be spatially expanded to sound as if it is coming from the direction the car was detected.

[0113] XR applications may also augment video content rather than the real environment. The video is displayed on a rendering device, and when timed events are detected in the video, virtual objects described in a node tree are overlaid. In such a context, the node tree contains only virtual object descriptions.

[0114] The exemplary embodiments are described within the MPEG-I scene description framework, using the Khronos glTF extension mechanism to support additional scene description features such as node trees, but the principles described herein are not limited to any particular scene description framework.

[0115] In an exemplary embodiment, the glTF scene description is extended to support interactivity. The interactivity extension is applied at the glTF scene level and is called MPEG_scene_interactivity. The corresponding semantics are presented in Table 1.

[0116] [Table 1]

[0117] In Table 1 and other semantic tables described herein, the "usage" column indicates a "M" for a "required" feature and an "O" for an "optional" feature. However, such features may be "required" or "optional" only depending on the particular proposed syntax. A feature marked "required" is not necessarily a feature required to implement an application. For example, in some embodiments, a feature marked "required" is present to meet the expectations of a particular type of parsing and rendering software, while in other embodiments, the feature may be optional or the feature may be omitted entirely, with the corresponding feature being implemented using default values ​​or not implemented at all without departing from the scope of this disclosure.

[0118] 5A-5D show non-limiting examples of augmented reality scene descriptions according to some embodiments.

[0119] In the first example presented in Figures 5A-5D, a virtual 3D object is continuously displayed and transformed during the media sequence. When the user's left hand is detected, the virtual 3D object is placed at the user's left hand and continues to follow it.

[0120] As an example of an interactive virtual object, according to some embodiments, a virtual 3D advertising object may be continuously displayed and transformed during a defined period of an MPEG media sequence (e.g., between 20 and 40 seconds). In this example, when a user's left hand is detected, a virtual 3D object is placed on the user's left hand, and the virtual 3D object continues to follow the user's hand.

[0121] In this example, two behaviors are defined to support this interactivity scenario: The first behavior has the following parameters: · The first trigger, related to a time sequence of MPEG media between 20 and 40 seconds, which fires as long as the condition is met (ACTIVATE_ON). Two sequential actions for enabling and transforming the virtual 3D object (node ​​0). The second behavior has the following parameters: A trigger combination that activates as long as the condition is met (ACTIVATE_ON), ​​with a second trigger associated with the detection of the user's left hand and a third trigger associated with the absence of the user's right hand. A single action to place a virtual object in the user's left hand (node ​​0).

[0122] The two behaviors define the same interrupt action, which disables a virtual object (node ​​0). Because the two behaviors affect the same virtual object (node ​​0), a higher priority is set to the second behavior associated with the user's gesture (left hand pose) to execute the desired interactivity scenario. In this example, the desired behavior can be implemented using scene interactivity information, which can be provided in JSON format, such as the examples shown in Figures 5A-5D.

[0123] In some embodiments, MPEG media scene descriptions may be used to combine the scene structure with a binary representation of the media content.

[0124] While the MPEG-I scene description framework ensures that timed media and corresponding associated virtual content are always available, it does not describe how a user interacts with scene objects at runtime for an immersive XR experience. European Patent Application No. EP22306405 (filed January 12, 2022) (the "024 Application") discusses augmenting time-evolving scene descriptions by adding behaviors. These behaviors are associated with pre-defined virtual objects that allow runtime interactivity for user-specific XR experiences.

[0125] A behavior is composed of a set of triggers that define the conditions that must be met for activation, and a set of actions that should be followed when the trigger is activated. The triggers control how parameters are defined to allow logical operations between the trigger and the activation policy.

[0126] Regarding the activation policy, a single Boolean flag (ActivateOnce) is specified for each trigger to indicate whether the trigger should be activated (i) every time the condition is met, or (ii) only once after the condition is met. Table 2 shows an example syntax for a trigger configured in this way.

[0127] Such activation mechanisms may not be well-suited when multiple triggers are referenced by a set of trigger behaviors. Depending on how those triggers are combined by logical operations (e.g., AND, OR, NOT, or other logical operators), the activation flags may not be consistent among all triggers in the following combination: a proximity trigger combined (logically ANDed) with a visibility trigger, with both ActivateOnce flags set to true. This trigger combination can never occur. In one scenario, the visibility trigger will only be activated once and will no longer be activated when the proximity trigger is activated. For example, the initial visibility trigger may only be activated once and will no longer be activated, even if the proximity trigger is subsequently activated multiple times.

[0128] A trigger combination may be a combination of conditions, rather than the activation state of each trigger. Furthermore, as defined in the '024 application, the activation flags may not allow for additional cases that may occur, such as, for example: Triggers fire when a condition is no longer met. Firing a trigger only n times during the entire scene timeline, or once each time a condition is met (or not met). The value of n can range from 1 to infinity.

[0129] According to some embodiments, this application defines a new Activate flag at the behavior level rather than the trigger level. This flag specifies the activation mechanism of a combination of triggers by defining several scenarios for activating the triggers. The conditions of each referenced trigger are evaluated and combined following a trigger control parameter. This control parameter allows describing a logical combination between the referenced triggers, which can be a string describing this combination. The result of this evaluation (satisfied or not satisfied) and the Activate flag tell the presentation engine when to execute the behavior's action. Applications introduce new data into the behavior-based interactive scene description. These data fields are used by the runtime processing model to control the behavior.

[0130] This application details the scope of the MPEG-I scene description framework using the Khronos glTF extension mechanism ("Khronos Group") to support the addition of scene description functionality. This application modifies the MPEG_scene_interactivity gltf extension defined in the '024 application.

[0131] This modification affects the definition of triggers (in Tables 2 and 3) and behaviors (in Tables 5 and 6). -Removed parameter for ActivateOnce trigger. -New Activate behavior parameter. Modification of the triggersControl parameter to allow multiple operations (e.g., AND, OR, NOT, or other operators) to be used in one combination of triggers. It can be a string combining a trigger index in the trigger array with a logical operation. For example, The "#" tag to indicate a trigger index, "&" to indicate a logical AND operation, "|" for a logical OR operation, "~" for a NOT operation, and parentheses to group several operations. Such a syntax could give the following strings: "#1&~#2|(#3)" The default empty string may be understood as a logical OR between all triggers.

[0132] An exemplary process for activating a trigger or combination of triggers according to some embodiments is shown in FIG. 10 and discussed further below.

[0133] The parameters of an exemplary trigger structure are shown in Table 2. Table 2 shows the activateOnce Boolean value, which indicates how often to activate the trigger when a condition is met. This Boolean value has been removed in Table 3.

[0134] [Table 2-1]

[0135] [Table 2-2] In some embodiments, instead of using string parameters associated with the Khronos OpenXR Interaction Profile Path syntax to define user body parts and gestures for USER_INPUT triggers, other syntax formats may be used, such as the syntax format used to represent haptic objects, where an array of vertices (geometric model) and a binary mask (body part mask) are used to specify where the haptic effect should be applied.

[0136] An updated trigger structure, according to some embodiments, is shown in Table 3. Table 3 removes the activateOnce Boolean value shown in Table 2. The updated example trigger structure, described below in Table 3, corresponds to FIG. 5A.

[0137] [Table 3-1]

[0138] [Table 3-2]

[0139] Exemplary parameters for an action structure are shown in Table 4. The exemplary action structure described below in Table 4 corresponds to FIG. 5B.

[0140] [Table 4-1]

[0141] [Table 4-2]

[0142] Exemplary parameters for a behavior structure are shown in Table 5. Table 5 shows the triggersControl enumeration, which combines multiple triggers using a logical OR operator when the enumeration is 0 and a logical AND operator when the enumeration is 1.

[0143] [Table 5]

[0144] Exemplary parameters for an updated behavior structure, according to some embodiments, are shown in Table 6. Table 6 shows an activate enumeration, which lists various activation states for how and when various triggers should be activated. Table 6 also shows several triggersControl strings, which use string values ​​to indicate logical operators to apply when combining multiple triggers. The updated exemplary behavior structure, described below in Table 6, corresponds to Figure 5C.

[0145] [Table 6]

[0146] In the first example presented in Figures 5A-5D, a virtual 3D object is continuously displayed and transformed during the media sequence. When the user's left hand is detected, the virtual 3D object is positioned at the user's left hand, and the media sequence continues to follow the virtual 3D object.

[0147] FIG. 5A is a diagram of a data structure illustrating an example set of triggers for an augmented reality scene description, according to some embodiments. FIG. 5A shows a data structure 502 in which a header indicates that interactivity metadata belongs to the scene description. Three triggers for two example behaviors are described. Triggers may be listed within a behavior field (such as the example shown in FIG. 5C). Listing them in separate arrays allows a method or apparatus to use the same trigger for multiple behaviors. Parameters for an example trigger structure are shown in Table 3. The "activateOnce" field shown in the data structure in Table 2 is absent from the data structures in Table 3 and FIG. 5A. FIG. 5A corresponds to Table 3. Returning to the example listed above, the first trigger in FIG. 5A is a time sequence of MPEG media between 20 and 40 seconds. The second trigger in FIG. 5A is a user input associated with the user's left hand. The third trigger in FIG. 5A is a user input associated with the user's right hand.

[0148] 5B is a diagram of a data structure showing an example set of actions for an augmented reality scene description, according to some embodiments. Data structure 504 has an "actions" field that contains descriptions of three actions needed to execute the two behaviors of the illustrative example. The first action, which enables the object at node 0 of the node tree, has index 0 because it is the first action in the action array. The second action, which places the object at node 0 in the user's left hand, has index 1, and the third action, which transforms the object at node 0 according to the transformation matrix in this example, has index 2. The fourth action, which disables the object at node 0, at index 3, is an interrupt action common to the two behaviors. FIG. 5B corresponds to Table 4.

[0149] FIG. 5C is a data structure illustrating an exemplary set of behaviors for an augmented reality scene description, according to some embodiments. Data structure 506 has a "behaviors" field containing descriptions of two illustrative behaviors. The list of triggers and actions is indicated by the indexes in the trigger array in FIG. 5A and the action array in FIG. 5B. The interrupt action of the two behaviors points to the fourth action with an index number of 3 in the action array. The second behavior has a priority of 2, which is higher than the first behavior, which has a priority of 1. Because the two behaviors apply to the same node 0 in the node tree, the second behavior is selected if both behaviors are active at the same time. Example parameters for the behavior structure are shown in Table 6. The "activate" field shown in Table 6 and FIG. 5C is not present in the data structure of Table 5. The "triggersControl" field shown in the data structure of Table 5 is changed in the data structure of Table 6 and FIG. 5C to a string describing a set of logical operations to be performed on the list of triggers. FIG. 5C corresponds to Table 6. For the first behavior (with an index of 0), the triggersControl field indicates that only the first trigger is used. This trigger is active as long as the condition is met, which means the time sequence is between 20 and 40 seconds. When the first behavior is applied, the first action (activating / enabling the virtual object) and the third action (transforming the virtual object) are performed. For the second behavior, the trigger condition is met when a left hand is detected and a right hand is not detected. When the second behavior is applied, the second action (placing the virtual object in the user's left hand) is performed.

[0150] 5D is a diagram of a data structure illustrating an exemplary set of information for an augmented reality scene description, according to some embodiments. Data structure 508 has a "nodes" field that contains exemplary descriptions of nodes. For example, in a scene, nodes may be named. Triggers, actions, and behaviors may also have unique ID numbers or unique names. Therefore, when a scene description is updated, it is easy to detect whether an ongoing behavior or node belongs to the new scene description.

[0151] FIG. 6 is a data syntax diagram illustrating an example syntax for a data stream encoding an augmented reality (XR) scene description, according to some embodiments. The structure consists of containers that organize the stream into independent syntactic elements. The structure may include a header section 602, which is a set of data common to all syntactic elements of the stream. For example, the header section includes some metadata about the syntactic elements that describes their respective nature and role. The structure also includes a payload that includes syntactic element 604 and syntactic element 606. Syntactic element 604 includes data representing a media content item described in a scene graph node associated with a virtual element. Images, meshes, and other raw data may be compressed depending on the compression method. Syntactic element 606 is part of the payload of the data stream and includes data encoding the scene description described in connection with FIGS. 5A through 5D.

[0152] FIG. 7 is a flowchart illustrating a method performed according to some embodiments. Scene description data is obtained for a 3D scene. The scene description data may be in GLTF format or another format. Process 700 includes obtaining (702) scene description data, which may include: scene element information describing each of a plurality of scene elements in the scene; trigger information describing at least one trigger condition; action information describing at least one action to perform on one or more scene elements associated with the action; and behavior information, where the behavior information associates at least one of the trigger conditions with at least one of the actions. In some embodiments, the action is performed on a scene element associated with a corresponding node in a hierarchical scene description graph. Alternatively, or in addition, the action may be performed on a scene element not associated with a specific node, such as an animation or MPEG_media element included in the scene description.

[0153] In some embodiments, characteristics of user interactivity are monitored to detect whether a trigger condition is met 704. The monitored characteristics may include, for example, user camera or viewpoint position (determined by user input and / or by sensors such as gyroscopes and accelerometers, among other possibilities), user gestures (detected by a camera, wrist-mounted or handheld accelerometer, or other sensors), or other user input.

[0154] In response to determining (706) that at least one of the trigger conditions is satisfied and that at least a first of the actions is associated with the trigger condition by the behavioral information, a first action is performed (708) on at least a first scene element associated with the first action.

[0155] In some embodiments, the action results in a modification to one or more scene elements of a scene description of the 3D scene. In some embodiments, the method includes rendering the 3D scene according to the modified scene description. In other embodiments, the modified scene description is provided to another renderer, which renders the 3D scene by conventional 3D rendering techniques. The 3D scene may be displayed on any display device, such as a display device described herein or others. In some embodiments, the 3D scene may be displayed as an overlay with a real-world scene using an optical see-through display or a video see-through display. Note that displaying a 3D scene as referred to herein includes displaying a 2D image of the 3D scene using a 2D display device.

[0156] 8 is a flowchart illustrating an exemplary process for invoking a trigger or combination of triggers, according to some embodiments. During runtime, the application iterates through each defined behavior and checks the fulfillment of the associated trigger according to the procedure shown in FIG.

[0157] As mentioned above, this application, according to some embodiments, defines a new activation flag at the behavior level rather than the trigger level. See Table 5. This flag specifies the activation mechanism of a combination of triggers by defining several scenarios for activating the trigger. The conditions of each referenced trigger are evaluated and combined following a trigger control parameter. This control parameter allows describing a logical combination between the referenced triggers. It can be a string describing this combination. The result of this evaluation (satisfied or not satisfied) and the Activate flag tell the presentation engine when to execute the behavior's action. The application introduces new data into the behavior-based interactive scene description. These data fields are used by the runtime processing model to control the behavior.

[0158] This application is detailed within the MPEG-I scene description framework using the Khronos glTF extension mechanism ("Khronos Group") to support the addition of scene description functionality. This application modifies the MPEG_scene_interactivity gltf extension defined in the '024 application.

[0159] This modification affects the definition of triggers (in Tables 2 and 3) and behaviors (in Tables 5 and 6). -Removed parameter for ActivateOnce trigger. -New Activate behavior parameter. Modification of the triggersControl parameter to allow multiple operations (e.g., AND, OR, NOT, or other operators) to be used in one combination of triggers. It can be a string combining a trigger index in the trigger array with a logical operation. For example, The "#" tag to indicate a trigger index, "&" to indicate a logical AND operation, "|" for a logical OR operation, "~" for a NOT operation, and parentheses to group several operations. Such a syntax could give the following strings: "#1&~#2|(#3)" The default empty string may be understood as a logical OR between all triggers.

[0160] Returning to the discussion of Figure 8, Figure 8 represents the state of a trigger combination ({T}xxxx). A condition combination ({Condition}) may be evaluated sequentially: each condition may be evaluated and the results of all evaluations may be combined according to the triggersControl parameter. Depending on the state and value of the trigger flag, the referenced action is executed. S1: Once when the condition is first met (used when Activate=ACTIVATE_FIRST_ENTER) S2: Once each time the condition is met (used when Activate=ACTIVATE_EACH_ENTER) S3: As long as the condition is met (used when Activate=ACTIVATE_ON) S4: Once the first time the condition is no longer met (used when Activate=ACTIVATE_FIRST_EXIT) S5: Once each time the condition is no longer met (used when Activate=ACTIVATE_EACH_EXIT) · S6: Unless the condition is met (used when Activate=ACTIVATE_OFF).

[0161] After the scene is loaded (814), the behavior is loaded, each combination of triggers is evaluated, initialized to state {T}off 810 or state {T}on 804, and the referenced action is executed (826 for S6 or 820 for S3, respectively).

[0162] From state {T}off 810, if the combination of conditions is met (812), the trigger enters state {T}enter 802 and the referenced action is executed: either the first time (816 for S1) or every time (818 for S2).

[0163] From state {T}enter, the trigger automatically proceeds to state {T}on 804, where the referenced action is executed (820 in the case of S3).

[0164] As long as the condition 806 is met, the trigger is in state {T}on 804 and the referenced action is executed (820 for S3).

[0165] From state {T}on 804, when condition 806 is no longer satisfied, the trigger enters state {T}exit 808 and the referenced action is executed: either only the first time (822 in the case of S4) or every time (824 in the case of S5).

[0166] From state {T}exit, the trigger automatically proceeds to state {T}off 810 and the referenced action is executed (826 in the case of S6).

[0167] Alternatively, if S1 816 or if S4 822 may be replaced by if (S1-n) or if (S4-n), respectively, which fires the trigger for the first n times when the trigger enters state {T}enter 802 or state {T}exit 808, respectively. The value of n may be specified in the additional behavior parameters.

[0168] Alternatively, a graph of triggers may be applied outside the scope of the above behavioral mechanisms: a combination of triggers may be specified for a scene, and S1 to S6 may be considered as events that are fired to execute predetermined callback functions.

[0169] The MPEG_scene_interactivity extended semantics may be added to the MPEG-I SD standard.

[0170] FIG. 9 is a flowchart illustrating an exemplary process for rendering an augmented reality scene with an updated trigger mechanism, according to some embodiments. In some embodiments of process 900, scene description data for a 3D scene is obtained (902). The scene description data may include trigger information, action information, and behavior information. In some embodiments, the scene description data may include data according to one or more of the data structures shown in Tables 3, 4, and 6 and FIGS. 5A, 5B, 5C, and 5D. The method may further include monitoring (904) one or more trigger conditions associated with the action. One or more trigger conditions associated with the action may be logically combined (906). For example, in the case of a single trigger, no trigger combination is performed; however, according to some embodiments, a NOT operation may be performed on the trigger. In another example, a first trigger state may be inverted (operated with a NOT logical operator), and the inverted first trigger may be logically ORed with a second trigger state. Next, the method determines whether the logical combination of one or more triggers produces a TRUE result (908). If the logical combination produces a TRUE result, the method may determine whether the activation state allows the associated action to be performed (910). If so, the associated action is performed for the associated node (912). For example, the activation state may be ACTIVATE_EACH_ENTER. If the device performing the method is currently in the S2 state (described in connection with FIG. 8), the associated action may be performed each time the logical combination of one or more triggers produces a TRUE result. If the activation state does not allow the action to be performed, the method may return to monitoring one or more trigger conditions associated with the action.

[0171] In some embodiments, a check may be made to determine that the logical combination produces a result (which may be a true or false result). If the logical combination produces a result, a check may be made to determine whether the activation state allows the action to be performed. If the activation state allows the action to be performed, the action may be performed on the node.

[0172] 10 is a flowchart illustrating an exemplary process for a trigger activation mechanism in a time-evolving scene description, according to some embodiments. In some embodiments, the exemplary process 1000 may include obtaining 1002 scene description data for a 3D scene. In some embodiments, the scene description data may include scene element information describing each of a plurality of scene elements in the scene, trigger information describing at least one trigger condition, action information describing at least one action to perform on one or more scene elements associated with the action, and behavioral information, the behavioral information associating at least one of the trigger conditions with at least one of the actions. In some embodiments, the exemplary process may include performing 1004 a first action on at least a first node associated with the first action in response to the logical determination that (i) at least one combination of the trigger conditions is satisfied, and (ii) at least a first one of the actions is associated with the combination of trigger conditions by the behavioral information.

[0173] Although methods and systems according to some embodiments are generally discussed in the context of extended reality (XR), some embodiments may be applied to any XR context, e.g., virtual reality (VR) context, mixed reality (MR) context, augmented reality (AR) context, etc. Although the term "head-mounted display (HMD)" is used herein in accordance with some embodiments, some embodiments may be applied to, for example, a wearable device (which may or may not be head-mounted) capable of XR, VR, AR, and / or MR in some embodiments.

[0174] An exemplary method according to some embodiments includes: obtaining scene description data for a 3D scene, the scene description data may include: scene element information describing each of a plurality of scene elements in the scene; trigger information describing at least one trigger condition; action information describing at least one action to perform on one or more scene elements associated with an action; and behavior information, the behavior information including: a list of at least one trigger; trigger combination information indicating how at least one trigger condition corresponding to a trigger in the list is combined with other trigger conditions corresponding to other triggers in the list; The method includes a step of obtaining scene description data of the 3D scene, the scene description data may include information, the information may include activation information indicating when a combination of at least one trigger from the list with other triggers from the list should be performed; and further, in response to the logical determination that (i) a combination of a trigger condition of at least one trigger from the list with a trigger condition of other triggers from the list produces a true result, and (ii) the activation information indicates that a combination of at least one trigger from the list with other triggers from the list should be performed, a first action of the at least one action may be performed on one or more scene elements associated with the first action.

[0175] In some embodiments of the example method, the list may include one trigger and no other triggers in the list, the trigger combination information may include a unary operator that operates on the one trigger, and the activation information may indicate when to execute the unary operator on the one trigger.

[0176] In some embodiments of the example method, the list may include at least two triggers, the trigger combination information may indicate how the at least two triggers should be combined, and the activation information may indicate when to execute the at least two trigger combinations of the list.

[0177] In some embodiments of the example method, performing the first action may include performing the first action one or more times based on the activation information.

[0178] In some embodiments of the example method, determining a combination of the trigger conditions of at least one trigger in the list with the trigger conditions of other triggers in the list may include performing a logical operation on at least one of the trigger conditions.

[0179] In some embodiments of the example method, determining a combination of the trigger conditions of at least one trigger in the list with the trigger conditions of other triggers in the list may include performing a logical OR operation of at least two trigger conditions.

[0180] In some embodiments of the example method, at least a first of the trigger conditions may be a visibility condition that is satisfied when a specified scene element is visible to a specified camera node.

[0181] In some embodiments of the example method, at least a first of the trigger conditions may be a proximity condition that is met when the distance from the user camera to a specified scene element is within a specified range.

[0182] In some embodiments of the example method, at least a first of the trigger conditions may be a user input condition that is satisfied when a specified user interaction is detected.

[0183] In some embodiments of the example method, at least a first of the trigger conditions may be a timed condition that is met during a specified period of time.

[0184] In some embodiments of the example method, at least a first of the trigger conditions may be a collider condition that is satisfied in response to detecting a collision between designated scene elements.

[0185] In some embodiments of the example method, the action information may describe at least two actions, and the behavior information may include, for at least one behavior, information indicating an order in which at least two of the at least two actions should be performed.

[0186] In some embodiments of the example method, the action information may describe at least two actions, and the behavior information may include information indicating that at least two of the at least two actions are to be performed simultaneously.

[0187] In some embodiments of the example method, at least one of the scene elements in the scene may be a virtual object.

[0188] In some embodiments, the example method may further include rendering the 3D scene according to the scene description data computed by the first action.

[0189] In some embodiments of the example method, the trigger information may include an array of two or more triggers in the 3D scene.

[0190] In some embodiments of the example method, the action information may include an array of two or more actions in the 3D scene.

[0191] In some embodiments of the example method, the behavior information may include an array of two or more behaviors in the 3D scene.

[0192] In some embodiments of the example method, the scene description data may be provided in JSON format.

[0193] In some embodiments of the exemplary method, the scene description data may be provided in GLTF format.

[0194] A further exemplary method, according to some embodiments, includes: obtaining scene description data of a 3D scene, wherein the scene description data may include: scene element information describing each of a plurality of scene elements in the scene, trigger information describing at least one trigger condition, action information describing at least one action to perform on one or more scene elements associated with the action, and behavioral information, wherein at least one of the trigger conditions may be associated with at least one of the actions; and further, in response to the logical determination that (i) at least one combination of the trigger conditions is satisfied, and (ii) at least a first of the actions is associated with the combination of trigger conditions by the behavioral information, performing a first action on at least a first node associated with the first action.

[0195] In some embodiments of the further exemplary method, the logical determining may further include determining that at least one trigger has been activated.

[0196] In some embodiments of the further exemplary method, the logical determination may further include determining an activation state of a combination of at least one trigger, and performing the first action may include performing the first action one or more times based on the combined activation state.

[0197] In some embodiments of the further exemplary method, determining the combination of the at least one trigger condition may include performing a logical operation on at least one of the trigger conditions.

[0198] In some embodiments of the further exemplary method, determining the combination of the at least one trigger condition may include performing a logical OR operation of at least two trigger conditions.

[0199] In some embodiments of the further exemplary method, at least a first of the trigger conditions may be a visibility condition that is satisfied when a specified scene element is visible to a specified camera node.

[0200] In some embodiments of further exemplary methods, at least a first of the trigger conditions may be a proximity condition that is met when the distance from the user camera to a specified scene element is within a specified range.

[0201] In some embodiments of the further exemplary method, at least a first of the trigger conditions may be a user input condition that is satisfied when a specified user interaction is detected.

[0202] In some embodiments of the further exemplary method, at least a first of the trigger conditions may be a timed condition that is satisfied during a specified period of time.

[0203] In some embodiments of the further exemplary method, at least a first of the trigger conditions may be a collider condition that is satisfied in response to detecting a collision between designated scene elements.

[0204] In some embodiments of further exemplary methods, the behavior information may identify at least one behavior, and the behavior information for each behavior may identify at least one of the triggers and at least one of the actions.

[0205] In some embodiments of further exemplary methods, the action information may describe at least two actions, and the behavior information may include, for at least one behavior, information indicating an order in which at least two of the at least two actions should be performed.

[0206] In some embodiments of further exemplary methods, the action information may describe at least two actions, and the behavior information may include information indicating that at least two of the at least two actions should be performed simultaneously.

[0207] In some embodiments of the further exemplary method, at least one of the scene elements in the scene may be a virtual object.

[0208] In some embodiments, further exemplary methods may further include rendering a 3D scene according to the scene description data computed by the first action.

[0209] In some embodiments of the further exemplary method, the trigger information may include an array of two or more triggers in the 3D scene.

[0210] In some embodiments of the further exemplary method, the action information may include an array of two or more actions in the 3D scene.

[0211] In some embodiments of the further exemplary method, the behavior information may include an array of two or more behaviors in the 3D scene.

[0212] In some embodiments of the further exemplary method, the scene description data may be provided in JSON format.

[0213] In some embodiments of the further exemplary method, the scene description data may be provided in GLTF format.

[0214] Another exemplary method of rendering an augmented reality scene for a user in a timed environment according to some embodiments may include steps of obtaining a description of the augmented reality scene, the description including: a scene tree connecting nodes describing timed objects, virtual objects, or relationships between objects; and behavior data items, the behavior data items including: at least trigger control parameters, where the trigger control parameters are descriptions of conditions associated with one or more triggers; activation conditions associated with the trigger control parameters; and at least actions, where the actions are descriptions of processing to be performed by the augmented reality engine on the objects described by the nodes of the scene tree; and steps of applying actions of the behaviors to the associated objects, conditional on at least one logical combination of the triggers of the behavior data items being triggered and the activation conditions associated with the trigger control parameters of the behaviors being satisfied.

[0215] In some embodiments of another exemplary method, the logical combination of the at least one trigger may include a logical operation on at least one of the trigger conditions.

[0216] In some embodiments of another exemplary method, the logical combination of the at least one trigger may include a logical OR operation of at least two trigger conditions.

[0217] Another further exemplary method for runtime updating a first description of an augmented reality scene, which may include behavioral data items, with a second description of the augmented reality scene, according to some embodiments, may include, for each ongoing behavioral data item of the first description, if the ongoing behavioral data item is not applicable to the second description: processing an interrupt action if one exists for the ongoing application of the first description; stopping the ongoing behavior; and applying the second description.

[0218] An exemplary device for rendering an augmented reality scene for a user in a timed environment, according to some embodiments, may include a processor configured to obtain a description of the augmented reality scene, the description including: a scene tree connecting nodes describing timed objects, virtual objects, or relationships between objects; and behavioral data items, the behavioral data items may include: at least triggers that are descriptions of conditions; triggers that are activated when the conditions are detected in the timed environment; and at least actions that are descriptions of processing to be performed by the augmented reality engine on the objects described by the nodes of the scene tree; and further may include memory associated with the processor configured to apply the behavioral actions to the associated objects, conditional on the trigger of the behavioral data item being activated.

[0219] In some embodiments of the exemplary device, the processor may be further configured to attribute an activation state set to false to at least one trigger of the description when the description of the augmented reality scene is obtained; to set the activation state of the trigger to true when the condition of the at least one trigger is met for the first time; and to activate the trigger when the condition of the at least one trigger is met.

[0220] In some embodiments of the exemplary device, the processor may be further configured to, when a condition of at least one trigger is satisfied, if the activation state of the trigger is set to true, activate the trigger only if a description of the trigger allows a second activation.

[0221] A further exemplary device for runtime updating a first description of an augmented reality scene including behavioral data items with a second description of the augmented reality scene according to some embodiments may include a memory associated with a processor configured to, for each ongoing behavioral data item of the first description, if the ongoing behavioral data item is not applicable to the second description: process an interrupt action if present for the ongoing application of the first description; stop the ongoing behavior; and apply the second description.

[0222] An exemplary apparatus according to some embodiments may include one or more processors configured to perform the above methods.

[0223] An exemplary computer-readable medium according to some embodiments may include instructions for causing one or more processors to perform any of the methods listed above.

[0224] In some embodiments of the exemplary computer-readable medium, the computer-readable medium may be a non-transitory storage medium.

[0225] An exemplary computer program product according to some embodiments may include instructions that, when executed by one or more processors, cause the one or more processors to perform any of the methods listed above.

[0226] An exemplary signal includes scene description data of a 3D scene according to some embodiments, where the scene description data may include: scene element information describing each of a plurality of scene elements in the scene; trigger information describing at least one trigger condition; action information describing at least one action to perform on one or more scene elements associated with the action; and behavior information, where the behavior information may associate at least one of the trigger conditions with at least one of the actions.

[0227] A further exemplary computer-readable medium includes, according to some embodiments, scene description data for a 3D scene, wherein the scene description data may include: scene element information describing each of a plurality of scene elements in the scene; trigger information describing at least one trigger condition; action information describing at least one action to perform on one or more scene elements associated with the action; and behavior information, wherein the behavior information associates at least one of the trigger conditions with at least one of the actions.

[0228] A first exemplary method according to some embodiments includes: obtaining scene description data for a 3D scene, wherein the scene description data may include: scene element information describing each of a plurality of scene elements in the scene; trigger information describing at least one trigger condition; action information describing at least one action to perform on one or more scene elements associated with at least one action; and behavior information, wherein the behavior information includes: a trigger list of at least one trigger; and an action list of at least one action; and trigger combination information, wherein a first trigger condition corresponds to a first trigger in the trigger list and other triggers corresponding to other triggers in the trigger list. and activation information indicating when at least one action should be performed in response to a logical determination that: (i) a combination of a first trigger condition corresponding to a first trigger in the trigger list and other trigger conditions corresponding to other triggers in the trigger list produces a result, and (ii) the activation information indicates that the result fires the first and other triggers.

[0229] In some embodiments of the first exemplary method, the trigger list may include one trigger, and there are no other triggers in the trigger list, the trigger combination information may include a unary operator that operates on the one trigger, and the activation information may indicate when to execute the unary operator on the one trigger.

[0230] In some embodiments of the first exemplary method, the trigger list may include at least two triggers, the trigger combination information may indicate how the at least two trigger conditions should be combined, and the activation information may indicate when to perform at least one action depending on the result of the trigger combination information.

[0231] In some embodiments of the first exemplary method, performing the first action may include performing the first action one or more times based on the activation information.

[0232] In some embodiments of the first exemplary method, determining a combination of the trigger conditions of at least one trigger in the trigger list with the trigger conditions of other triggers in the trigger list may include performing a logical operation on at least one of the trigger conditions.

[0233] In some embodiments of the first exemplary method, determining a combination of the trigger conditions of at least one trigger in the trigger list with the trigger conditions of other triggers in the trigger list may include performing a logical OR operation of at least two trigger conditions.

[0234] In some embodiments of the first exemplary method, at least a first of the trigger conditions is a visibility condition that is satisfied when a specified scene element is visible to a specified camera node.

[0235] In some embodiments of the first exemplary method, at least a first of the trigger conditions is a proximity condition that is met when the distance from the user camera to a specified scene element is within a specified range.

[0236] In some embodiments of the first exemplary method, at least a first of the trigger conditions is a user input condition that is satisfied when a specified user interaction is detected.

[0237] In some embodiments of the first exemplary method, at least a first of the trigger conditions is a timed condition that is satisfied during a specified period of time.

[0238] In some embodiments of the first exemplary method, at least a first of the trigger conditions is a collider condition that is satisfied in response to detecting a collision between designated scene elements.

[0239] In some embodiments of the first exemplary method, the action information may describe at least two actions, and the behavior information may include, for at least one behavior, information indicating an order in which at least two of the at least two actions should be performed.

[0240] In some embodiments of the first exemplary method, the action information may describe at least two actions, and the behavior information may include information indicating that at least two of the at least two actions are to be performed simultaneously.

[0241] In some embodiments of the first exemplary method, at least one of the scene elements in the scene is a virtual object.

[0242] Some embodiments of the first exemplary method may further include rendering the 3D scene according to the scene description data computed by the first action.

[0243] In some embodiments of the first exemplary method, the trigger information may include an array of two or more triggers in the 3D scene.

[0244] In some embodiments of the first exemplary method, the action information may include an array of two or more actions in the 3D scene.

[0245] In some embodiments of the first exemplary method, the behavior information may include an array of two or more behaviors in the 3D scene.

[0246] In some embodiments of the first exemplary method, the scene description data may be provided in JSON format.

[0247] In some embodiments of the first exemplary method, the scene description data may be provided in GLTF format.

[0248] A first exemplary method / apparatus according to some embodiments may include a processor and a non-transitory computer-readable medium storing instructions that, when executed by the processor, function to cause the apparatus to perform any one of the methods set forth above.

[0249] A second exemplary method / apparatus according to some embodiments may include: acquiring scene description data of a 3D scene, where the scene description data may include: scene element information describing each of a plurality of scene elements in the scene, trigger information describing at least one trigger condition, action information describing at least one action to perform on one or more scene elements associated with the at least one action, and behavioral information associating at least one of the trigger conditions with at least one of the actions; and performing a first action on at least a first node associated with the first action in response to a logical determination that (i) at least one combination of the trigger conditions is satisfied, and (ii) a first action of the at least one action is associated with the combination of trigger conditions by the behavioral information.

[0250] In some embodiments of the second exemplary method, the logical determining may further include determining that a trigger associated with the at least one trigger condition has been activated.

[0251] In some embodiments of the second exemplary method, the logical determination may further include determining an activation state of the combination of the at least one trigger condition, and performing the first action may include performing the first action one or more times based on the activation state.

[0252] In some embodiments of the second exemplary method, determining the combination of the at least one trigger condition may include performing a logical operation on at least one of the trigger conditions.

[0253] In some embodiments of the second exemplary method, determining the combination of the at least one trigger condition may include performing a logical OR operation on the at least two trigger conditions.

[0254] In some embodiments of the second exemplary method, at least a first of the at least one trigger condition is a visibility condition that is satisfied when a specified scene element is visible to a specified camera node.

[0255] In some embodiments of the second exemplary method, at least a first of the at least one trigger condition is a proximity condition that is satisfied when the distance from the user camera to a specified scene element is within a specified range.

[0256] In some embodiments of the second exemplary method, at least a first of the at least one trigger condition is a user input condition that is satisfied when a specified user interaction is detected.

[0257] In some embodiments of the second exemplary method, at least a first of the at least one trigger condition is a timed condition that is satisfied during a specified period of time.

[0258] In some embodiments of the second exemplary method, at least a first of the at least one trigger condition is a collider condition that is satisfied in response to detecting a collision between designated scene elements.

[0259] In some embodiments of the second exemplary method, the behavioral information may identify at least one behavior, and the behavioral information for each of the at least one behavior identifies a trigger associated with one of the at least one trigger condition and one of the at least one action.

[0260] In some embodiments of the second exemplary method, the action information may describe at least two actions, and the behavior information may include, for at least one behavior, information indicating an order in which at least two of the at least two actions should be performed.

[0261] In some embodiments of the second exemplary method, the action information may describe at least two actions, and the behavior information may include information indicating that at least two of the at least two actions are to be performed simultaneously.

[0262] In some embodiments of the second exemplary method, at least one of the scene elements in the scene is a virtual object.

[0263] Some embodiments of the second exemplary method may further include rendering the 3D scene according to the scene description data computed by the first action.

[0264] In some embodiments of the second exemplary method, the trigger information may include an array of two or more triggers in the 3D scene.

[0265] In some embodiments of the second exemplary method, the action information may include an array of two or more actions in the 3D scene.

[0266] In some embodiments of the second exemplary method, the behavior information may include an array of two or more behaviors in the 3D scene.

[0267] In some embodiments of the second exemplary method, the scene description data may be provided in JSON format.

[0268] In some embodiments of the second exemplary method, the scene description data may be provided in GLTF format.

[0269] A second exemplary method / apparatus according to some embodiments may include a processor and a non-transitory computer-readable medium storing instructions that, when executed by the processor, function to cause the apparatus to perform any one of the methods listed above.

[0270] A third exemplary method for rendering an augmented reality scene for a user in a timed environment according to some embodiments may include steps of obtaining a description of the augmented reality scene, the description including: a scene tree connecting nodes describing timed objects, virtual objects, or relationships between objects; and behavior data items, the behavior data items including: at least trigger control parameters, where the trigger control parameters are descriptions of conditions associated with one or more triggers; activation conditions associated with the trigger control parameters; and at least actions, where the actions are descriptions of processing to be performed by the augmented reality engine on the objects described by the nodes of the scene tree; and steps of applying the actions of the behavior data items to the associated objects, conditional on at least one logical combination of the triggers of the behavior data items being triggered and the activation conditions associated with the trigger control parameters of the behavior data items being satisfied.

[0271] In some embodiments of the third exemplary method, the logical combination of the at least one trigger may include a logical operation on at least one of the conditions associated with the one or more triggers.

[0272] In some embodiments of the third exemplary method, the logical combination of the at least one trigger may include a logical OR operation of at least two conditions associated with the one or more triggers.

[0273] A third example method / apparatus according to some embodiments may include a processor and a non-transitory computer-readable medium storing instructions that, when executed by the processor, function to cause the apparatus to perform any one of the methods listed above.

[0274] A fourth exemplary method, according to some embodiments, for updating a first description of an augmented reality scene including behavior data items at runtime with a second description of the augmented reality scene may include, for each ongoing behavior data item of the first description, if the ongoing behavior data item is not applicable to the second description: processing an interrupt action if one exists for the ongoing application of the first description; stopping the ongoing behavior; and applying the second description.

[0275] A fifth exemplary apparatus, according to some embodiments, that is a device for rendering an augmented reality scene for a user in a timed environment, may include a processor configured to obtain a description of the augmented reality scene, the description including: a scene tree connecting nodes that describe timed objects, virtual objects, or relationships between objects; and behavioral data items, which may include: at least triggers that are descriptions of conditions; triggers that are activated when the conditions are detected in the timed environment; and at least actions that are descriptions of processing to be performed by the augmented reality engine on the objects described by the nodes of the scene tree; and further include memory associated with the processor configured to apply the behavioral actions to the associated objects, conditional on the trigger of the behavioral data item being activated.

[0276] In some embodiments of the fifth exemplary apparatus, the processor is further configured to attribute an activation state set to false to at least one trigger of the description when the description of the augmented reality scene is obtained; to set the activation state of the trigger to true when the condition of the at least one trigger is met for the first time; and to activate the trigger when the condition of the at least one trigger is met.

[0277] In some embodiments of the fifth exemplary apparatus, the processor is further configured to, when a condition of at least one trigger is satisfied, if an activation state of the trigger is set to true, activate the trigger only if a description of the trigger allows a second activation.

[0278] A sixth exemplary apparatus, according to some embodiments, that is a device for runtime updating a first description of an augmented reality scene including behavioral data items with a second description of the augmented reality scene may include memory associated with a processor configured to, for each ongoing behavioral data item of the first description, if the ongoing behavioral data item is not applicable to the second description: process an interrupt action if present for the ongoing application of the first description; stop the ongoing behavior; and apply the second description.

[0279] A seventh exemplary apparatus according to some embodiments may include one or more processors configured to perform any one of the methods listed above.

[0280] An eighth exemplary apparatus according to some embodiments may include a computer-readable medium including instructions for causing one or more processors to perform any one of the methods listed above.

[0281] In some embodiments of the eighth exemplary apparatus, the computer-readable medium is a non-transitory storage medium.

[0282] A tenth exemplary apparatus according to some embodiments may include a computer program product including instructions that, when executed by one or more processors, cause the one or more processors to perform any one of the methods listed above.

[0283] An eleventh exemplary apparatus according to some embodiments may include a signal including scene description data of a 3D scene, wherein the scene description data may include: scene element information describing each of a plurality of scene elements in the scene; trigger information describing at least one trigger condition; action information describing at least one action to perform on one or more scene elements associated with the action; and behavior information, wherein the behavior information associates at least one of the trigger conditions with at least one of the actions.

[0284] A twelfth exemplary method / apparatus according to some embodiments may include a computer-readable medium including scene description data for a 3D scene, wherein the scene description data may include: scene element information describing each of a plurality of scene elements in the scene; trigger information describing at least one trigger condition; action information describing at least one action to perform on one or more scene elements associated with the action; and behavior information, wherein the behavior information associates at least one of the trigger conditions with at least one of the actions.

[0285] A further exemplary computer-readable medium includes, according to some embodiments, scene description data for a 3D scene, wherein the scene description data may include: scene element information describing each of a plurality of scene elements in the scene; trigger information describing at least one trigger condition; action information describing at least one action to perform on one or more scene elements associated with the action; and behavior information, wherein the behavior information associates at least one of the trigger conditions with at least one of the actions.

[0286] This disclosure describes various aspects, including tools, features, embodiments, models, techniques, and the like. Many of these aspects are described with particularity, often in a manner that may sound limiting, to at least indicate their individual characteristics. However, this is for clarity of description and does not limit the disclosure or scope of these aspects. In fact, all of the different aspects can be combined and interchanged to provide further aspects. Furthermore, aspects can also be combined and interchanged with aspects described in prior applications.

[0287] The aspects described and contemplated in this disclosure can be implemented in many different forms. While some embodiments are specifically illustrated, other embodiments are contemplated, and discussion of a particular embodiment is not intended to limit the breadth of implementations. At least one aspect generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting a generated or encoded bitstream. These and other aspects can be implemented as a method, an apparatus, a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to any of the described methods, and / or a computer-readable storage medium having stored thereon a bitstream generated according to any of the described methods.

[0288] In this disclosure, the terms "reconstructed" and "decoded" may be used interchangeably, the terms "pixel" and "sample" may be used interchangeably, and the terms "image," "picture," and "frame" may be used interchangeably. Typically, but not necessarily, the term "reconstructed" is used on the encoder side and "decoded" is used on the decoder side.

[0289] The terms HDR (high dynamic range) and SDR (standard dynamic range) often convey specific values ​​of dynamic range to those skilled in the art. However, additional embodiments are also contemplated in which reference to HDR is understood to mean "higher dynamic range" and reference to SDR to mean "lower dynamic range." Such additional embodiments are not constrained by any specific values ​​of dynamic range that are sometimes associated with the terms "high dynamic range" and "standard dynamic range."

[0290] Various methods are described herein, each of which includes one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for proper implementation of the method, the order and / or use of specific steps and / or actions may be modified or combined. Terms such as "first," "second," and the like may be used in various embodiments to modify elements, components, steps, operations, etc., e.g., "first decode" and "second decode." The use of such terms does not imply a modified ordering of operations unless specifically required. Thus, in this example, the first decode need not be performed before the second decode, but may occur before, during, or in a period overlapping with the second decode.

[0291] Various numerical values ​​may be used, for example, in this disclosure, and the specific values ​​are for illustrative purposes only and the described aspects are not limited to these particular values.

[0292] The embodiments described herein may be performed by computer software implemented by a processor or other hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The processor may be of any type appropriate to the technology environment, and may include, by way of non-limiting examples, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0293] Various embodiments involve decoding. "Decoding," as used in this disclosure, can encompass all or part of the processing performed on a received encoded sequence, e.g., to generate a final output suitable for display. In various embodiments, such processing includes one or more of the processing typically performed by a decoder (e.g., entropy decoding, inverse quantization, inverse transform, and differential decoding). In various embodiments, such processing also, or alternatively, includes processing performed by the decoder of various implementations described in this disclosure, including, for example, extracting a picture from a tiled (packed) picture, determining an upsampling filter to use and then upsampling the picture, and reversing the picture to its intended orientation.

[0294] As a further example, in one embodiment, "decoding" refers only to entropy decoding, in another embodiment, "decoding" refers only to differential decoding, and in another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" refers specifically to a subset of operations or generally to the broader decoding process will become clear based on the context of a particular description.

[0295] Various embodiments include encoding. Similar to the above discussion of "decoding," "encoding," as used in this disclosure, can encompass all or part of the processing performed on an input video sequence to, for example, generate an encoded bitstream. In various embodiments, such processing includes one or more of the processing typically performed by an encoder (e.g., segmentation, differential encoding, transform, quantization, and entropy coding). In various embodiments, such processing also, or alternatively, includes processing performed by the encoder of various implementations described in this disclosure.

[0296] As a further example, in one embodiment, "encoding" refers only to entropy encoding, in another embodiment, "encoding" refers only to differential encoding, and in another embodiment, "encoding" refers to a combination of differential and entropy encoding. Whether the phrase "encoding process" refers specifically to a subset of operations or generally to a broad range of encoding processes will become clear based on the context of a particular description.

[0297] Various embodiments refer to rate-distortion optimization. In particular, during the encoding process, a balance or trade-off between rate and distortion is typically considered, often subject to computational complexity constraints. Rate-distortion optimization is typically formulated as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. There are different approaches to solving the rate-distortion optimization problem. For example, the approach may be based on extensive testing of all encoding options, including all possible modes or coding parameter values, along with a full evaluation of the coding cost and associated distortion of the reconstructed signal after coding and decoding. Faster approaches to reduce encoding complexity may also be used, particularly with calculation of approximate distortion based on a predicted signal or a predicted residual signal rather than the reconstructed signal. A mixture of these two approaches may also be used, such as by using approximate distortion for only some of the possible encoding options and full distortion for others. Other approaches evaluate only a subset of the possible encoding options. More generally, many approaches employ any of a variety of techniques to perform optimization, but the optimization does not necessarily involve a full evaluation of both the coding cost and the associated distortion.

[0298] When a diagram is presented as a flow diagram, it should be understood that it also provides a block diagram of the corresponding apparatus. Similarly, when a diagram is presented as a block diagram, it should be understood that it also provides a flow diagram of the corresponding method / process.

[0299] The embodiments and aspects described herein may be implemented in, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Even if discussed in the context of only a single form of embodiment (e.g., only a method), the discussed functional implementation may also be implemented in other forms (e.g., an apparatus or a program). An apparatus may be implemented in, for example, appropriate hardware, software, or firmware. A method may be implemented in a processor, which generally refers to a processing device including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, computers, cell phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate the transfer of information between end users.

[0300] Reference to "one embodiment" or "embodiment" or "one implementation" or "embodiment," as well as other variations thereof, means that a particular feature, structure, characteristic, etc. described in connection with that embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in one embodiment" or "in an embodiment" in various places throughout this disclosure, as well as any other variations, are not necessarily all referring to the same embodiment.

[0301] Additionally, this disclosure may refer to "determining" various pieces of information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from memory.

[0302] Additionally, this disclosure may refer to "accessing" various pieces of information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, transferring information, copying information, calculating information, determining information, predicting information, or estimating information.

[0303] Additionally, this disclosure may refer to "receiving" various pieces of information. Receiving, like "accessing," is a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Furthermore, "receiving" is typically involved in some way in an operation such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0304] For example, in the cases of "A / B," "A and / or B," and "at least one of A and B," the use of any of the following " / ," "and / or," and "at least one of" should be understood to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of both alternatives (A and B). As a further example, in the cases of "A, B, and / or C" and "at least one of A, B, and C," such phrases encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of only the third listed alternative (C), or the selection of only the first and second alternatives (A and B), or the selection of only the first and third alternatives (A and C), or the selection of only the second and third alternatives (B and C), or the selection of all three alternatives (A, B, and C). This can be expanded as many times as there are listed items.

[0305] Also, as used herein, the word "signal" refers, among other things, to indicating something to a corresponding decoder. For example, in some embodiments, an encoder signals a specific one of multiple parameters for region-based filter parameter selection for de-artifact filtering. In this manner, in embodiments, the same parameters are used on both the encoder and decoder sides. Thus, for example, an encoder can transmit specific parameters to a decoder (explicit signaling) so that the decoder can use the same specific parameters. Conversely, if the decoder already has the specific parameters as well as others, signaling can be used without transmission (implicit signaling) to simply enable the decoder to know and select the specific parameters. By avoiding the transmission of any actual functions, bit savings are realized in various embodiments. It should be understood that signaling can be achieved in various ways. For example, one or more syntax elements, flags, etc. are used in various embodiments to signal information to a corresponding decoder. Although the above relates to the verb form of the word "signal," the word "signal" can also be used as a noun herein.

[0306] In embodiments, various signals formatted to carry information may be generated, for example, that can be stored or transmitted. The information may include, for example, instructions for performing a method or data generated by one of the described embodiments. For example, a signal may be formatted to carry a bit stream of the described embodiments. Such a signal may be formatted, for example, as an electromagnetic wave (e.g., using a high frequency portion of the spectrum) or as a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. The signal may be transmitted over a variety of different wired or wireless links, as is known. The signal may be stored on a processor-readable medium.

[0307] Several embodiments are described. Features of these embodiments can be provided alone or in any combination across various claim categories and types. Furthermore, embodiments can include one or more of the following features, devices, or aspects, alone or in any combination across various claim categories and types. A bitstream or signal containing one or more of the described syntax elements or variations thereof. A bitstream or signal containing syntax conveying information generated by any of the described embodiments. Creating and / or transmitting and / or receiving and / or decoding a bitstream or signal that includes one or more of the described syntax elements or variations thereof. · Creating and / or transmitting and / or receiving and / or decoding according to any of the described embodiments. · A method, process, apparatus, instruction storage medium, data storage medium, or signal according to any of the described embodiments.

[0308] It should be noted that various hardware elements of one or more of the described embodiments are referred to as “modules” that perform (i.e., implement, execute, etc.) various functions, and various functions are described herein with reference to the respective modules. As used herein, a module includes any hardware deemed appropriate for a given implementation (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more memory devices). It should be noted that each described module may also include executable instructions to perform one or more functions described as being performed by the respective module, which may take the form of or include hardware (i.e., hardwired) instructions, firmware instructions, software instructions, and / or the like, and may be stored on any suitable non-transitory computer-readable medium, such as commonly referred to as RAM, ROM, etc.

[0309] Although features and elements are described above in particular combinations, each feature or element can be used alone or in any combination with the other features and elements. In addition, the methods described herein may be implemented in a computer program, software, or firmware embodied in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random-access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and digital versatile disks (DVDs). A processor in association with software may be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.

[0310] It should be noted that various hardware elements of one or more of the described embodiments are referred to as “modules” that perform (i.e., implement, execute, etc.) various functions, and various functions are described herein with reference to the respective modules. As used herein, a module includes hardware deemed appropriate by one of ordinary skill in the art for a given implementation (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more memory devices). It should be noted that each described module may also include executable instructions to perform one or more functions described as being performed by the respective module, which may take the form of or include hardware (i.e., hardwired) instructions, firmware instructions, software instructions, and / or the like, and may be stored on any suitable non-transitory computer-readable medium, such as commonly referred to as RAM, ROM, etc.

[0311] While features and elements are described above in particular combinations, those skilled in the art will understand that each feature or element can be used alone or in any combination with the other features and elements. In addition, the methods described herein may be implemented in a computer program, software, or firmware embodied in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, ROM, RAM, registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and DVDs. A processor in association with software may be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. obtaining scene description data for a 3D scene, The scene description data is scene element information describing each of a plurality of scene elements within a scene; trigger information describing at least one trigger condition; action information describing at least one action to be performed on one or more scene elements associated with the at least one action; Behavioral information and Including, The behavior information is a trigger list of at least one trigger; an action list of the at least one action; trigger combination information indicating a combination of a first trigger condition corresponding to a first trigger in the trigger list and another trigger condition corresponding to another trigger in the trigger list; activation information indicating when the at least one action should be performed depending on the result of the trigger combination information; and (i) in response to a logical determination that a combination of a first trigger condition corresponding to a first trigger in the trigger list and other trigger conditions corresponding to other triggers in the trigger list produces a result, and the activation information indicates that the result is firing the first trigger and the other triggers, performing the at least one action on one or more scene elements associated with the at least one action; A method comprising:

2. the trigger list includes one trigger and no other triggers in the trigger list are present; the trigger combination information includes a unary operator that operates on the one trigger; The activation information indicates when to execute the unary operator for the one trigger. The method of claim 1.

3. the trigger list includes at least two triggers; the trigger combination information indicates how at least two trigger conditions should be combined; the activation information indicates when the at least one action should be performed depending on the result of the trigger combination information; The method of claim 1.

4. The method of claim 1 , wherein performing a first action comprises performing the first action one or more times based on the activation information.

5. 5. The method of claim 1, wherein determining a combination of a trigger condition of at least one trigger in the trigger list with a trigger condition of another trigger in the trigger list comprises performing a logical operation on at least one of the trigger conditions.

6. 5. The method of claim 1, wherein determining a combination of a trigger condition of at least one trigger of the trigger list with a trigger condition of another trigger of the trigger list comprises performing a logical OR operation of at least two trigger conditions.

7. The method of claim 1 , wherein at least a first one of the trigger conditions is a visibility condition that is satisfied when a specified scene element is visible to a specified camera node.

8. 8. The method of claim 1, wherein at least a first one of the trigger conditions is a proximity condition that is satisfied when a distance from a user camera to a specified scene element is within a specified range.

9. The method of claim 1 , wherein at least a first one of the trigger conditions is a user input condition that is satisfied when a specified user interaction is detected.

10. 10. The method of claim 1, wherein at least a first one of the trigger conditions is a timed condition that is satisfied during a specified period of time.

11. The method of claim 1 , wherein at least a first one of the trigger conditions is a collider condition that is satisfied in response to detecting a collision between specified scene elements.

12. the action information describes at least two actions; the behavior information includes, for at least one behavior, information indicating an order in which at least two of at least two actions should be performed; 12. The method according to any one of claims 1 to 11.

13. the action information describes at least two actions; the behavior information includes information indicating that at least two of the at least two actions should be performed simultaneously; 12. The method according to any one of claims 1 to 11.

14. The method of claim 1 , wherein at least one of the scene elements in the scene is a virtual object.

15. The method of claim 1 , further comprising rendering the 3D scene according to scene description data computed by a first action.

16. The method of claim 1 , wherein the trigger information comprises an array of two or more triggers in the 3D scene.

17. The method of claim 1 , wherein the action information comprises an array of two or more actions in the 3D scene.

18. The method of claim 1 , wherein the behavior information comprises an array of two or more behaviors in the 3D scene.

19. 19. The method of any one of claims 1 to 18, wherein the scene description data is provided in JSON format.

20. 20. The method of any one of claims 1 to 19, wherein the scene description data is provided in GLTF format.

21. a processor; a non-transitory computer-readable medium storing instructions that, when executed by the processor, cause an apparatus to perform the method of any one of claims 1 to 20; A device comprising:

22. obtaining scene description data for a 3D scene, The scene description data is scene element information describing each of a plurality of scene elements within a scene; trigger information describing at least one trigger condition; action information describing at least one action to perform on one or more scene elements associated with the at least one action; behavioral information that associates at least one of the trigger conditions with at least one of the actions; and In response to a logical determination that (i) at least one combination of the trigger conditions is satisfied, and (ii) a first action of the at least one action is associated with the combination of trigger conditions by the behavior information, executing the first action on at least a first node associated with the first action; A method comprising:

23. 23. The method of claim 22, wherein the logical determining further comprises determining that a trigger associated with the at least one trigger condition has been activated.

24. the logical determination further includes determining an activation state of the combination of the at least one trigger condition; performing the first action includes performing the first action one or more times based on the activation state.

24. The method of claim 22 or claim 23.

25. 25. The method of any one of claims 22 to 24, wherein determining the combination of the at least one trigger condition comprises performing a logical operation on at least one of the trigger conditions.

26. 25. The method of any one of claims 22 to 24, wherein determining the combination of the at least one trigger condition comprises performing a logical OR operation on at least two trigger conditions.

27. 27. The method of any one of claims 22 to 26, wherein at least a first of the at least one trigger condition is a visibility condition that is satisfied when a specified scene element is visible to a specified camera node.

28. 28. The method of any one of claims 22 to 27, wherein at least a first of the at least one trigger condition is a proximity condition that is satisfied when a distance from a user camera to a specified scene element is within a specified range.

29. 27. The method of any one of claims 22 to 26, wherein at least a first of the at least one trigger condition is a user input condition that is satisfied when a specified user interaction is detected.

30. 27. The method of any one of claims 22 to 26, wherein at least a first of the at least one trigger condition is a timed condition that is satisfied during a specified period of time.

31. 27. The method of any one of claims 22 to 26, wherein at least a first of the at least one trigger condition is a collider condition that is satisfied in response to detecting a collision between specified scene elements.

32. the behavior information identifies at least one behavior; 32. The method of claim 22, wherein the behavior information for each of at least one behavior identifies a trigger associated with one of at least one trigger condition and one of at least one action.

33. the action information describes at least two actions; the behavior information includes, for at least one behavior, information indicating an order in which at least two of at least two actions should be performed; 33. The method of any one of claims 22 to 32.

34. the action information describes at least two actions; the behavior information includes information indicating that at least two of the at least two actions should be performed simultaneously; 33. The method of any one of claims 22 to 32.

35. 35. The method of any one of claims 22 to 34, wherein at least one of the scene elements in the scene is a virtual object.

36. 36. The method of any one of claims 22 to 35, further comprising rendering the 3D scene according to scene description data computed by the first action.

37. 37. The method of any one of claims 22 to 36, wherein the trigger information comprises an array of two or more triggers in the 3D scene.

38. 38. The method of any one of claims 22 to 37, wherein the action information comprises an array of two or more actions in the 3D scene.

39. 39. The method of any one of claims 22 to 38, wherein the behavior information comprises an array of two or more behaviors in the 3D scene.

40. 40. The method of any one of claims 22 to 39, wherein the scene description data is provided in JSON format.

41. 41. A method according to any one of claims 22 to 40, wherein the scene description data is provided in GLTF format.

42. a processor; a non-transitory computer-readable medium storing instructions that, when executed by the processor, cause an apparatus to perform the method of any one of claims 22 to 41; A device comprising:

43. 1. A method of rendering an augmented reality scene for a user in a timed environment, comprising: obtaining a description of the augmented reality scene, said description comprising: a scene tree connecting nodes that describe timed objects, virtual objects, or relationships between objects; A behavior data item, the behavior data item comprising: at least trigger control parameters, the trigger control parameters being descriptions of conditions associated with one or more triggers; a trigger condition associated with the trigger control parameter; at least actions, where the actions are descriptions of operations to be performed by the augmented reality engine on the objects described by the nodes of the scene tree; behavioral data items, including: and applying an action of the behavior data item to an associated object, conditioned on at least one logical combination of triggers of the behavior data item being triggered and on activation conditions associated with trigger control parameters of the behavior data item being satisfied; A method comprising:

44. 44. The method of claim 43, wherein the logical combination of the at least one trigger comprises a logical operation on at least one of the conditions associated with the one or more triggers.

45. 44. The method of claim 43, wherein the logical combination of at least one trigger comprises a logical OR operation of at least two conditions associated with one or more triggers.

46. a processor; a non-transitory computer readable medium storing instructions that, when executed by the processor, cause an apparatus to perform the method of any one of claims 43 to 45; A device comprising:

47. 1. A method for updating at runtime a first description of an augmented reality scene including behavior data items with a second description of the augmented reality scene, the method comprising: for each ongoing behavior data item of said first description, if the ongoing behavior data item is not applicable to the second description; processing an interrupt action, if any, for an ongoing application of the first description; Stopping an ongoing behavior; and applying the second statement; A method comprising:

48. 1. An apparatus for rendering an augmented reality scene for a user in a timed environment, comprising: obtaining a description of the augmented reality scene, said description comprising: a scene tree connecting nodes that describe timed objects, virtual objects, or relationships between objects; A behavior data item, the behavior data item comprising: At least a trigger, which is a description of the condition, and a trigger that is activated when the condition is detected within the timed environment; - at least actions, which are descriptions of operations to be performed by an augmented reality engine on objects described by nodes of said scene tree; behavioral data items, including: and applying a behavioral action to an associated object, conditional on the trigger of said behavioral data item being activated; 1. An apparatus comprising a processor and associated memory configured to execute

49. The processor: attributing a trigger state set to False to at least one description trigger when the description of the augmented reality scene is obtained; setting an activated state of at least one trigger to true when a condition of the trigger is met for the first time; activating the at least one trigger when a condition of the trigger is satisfied; and 49. The apparatus of claim 48, further configured to:

50. 50. The apparatus of claim 49, wherein the processor is further configured to, when a condition of the at least one trigger is satisfied, if an activation state of the trigger is set to true, activate the trigger only if a description of the trigger allows for a second activation.

51. 1. An apparatus for updating, at runtime, a first description of an augmented reality scene including behavior data items with a second description of the augmented reality scene, the apparatus comprising: for each ongoing behavior data item of the first description, if the ongoing behavior data item is not applicable to said second description; processing an interrupt action, if any, for an ongoing application of the first description; stopping the ongoing behavior; and applying the second statement; 1. An apparatus comprising a memory associated with a processor configured to execute

52. 48. An apparatus comprising one or more processors configured to perform the method of any one of claims 1 to 20, 22 to 41, 43 to 45 and 47.

53. 48. A computer readable medium containing instructions for causing one or more processors to perform the method of any one of claims 1-20, 22-41, 43-45, and 47.

54. 54. The computer-readable medium of claim 53, wherein the computer-readable medium is a non-transitory storage medium.

55. 48. A computer program product comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 20, 22 to 41, 43 to 45 and 47.

56. 1. A signal comprising scene description data for a 3D scene, the scene description data comprising: scene element information describing each of a plurality of scene elements within a scene; trigger information describing at least one trigger condition; action information describing at least one action to perform on one or more scene elements associated with the action; behavioral information, the behavioral information associating at least one of the trigger conditions with at least one of the actions; A signal containing

57. 1. A computer-readable medium containing scene description data for a 3D scene, the scene description data comprising: scene element information describing each of a plurality of scene elements within a scene; trigger information describing at least one trigger condition; action information describing at least one action to perform on one or more scene elements associated with the action; behavioral information, the behavioral information associating at least one of the trigger conditions with at least one of the actions; 1. A computer-readable medium comprising:

Images