Shared event-based updates in scene descriptions

Abstract

Some embodiments of a method may include obtaining scene description data of a three-dimensional (3D) scene, where the scene description data includes behavioral information, the behavior information includes trigger information describing at least one trigger condition, action information describing actions to be performed on scene elements in the 3D scene, and parameters indicating whether a behavior is to be shared with at least one other device; and in response to determining that (i) the parameter indicates that the behavior is to be shared with the at least one other user device, and (ii) a trigger condition of the at least one trigger condition has occurred, performing a trigger process, including: performing an action locally on the scene element; and transmitting behavior information describing the behavior to at least one other device. And additional changes may be performed according to scene updates.

Description

[0001] Cross-references to related applications This application claims priority to European Patent Application No. EP23306138 entitled “SHARED EVENT-BASED UPDATE IN SCENE DESCRIPTION”, filed on 6 July 2023, which is incorporated herein by reference in its entirety. Background Technology

[0002] In extended reality (XR) applications, scene descriptions are used to combine a clear and easily parsed description of the scene structure with some binary representation of the media content.

[0003] In time-based media streaming, the scene description itself can evolve over time to provide relevant virtual content for each sequence of the media stream. For example, for advertising purposes, a virtual bottle could be displayed on a table during a video sequence in which people are sitting around it.

[0004] This behavior can be achieved by relying on the framework defined in the document Information Technology – Coded Representation of Immersive Media – Part 14: Scene Description for MPEG Media, ISO / IEC DIS23090-14:2021 (E) (“MPEG Scene Description”).

[0005] Although the MPEG-I scene description framework ensures that timing media and corresponding associated virtual content are available at all times, it does not describe how scenes can be updated based on runtime interactivity. Summary of the Invention

[0006] The embodiments described herein include methods used in video encoding and decoding (collectively, “coding”).

[0007] Example methods according to some embodiments may include: obtaining scene description data of a three-dimensional (3D) scene, wherein the scene description data includes behavioral information, the behavioral information including: trigger information describing at least one triggering condition, action information describing an action to be performed on scene elements in the 3D scene, and parameters indicating whether the behavior should be shared with at least one other device; and in response to determining that: (i) the parameters indicate that the behavior should be shared with at least one other user device, (ii) a triggering condition in at least one triggering condition has occurred, performing a triggering process, including: locally performing an action on a scene element; and transmitting the behavioral information describing the behavior to at least one other device.

[0008] For some embodiments of the example method, performing the triggering procedure is further in response to determining that the action information indicates that the action is to be performed.

[0009] For some embodiments of the example method, the triggering procedure can further include transmitting the behavior information to a server.

[0010] For some embodiments of the example method, the behavior information can include additional information for performing the action.

[0011] For some embodiments of the example method, the additional information can include information indicating a location at which the gesture was detected.

[0012] For some embodiments of the example method, the action information can include a list of at least one set of actions, and a set of action information.

[0013] For some embodiments of the example method, the action information can include an update action, and performing the action can include updating the scene element based on the update action.

[0014] Some embodiments of the example method can further include receiving a scene update message from a server.

[0015] For some embodiments of the example method, performing the action can include performing a temporary update to the scene element.

[0016] Some embodiments of the example method can further include receiving a scene update message from a server; undoing the temporary update to the scene element; and performing a second-level action to the scene element, wherein the second-level action is indicated in the scene update message.

[0017] Some embodiments of the example method can further include receiving a scene update message from a server; merging the temporary update with the second-level action to generate a third-level action; and performing the third-level action to the scene element, wherein the second-level action is indicated in the scene update message.

[0018] For some embodiments of the example method, determining that the triggering condition has occurred can include determining that the triggering condition has produced a true result.

[0019] For some embodiments of the example method, determining that the parameter indicates that the behavior is to be shared with at least one other user device can include determining that the parameter indicates a true state.

[0020] For some embodiments of the example method, performing the action locally to the scene element is performed prior to transmitting the action information describing the action to the at least one other device.

[0021] For some embodiments of the example method, the parameter can include a shared parameter identified as “shared” according to the MPEG-I scene description framework.

[0022] For some embodiments of the example method, the parameters can include shared parameters identified as "shared" according to the glTF scene description format.

[0023] An example apparatus according to some embodiments can include a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to perform any of the above-described methods.

[0024] A further example method according to some embodiments can include performing an update to a 3D scene, wherein the 3D scene is shared with at least a first device and a second device, the first device and the second device each connected to the 3D scene, wherein the update is performed first on the first device locally and then on the second device, and wherein the first device is different from the second device.

[0025] For some embodiments of the further example method, sharing the 3D scene with at least the first device and the second device is responsive to a parameter indicating whether the behavior is to be shared with at least one other device.

[0026] A further example apparatus according to some embodiments can include a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to perform any of the above-described methods.

[0027] An additional example method according to some embodiments can include obtaining scene description data for a three-dimensional (3D) scene, wherein the scene description data includes behavior information that can include trigger information describing at least one trigger condition, action information describing a list of actions to be performed on a scene element in the 3D scene, and a parameter indicating whether the behavior is to be shared with at least one other device; and responsive to determining that (i) the parameter indicates that the behavior is to be shared with at least one other user device, and (ii) a trigger condition of the at least one trigger condition has occurred, performing a triggering process including: if the list of actions is not empty, performing at least one action in the list of actions on the scene element locally; and transmitting behavior information describing the behavior to the at least one other device.

[0028] An additional example apparatus according to some embodiments can include a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to perform the above-listed method.

[0029] A further additional example method according to some embodiments can include obtaining scene description data for a three-dimensional (3D) scene, wherein the scene description data includes behavior information including: trigger information describing at least one trigger condition, action information describing a list of actions to be performed on a scene element in the 3D scene, and a parameter indicating whether the behavior is to be shared with at least one other device; and in response to determining that: (i) the parameter indicates that the behavior is not to be shared with the at least one other device, and (ii) a trigger condition of the at least one trigger condition has occurred, performing the list of actions locally on the scene element.

[0030] A further additional example apparatus according to some embodiments can include a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to perform any of the methods listed above.

[0031] A first example method according to some embodiments can include obtaining, at a first device, scene description data for a three-dimensional (3D) scene; in response to determining that a first parameter indicates that a first behavior is to be performed locally only on the first device, performing a first update to the 3D scene locally only on the first device; and in response to determining that a second parameter indicates that a second behavior is to be shared with at least one other user device, performing an update procedure including: first performing a second update to the 3D scene locally on the first device; and thereafter performing the second update to the 3D scene on a second device, wherein the 3D scene is shared with at least the first device and the second device, the first device and the second device are each connected to the 3D scene, and wherein the first device is different from the second device.

[0032] A first example apparatus according to some embodiments can include at least one processor configured to perform any of the methods listed above.

[0033] A second example apparatus according to some embodiments can include a computer-readable medium storing instructions for causing one or more processors to perform any of the methods listed above.

[0034] A third example apparatus according to some embodiments can include at least one processor and at least one non-transitory computer-readable medium storing instructions for causing the at least one processor to perform any of the methods listed above.

[0035] An example bitstream according to some embodiments can include a bitstream of scene description data generated according to any of the methods listed above.

[0036] In additional embodiments, encoder and decoder apparatuses are provided to perform the methods described herein. An encoder or decoder apparatus can include a processor configured to perform the methods described herein. The apparatus can include a computer readable medium (e.g., non-transitory medium) storing instructions for performing the methods described herein. In some embodiments, the computer readable medium (e.g., non-transitory medium) stores video encoded using any of the methods described herein.

[0037] One or more of the present embodiments also provide a computer readable storage medium having stored thereon instructions for performing bidirectional optical flow, encoding or decoding video data according to any of the methods described above. The present embodiments also provide a computer readable storage medium having stored thereon a bitstream generated according to the methods described above. The present embodiments also provide a method and apparatus for transmitting a bitstream generated according to the methods described above. The present embodiments also provide a computer program product comprising instructions for performing any of the methods described. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1A FIG. 1 is a schematic side view illustrating an example waveguide display that can be used with extended reality (XR) applications, according to some embodiments.

[0039] Figure 1A FIG. 2 is a schematic side view illustrating an example alternative display type that can be used with extended reality applications, according to some embodiments.

[0040] Figure 1A FIG. 3 is a schematic side view illustrating an example alternative display type that can be used with extended reality applications, according to some embodiments.

[0041] Figure 1A FIG. 4 is a system diagram illustrating a set of example interfaces of a system, according to some embodiments.

[0042] Figure 1A FIG. 5 is a system diagram illustrating a set of example interfaces of an ISOBMFF file with scene description (stored as an item in glTF.json), three video tracks, one audio track, and one JSON patch update track, according to some embodiments.

[0043] Figure 2 FIG. 6 is a system diagram illustrating a set of example interfaces of an MPEG-I node hierarchy of elements that support scene interactivity, according to some embodiments.

[0044] Figure 3 FIG. 7 is a block diagram showing an example of logical relationships between trigger information, action information, and behavior information, according to some embodiments.

[0045] Figure 4 This is a schematic plan view illustrating example relationships of objects describing an extended reality scene according to some embodiments.

[0046] Figure 5 This is a message sequence diagram illustrating an example process for event-based scene updates according to some embodiments.

[0047] Figure 6 This is a flowchart illustrating an example of the processing of shared parameters according to some embodiments.

[0048] Figure 7 This is a flowchart illustrating an example process for updating a multi-user three-dimensional (3D) scene according to some embodiments.

[0049] Figure 8 This is a flowchart illustrating an example process for updating a multi-user 3D scene according to some embodiments.

[0050] Figure 9 This is a flowchart illustrating an example process for updating a multi-user 3D scene according to some embodiments.

[0051] Entities, connections, arrangements, and such depictions in various figures, as well as those described in conjunction with various figures, are presented by way of example rather than by way of limitation. Therefore, any and all statements or other indications regarding what a particular figure “depicts,” what a particular element or entity in a particular figure “is” or “has,” and any and all similar statements (which may be understood in isolation and out of context as absolute and therefore limiting) may only be correctly understood as being preceded by a clause such as “In at least one embodiment, …”. For the sake of brevity and clarity, this implicit introductory clause is not tiresomely repeated in the detailed description. Detailed Implementation

[0052] Figure 1A This illustration shows a schematic side view of an example waveguide display that can be used with extended reality (XR) applications according to some embodiments. The image is projected by an image generator 102. The image generator 102 can project the image using one or more of a variety of technologies. For example, the image generator 102 can be a laser beam scanning (LBS) projector, a liquid crystal display (LCD), a light-emitting diode (LED) display (including organic LED (OLED) or micro LED (µLED) displays), a digital light processor (DLP), a liquid crystal on silicon (LCoS) display, or other types of image generators or light engines.

[0053] The light representing image 112 generated by image generator 102 is coupled into waveguide 104 via diffraction in-coupler 106. In-coupler 106 diffracts the light representing image 112 into one or more diffraction orders. For example, ray 108 representing a portion of the bottom of the image is diffracted by in-coupler 106, and one of the diffraction orders 110 (e.g., the second order) is at an angle capable of propagating through waveguide 104 via total internal reflection. Image generator 102 displays the image according to instructions from control module 124, which operates to render image data, video data, point cloud data, or other displayable data.

[0054] At least a portion of the light 110, already coupled into waveguide 104 by diffraction-in coupler 106, is coupled out of the waveguide by diffraction-out coupler 114. At least some of the light coupled out of waveguide 104 replicates the angle of incidence of the light coupled into the waveguide. For example, in the illustration, out-coupled rays 116a, 116b, and 116c replicate the angle of the input coupled ray 108. Because the light leaving the out-coupler replicates the direction of the light entering the in-coupler, the waveguide essentially replicates the original image 112. The user's eye 118 can focus on the replicated image.

[0055] exist Figure 1A In the example, the out-coupler 114 outputs only a portion of the coupled light, where each reflection allows a single input beam (such as beam 108) to generate multiple parallel output beams (such as beams 116a, 116b, and 116c). Thus, even if the user's eye is not perfectly aligned with the center of the out-coupler, at least some of the light originating from each part of the image may reach the user's eye. For example, if eye 118 moves downwards, beam 116c may enter the eye even if beams 116a and 116b do not, so the user can still perceive the bottom of image 112 despite the positional shift. Therefore, the out-coupler 114 partially functions as an exit pupil expander in the vertical direction. The waveguide may also include one or more additional exit pupil expanders (…). Figure 1A (not shown in the image) to expand the exit pupil in the horizontal direction.

[0056] In some embodiments, the waveguide 104 is at least partially transparent to light originating outside the waveguide display. For example, at least some of the light 120 from real world objects, such as object 122, passes through the waveguide 104, allowing the user to see the real world objects while using the waveguide display. Since the light 120 from the real world objects also passes through the diffraction grating 114, there will be multiple diffraction orders, and thus multiple images. To minimize the visibility of the multiple images, it is desirable for the zeroth order of diffraction (without the bias of 114) to have a large diffraction efficiency for the light 120 and the zeroth order, while higher diffraction orders are lower in energy. Thus, in addition to expanding and out-coupling the virtual image, the out-coupler 114 is preferably configured to pass the zeroth order of the real image. In such embodiments, the image displayed by the waveguide display can appear to be superimposed on the real world.

[0057] Figure 1A is a schematic side view illustrating an example alternative display type that can be used with extended reality applications, according to some embodiments. In XR head-mounted display device 130, control module 132 controls display 134 (which can be an LCD) to display an image. The head-mounted display includes partially reflective surface 136, which reflects (and in some embodiments, both reflects and focuses) the image displayed on the LCD to make the image visible to the user. Partially reflective surface 136 also allows at least some external light to pass through, allowing the user to see their surroundings.

[0058] Figure 1A is a schematic side view illustrating an example alternative display type that can be used with extended reality applications, according to some embodiments. In XR head-mounted display device 140, control module 142 controls display 144 (which can be an LCD) to display an image. The image is focused by one or more lenses of display optics 146 to make the image visible to the user. In Figure 1A In the example of, external light does not directly reach the user’s eye. However, in some such embodiments, external camera 148 can be used to capture images of the external environment, and such images are displayed on display 144 along with any virtual content that can also be displayed.

[0059] Embodiments described herein are not limited to any particular type or structure of XR display device.

[0060] Figure 1A is a system diagram illustrating a set of example interfaces of a system, according to some embodiments. The system can be used with a system such as Figure 1ASystems, such as the system 150, can be used to implement an extended reality display device and control electronics therefor. The system 150 can be implemented as a device including various components described below and configured to perform one or more of the aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 150, singly or in combination, can be implemented in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 150 are distributed across multiple ICs and / or discrete components. In various embodiments, system 150 is communicatively coupled to one or more other systems, or other electronic devices, via, for example, a communications bus or through dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more of the aspects described in this document.

[0061] The system 150 includes at least one processor 152 configured to execute instructions loaded therein for implementing, for example, the aspects described in this document. Processor 152 can include embedded memory, input output interface, and various other circuitries as known in the art. The system 150 includes at least one memory 154 (e.g., a volatile memory device and / or a non-volatile memory device). The system 150 can include a storage device 158, which can include non-volatile memory and / or volatile memory, including, but not limited to, Electrically Erasable Programmable Read-Only Memory (EEPROM), Read-Only Memory (ROM), Programmable Read-Only Memory (PROM), Random Access Memory (RAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), flash, magnetic disk drive, and / or optical disk drive. By way of non-limiting example only, the storage device 158 can include an internal storage device, an attached storage device (including a removable and non-removable storage device) and / or a network accessible storage device.

[0062] System 150 includes an encoder / decoder module 156 that is configured, for example, to process data to provide encoded video or decoded video, and the encoder / decoder module 156 can include its own processor and memory. The encoder / decoder module 156 represents the module(s) that can be included in a device to perform the encoding and / or decoding functions. As is known, a device can include one or both of the encoding and decoding modules. Additionally, the encoder / decoder module 156 can be implemented as a separate element of system 150 or can be incorporated as a combination of hardware and software as known to those skilled in the art into processor 152.

[0063] Program code to be loaded onto processor 152 or encoder / decoder 156 to perform aspects described in this document can be stored in storage device 158 and then loaded onto memory 154 for execution by processor 152. According to various embodiments, one or more of processor 152, memory 154, storage device 158, and encoder / decoder module 156 can store one or more of the various items during the execution of the processes described in this document. Such stored items can include, without limitation, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from processing equations, formulas, operations, and operational logic.

[0064] In some embodiments, the memory internal to processor 152 and / or encoder / decoder module 156 is used to store instructions as well as provide working memory for processing needed during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device can be processor 152 or encoder / decoder module 152) is used for one or more of these functions. The external memory can be memory 154 and / or storage device 158, such as dynamic volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store, for example, the operating system of a television. In at least one embodiment, fast external dynamic volatile memory, such as RAM, is used as working memory for video encoding and decoding operations, such as for MPEG-2 (MPEG refers to the Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard developed by the Joint Video Expert Team, JVET).

[0065] Input to elements of system 150 can be provided through various input devices as indicated in block 172. Such input devices include, without limitation: (i) a radio frequency (RF) portion that receives RF signals transmitted, for example, over the air by a broadcaster, (ii) a component (COMP) input terminal (or set of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Figure 1A Other examples, not shown in FIG. 1, include composite video.

[0066] In various embodiments, the input devices of block 172 have associated respective input processing elements as known in the art. For example, the RF portion can be associated with elements suitable to: (i) select a desired frequency (also referred to as selecting a signal, or band-limiting a signal to a band of frequencies), (ii) down-convert the selected signal, (iii) band-limit again to a narrower band of frequencies to select a signal frequency band which may, in certain embodiments, be referred to as a channel, (iv) demodulate the down-converted and band-limited signal, (v) perform error correction, and (vi) demultiplex to select a desired stream of data packets. The RF portion of various embodiments includes one or more elements to perform these functions, for example, frequency selectors, signal selectors, band-limiters, channel selectors, filters, down-converters, demodulators, error correctors, and demultiplexers. The RF portion can include a tuner that performs various ones of these functions, including, for example, down-converting received signals to lower frequencies (e.g., intermediate or near-baseband frequencies) or to baseband. In one set-top box embodiment, the RF portion and its associated input processing elements receive RF signals transmitted through a wired (e.g., cable) medium, and perform frequency selection by filtering, down-converting, and filtering again to a desired frequency band. Various embodiments rearrange the order of the elements described above (and others), remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements can include inserting elements in between existing elements, such as, for example, inserting an amplifier and an analog-to-digital converter. In various embodiments, the RF portion includes an antenna.

[0067] Additionally, the USB and / or HDMI terminals can include respective interface processors for connecting the system 150 to other electronic devices across USB and / or HDMI connections. It is to be understood that various aspects of input processing (e.g., Reed-Solomon error correction) can be implemented as desired, for example, within a separate input processing IC or within the processor 152. Similarly, various aspects of USB or HDMI interface processing can be implemented as desired, for example, within a separate interface IC or within the processor 152. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, the processor 152 and the encoder / decoder 156, which operate in conjunction with memory and storage elements to process the data stream as desired for presentation on an output device.

[0068] The various elements of the system 150 can be provided within an integrated housing, within which the various elements can be interconnected and transmit data therebetween using a suitable connection arrangement 174, for example, internal buses as known in the art, including Inter-IC (I2C) buses, wiring, and printed circuit boards.

[0069] The system 150 includes a communication interface 160 that enables communication with other devices via a communication channel 162. The communication interface 160 can include, without limitation, a transceiver configured to transmit and receive data over the communication channel 162. The communication interface 160 can include, without limitation, a modem or network card, and the communication channel 162 can be implemented, for example, within wired and / or wireless media.

[0070] In various embodiments, wireless networks, such as Wi-Fi networks (e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers)) are used to stream or otherwise provide data to the system 150. The Wi-Fi signals of these embodiments are received through the communication channel 162 and the communication interface 160, which are suitable for Wi-Fi communications. The communication channel 162 of these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box to provide streamed data to the system 150, which delivers the data through the HDMI connection of the input block 172. Still other embodiments use the RF connection of the input block 172 to provide streamed data to the system 150. As indicated above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, for example, cellular networks or Bluetooth networks.

[0071] The system 150 can provide output signals to various output devices, including a display 176, speakers 178, and other peripheral devices 180. The display 176 of various embodiments includes one or more of, for example, a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. The display 176 can be used in a television, a tablet computer, a laptop computer, a cellular phone (mobile phone), or other device. The display 176 can also be integrated with other components (e.g., as in a smartphone), or separate (e.g., an external monitor for a laptop computer). The other peripheral devices 180 include one or more of, in various examples of embodiments, a standalone digital video disc (or digital versatile disc) (DVR, for both terms), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more of the peripheral devices 180 to provide functionality based on the output of the system 150. For example, the disc player performs the functionality of playing the output of the system 150.

[0072] In various embodiments, control signals are communicated between the system 150 and the display 176, speakers 178, or other peripheral devices 180 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that enable device-to-device control with or without user intervention. The output devices can be communicatively coupled to the system 1000 via dedicated connections through the respective interfaces 164, 166, and 168. Alternatively, the output devices can be connected to the system 150 using the communication channel 162 via the communication interface 160. In an electronic device such as, for example, a television, the display 176 and speakers 178 can be integrated in a single unit with the other components of the system 150. In various embodiments, the display interface 164 includes a display driver such as, for example, a timing controller (T Con) chip.

[0073] For example, if the RF portion of the input 172 is part of a separate set-top box, the display 176 and speakers 178 can instead be separate from one or more of the other components. In various embodiments in which the display 176 and speakers 178 are external components, the output signals can be provided via dedicated output connections including, for example, HDMI ports, USB ports, or COMP outputs.

[0074] The system 150 can include one or more sensor devices 168. Examples of sensor devices that can be used include one or more GPS sensors, gyroscope sensors, accelerometers, light sensors, cameras, depth cameras, microphones, and / or magnetometers. Such sensors can be used to determine information such as the location and orientation of a user. In the case that the system 150 is used as a control module for an extended reality display, such as the control modules 124, 132, the location and orientation of the user can be used in determining how to render image data, so that the user perceives the correct portion of a virtual object or virtual scene from the correct perspective. In the case of a head-mounted display device, the location and orientation of the device itself can be used to determine the location and orientation of the user for purposes of rendering virtual content. In the case of other display devices, such as a phone, tablet computer, computer monitor, or television, other inputs can be used to determine the location and orientation of the user for purposes of rendering content. For example, the user can use a touch screen, keypad or keyboard, trackball, joystick, or other input to select and / or adjust a desired viewpoint and / or viewing direction. In the case that the display device has sensors such as accelerometers and / or gyroscopes, the viewpoint and orientation for purposes of rendering content can be selected and / or adjusted based on motion of the display device.

[0075] Embodiments can be performed by computer software implemented by the processor 152, or by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. As a non-limiting example, the memory 154 can be of any type appropriate for the technology environment and can be implemented using any appropriate data storage technology, such as optical, magnetic, semiconductor-based, fixed, removable, or circuit-based memory devices. As a non-limiting example, the processor 152 can be of any type appropriate for the technology environment, and can encompass one or more of microprocessors, general purpose computers, special purpose computers, and processors based on a multi-core architecture, as non-limiting examples.

[0076] Scene description framework for XR The present principles relate generally to the field of extended reality scene description and rendering of extended reality renderings. The document is also understood in the context of formatting and playing out of extended reality applications, when rendered on end-user devices such as mobile devices or head-mounted displays (HMDs).

[0077] In XR applications, scene description is used to combine an explicit and easily parsable description of the scene structure with some binary representation of the media content.

[0078] In time-based media streams, the scene description itself can evolve over time in order to provide relevant virtual content for each sequence of the media stream. For example, for advertising purposes, a virtual bottle can be displayed during a video sequence where people are drinking.

[0079] This behavior can be achieved by relying on the framework defined in the scene description of MPEG media documents, Information technology - Coded representation of immersive media - Part 14: Scene Description for MPEG media, ISO / IEC DIS 23090-14: 2021 (E). A scene update mechanism based on the JSON patch protocol as defined in IETF RFC 6902 can be used to synchronize virtual content with MPEG media streams.

[0080] Figure 1A Figure 1 is a system diagram illustrating a set of example interfaces in an ISOBMFF file 182, a scene description (stored as an item in glTF.json), three video tracks, one audio track, and one JSON patch update track, according to some embodiments.

[0081] While the MPEG-I scene description framework ensures that timed media and corresponding relevant virtual content are available at any time, it does not provide a description of how users can interact with scene objects at runtime to obtain an immersive XR experience. As a result, user-specific XR experiences are not supported to use immersive media.

[0082] Example embodiments as described herein can be used to provide a scene description including virtual objects or light sources, but even if available, it does not necessarily display or render the virtual objects or light sources. In some embodiments, one or more of the following aspects can be considered when determining whether to display a virtual object or light source.

[0083] Spatial aspects can be considered when determining whether to display a virtual object or light source. For example, a virtual object or light source can not be displayed if the user environment is not suitable (e.g., the user is too far away from the location of the rendered timed media), or if the user is not looking in the right direction, or if the virtual object should be displayed on a user-specific area (e.g., above his left hand which has not been detected yet).

[0084] Temporal aspects can be considered when determining whether to display a virtual object or light source. For example, a virtual object or light source can not be displayed if the user is not ready or wants to trigger the display of the object by himself (e.g., using a specific gesture) until an appropriate trigger is detected.

[0085] In some embodiments, the scene description specifies which objects or light sources the user is allowed to manipulate or interact with via potential haptic feedback.

[0086] Runtime interactivity Figure 2 This is a system diagram illustrating a set of example interfaces of an MPEG-I node hierarchy 200 supporting scene interactivity elements according to some embodiments. According to this principle, in addition to as per [the relevant provisions]... Figure 3 In addition to the described node tree, behavioral metadata items (referred to herein as "behaviors") are added to the scene description. In the example embodiment, the scene description, which evolves over time, is enhanced by adding information identifying behaviors. These behaviors may be associated with predefined virtual objects on which interactivity is permitted for a user-specific XR experience when running.

[0087] In some embodiments, these behaviors evolve over time. In such embodiments, the behaviors can be updated using an existing scene description update mechanism.

[0088] In the example embodiment, the behavior is characterized by one or more of the following properties: • One or more triggers, which define the conditions that must be met for activation.

[0089] • Trigger control parameters define the logical operations between defined triggers.

[0090] • Actions implemented in response to activation.

[0091] • Action control parameters define the order in which the defined actions are executed.

[0092] • Priority numbering enables the selection of the highest priority behavior when several behaviors occur simultaneously on the same virtual object.

[0093] • Optional interrupt action, specifying how to terminate the behavior if it is no longer defined in a newly received scene update. For example, the behavior is no longer defined if the relevant object has been removed, or if the behavior is no longer associated with the current media (e.g., audio or video) sequence.

[0094] By adding these behaviors, time-related user interactivity can be defined in immersive content for XR experiences.

[0095] When the second scene description is received, some of the behaviors in the first scene description may be "in progress," meaning they have been triggered and their actions are running. The second scene description can be provided as updated metadata (i.e., metadata describing the differences between the first and second scene descriptions). The second scene description includes a node tree describing objects that may be the same as or different from those in the first scene description. Objects in the node tree of the first scene description may no longer exist in the second description. If objects related to the running actions of an ongoing behavior are missing in the second scene description, then those ongoing behaviors are no longer applicable. Similarly, if an ongoing behavior is not defined in the second description, then the ongoing behavior is no longer applicable. The interrupt action field describes how to properly interrupt the running actions of an ongoing behavior.

[0096] Figure 3 This is a block diagram illustrating an example of the logical relationship between trigger information (describing triggers 1 to n), action information (describing actions 1 to m), and behavior information (describing the relationship between triggers and actions) according to some embodiments, wherein triggers and actions may refer to one or more nodes in a scene description (such as a hierarchical scene graph).

[0097] In XR applications, scene descriptions are used to combine a clear and easily parsed description of the scene structure with some binary representation of the media content. The above section describes the action mechanism of scene descriptions. These behaviors are associated with predefined virtual objects, on which interactivity is permitted for a user-specific XR experience. Figure 3 The structure of the example behavior mechanism is illustrated. Example structure 300 shows example trigger information 302 and action information 304. Within trigger information 302 are example triggers 1 (306), 2 (308), ..., n (310). Within action information 304 are example triggers 1 (312), 2 (314), ..., n (316). Triggers 306, 308, 310 and actions 312, 314, 316 are shown as having example relationships with each node 318.

[0098] Figure 4This is a schematic plan view illustrating example relationships between objects describing an extended reality scene according to some embodiments. In this example, scene graph 400 includes descriptions of real-world objects 412, such as a “flat horizontal surface” (which could be a table, floor, or plate), and descriptions of virtual objects 414, such as an animation of a walking character. Scene graph node 414 is associated with media content item 416, which is an encoding (e.g., as a textured animated 3D mesh) of data used to render and display the walking character. Scene graph 400 also includes node 410, which describes the spatial relationships between the real-world objects described in node 412 and the virtual objects described in node 414. In this example, node 410 describes the spatial relationships that allow the character to walk on a flat surface. When an XR application is started, media content item 416 is loaded, rendered, and buffered for display upon triggering. When a flat surface is detected in the real environment by a sensor (or, in some embodiments, a camera), the application displays the buffered media content item, as described in node 410. Timing is managed by the application based on features detected in the real environment and the timing of animations. Nodes in the scene graph may also omit descriptions and simply serve as parent nodes to child nodes.

[0099] XR applications are diverse and can be applied to various contexts and real or virtual environments. For example, in industrial XR applications, when a reference object (part B of the engine) is detected in a real environment by a camera mounted on a head-mounted display, a virtual 3D content item (e.g., part A of the engine) is displayed. The 3D content item is positioned in the real world with a position and scale defined relative to the detected reference object.

[0100] For example, in an XR application for interior design, a 3D model of furniture is displayed when a given image from a catalog is detected in the input camera view. The 3D content is positioned in the real world with a location and scale defined relative to the detected reference image. In another application, some audio files might start playing when a user enters an area near a church (real or virtual rendered in an expanded real-world environment). In yet another example, an advertising jingle sound file could play when a user sees a given can of soda in a real-world environment. In outdoor gaming applications, various virtual characters may appear based on the semantics of the scene observed by the user. For example, bird characters are suitable for trees, so if the XR device's sensors detect a real object described by the semantic label "tree," birds flying around a tree could be added. In a companion application implemented with smart glasses, car noise could be played in the user's headphones when a car is detected within the user's camera's field of view to warn them of potential danger; furthermore, the sound could be spatialized so that it travels from the direction of the detected car.

[0101] XR applications can also enhance video content rather than the real environment. The video is displayed on a rendering device, and when a timed event is detected in the video, virtual objects described in the node tree are overwritten. In this context, the node tree only includes descriptions of the virtual objects.

[0102] Referring to an example embodiment of the range description of an MPEG-I scene description framework using the Khronos glTF extension mechanism, which supports additional scene description features such as node trees, the principles described herein are not limited to any particular scene description framework.

[0103] In the example implementation, the glTF scene description is extended to support interactivity. This interactivity extension is applied at the glTF scene level and is referred to as MPEG_scene_interactivity. The corresponding semantics are provided in Table 1. Name Type Usage Default Description triggers Array M [] Contains the definition of all triggers used in this scene actions Array M [] Contains the definition of all actions used in this scene behaviors Array M [] Contains the definition of all behaviors used in this scene. A behavior consists of a pair (trigger, action), the control parameters of the trigger and action, a priority weight and optionally an interrupt action Table 1: Semantics of the example MPEG_scene_interactivity extension.

[0104] In Table 1 and other semantic tables described herein, the “Use” column indicates “M” for “mandatory” features and “O” for “optional” features. However, such features may be “mandatory” or “optional” only depending on the specific proposed syntax. Features marked as “mandatory” are not necessarily required for implementing the application. For example, in some embodiments, features marked as “mandatory” exist to meet the expectations of a particular type of parsing and rendering software; however, in other embodiments, the feature may be optional, or the feature may be omitted entirely, wherein the corresponding functionality is implemented using default values ​​or is not implemented at all without departing from the scope of this disclosure.

[0105] In XR applications, scene descriptions are used to combine a clear and easily parsed description of the scene structure with some binary representation of the media content.

[0106] In time-based media streaming, the scene description itself can evolve over time to provide relevant virtual content for each sequence of the media stream. For example, for advertising purposes, a virtual bottle could be displayed on a table during a video sequence in which people are sitting around it.

[0107] This behavior can be achieved by relying on the framework defined in the document Information Technology – Coded Representation of Immersive Media – Part 14: Scene Description for MPEG Media, ISO / IEC DIS23090-14:2021 (E) (“MPEG Scene Description”).

[0108] Although the MPEG-I scene description framework ensures that timing media and corresponding associated virtual content are available at all times, it does not describe how scenes can be updated based on runtime interactivity.

[0109] Scene update in MPEG-SD Building upon the glTF description language and interactivity framework introduced in European application EP22305024.6 (“'024 application”), European application EP22306885.9 (“'885 application”) introduces new action types related to scene updates. For some embodiments, these new action types are introduced to address the description of event-based updates in scene description documents. Behavior objects may include a set of triggers (with the conditions that their activation must satisfy) and a set of actions to be performed when a trigger is activated.

[0110] Setting an action allows updating glTF elements in the scene description document. This element can be addressed using a JSON pointer, which is available in the datatracker. <dot>ietf <dot>The document JAVASCRIPT OBJECT NOTIFICATION (JSON) POINTER, IETF RFC 6901, available at https: / / tools.ietf.org / doc / html / rfc6901, describes this setting action. This setting action can replace other actions that only update a single element of the glTF tree (e.g., a SET_MATERIAL action).

[0111] Based on the JSON patch used in the MPEG scene description document, the update action allows to perform more extensive (and complex) updates. The update action contains a scene update patch that includes an array of operations (add, remove, replace, move or copy, the replace operation is similar to the setting action). Each operation specifies its type ("op"), the JSON element to address ("path"), and (if needed) the value to apply ("value") or the source JSON element ("from"). The example JSON patch operation given below in code listing 1 replaces (or sets) the scale value of node 10 to "[2, 2, 2]." Code listing 1.

[0112] For the setting and update actions, a place parameter (placeDescription) can be set to get a position from a user input (hand position or finger position on a touchscreen...). This position can be used to compute the new value of the setting action, or the position of the new node (or tree of nodes) added by the update action.

[0113] Moreover, for the setting and update actions, a parameter (e.g., a shared parameter, e.g., "shared") can be set to allow to update all the devices connected to the same session and sharing the same three-dimensional (3D) scene.

[0114] The above parameter (e.g., shared parameter) is introduced to address multi-user use cases. However, for some scenarios, the handling of this parameter lacks some details: for example, the action is first executed locally before notifying other users Moreover, this parameter can be specified at the behavior object level, for example, instead of at the action object level. It can be possible to have two behaviors that specify a reference to the same action object. For example, one behavior can have scene updates to synchronize, while the other behavior can have only local updates: For example, an action can be configured to add a new object to the scene. A behavior can be configured to associate a user input trigger (e.g., a user touching a virtual button) to this action. The new object can be shared with other users.

[0115] For some embodiments, a user can want to preview the effect of a new object in the scene before sharing it (e.g., by touching another virtual button). In that case, the second behavior can be associated with the same action, but can be executed locally only.

[0116] Based on the MPEG-I scene description framework found in the MPEG scene description document, according to some embodiments, the present application discloses herein enhancing the interactivity framework of the '024 application by specifying new parameters to address event-based scene updates in a multi-user environment. For some embodiments, the use case can be a multi-user application, such as a game, where users are connected to a server (e.g., an application server) that handles the scene synchronization of the users. The application server can be associated with one of the users that has the role of the server. As specified in the '024 application, an event can include the activation of a set of triggers associated with a behavior. Some actions can be associated with the set of triggers, and can be related to a scene update to be applied to one or more (or, for some embodiments, to all) users.

[0117] Figure 5 is a sequence diagram illustrating an example process of event-based scene updates according to some embodiments. Figure 5 A sequence diagram 500 describing a shared event scene update is described. For some embodiments, the sequence of an event can occur if the "shared" parameter is true. Each user 502, 506 receives 508, 510 [item 1] an initial scene description file from the application server 504, which includes, for example, the description of 3D scene objects and interactivity elements.

[0118] When it is checked that a set of triggers has fired 512 [item 2] at one user side, the update actions can be first executed 514 [item 3] locally to avoid latency between user interaction and corresponding scene modification. For example, the scene modification can be a change of shape or color of an object when the user touches it. The behavior information is then sent 516 [item 4] to the application server 504.

[0119] For some embodiments, the behavior information can include the index of the behavior in the array of behaviors indicated in the scene description file. The application server knows the scene and its interactivity features, and the application server 504 is able to launch 518 [item 5] the related actions and generate the scene update based on the specified behavior.

[0120] For some embodiments, the behavior information can include additional information for executing the update. For example, for a user input trigger, the additional information can include, for example, pose information related to the location where the gesture is detected, which can include the two-dimensional (2D) location of the touch on the surface or the 3D location of the user's hand.

[0121] The application server uses this information to generate scene updates 520, 522 [item 6] and forwards them to other users 502, 506. Scene updates may include patches, actions, new states, and / or identifiers. In some embodiments, a patch may be, for example, a JSON patch for updating the glTF scene graph to be applied to the scene graph to add, delete, or modify some nodes. In some embodiments, an action may be something to be performed by each user. For example, such an action may be playing or stopping a media file or animation, or activating a node. An action may be a string describing an action as specified in application '024, or an index of an action object in an array of actions as indicated in the scene description file. In some embodiments, a new state may be applied to one or more objects in the scene graph and may be, for example, activating, enabling, or pausing. In some embodiments, an identifier may be information identifying a trigger that caused the update.

[0122] Upon receipt, all users can apply scenario 524 [item 7] for updates.

[0123] glTF semantics Table 2 illustrates a new parameter, referred to as "Shared," added to the behavior object to allow the action list to be an optional parameter (e.g., the action list can be an empty set). For example, in some embodiments, the behavior's "Action" parameter can be an optional parameter that can specify an empty list of actions (e.g., if the "Shared" parameter is true). According to some embodiments, if the "Shared" parameter is true, the behavior's "Action" parameter specifies an empty list of actions. Name Type Usage Default Description triggers Array M Index of the trigger in the array of triggers considered for this behavior actions Array O [] Index of the action in the array of actions considered for this behavior. shared Boolean O False Indicates whether this behavior is only handled locally (false) or whether this behavior is handled by other connected users (true). … Table 2: Semantics of behavior.

[0124] Figure 6 This is a flowchart illustrating an example of processing shared parameters ("shared") according to some embodiments. Figure 6 This shows a state diagram illustrating how the rendering engine processes the "shared" and "action" parameters. Of course, Figure 6 Example process 600 illustrates one sequence of steps that can be performed therein, and according to some embodiments, other processes and sequences are possible.

[0125] For some embodiments, such as Figure 6 As shown, the rendering engine can check the 602 trigger activation state to see if the trigger is active, and then check (e.g., by parsing the behavior object) to see if the "share" parameter is set to true. In some embodiments, the rendering engine can parse the 604 behavior object to see if the "share" parameter is set to true (see [link to documentation]). Figure 5 (The sequence diagram in the image), and then check the triggered activation state.

[0126] If the trigger is active / becomes active and the sharing parameter is set to true, the presentation engine can look to see 606 if actions exist.

[0127] If the list of actions is not empty, the presentation engine can execute 614 each action. The actions can be related to the update to be shared: for some embodiments, the update is a temporary update, which allows the user to see the immediate effect of interacting with the scene.

[0128] The presentation engine can send 610 behavior information to the application server. For some embodiments, the behavior information can include the index of the behavior in the array of behaviors indicated in the scene description file. According to some embodiments, the application server is aware of the scene and its interactive features, and the application server is able to initiate relevant actions and generate scene updates based on the specified behavior.

[0129] For some embodiments, the behavior information can include additional information for performing the update. For example, for a user input trigger, the additional information can include pose information related to the location where the gesture was detected, which can include the 2D position of the touch on the surface or the 3D position of the user’s hand.

[0130] According to some embodiments, if the presentation engine receives a scene update message from the application server, the presentation engine updates the scene graph accordingly. For some embodiments, if the presentation engine previously performed a temporary update, the presentation engine handles such cases first. For example, the presentation engine can undo the temporary action, or can merge the temporary action with the received update.

[0131] For some embodiments, if the presentation engine parses 604 a behavior object that contains a "share” parameter set to false or no "share” parameter, the presentation engine can check the activation state of the reference trigger. For some embodiments, the presentation engine can check to see if the trigger is active, and then check to see if the "share” parameter is set to false. If the trigger is active / becomes active and the sharing parameter is set to false, the presentation engine can look to see 608 if actions exist. If the list of actions is empty, the presentation engine raises 612 an error. If the list of actions is not empty, the presentation engine executes 616 each action that causes only local changes.

[0132] Figure 7 is a flowchart illustrating an example process for updating a multi-user 3D scene, in accordance with some embodiments. For some embodiments, the example process 700 can include obtaining 702 scene description data for a 3D scene. For some embodiments of the example process 700, the scene description data can include 704 behavior information that can include: trigger information describing at least one trigger condition, action information describing a list of actions to be performed on a scene element in the 3D scene, and a parameter indicating whether the behavior is to be shared with at least one other device. For some embodiments, the example process 700 can further include, in response to determining 706 that: (i) the parameter indicates that the behavior is to be shared with at least one other user device, and (ii) a trigger condition of the at least one trigger condition has occurred, performing a trigger process. For some embodiments of the example process 700, the trigger process can further include locally performing the list of actions on the scene element; and transmitting the behavior information describing the behavior to the at least one other device.

[0133] Figure 8 is a flowchart illustrating an example process for updating a multi-user 3D scene, in accordance with some embodiments. For some embodiments, the example process 800 can include obtaining 802 scene description data for a three-dimensional (3D) scene. For some embodiments of the example process 800, the scene description data can include 804 behavior information that can include: trigger information describing at least one trigger condition, action information describing a list of actions to be performed on a scene element in the 3D scene, and a parameter indicating whether the behavior is to be shared with at least one other device. For some embodiments, the example process 800 can further include, in response to determining 806 that: (i) the parameter indicates that the behavior is to be shared with at least one other user device, and (ii) a trigger condition of the at least one trigger condition has occurred, performing a trigger process. For some embodiments of the example process 800, the trigger process can further include, if the list of actions is not empty, locally performing at least one action of the list of actions on the scene element; and transmitting the behavior information describing the behavior to the at least one other device.

[0134] Figure 9 is a flowchart illustrating an example process for updating a multi-user 3D scene, in accordance with some embodiments. For some embodiments, the example process 900 can include obtaining 902 scene description data for a three-dimensional (3D) scene. For some embodiments of the example process 900, the scene description data can include 904 behavior information, which can include: trigger information describing at least one trigger condition, action information describing a list of actions to be performed on a scene element in the 3D scene, and a parameter indicating whether the behavior is to be shared with at least one other device. For some embodiments, the example process 900 can further include locally performing the list of actions on the scene element in response to determining 906 that: (i) the parameter indicates that the behavior is to be performed locally only, and (ii) a trigger condition of the at least one trigger condition has occurred.

[0135] For some embodiments, the term "locally" can mean performing the behavior on only the device that processed the scene description data.

[0136] While the methods and systems according to some embodiments are generally discussed in the context of extended reality (XR), some embodiments can apply to any XR context, such as, for example, a virtual reality (VR) / mixed reality (MR) / augmented reality (AR) context. Moreover, although the term "head-mounted display (HMD)" is used herein according to some embodiments, some embodiments can apply to, for example, wearable devices with XR, VR, AR, and / or MR capabilities, which can or can not be attached to a head.

[0137] An example method according to some embodiments can include obtaining scene description data for a three-dimensional (3D) scene, wherein the scene description data includes behavior information, the behavior information including: trigger information describing at least one trigger condition, action information describing an action to be performed on a scene element in the 3D scene, and a parameter indicating whether the behavior is to be shared with at least one other device; and in response to determining that: (i) the parameter indicates that the behavior is to be shared with at least one other user device, (ii) a trigger condition of the at least one trigger condition has occurred, performing a trigger process including: locally performing the action on the scene element; and transmitting behavior information describing the behavior to the at least one other device.

[0138] For some embodiments of the example method, performing the trigger process is further in response to determining that the action information indicates that the action is to be performed.

[0139] For some embodiments of the example method, the trigger process can further include transmitting the behavior information to a server.

[0140] For some embodiments of the example method, the behavior information can include additional information for performing the action.

[0141] For some embodiments of the example method, the additional information can include information indicating a location at which the gesture was detected.

[0142] For some embodiments of the example method, the action information can include a list of at least one group of actions and a group of action information.

[0143] For some embodiments of the example method, the action information can include an update action, and performing the action can include updating the scene element based on the update action.

[0144] Some embodiments of the example method can further include receiving a scene update message from the server.

[0145] For some embodiments of the example method, performing the action can include performing a temporary update to the scene element.

[0146] Some embodiments of the example method can further include receiving a scene update message from the server; undoing the temporary update to the scene element; and performing a second-level action to the scene element, wherein the second-level action is indicated in the scene update message.

[0147] Some embodiments of the example method can further include receiving a scene update message from the server; merging the temporary update with the second-level action to generate a third-level action; and performing the third-level action to the scene element, wherein the second-level action is indicated in the scene update message.

[0148] For some embodiments of the example method, determining that the trigger condition has occurred can include determining that the trigger condition has produced a true result.

[0149] For some embodiments of the example method, determining that the parameter indicates that the behavior is to be shared with at least one other user device can include determining that the parameter indicates a true state.

[0150] For some embodiments of the example method, performing the action locally to the scene element is performed prior to transmitting the action information describing the action to the at least one other device.

[0151] For some embodiments of the example method, the parameter can include a shared parameter identified as "shared" according to the MPEG-I scene description framework.

[0152] For some embodiments of the example method, the parameter can include a shared parameter identified as "shared" according to the glTF scene description format.

[0153] An example apparatus according to some embodiments can include a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to perform any of the above-described methods.

[0154] Further example methods according to some embodiments may include: performing an update on a 3D scene, wherein the 3D scene is shared with at least a local first device and a second device, each of which is connected to the 3D scene, wherein the update is first performed on the local first device and then on the second device, and wherein the local first device is different from the second device.

[0155] In some embodiments of further example methods, the 3D scene is shared with at least a local first device and a second device in response to parameters indicating whether to share with at least one other device.

[0156] Further example apparatuses according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to perform any of the methods described above.

[0157] Additional example methods according to some embodiments may include: obtaining scene description data of a three-dimensional (3D) scene, wherein the scene description data includes behavioral information, the behavioral information including: trigger information describing at least one triggering condition, action information describing a list of actions to be performed on scene elements in the 3D scene, and parameters indicating whether the behavior should be shared with at least one other device; and in response to determining that: (i) the parameters indicate that the behavior should be shared with at least one other user device, and (ii) a triggering condition in at least one triggering condition has occurred, performing a triggering process, including: if the list of actions is not empty, performing at least one action in the list of actions locally on the scene element; and transmitting the behavioral information describing the behavior to at least one other device.

[0158] Additional example apparatuses according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to perform the methods listed above.

[0159] Further additional example methods according to some embodiments may include: obtaining scene description data of a three-dimensional (3D) scene, wherein the scene description data includes behavioral information, the behavioral information including: trigger information describing at least one triggering condition, action information describing a list of actions to be performed on scene elements in the 3D scene, and a parameter indicating whether the behavior should be shared with at least one other device; and in response to determining that: (i) the parameter indicates that the behavior will not be shared with at least one other device, and (ii) a triggering condition in at least one triggering condition has occurred, performing a list of actions locally on the scene elements.

[0160] A further additional example apparatus according to some embodiments can include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to perform any of the methods listed above.

[0161] A first example method according to some embodiments can include: obtaining, at a first device, scene description data for a three-dimensional (3D) scene; in response to determining that a first parameter indicates that a first behavior is to be performed locally only on the first device, performing the first update to the 3D scene locally only on the first device; and in response to determining that a second parameter indicates that a second behavior is to be shared with at least one other user device, performing an update procedure including: first performing the second update to the 3D scene locally on the first device; and then performing the second update to the 3D scene on a second device, wherein the 3D scene is shared with at least the first device and the second device, which are each connected to the 3D scene, and wherein the first device is different from the second device.

[0162] A first example apparatus according to some embodiments can include at least one processor configured to perform any of the methods listed above.

[0163] A second example apparatus according to some embodiments can include a computer- readable medium storing instructions for causing one or more processors to perform any of the methods listed above.

[0164] A third example apparatus according to some embodiments can include at least one processor and at least one non-transitory computer-readable medium storing instructions for causing the at least one processor to perform any of the methods listed above.

[0165] An example bitstream according to some embodiments can include a bitstream of scene description data generated according to any of the methods listed above.

[0166] This disclosure describes various aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are specifically described, and are generally described in a manner that can sound limiting at least to illustrate individual characteristics. However, this is for purposes of clarity of description, and does not limit the disclosure or scope of those aspects. In fact, all the different aspects can be combined and interchanged to provide further aspects. Moreover, the described aspects can also be combined and interchanged with aspects described in earlier filings.

[0167] The aspects described and contemplated in this disclosure can be implemented in many different forms. While some embodiments are specifically illustrated, other embodiments are contemplated, and the discussion of specific embodiments does not limit the breadth of implementation. At least one of the described aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting generated or encoded bitstreams. These and other aspects can be implemented as a method, an apparatus, a computer-readable storage medium having instructions stored thereon for encoding or decoding video data according to any of the described methods, and / or a computer-readable storage medium having a bitstream generated according to any of the described methods stored thereon.

[0168] In this disclosure, the terms "reconstruction" and "decoding" can be used interchangeably, the terms "pixel" and "sample" can be used interchangeably, and the terms "image," "picture," and "frame" can be used interchangeably. Generally, but not necessarily, the term "reconstruction" is used at the encoder side, while "decoding" is used at the decoder side.

[0169] The terms HDR (high dynamic range) and SDR (standard dynamic range) generally convey to one of ordinary skill in the art specific values of dynamic range. However, additional embodiments are also intended, in which a reference to HDR is understood to mean "higher dynamic range," and a reference to SDR is understood to mean "lower dynamic range." Such additional embodiments are not bound to any specific values of dynamic range that can often be associated with the terms "high dynamic range" and "standard dynamic range."

[0170] Various methods are described herein, and each of the described methods includes for implementing one or more steps or actions of the described methods. The order and / or use of a particular step and / or action can be modified or combined unless a particular order of steps or actions is required for proper operation of the method. Furthermore, terms such as "first," "second," etc. can be used to modify elements, components, steps, operations, etc. in various embodiments, such as, for example, "first decoding" and "second decoding." The use of such terms does not imply an ordering of the modified operations unless specifically required. Thus, in this example, the first decoding need not be performed before the second decoding, and can occur, for example, in a time period before, during, or overlapping the second decoding.

[0171] For example, various numerical values can be used in this disclosure. The specific values are for purposes of example, and the described aspects are not limited to these specific values.

[0172] The embodiments described herein can be implemented by computer software or by hardware and software in combination. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. As a non-limiting example, the processor can be any type of processor suitable for the technology environment, and can encompass one or more of a microprocessor, a general purpose computer, a special purpose computer, and a processor based on a multi-core architecture.

[0173] Various implementations relate to decoding. As used in this disclosure, "decoding" can encompass all or part of a process performed on a received encoded sequence in order to produce a final output suitable for display, for example. In various embodiments, such a process includes one or more of the processes typically performed by a decoder, e.g., entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such a process also includes or instead includes processes performed by decoders of various implementations described in this disclosure, e.g., extracting a picture from a tiled (packed) picture, determining an up-sampling filter to use, and then up-sampling the picture, and flipping the picture back to its intended orientation.

[0174] As further examples, in one embodiment "decoding" refers only to entropy decoding, in another embodiment "decoding" refers only to differential decoding, and in another embodiment "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or generally to a broader decoding process will be clear based on the context of the specific description.

[0175] Various implementations relate to encoding. In a similar manner as discussed above with respect to "decoding," as used in this disclosure "encoding" can encompass all or part of a process performed on an input video sequence in order to produce an encoded bitstream, for example. In various embodiments, such a process includes one or more of the processes typically performed by an encoder, e.g., partitioning, differential encoding, transform, quantization, and entropy encoding. In various embodiments, such a process also includes or instead includes processes performed by encoders of various implementations described in this disclosure.

[0176] As further examples, in one embodiment "encoding" refers only to entropy encoding, in another embodiment "encoding" refers only to differential encoding, and in another embodiment "encoding" refers to a combination of differential encoding and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or generally to a broader encoding process will be clear based on the context of the specific description.

[0177] Various embodiments relate to rate-distortion optimization. In particular, during the encoding process, a balance or trade-off between the rate and the distortion is typically considered, often given a constraint on the computational complexity. Rate-distortion optimization is typically formulated as minimizing a rate-distortion function, which is a weighted sum of the rate and the distortion. There are different approaches to solve the rate-distortion optimization problem. For example, the approach can be based on extensive testing of all encoding options, including all considered modes or encoding parameter values, and full evaluation of their encoding cost and the associated distortion of the reconstructed signal after encoding and decoding. Faster approaches can also be used to save encoding complexity, in particular by means of computing an approximate distortion based on a prediction or a prediction residual signal instead of the reconstructed signal. A mix of the two approaches can also be used, such as by using the approximate distortion for only some of the possible encoding options, and the full distortion for the other encoding options. Other approaches evaluate only a subset of the possible encoding options. More generally, many approaches employ any of a variety of techniques to perform the optimization, but the optimization is not necessarily a full evaluation of both the encoding cost and the associated distortion.

[0178] When an apparatus is presented in flow diagram form, it is to be understood that each box of the flow diagram also provides a corresponding block of a block diagram of the apparatus. Similarly, when an apparatus is presented in block diagram form, it is to be understood that each block of the block diagram also provides a corresponding flow diagram of a method / process.

[0179] Implementations and aspects described herein can be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method), implementation of the discussed features is not limited to one form or implementation. Apparatuses can be implemented in, for example, appropriate hardware, software, and firmware. The methods can be implemented in, for example, an apparatus such as a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, computers, cell phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end-users.

[0180] Reference to "one embodiment" or "an embodiment" or "one implementation" or "an implementation", as well as other variants, means that a particular feature, structure, characteristic, and so forth being described in connection with an embodiment is included in at least one embodiment. Therefore, the appearance of the phrase "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation", as well any other variations, appearing in various places throughout this disclosure are not necessarily all referring to the same embodiment.

[0181] Additionally, the disclosure can relate to "determining" various pieces of information. Determining information can include, for example, one or more of: estimating information, calculating information, predicting information, or retrieving information from memory.

[0182] Further, the present disclosure can relate to "accessing" pieces of various information. Accessing information can include one or more of, for example: receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0183] Additionally, the present disclosure can relate to "receiving" pieces of various information. As with "accessing," receiving is intended to be a broad term. Receiving information can include one or more of, for example: accessing information or retrieving information (e.g., from memory). Further, "receiving" is typically involved, in one way or another, during operations such as, for example: storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0184] It is to be understood that, in cases such as "A / B," "A and / or B," and "at least one of A and B," the use of " / ", "and / or", and "at least one of" is intended to encompass only the selection of the first listed option (A) or the only the selection of the second listed option (B) or the selection of both options (A and B). As a further example, in cases such as "A, B, and / or C" and "at least one of A, B, and C," the use of " / ", "and / or", and "at least one of" is intended to encompass only the selection of the first listed option (A), or the only the selection of the second listed option (B), or the only the selection of the third listed option (C), or the selection of the first and second listed options (A and B), or the selection of the first and third listed options (A and C), or the selection of the second and third listed options (B and C), or the selection of all three options (A and B and C). This can be extended to lists of as many items as desired.

[0185] Furthermore, among other things, as used herein the word "signaling" refers to indicating something to a corresponding decoder. For example, in certain embodiments, an encoder signals a particular one of a plurality of parameters for region-based filter parameter selection for de-artifact filtering. As such, in embodiments, the same parameter is used at both the encoder side and the decoder side. Thus, for example, the encoder can transmit (explicitly signal) a particular parameter to the decoder so that the decoder can use the same particular parameter. Conversely, if the decoder already has the particular parameter along with other parameters, signaling can be used without transmission (implicit signaling) to simply allow the decoder to know and select the particular parameter. By avoiding transmission of any actual functionality, bit savings are achieved in various embodiments. It is to be understood that signaling can be implemented in a variety of ways. For example, in various embodiments, information is signaled to a corresponding decoder using one or more syntax elements, flags, and the like. While the foregoing involves the verb form of the word "signaling," the word "signal" can also be used as a noun herein.

[0186] Implementations can produce a variety of signals formatted to carry information that can be, for example, stored or transmitted. The information can include, for example, instructions for performing a method or data generated by one of the described implementations. For example, a signal can be formatted to carry a bitstream of a described embodiment. Such a signal can be formatted, for example, as an electromagnetic wave (e.g., using a radio frequency portion of spectrum) or as a baseband signal. The formatting can include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that the signal carries can be, for example, analog or digital information. The signal can be transmitted via a variety of different wired or wireless links, as is known. The signal can be stored on a processor-readable medium.

[0187] We describe a number of embodiments. Features of the embodiments can be provided individually or in any combination across various claim classes and types. Further, embodiments can include one or more of the following features, devices or aspects across various claim classes and types, individually or in any combination: • a bitstream or signal comprising one or more of the described syntax elements or variations thereof.

[0188] • a bitstream or signal comprising a syntax conveying information generated according to any of the described embodiments.

[0189] • creating and / or transmitting and / or receiving and / or decoding a bitstream or signal comprising one or more of the described syntax elements or variations thereof.

[0190] • creating and / or transmitting and / or receiving and / or decoding according to any of the described embodiments.

[0191] • a method, a process, an apparatus, a medium storing instructions, a medium storing data, or a signal according to any of the described embodiments.

[0192] Note that various hardware elements of one or more of the described embodiments are referred to as "modules," which perform (i.e., execute, carry out, and the like) various functions described herein in connection with the respective modules. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more memory devices) deemed suitable by those of skill in the relevant art for a given implementation. Each of the described modules can also include executable instructions for performing one or more functions described as being performed by the respective module, and note that those instructions can take the form of, or include, hardware (i.e., hardwired) instructions, firmware instructions, software instructions, and / or the like, and can be stored in any suitable non-transitory computer readable medium (such as generally referred to as RAM, ROM, and the like).

[0193] Although features and elements are described above in particular combinations, one of ordinary skill in the art will appreciate that each feature or element can be used alone or in any combination with the other features and elements. In addition, the methods described herein can be implemented in a computer program, software, or firmware incorporated in a computer- readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random-access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and digital versatile disks (DVDs). The processor can be a hardware processor or a combination of hardware and software devices that execute software. The processor can be a radio-frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.< / dot> < / dot>

Claims

1. A method comprising: Obtain scene description data for a three-dimensional (3D) scene. The scene description data includes behavioral information, which includes: Trigger information describing at least one trigger condition. Describes the action information to be performed on scene elements in the 3D scene, and Parameters indicating whether the behavior should be shared with at least one other device; and In response to determining that: (i) the parameter indicates that the behavior should be shared with at least one other user device, and (ii) at least one of the trigger conditions has occurred, a triggering procedure is executed, including: Execute actions locally on scene elements; and Transmit behavioral information describing the behavior to at least one other device.

2. The method of claim 1, wherein the execution triggering process further responds to determining action information indicating that an action should be performed.

3. The method according to any one of claims 1-2, wherein the triggering process further includes transmitting behavioral information to the server.

4. The method of claim 3, wherein the behavioral information includes additional information about the action to be performed.

5. The method of claim 4, wherein the additional information includes information indicating the location where the gesture was detected.

6. The method according to any one of claims 1-5, wherein the action information includes: A list of at least one set of actions, and A set of action information.

7. The method according to any one of claims 1-6, The action information includes update actions, and The actions performed include updating scene elements based on the update actions.

8. The method of claim 7, further comprising receiving a scene update message from a server.

9. The method according to any one of claims 1-8, wherein performing the action includes performing a temporary update on the scene elements.

10. The method of claim 9, further comprising: Receive scene update messages from the server; Cancel temporary updates to scene elements; as well as Perform a second-level action on the scene element. The second-level action is indicated in the scene update message.

11. The method of claim 9, further comprising: Receive scene update messages from the server; The temporary update is merged with the second-level action to generate the third-level action; as well as Perform a third-level action on a scene element. The second-level action is indicated in the scene update message.

12. The method according to any one of claims 1-11, wherein determining that the triggering condition has occurred includes determining that the triggering condition has produced a true result.

13. The method according to any one of claims 1-12, wherein determining that parameter indication behavior is to be shared with at least one other user equipment includes determining the parameter indication true state.

14. The method according to any one of claims 1-13, wherein, Perform the action locally on the scene element before transmitting the action information describing the action to at least one other device.

15. The method according to any one of claims 1-14, wherein the parameters include shared parameters identified as "shared" according to the MPEG-I scene description framework.

16. The method according to any one of claims 1-14, wherein the parameters include shared parameters identified as "shared" according to the glTF scene description format.

17. An apparatus comprising: processor; as well as A non-transitory computer-readable medium storing instructions that, when executed by a processor, are operable to cause the apparatus to perform any one of the methods described in claims 1 to 16.

18. A method comprising: Perform an update on the 3D scene. The 3D scene is shared with at least a first local device and a second local device, each of which is connected to the 3D scene. The update is first performed on the local first device, and then on the second device. The first local device is different from the second device.

19. The method of claim 18, wherein the 3D scene is shared with at least a local first device and a second device in response to a parameter indicating whether to share it with at least one other device.

20. An apparatus comprising: processor; as well as A non-transitory computer-readable medium storing instructions that, when executed by a processor, are operable to cause the apparatus to perform any one of the methods described in claims 18 to 19.

21. A method comprising: Obtain scene description data for a three-dimensional (3D) scene. The scene description data includes behavioral information, which includes: Trigger information describing at least one trigger condition. Action information describing a list of actions to be performed on scene elements in the 3D scene, and Parameters indicating whether the behavior should be shared with at least one other device; and In response to determining that: (i) the parameter indicates that the behavior should be shared with at least one other user device, and (ii) at least one trigger condition has occurred, a triggering procedure is executed, including: If the list of actions is not empty, then at least one action from the list is executed locally on the scene element; and Transmit behavioral information describing the behavior to at least one other device.

22. An apparatus comprising: processor; as well as A non-transitory computer-readable medium storing instructions that, when executed by a processor, are operable to cause the apparatus to perform the method of claim 21.

23. A method comprising: Obtain scene description data for a three-dimensional (3D) scene. The scene description data includes behavioral information, which includes: Trigger information describing at least one trigger condition. Action information describing a list of actions to be performed on scene elements in the 3D scene, and Parameters indicating whether the behavior should be shared with at least one other device; and In response to determining that (i) the parameter indicates that the behavior will not be shared with at least one other device, and (ii) at least one of the trigger conditions has occurred, a list of actions is performed locally on the scene element.

24. An apparatus comprising: processor; as well as A non-transitory computer-readable medium storing instructions that, when executed by a processor, are operable to cause the apparatus to perform the method of claim 23.

25. A method comprising: Obtain scene description data of a three-dimensional (3D) scene on a first device; In response to determining that the first parameter indicates that the first action will be performed locally only on the first device, the first update to the 3D scene will be performed locally only on the first device; as well as In response to determining that the second parameter indicates that the second action should be shared with at least one other user equipment, an update process is performed, including: First, perform a second update to the 3D scene locally on the first device; as well as Then, a second update to the 3D scene is performed on the second device. The 3D scene is shared at least with the first device and the second device, and the first device and the second device are each connected to the 3D scene. The first device is different from the second device.

26. An apparatus comprising at least one processor configured to perform the method of any one of claims 1-16, 21, 23 and 25.

27. An apparatus comprising a computer-readable medium storing instructions for causing one or more processors to perform the method of any one of claims 1-16, 21, 23 and 25.

28. An apparatus comprising at least one processor and at least one non-transitory computer-readable medium storing instructions for causing at least one processor to perform the method of any one of claims 1-16, 21, 23 and 25.

29. A signal comprising a bitstream of scene description data generated according to any one of claims 1-16, 21, 23 and 25.

Images