System and method for providing interactivity with light sources in scene description

Abstract

Scene description data for a 3D scene is obtained. The scene description data includes: scene element information describing each of a plurality of scene elements in the scene; trigger information describing at least one trigger condition; action information describing at least one action to be performed on one or more scene elements associated with the action; and behavior information, wherein the behavior information associates at least one of the trigger conditions with at least one of the actions. If at least one of the trigger conditions has been satisfied and an action is associated with the trigger condition by the behavior information, the action is performed on at least one scene element associated with the action. In some embodiments, at least one of the scene elements is a virtual light source, and at least one of the actions is an action performed on the virtual light source, such as changing the intensity, color, or size of the virtual light source.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to European patent application No. EP22305880.1 filed on June 16, 2022, the entire contents of which are incorporated herein by reference. Background Art

[0003] Various techniques can be used to generate, process and render virtual three-dimensional (3D) scenes. Information characterizing a 3D scene, called a scene description, can be time-dependent, allowing the 3D scene to change in a manner similar to video playback. This behavior can be achieved by relying on the framework defined in the following scene description for MPEG media document: Information technology—Coded representation of immersive media—Part 14: Scene description for MPEG media, ISO / IEC DIS23090-14:2021(E). A scene update mechanism based on the JSON patch protocol as defined in IETF RFC 6902 can be used to synchronize virtual content to MPEG media streams.

[0004] Although the MPEG-I scene description framework ensures that timed media and the corresponding related virtual content are available at any time, the framework does not provide a description of how users can interact with scene objects at runtime to obtain an immersive XR experience. Therefore, the framework does not provide support for user-specific XR experiences based on immersive media. Summary of the invention

[0005] A method according to some embodiments includes obtaining scene description data of a 3D scene. The scene description data includes: scene element information, which describes each of a plurality of scene elements in the scene; trigger information, which describes at least one trigger condition; action information, which describes at least one action to be performed on one or more scene elements associated with the action; and behavior information, wherein the behavior information associates at least one of the trigger conditions with at least one of the actions. In response to a determination that at least one of the trigger conditions has been met and at least a first action of the actions is associated with the trigger condition through the behavior information, the first action is performed on at least a first scene element associated with the first action. In some embodiments, at least one of the scene elements is a virtual light source, and at least one of the actions is an action performed on the virtual light source.

[0006] According to some embodiments, actions that can be performed on a virtual light source include one or more of the following: multiplying the intensity of the virtual light source by a specified value; setting a value indicating whether the virtual light source casts a shadow; changing the color of the virtual light source; setting a distance cutoff for the virtual light source; defining an inner cone angle at which the intensity attenuation of the virtual point light begins; defining an outer cone angle at which the intensity attenuation of the virtual point light ends; changing the width of the virtual surface light source; changing the height of the virtual surface light source; applying a specified rotation to the virtual cube map light source; applying a multiplier to the irradiance coefficient of the virtual cube map light source; changing the resolution of the image of the virtual cube map light source; and defining the image to be used by the virtual cube map light source.

[0007] In some embodiments, the trigger condition may be satisfied under one or more of the following conditions: a visibility condition satisfied when a specified node in the scene description is visible to a specified camera node; a proximity condition satisfied when the distance from a reference node (e.g., a user camera) to the specified node is within specified limits; a user input condition satisfied when a specified user interaction is detected; a timing condition satisfied during a specified time period; a collider condition satisfied in response to detecting a collision between specified nodes.

[0008] In some embodiments, the behavior information identifies at least one behavior, and the behavior information for each behavior identifies at least one of the triggers and at least one of the actions. In some embodiments, the behavior information includes information indicating, for at least one behavior, whether the identified action will be performed in response to at least one of the identified triggers or whether the identified action will be performed only in response to all of the identified triggers. In some embodiments, the behavior information includes information indicating, for at least one behavior, the order in which the identified actions will be performed. In some embodiments, the behavior information includes information indicating that the identified actions will be performed simultaneously.

[0009] Example embodiments also include an apparatus comprising one or more processors configured to perform any of the methods described herein.

[0010] Example embodiments also include a computer readable medium including instructions for causing one or more processors to perform any of the methods described herein. The computer readable medium may be a non-transitory storage medium.

[0011] Example embodiments also include a computer program product comprising instructions which, when executed by one or more processors, cause the one or more processors to perform any of the methods described herein.

[0012] A signal according to some embodiments includes scene description data of a 3D scene. The scene description data includes: scene element information, the scene element information describing each of a plurality of scene elements in the scene; trigger information, the trigger information describing at least one trigger condition; action information, the action information describing at least one action to be performed on one or more scene elements associated with the action; and behavior information, wherein the behavior information associates at least one of the trigger conditions with at least one of the actions. In some embodiments, at least one of the scene elements is a virtual light source, and at least one of the actions is associated with the virtual light source.

[0013] A computer-readable medium according to some embodiments includes scene description data of a 3D scene. The scene description data includes: scene element information, the scene element information describing each of a plurality of scene elements in the scene; trigger information, the trigger information describing at least one trigger condition; action information, the action information describing at least one action to be performed on one or more scene elements associated with the action; and behavior information, wherein the behavior information associates at least one of the trigger conditions with at least one of the actions. In some embodiments, at least one of the scene elements is a virtual light source, and at least one of the actions is associated with the virtual light source. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1A is a cross-sectional schematic diagram of a waveguide display that may be used with extended reality applications according to some embodiments.

[0015] Figure 1B to Figure 1C is a cross-sectional schematic diagram of an alternative display type that may be used with an extended reality application according to some embodiments.

[0016] Figure 1D is a functional block diagram of a system used in some embodiments described herein.

[0017] Figure 1E An example illustrating the relationship between a scene description in an ISOBMFF file (stored as items in glTF.json), three video tracks, an audio track, and a JSON patch update track.

[0018] Figure 2 An example of an MPEG-I node hierarchy supporting scene interactivity elements according to some embodiments is illustrated.

[0019] Figure 3 is a flow chart illustrating the operation of an "activate once" parameter according to some embodiments.

[0020] Figure 4 is a flow chart illustrating the processing of interactive behavior in response to changes in scene description.

[0021] Figure 5 An example of a logical relationship between trigger information (describing triggers 1 to n), action information (describing actions 1 to m), and behavior information (describing the relationship between triggers and actions) is schematically illustrated. Triggers and actions may refer to one or more nodes or other scene elements in a scene description (such as a hierarchical scene graph).

[0022] Figure 6 is a flow chart illustrating a method performed in some embodiments. DETAILED DESCRIPTION

[0023] Extended Reality (XR) Display Devices .

[0024] Figure 1A An example extended reality (XR) display device is illustrated in FIG. Figure 1A 1 is a schematic cross-sectional side view of a waveguide display device in operation. The image is projected by an image generator 102. The image generator 102 may project the image using one or more of a variety of techniques. For example, the image generator 102 may be a laser beam scanning (LBS) projector, a liquid crystal display (LCD), a light emitting diode (LED) display (including an organic LED (OLED) or micro LED (μLED) display), a digital light processor (DLP), a liquid crystal on silicon (LCoS) display, or other type of image generator or light engine.

[0025] Light representing an image 112 generated by the image generator 102 is coupled into the waveguide 104 via a diffractive inner coupler 106. The inner coupler 106 diffracts the light representing the image 112 into one or more diffraction orders. For example, one of the light rays 108, which is a portion of the bottom portion of the image, is diffracted by the inner coupler 106, and one of the diffraction orders 110 (e.g., the second order) is at an angle that enables propagation through the waveguide 104 via total internal reflection. The image generator 102 displays the image as directed by the control module 124, which operates to render image data, video data, point cloud data, or other displayable data.

[0026] At least a portion of the light 110 coupled into the waveguide 104 by the diffractive inner coupler 106 is coupled out of the waveguide by the diffractive outer coupler 114. At least some of the light coupled out of the waveguide 104 replicates the angle of incidence of the light coupled into the waveguide. For example, in the illustration, the outcoupled light rays 116a, 116b, and 116c replicate the angle of the incoupled light ray 108. Because the light exiting the outer coupler replicates the direction of the light entering the inner coupler, the waveguide substantially replicates the original image 112. The user's eye 118 can focus on the replicated image.

[0027] exist Figure 1A In the example of , the outcoupler 114 allows a single input beam (such as beam 108) to generate multiple parallel output beams (such as beams 116a, 116b, and 116c) by reflecting only a portion of the outcoupled light at a time. In this way, at least some light originating from each portion of the image may reach the user's eye even if the eye is not perfectly aligned with the center of the outcoupler. For example, if the eye 118 moves downward, beam 116c may enter the eye even if beams 116a and 116b do not enter the eye, so the user can still perceive the bottom of the image 112 despite the positional shift. Thus, the outcoupler 114 operates in part as an exit pupil expander in the vertical direction. The waveguide may also include one or more additional exit pupil expanders ( Figure 1A ), to expand the exit pupil in the horizontal direction.

[0028] In some embodiments, the waveguide 104 is at least partially permeable with respect to light originating from outside the waveguide display. For example, at least some light 120 from a real-world object (such as object 122) passes through the waveguide 104, allowing the user to see the real-world object when using the waveguide display. Since the light 120 from the real-world object also passes through the diffraction grating 114, there will be multiple diffraction orders and therefore multiple images. In order to minimize the visibility of multiple images, it is expected that the zeroth order diffraction (not biased by 114) has a large diffraction efficiency for the light 120 as well as the zeroth order, while the higher diffraction orders are lower in energy. Therefore, in addition to expanding and out-coupling the virtual image, the outcoupler 114 is preferably configured to pass the zeroth order of the actual image. In such embodiments, the image displayed by the waveguide display may appear to be superimposed on the real world.

[0029] Figure 1BSchematically illustrates an alternative type of extended reality head mounted display that may be used in some embodiments. In the XR head mounted display device 1250, the control module 1254 controls the display 1256, which may be an LCD, to display an image. The head mounted display includes a partially reflective surface 1258 that reflects (and in some embodiments, both reflects and focuses) an image displayed on the LCD to make the image visible to the user. The partially reflective surface 1258 also allows at least some external light to pass through, thereby allowing the user to see their surroundings.

[0030] Figure 1C Schematically illustrates an alternative type of extended reality head mounted display that may be used in some embodiments. In the XR head mounted display device 1260, the control module 1264 controls the display 1266, which may be an LCD, to display an image. The image is focused by one or more lenses of the display optics 1268 to make the image visible to the user. Figure 1C In the example of , the external light does not directly reach the user's eyes. However, in some such embodiments, the external camera 1270 can be used to capture images of the external environment and display such images on the display 1266 along with any virtual content that can also be displayed.

[0031] The embodiments described herein are not limited to any particular type or structure of XR display device.

[0032] The extended reality display device together with its control electronics can use Figure 1D It can be achieved by a system of systems. Figure 1D It is a block diagram of an example of a system that implements various aspects and embodiments. System 1000 may be embodied as a device including various components described below, and is configured to perform one or more aspects of the aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances and servers. The elements of system 1000 may be embodied in a single integrated circuit (IC), multiple ICs and / or discrete components individually or in combination. For example, in at least one embodiment, the processing element and encoder / decoder element of system 1000 are distributed across multiple ICs and / or discrete components. In various embodiments, system 1000 is coupled to one or more other systems or other electronic devices via, for example, a communication bus or by a dedicated input and / or output port communication. In various embodiments, system 1000 is configured to implement one or more aspects of the aspects described in this document.

[0033] The system 1000 includes at least one processor 1010 configured to execute instructions loaded therein for implementing, for example, various aspects described in this document. The processor 1010 may include embedded memory, input and output interfaces, and various other circuits as known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). The system 1000 includes a storage device 1040, which may include a non-volatile memory and / or a volatile memory, including but not limited to an electrically erasable programmable read-only memory (EEPROM), a read-only memory (ROM), a programmable read-only memory (PROM), a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a flash memory, a magnetic disk drive, and / or an optical disk drive. As non-limiting examples, the storage device 1040 may include an internal storage device, an attached storage device (including a removable and non-removable storage device), and / or a network accessible storage device.

[0034] The system 1000 includes an encoder / decoder module 1030, which is configured to process data, for example, to provide encoded video or decoded video, and the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 represents one or more modules that may be included in a device to perform encoding and / or decoding functions. As is well known, a device may include one or both of an encoding module and a decoding module. Additionally, the encoder / decoder module 1030 may be implemented as an independent element of the system 1000, or may be combined in the processor 1010 as a combination of hardware and software known to those skilled in the art.

[0035] Program code to be loaded onto the processor 1010 or the encoder / decoder 1030 to perform various aspects described in this document may be stored in the storage device 1040 and subsequently loaded onto the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of the various items during the execution of the processes described in this document. Such stored items may include, but are not limited to, input video, decoded video or partially decoded video, bitstreams, matrices, variables, and intermediate or final results of processing equations, formulas, operations, and operation logic.

[0036] In some embodiments, memory internal to the processor 1010 and / or encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be memory 1020 and / or storage device 1040, such as dynamic volatile memory and / or non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store, for example, the operating system of the television. In at least one embodiment, a fast external dynamic volatile memory such as RAM is used as working memory for video coding and decoding operations, such as MPEG-2 (MPEG refers to Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard developed by the Joint Video Experts Team (JVET)).

[0037] Input to the elements of system 1000 may be provided through various input devices as indicated in block 1130. Such input devices include, but are not limited to: (i) a radio frequency (RF) section that receives an RF signal transmitted, for example, over the air by a broadcaster, (ii) a component (COMP) input terminal (or a set of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Figure 1C Other examples not shown include composite video.

[0038] In various embodiments, the input device of block 1130 has associated corresponding input processing elements as known in the art. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also referred to as selecting a signal, or band limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) again band-limiting to a narrower frequency band to select a signal band that may be referred to as a channel in certain embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired packet stream. The RF section of various embodiments includes one or more elements for performing these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a near-baseband frequency) or to baseband. In a set-top box embodiment, the RF part and its associated input processing element receive the RF signal that sends by wired (for example, cable) medium, and by filtering, down-conversion and filtering to required frequency band again to perform frequency selection.Each embodiment rearranges the order of (and other) element described above, removes some elements in these elements, and / or adds other elements that perform similar or different functions.Adding element can be included in and inserts element between existing element, for example, inserts amplifier and analog-to-digital converter.In various embodiments, the RF part comprises antenna.

[0039] In addition, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 1000 to other electronic devices across the USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) may be implemented, for example, in a separate input processing IC or in the processor 1010, if necessary. Similarly, USB or HDMI interface processing aspects may be implemented, if necessary, in a separate interface IC or in the processor 1010. The demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, the processor 1010 and the encoder / decoder 1030, which operate in conjunction with memory and storage elements to process the data streams as needed for presentation on an output device.

[0040] The various components of system 1000 may be disposed within an integrated housing. Within the integrated housing, the various components may be interconnected and transmit data between the components using a suitable connection arrangement 1140 (e.g., an internal bus known in the art, including an inter-chip (I2C) bus, wiring, and a printed circuit board).

[0041] The system 1000 includes a communication interface 1050 that enables communication with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to send and receive data through the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or a network card, and the communication channel 1060 may be implemented, for example, within a wired and / or wireless medium.

[0042] In various embodiments, data is streamed or otherwise provided to the system 1000 using a wireless network such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signals of these embodiments are received by a communication channel 1060 and a communication interface 1050 suitable for Wi-Fi communication. The communication channel 1060 of these embodiments is typically connected to an access point or router that provides access to an external network (including the Internet) for allowing streaming applications and other over-the-top communications. Other embodiments provide streaming data to the system 1000 using a set-top box that delivers data via an HDMI connection of an input box 1130. Still other embodiments provide streaming data to the system 1000 using an RF connection of an input box 1130. As described above, various embodiments provide data in a non-streaming manner. In addition, various embodiments use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.

[0043] System 1000 can provide output signals to various output devices, including display 1100, speaker 1110 and other peripheral devices 1120. Display 1100 of various embodiments includes one or more of, for example, a touch screen display, an organic light emitting diode (OLED) display, a curved display and / or a foldable display. Display 1100 can be used for television, tablet computer, laptop computer, cellular phone (mobile phone) or other devices. Display 1100 can also be integrated with other components (for example, as in a smart phone), or it can be independent (for example, an external monitor for a laptop computer). In various examples of embodiments, other peripheral devices 1120 include one or more of an independent digital video disc (or digital versatile disc) (DVR, a term for both), a CD player, a stereo system and / or a lighting system. Various embodiments use one or more peripheral devices 1120, which provide functions based on the output of system 1000. For example, a CD player performs the function of playing the output of system 1000.

[0044] In various embodiments, control signals are transmitted between the system 1000 and the display 1100, speaker 1110, or other peripheral device 1120 using signaling such as AV.Link, consumer electronics control (CEC), or other communication protocols that enable device-to-device control with or without user intervention. Output devices can be communicatively coupled to the system 1000 via dedicated connections through corresponding interfaces 1070, 1080, and 1090. Alternatively, the output devices can be connected to the system 1000 via the communication interface 1050 using the communication channel 1060. The display 1100 and the speaker 1110 can be integrated into a single unit with other components of the system 1000 in an electronic device (such as, for example, a television). In various embodiments, the display interface 1070 includes a display driver, such as, for example, a timing controller (T Con) chip.

[0045] For example, if the RF portion of input 1130 is part of a stand-alone set-top box, the display 1100 and speaker 1110 may alternatively be independent of one or more of the other components. In various embodiments where the display 1100 and speaker 1110 are external components, the output signal may be provided via a dedicated output connection including, for example, an HDMI port, a USB port, or a COMP output.

[0046] The system 1000 may include one or more sensor devices 1095. Examples of sensor devices that can be used include one or more GPS sensors, gyro sensors, accelerometers, light sensors, cameras, depth cameras, microphones, and / or magnetometers. Such sensors can be used to determine information such as the position and orientation of the user. In the case where the system 1000 is used as a control module (such as control module 124, 1254) of an extended reality display, the position and orientation of the user can be used to determine how to render image data so that the user perceives the correct part of the virtual object or virtual scene from the correct viewpoint. In the case of a head-mounted display device, the position and orientation of the device itself can be used to determine the position and orientation of the user in order to render virtual content. In the case of other display devices such as phones, tablets, computer monitors, or televisions, other inputs can be used to determine the position and orientation of the user for the purpose of rendering content. For example, a user can select and / or adjust a desired viewpoint and / or viewing direction by using a touch screen, a keypad or keyboard, a trackball, a joystick, or other inputs. In the case where the display device has sensors such as accelerometers and / or gyroscopes, the viewpoint and orientation for the purpose of rendering content can be selected and / or adjusted based on the movement of the display device.

[0047] These embodiments may be performed by computer software implemented by the processor 1010 or by hardware or by a combination of hardware and software. As a non-limiting example, these embodiments may be implemented by one or more integrated circuits. As a non-limiting example, the memory 1020 may be of any type suitable for the technical environment and may be implemented using any appropriate data storage technology (such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory). As a non-limiting example, the processor 1010 may be of any type suitable for the technical environment and may include one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0048] XR scene description framework .

[0049] In XR applications, scene descriptions are used to combine an explicit and easily parseable description of the scene structure with some binary representation of the media content.

[0050] In time-based media streaming, the scene description itself can evolve over time to provide relevant virtual content for each sequence of the media stream. For example, for advertising purposes, a virtual bottle can be shown during a video sequence of people drinking.

[0051] This behavior can be achieved by relying on the framework defined in the following scene description for MPEG media document: Information technology — Coded representation of immersive media — Part 14: Scene description for MPEG media, ISO / IEC DIS 23090-14:2021(E). A scene update mechanism based on the JSON patch protocol as defined in IETF RFC 6902 can be used to synchronize virtual content to the MPEG media stream.

[0052] Figure 1E An example illustrating the relationship between a scene description in an ISOBMFF file (stored as items in glTF.json), three video tracks, an audio track, and a JSON patch update track.

[0053] Although the MPEG-I scene description framework ensures that timed media and the corresponding related virtual content are available at any time, the framework does not provide a description of how users can interact with scene objects at runtime to obtain an immersive XR experience. Therefore, user-specific XR experiences are not supported to consume immersive media.

[0054] Example embodiments as described herein may be used to provide a scene description that includes a virtual object or light source, but even if available, the virtual object or light source may not necessarily be displayed or rendered. In some embodiments, one or more of the following aspects may be considered when determining whether to display a virtual object or light source.

[0055] Spatial aspects may be considered when determining whether to display a virtual object or light source. For example, a virtual object or light source may not be displayed if the user environment is not suitable (e.g., the user is too far away from the location of the rendered timed media), or if the user is not looking in the correct direction, or if the virtual object should be displayed on a specific area of ​​the user (e.g., above his left hand which has not yet been detected).

[0056] Time aspects may be considered when determining whether to display a virtual object or light source. For example, if the user is not yet ready or willing to trigger the display of the object themselves (e.g., using a specific gesture), the virtual object or light source may not be displayed until the appropriate trigger is detected.

[0057] In some embodiments, it is specified in the scene description which objects or light sources the user is allowed to manipulate or interact with, via potential tactile feedback.

[0058] Runtime interactivity .

[0059] In an example embodiment, the scene description that evolves over time is enhanced by adding information identifying behaviors. These behaviors may be associated with predefined virtual objects, on which runtime interactivity is allowed to achieve user-specific XR experiences. Figure 2 An example of an MPEG-I node hierarchy supporting scene interactivity elements is illustrated in .

[0060] In some embodiments, these behaviors can evolve over time. In such embodiments, the behaviors can be updated through an existing scene description update mechanism.

[0061] In an example embodiment, behavior is characterized by one or more of the following attributes:

[0062] -One or more triggers that define the conditions to be met for activation.

[0063] - Trigger control parameters that define logical operations between defined triggers.

[0064] -An action implemented in response to activation of a trigger.

[0065] - Action control parameters that define the execution order of the defined actions.

[0066] - A priority number that enables selection of the behavior with the highest priority in case several behaviors on the same virtual object occur simultaneously.

[0067] - Optional interrupt actions specifying how to terminate the behavior when it is no longer defined in a newly received scene update. For example, the behavior is no longer defined if the associated object has been deleted or if the behavior is no longer relevant to the current media (e.g., audio or video) sequence.

[0068] By adding these behaviors, you define how users can interact with the immersive content of an XR experience at runtime.

[0069] Example implementations are described with reference to the scope of the MPEG-I scene description framework using the Khronos glTF extension mechanism, which supports additional scene description features. However, the principles described herein are not limited to a particular scene description framework.

[0070] In an example implementation, the glTF scene description is extended to support interactivity. The interactivity extension is applied at the glTF scene level and is called MPEG_scene_interactivity. The corresponding semantics are provided in Table 1.

[0071]

[0072] Table 1: Semantics of an example MPEG_scene_interactivity extension .

[0073] #

[0074] In Table 1 and other semantic tables described herein, the "Usage" column indicates that "M" represents a "mandatory" feature, and "O" represents an "optional" feature. However, it should be understood that such features can be "mandatory" or "optional" solely based on the specific syntax proposed. Features marked as "mandatory" are not necessarily features required to implement the present invention. For example, in some embodiments, there are features marked as "mandatory" to meet the expectations of a particular type of parsing and rendering software; however, in other embodiments, without departing from the scope of the present disclosure, the feature may be optional, or may be omitted entirely, with the corresponding functionality being implemented using default values ​​or not implemented at all.

[0075] Table 2 shows parameters of an example of a trigger.

[0076]

[0077]

[0078]

[0079] Table 2: Trigger semantics .

[0080] #

[0081] In some embodiments, instead of using string parameters associated with the Khronos OpenXR Interaction Profile path syntax to define user body parts and gestures for USER_INPUT triggers, other syntax formats may be used, such as a syntax format for representation of haptic objects, where vertex arrays (geometric models) and binary masks (body part masks) are used to specify where haptic effects should be applied.

[0082] Example parameters for the actions are shown in Table 3.

[0083]

[0084]

[0085]

[0086] Table 3: Action semantics .

[0087] #

[0088] Example parameters for the behaviors are shown in Table 4.

[0089]

[0090] Table 4: Behavior semantics .

[0091] #

[0092] During runtime, the application can iterate over each defined behavior and follow Figure 3 The process illustrated in the example checks the implementation of the relevant triggers to operate. Figure 3 is a flow chart of a processing model for activating a trigger. At 302, a determination is made as to whether a trigger condition of a related trigger is satisfied. If the trigger condition is not satisfied, the result of the determination is stored, for example, an activation state variable is set to false at 304. If the trigger condition is satisfied, a determination is made at 306 as to whether the trigger has been activated, for example, by checking whether the activation state has been set to true. If not, the activation state of the trigger is set to true at 308, and the trigger is activated at 312. If the determination at 306 indicates that the trigger has been activated, a determination is made at 310 as to whether the related trigger is intended to be activated only once (as opposed to, for example, a trigger intended to be activated as long as the trigger condition exists). If the trigger is intended to be activated only once, and if the trigger has been activated (as determined at 306), the trigger will not be activated again. Otherwise, the trigger is activated again at 312.

[0093] In response to activation of a defined behavior trigger, a corresponding action is initiated. A behavior has an ongoing state between initiation and completion of its defined actions.

[0094] In some embodiments, when several behaviors occur simultaneously to affect the same node (or other scene element) at the same time, only the behavior with the highest priority is initiated.

[0095] In some embodiments, in response to receiving a new scene description, the application follows the Figure 4 The process shown. Figure 4 is a flow chart illustrating a processing model implemented in response to the receipt of a new scene description. After obtaining the new scene description (402), a determination is made as to whether the behavior has an ongoing state (404). In response to the behavior not having an ongoing state, the new scene description is applied (410). If the behavior does have an ongoing state, a determination is made (406) as to whether the behavior is still defined. For example, if the relevant object has been deleted or if the behavior is no longer relevant to the current media (e.g., audio or video) sequence, the behavior is no longer defined. If the behavior is no longer defined, an interrupt action (412) is processed and the ongoing behavior (414) is stopped before the new scene description (410) is applied. If it is determined at 406 that the behavior is still defined, the ongoing behavior (408) is continued when the new scene description (410) is applied.

[0096] Example interactive object .

[0097] As an example of an interactive virtual object according to some embodiments, a virtual 3D advertisement object may be continuously displayed and transformed during a defined period (e.g., between 20 seconds and 40 seconds) in an MPEG media sequence in between. In this example, once the user's left hand is detected, the virtual 3D object is placed on the user's left hand and continuously follows it.

[0098] In this example, two behaviors are defined to support this interactivity scenario. The first behavior has the following parameters:

[0099] o A single trigger associated with a time sequence of between 20 seconds and 40 seconds of continuous activation of MPEG media ("activateOnce" = False).

[0100] ○ Two sequential actions to enable and transform the virtual 3D object (Node 0).

[0101] The second line has the following parameters:

[0102] o A single trigger associated with detection of the user's left hand with continuous activation ("activateOnce" = false).

[0103] ○ A single action of placing a virtual object (Node 0) on the user's left hand.

[0104] Both behaviors define the same interrupt action, which disables the virtual object (node ​​0).

[0105] Since these two behaviors affect the same virtual object (node ​​0), a higher priority is set for the second behavior associated with the user gesture (left hand gesture) to execute the desired interactivity scene. In this example, to implement the desired behavior, the scene interactivity information can be used, which can be provided in a JSON format such as the following or in an alternative format.

[0106]

[0107]

[0108] Runtime interactivity for lights .

[0109] In XR applications, scene descriptions are used to combine an explicit and easily parsable description of the scene structure with some binary representation of the media content. The above section describes the action mechanism used for scene descriptions. These actions are associated with predefined virtual objects, based on which runtime interactivity is allowed for user-specific XR experiences. Figure 5 The structure of an example behavior mechanism is illustrated.

[0110] Some embodiments extend the interactivity of virtual objects as described above to provide lighting interactions, such as light intensity depending on the proximity of the user (e.g., triggered by a proximity trigger), light color depending on the time of day, and other types of interactivity. Example embodiments enable specialized lighting interactions for any light described by the scene.

[0111] In some embodiments, the action semantics of the scene description include a SET_LIGHT field for dedicated light actions.

[0112] In Table 5 below, the term cubemap refers to the array of image-based lights (also called IBL) described in the EXT_lights_image_based extension.

[0113] In an example embodiment, the scene description enhancements include one or more of the following:

[0114] ○ Parameter (number) by which the light node's current intensity level is multiplied.

[0115] ○ Parameter (boolean) that allows enabling and disabling shadow casting from the light.

[0116] ○ Parameters (array) that allow changing the light color.

[0117] o A parameter (a number) that defines the distance at which the light intensity can be considered to have reached zero.

[0118] ○ A parameter (number) that changes the width of the area light.

[0119] ○ A parameter (number) that changes the height of the area light.

[0120] Allows changing the cubemap's rotation parameters (quaternion).

[0121] ○Parameters (array) that set the irradiance coefficient.

[0122] ○ Parameter (number) used to change the resolution of the images that make up the cubemap.

[0123] ○ Parameters (array) of images used to replace the cubemap.

[0124] ○ Parameters (array) indicating which optical node is being referenced. In some embodiments,

[0125] This parameter may be mandatory.

[0126] In the Khronos GLTF example, there are two light extensions involved: the KHR_lights_punctual extension that supports directional, point and spot lights, and the EXT_lights_image_based extension that supports arrays of image-based lights (in other words, these are cubemaps that describe the specular radiance of the scene, l=2 spherical harmonic coefficients for diffuse irradiance, rotation and brightness values).

[0127] The current implementation of KHR_lights_punctual does not support area lights yet, but since this light type is useful for integrating virtual objects in a real environment and for describing real lights, and due to its presence in game engines like Unity or Unreal Engine, the glTF format is expected to add support for area lights.

[0128] This area light can be a square shaped, with the following parameters (added to every light parameter that already exists): width and height, and whether it will inherit the parent transform like other lights.

[0129] Example implementations are described with reference to the MPEG-I scene description architecture using the Khronos glTF extension mechanism to support additional scene description features. However, the principles described herein are not limited to any particular scene description framework.

[0130] In an example embodiment, a SET_LIGHT action is provided in addition to the actions already defined in Table 3. An example of the semantics of the SET_LIGHT action at the node level is provided in Table 5. The example SET_LIGHT action indicates a modification of at least one lighting property of a virtual light source, such as the intensity, color, size, range, orientation, and / or shadow properties of the virtual light source.

[0131]

[0132] Table 5: Semantics of example light actions .

[0133] As an example, information indicating runtime interactivity of a light source may be represented in a GLTF file using the JSON (JavaScript Object Notation) format, as shown below. However, the principles described herein are not limited to any particular syntax format.

[0134]

[0135]

[0136]

[0137] Figure 6 is a flowchart illustrating a method performed according to some embodiments. Scene description data for a 3D scene is obtained. The scene description information may be in GLTF format or another format. The scene description data may include: scene element information, which describes each of a plurality of scene elements in the scene; trigger information, which describes at least one trigger condition; action information, which describes at least one action to be performed on one or more scene elements associated with the action; and behavior information, wherein the behavior information associates at least one of the trigger conditions with at least one of the actions. In some embodiments, actions are performed on scene elements associated with corresponding nodes in a hierarchical scene description graph. Alternatively or additionally, one or more actions may be performed on scene elements that are not associated with a particular node, such as an animation or MPEG_media element contained in the scene description.

[0138] In such Figure 6In the illustrated example method, scene description data of a 3D scene is obtained (602). The scene description data includes scene element information, which describes each scene element of a plurality of scene elements in the scene, wherein at least one of the scene elements is a virtual light source. The scene description data also includes trigger information, which describes at least a first trigger condition. The scene description data also includes action information, which describes at least a first action and associates one or more scene elements (including a virtual light source) in the scene elements with at least the first action. The scene description data also includes behavior information. The behavior information associates at least the first trigger condition with at least the first action. The behavior information may also identify additional behaviors that associate other trigger conditions with other actions. At 604, the scene is rendered based on the scene description data. At 606 (which may be performed during rendering of the scene), the scene (and / or user interaction with the scene) is monitored for trigger conditions, including monitoring for at least the first trigger condition. At 608, a determination is made as to whether the trigger condition is met. If the trigger condition is met, then at 610, one or more actions associated with the trigger are identified. This identification may be made based on behavior information that associates a trigger condition with a corresponding action. At 612, the identified action is performed on the node (or other scene element) identified in the action. For example, where the action is associated with a virtual light source, the action is performed on the virtual light source. Rendering of scene 604 may continue, where the appearance or other features of the scene are updated according to the one or more actions performed (e.g., where properties of one or more virtual light sources have changed based on the action).

[0139] In some embodiments, characteristics of user interactivity are monitored to detect whether a trigger condition is met. The monitored characteristics may include, for example, the position of a user's camera or viewpoint (as determined by user input and / or by sensors such as gyroscopes and accelerometers, among other possibilities), user gestures (as detected by a camera, wrist-mounted or handheld accelerometer, or other sensor), or other user input.

[0140] In response to determining that at least one of the trigger conditions has been satisfied and at least a first action in the actions is associated with the trigger condition through the behavior information, the first action is performed on at least a first scene element associated with the first action. In some embodiments, at least one of the scene elements is a virtual light source, and at least one of the actions is an action performed on the virtual light source.

[0141] In some embodiments, the action results in the modification of one or more scene elements in the scene description of the 3D scene. The method in some embodiments includes rendering the 3D scene according to the changed scene description. In other embodiments, the modified scene description is provided to a separate renderer for rendering the 3D scene according to conventional 3D rendering techniques. The 3D scene can be displayed on any display device (such as the display device described herein or other display devices). In some embodiments, an optical perspective or video perspective display can be used to display the 3D scene as an overlay together with the real world scene. It should be noted that the display of the 3D scene referred to herein includes using a 2D display device to display a 2D projection of the 3D scene.

[0142] Further implementation plans .

[0143] The present disclosure describes various aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are specifically described, and at least individual characteristics are shown, usually in a manner that may sound limited. However, this is for clarity in the description and does not limit the disclosure or scope of those aspects. In fact, all different aspects can be combined and interchanged to provide further aspects. In addition, these aspects can also be combined and interchanged with aspects described in previous submissions.

[0144] The aspects described and contemplated in this disclosure can be implemented in many different forms. Although some embodiments are specifically shown, other embodiments are contemplated, and the discussion of specific embodiments does not limit the breadth of specific implementations. At least one of these aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to sending a generated or encoded bitstream. These and other aspects can be implemented as methods, devices, computer-readable storage media having instructions for encoding or decoding video data according to any of the methods, and / or computer-readable storage media having a bitstream generated according to any of the methods stored thereon.

[0145] Various methods are described herein, and each method in the method includes one or more steps or actions for implementing the method. Unless the correct operation method requires a specific order of steps or actions, the order and / or purposes of specific steps and / or actions can be modified or combined. In addition, terms such as "first", "second" etc. can be used to modify elements, parts, steps, operations, etc. in various embodiments, such as, for example, "first decoding" and "second decoding". Unless specifically required, the use of such terms does not imply the ordering of the modification operation. Therefore, in this example, the first decoding does not need to be performed before the second decoding, and can, for example, occur before, during, or in overlapping time periods of the second decoding.

[0146] For example, various numerical values ​​may be used in the present disclosure. Specific values ​​are for illustrative purposes, and the aspects are not limited to these specific values.

[0147] The embodiments described herein may be performed by a processor or other hardware or computer software implemented by a combination of hardware and software. As a non-limiting example, these embodiments may be implemented by one or more integrated circuits. As a non-limiting example, the processor may be any type suitable for the technical environment and may include one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0148] When the figures are presented as flow charts, it should be understood that they also provide block diagrams of corresponding devices. Similarly, when the figures are presented as block diagrams, it should be understood that they also provide flow charts of corresponding methods / processes.

[0149] The specific implementations and aspects described herein may be implemented in, for example, methods or processes, devices, software programs, data streams, or signals. Even if discussed only in the context of a single form of specific implementation (e.g., discussed only as a method), the specific implementation of the features discussed may also be implemented in other forms (e.g., devices or programs). The device may be implemented in, for example, appropriate hardware, software, and firmware. The method may be implemented in, for example, a processor generally referring to a processing device, which includes, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes a communication device, such as, for example, a computer, a cellular phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate the communication of information between end users.

[0150] Reference to "one embodiment" or "an embodiment" or "one implementation" or "an implementation" and other variations thereof mean that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearance of the phrase "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation" and any other variations thereof in various places throughout this disclosure are not necessarily all referring to the same embodiment.

[0151] Additionally, the present disclosure may refer to "determining" various pieces of information. Determining information may include, for example, one or more of estimating the information, calculating the information, predicting the information, or retrieving the information from a memory.

[0152] Additionally, the present disclosure may refer to "accessing" various pieces of information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0153] Furthermore, the present disclosure may refer to "receiving" various pieces of information. Like "accessing," receiving is intended to be a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Furthermore, "receiving" generally involves, in one way or another, during operations such as, for example, storing information, processing information, sending information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0154] It should be understood that, for example, in the case of "A / B," "A and / or B," and "at least one of A and B," the use of any of the following " / ," "and / or," and "at least one of" is intended to encompass selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such phrases are intended to encompass selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first listed option and the second listed option (A and B), or only the first listed option and the third listed option (A and C), or only the second listed option and the third listed option (B and C), or all three options (A and B and C). This can be extended to as many items as listed.

[0155] Moreover, as used herein, the word "signal" particularly indicates something to a corresponding decoder. For example, in some embodiments, the encoder signals

[0156] The filter parameter selection based on the region is used for a specific parameter in a plurality of parameters of the artifact removal filtering. In this way, in one embodiment, the same parameters are used on both the encoder side and the decoder side. Therefore, for example, the encoder can send (explicit signaling) specific parameters to the decoder so that the decoder can use the same specific parameters. On the contrary, if the decoder already has specific parameters and other parameters, signaling can be used without sending (implicit signaling) to simply allow the decoder to know and select specific parameters. By avoiding sending any actual function, bit saving is achieved in various embodiments. It should be understood that signaling can be implemented in various ways. For example, in various embodiments, one or more syntax elements, flags, etc. are used to signal the information to the corresponding decoder. Although the verb form of the word "signaling (signal)" is related to the above, the word "signal (signal)" can also be used as a noun in this article.

[0157] Specific implementations may generate various signals, formatting these signals to carry information that can be, for example, stored or transmitted. The information may include, for example, instructions for executing the method or data generated by one of the described specific implementations. For example, the signal may be formatted to carry a bit stream of the described embodiment. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is well known, the signal may be sent via a variety of different wired or wireless links. The signal may be stored on a processor readable medium.

[0158] A number of embodiments are described. Features of these embodiments may be provided individually or in any combination in various claim categories and types. Further, embodiments may include one or more of the following features, devices, or aspects, individually or in any combination in various claim categories and types:

[0159] • A bitstream or signal comprising one or more of the described syntax elements or variants thereof.

[0160] - A bitstream or signal comprising syntax conveying information generated according to any of the described embodiments.

[0161] • Creating and / or sending and / or receiving and / or decoding a bitstream or signal comprising one or more of the described syntax elements or variations thereof.

[0162] · Creating and / or sending and / or receiving and / or decoding according to any of the embodiments described.

[0163] A method, process, apparatus, medium storing instructions, medium storing data or signal according to any one of the embodiments described.

[0164] It should be noted that the various hardware elements of one or more of the described embodiments are referred to as "modules" that perform (i.e., execute, implement, etc.) the various functions described herein in conjunction with the corresponding modules. As used herein, a module includes hardware that is considered suitable for a given specific implementation (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more memory devices). Each of the modules described may also include executable instructions for performing one or more functions described as being performed by the corresponding module, and it should be noted that these instructions may take the form of or include the following instructions: hardware (i.e., hard-wired) instructions, firmware instructions, software instructions, etc., and may be stored in any suitable one or more non-transitory computer-readable media (such as commonly referred to as RAM, ROM, etc.).

[0165] Although features and elements are described above in particular combinations, each feature or element may be used alone or in any combination with other features and elements. In addition, the methods described herein may be implemented in a computer program, software, or firmware incorporated in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media (such as built-in hard disks and removable disks), magneto-optical media, and optical media (such as CD-ROM disks and digital versatile disks (DVDs)). A processor associated with software may be used to implement a radio frequency transceiver for a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. A method, comprising: Obtaining scene description data of a 3D scene, wherein the scene description data comprises: Scene element information, the scene element information describing each scene element of a plurality of scene elements in the scene, at least one scene element of the scene elements being a virtual light source; Trigger information, the trigger information describing at least a first trigger condition; action information describing at least a first action indicating a modification of at least one property of the virtual light source; and behavior information, wherein the behavior information associates at least the first trigger condition with at least the first action; and In response to a determination that at least the first trigger condition has been met, at least the first action is performed on at least the virtual light source. 2 . The method of claim 1 , wherein the first action comprises multiplying the intensity of the virtual light source by a specified value.

3. The method of claim 1 or 2, wherein the first action comprises setting a value indicating whether the virtual light source casts a shadow.

4. The method according to any one of claims 1 to 3, wherein the first action comprises changing the color of the virtual light source.

5. The method of any one of claims 1 to 4, wherein the first action comprises setting a distance cutoff for the virtual light source. 6 . The method according to claim 1 , wherein the virtual light source is a virtual point light source, and the first action comprises defining an inner cone angle at which intensity decay of the virtual point light source begins.

7. The method according to any one of claims 1 to 6, wherein the virtual light source is a virtual point light source, and the first action comprises defining an outer cone angle at which an intensity decay of the virtual point light source ends. 8 . The method according to claim 1 , wherein the virtual light source is a virtual surface light source, and the first action comprises changing a width or a height of the virtual surface light source.

9. The method of any one of claims 1 to 5, wherein the virtual light source is a virtual cubemap light source, and the first action comprises applying a specified rotation to the virtual cubemap light source.

10. A method according to any one of claims 1 to 5 or 9, wherein the virtual light source is a virtual cubemap light source and the first action comprises applying a multiplier to an irradiance coefficient of the virtual cubemap light source.

11. A method according to any one of claims 1 to 5 or 9 to 10, wherein the virtual light source is a virtual cubemap light source and the first action comprises changing the resolution of an image of the virtual cubemap light source.

12. A method according to any one of claims 1 to 5 or 9 to 11, wherein the virtual light source is a virtual cubemap light source and the first action comprises defining an image to be used by the virtual cubemap light source.

13. The method according to any one of claims 1 to 12, wherein the first trigger condition is a visibility condition that is satisfied when a specified scene element is visible to a specified camera node.

14. The method according to any one of claims 1 to 13, wherein the first trigger condition is a proximity condition that is satisfied when the distance from the reference node to the specified scene element is within a specified limit.

15. The method according to any one of claims 1 to 14, wherein the first trigger condition is a user input condition that is satisfied when a specified user interaction is detected.

16. The method according to any one of claims 1 to 15, wherein the first trigger condition is a timing condition satisfied during a specified time period.

17. The method according to any one of claims 1 to 16, wherein the first trigger condition is a collider condition satisfied in response to detecting a collision between designated scene elements.

18. A method according to any one of claims 1 to 17, wherein the behavior information associates the first action with multiple trigger conditions, and wherein the behavior information also includes information indicating whether the first associated action is (i) performed in response to a determination that all of the multiple trigger conditions have been met or (ii) performed in response to a determination that any one of the multiple trigger conditions has been met.

19. The method according to any one of claims 1 to 18, wherein the behavior information associates at least the first trigger condition with a plurality of actions, and wherein the behavior information further comprises information indicating an order in which the plurality of actions are to be performed.

20. The method according to any one of claims 1 to 19, wherein the behavior information associates at least the first trigger condition with a plurality of actions, and wherein the behavior information further comprises information indicating that the plurality of actions are to be performed simultaneously.

21. The method according to any one of claims 1 to 20, wherein at least one of the scene elements in the scene is a virtual object.

22. A method according to any one of claims 1 to 21, further comprising rendering the 3D scene in accordance with the scene description data as operated on by the first action.

23. The method according to any one of claims 1 to 22, wherein the trigger information comprises an array of all triggers in the 3D scene, each trigger describing a corresponding trigger condition.

24. The method according to any one of claims 1 to 23, wherein the action information comprises an array of all actions in the 3D scene, the action information comprising information associating each action with at least one corresponding scene element.

25. The method of any one of claims 1 to 24, wherein the behavior information comprises an array of all behaviors in the 3D scene.

26. The method according to any one of claims 1 to 25, wherein each action described in the action information comprises action type information, and wherein the action type information for at least the first action comprises information indicating a dedicated light action.

27. The method of any one of claims 1 to 26, further comprising rendering the 3D scene, wherein the determination that the first trigger condition has been met is made during rendering of the 3D scene.

28. An apparatus comprising one or more processors configured to perform the method according to any one of claims 1 to 27.

29. A computer readable medium comprising instructions for causing one or more processors to perform the method of any one of claims 1 to 27.

30. The computer-readable medium of claim 29, wherein the computer-readable medium is a non-transitory storage medium.

31. A computer program product comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 29.

32. A signal comprising scene description data of a 3D scene, wherein the scene description data comprises: Scene element information, the scene element information describing each scene element of a plurality of scene elements in the scene, at least one scene element of the scene elements being a virtual light source; Trigger information, the trigger information describing at least a first trigger condition; action information describing at least a first action, the first action indicating a modification of at least one property of the virtual light source; and Behavior information, wherein the behavior information associates at least the first trigger condition with at least the first action.

33. A computer readable medium comprising scene description data of a 3D scene, wherein the scene description data comprises: Scene element information, the scene element information describing each scene element of a plurality of scene elements in the scene, at least one scene element of the scene elements being a virtual light source; Trigger information, the trigger information describing at least a first trigger condition; action information describing at least a first action, the first action indicating a modification of at least one property of the virtual light source; and Behavior information, wherein the behavior information associates at least the first trigger condition with at least the first action.

Images