Timed lighting in XR experience

Abstract

An augmented reality may be created using one or more methods, devices, and / or systems. In augmented reality, novel and improved features need to be achieved, such as features relating to lighting and / or lighting evolution over time. In some cases, these features may be implemented by using lighting information. In some cases, the lighting information may follow a standardized format. In some cases, lighting information may be established from a standardized format.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of European Application No. 22305889.2 filed on June 17, 2022, the contents of which are incorporated herein by reference. Background Art

[0003] New and improved approaches are needed as they relate to extended realities, such as augmented reality, virtual reality, mixed reality, and correlated reality. Summary of the invention

[0004] One or more methods, devices, and / or systems may be used to create an extended reality. In an extended reality, there is a need to implement new and improved features, such as features related to lighting and / or the evolution of lighting over time. In some cases, these features may be implemented by using lighting information. In some cases, the lighting information may follow a standardized format. In some cases, the lighting information may be built from a standardized format. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] A more detailed understanding may be obtained from the following description given by way of example in conjunction with the accompanying drawings, in which like reference numerals indicate like elements, and in which:

[0006] Figure 1A is a system diagram illustrating an example communication system in which one or more disclosed embodiments may be implemented;

[0007] Figure 1B is an example of an embodiment in which Figure 1A A system diagram of an example wireless transmit / receive unit (WTRU) for use within an illustrated communication system;

[0008] Figure 1C is an example of an embodiment in which Figure 1A A system diagram of an example Radio Access Network (RAN) and an example Core Network (CN) used within the illustrated communication system;

[0009] Figure 1D is an example of an embodiment in which Figure 1A A system diagram of another example RAN and another example CN used within the illustrated communication system;

[0010] Figure 1E Illustrate an example hardware configuration for operating an XR application as described herein;

[0011] Figure 1F Illustrate an example hardware configuration for operating an XR application as described herein;

[0012] Figure 2 Example process illustrating XR applications;

[0013] Figure 3 Example illustrating the KHR_lights_punctual gltf extension;

[0014] Figure 4 Example illustrating the EXT_lights_image_based gltf extension;

[0015] Figure 5 illustrates an example of an MPEG_media element for time-evolving light properties;

[0016] Figure 6 Example of a JSON file used to control a GLTF light source;

[0017] Figure 7 Example illustrating the MPEG_lighting_timing extension at scene level;

[0018] Figure 8 Example illustrating the MPEG_lighting_timing extension at the light component level;

[0019] Fig. 9 Example of instantiating a video texture of EXT_lights_image_based lights;

[0020] Fig.10 illustrates an example of the MPEG_lighting_timing extension at light node level; and

[0021] Fig.11 An example process illustrating an XR application. DETAILED DESCRIPTION

[0022] The principles of the present invention will be described more fully below with reference to the accompanying drawings, in which examples of the principles of the present invention are shown. However, the principles of the present invention can be embodied in many alternative forms and should not be construed as being limited to the examples set forth herein. Therefore, although the principles of the present invention are susceptible to various modifications and alternative forms, specific examples thereof are shown by the examples in the accompanying drawings and will be described in detail herein. However, it should be understood that there is no intention to limit the principles of the present invention to the specific forms disclosed, but on the contrary, the present disclosure is used to cover all modifications, equivalents and alternatives that fall within the essence and scope of the principles of the present invention as defined by the claims.

[0023] The terms used herein are only for the purpose of describing specific examples, and are not intended to limit the principles of the present invention. As used herein, the singular forms "one", "an" and "the" are also intended to include plural forms, unless the context clearly states otherwise. It will be further understood that when used in this specification, the terms "include" and / or "comprise" specify the presence of the features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups. In addition, when an element is referred to as "responding" or "connected" to another element, it can directly respond or be connected to other elements, and there can also be intermediate elements. On the contrary, when an element is referred to as "directly responding" or "directly connected" to other elements, there is no intermediate element. As used herein, the term "and / or" includes any and all combinations of one or more of the listed associated items, and can be abbreviated as " / ".

[0024] It should be understood that although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, without departing from the teachings of the principles of the present invention, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element.

[0025] Although some illustrations include arrows on communication paths to show a primary communication direction, it is understood that communications can occur in the opposite direction to the depicted arrows.

[0026] Some examples are described with respect to block diagrams and operational flow charts in which each block represents a circuit element, module, or code portion, which includes one or more executable instructions for implementing a specified logical function. It should also be noted that in other specific implementations, the functions annotated in the blocks may not appear in the order annotated. For example, two blocks shown in succession may actually be executed substantially concurrently, or the blocks may sometimes be executed in reverse order depending on the functions involved.

[0027] References herein to "according to an example" or "in an example" mean that a particular feature, structure, or characteristic described in conjunction with the example can be included in at least one specific implementation of the present principles. The appearances of the phrases "according to an example" or "in an example" in various places in the specification are not necessarily all referring to the same example, nor are they necessarily separate or alternative examples that are mutually exclusive of other examples.

[0028] The reference numerals appearing in the claims are by way of illustration only and have no limiting effect on the scope of the claims.The present examples and variations may be employed in any combination or sub-combination although not explicitly described.

[0029] Figure 1A1 is a diagram illustrating an example communication system 100 in which one or more disclosed embodiments may be implemented. The communication system 100 may be a multiple access system that provides content such as voice, data, video, messaging, broadcast, etc. to multiple wireless users. The communication system 100 may enable multiple wireless users to access such content through sharing of system resources (including wireless bandwidth). For example, the communication system 100 may employ one or more channel access methods such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single carrier FDMA (SC-FDMA), zero tail unique word discrete Fourier transform spread OFDM (ZT-UW-DFT-S-OFDM), unique word OFDM (UW-OFDM), resource block filtered OFDM, filter bank multi-carrier (FBMC), etc.

[0030] like Figure 1A As shown, the communication system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, a radio access network (RAN) 104, a core network (CN) 106, a public switched telephone network (PSTN) 108, the Internet 110 and other networks 112, but it should be understood that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, 102d may be any type of device configured to operate and / or communicate in a wireless environment. By way of example, the WTRUs 102a, 102b, 102c, 102d (any of which may be referred to as a station (STA)) may be configured to send and / or receive wireless signals and may include user equipment (UE), a mobile station, a fixed or mobile subscriber unit, a subscription-based unit, a pager, a cellular phone, a personal digital assistant (PDA), a smart phone, a laptop, a netbook, a personal computer, a television, a projector, a 3D display, a 3D television, a 3D projector, a wireless sensor, a hotspot or Mi-Fi device, an Internet of Things (IoT) device, a watch or other wearable device, a head-mounted display (HMD), a vehicle, a drone, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in an industrial and / or automated process chain environment), a consumer electronic device, a device operating on a commercial and / or industrial wireless network, etc. Any of the WTRUs 102a, 102b, 102c, and 102d may be interchangeably referred to as a UE.

[0031] The communication system 100 may also include a base station 114a and / or a base station 114b. Each of the base stations 114a, 114b may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communication networks, such as the CN 106, the Internet 110, and / or other networks 112. By way of example, the base stations 114a, 114b may be a base transceiver station (BTS), a Node B, an evolved Node B (eNB), a Home Node B, a Home evolved Node B, a next generation Node B such as a gNode B (gNB), a new radio (NR) Node B, a site controller, an access point (AP), a wireless router, and the like. While the base stations 114a, 114b are each depicted as a single element, it will be appreciated that the base stations 114a, 114b may include any number of interconnected base stations and / or network elements.

[0032] The base station 114a may be part of the RAN 104, which may also include other base stations and / or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), a relay node, etc. The base station 114a and / or the base station 114b may be configured to send and / or receive wireless signals on one or more carrier frequencies, which may be referred to as a cell (not shown). These frequencies may be in a licensed spectrum, an unlicensed spectrum, or a combination of licensed and unlicensed spectrums. A cell may provide coverage of wireless services to a specific geographic area, which may be relatively fixed or may change over time. The cell may be further divided into cell sectors. For example, a cell associated with the base station 114a may be divided into three sectors. Therefore, in one embodiment, the base station 114a may include three transceivers, i.e., one transceiver for each sector of the cell. In one embodiment, the base station 114a may employ multiple-input multiple-output (MIMO) technology and may utilize multiple transceivers for each sector of the cell. For example, beamforming may be used to send and / or receive signals in a desired spatial direction.

[0033] The base stations 114a, 114b may communicate with one or more of the WTRUs 102a, 102b, 102c, 102d over an air interface 116, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 may be established using any suitable radio access technology (RAT).

[0034] More specifically, as noted above, the communication system 100 may be a multiple access system and may employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, etc. For example, the base station 114a in the RAN 104 and the WTRUs 102a, 102b, 102c may implement a radio technology, such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), that may use Wideband CDMA (WCDMA) to establish the air interface 116. WCDMA may include communication protocols, such as High Speed ​​Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA may include High Speed ​​Downlink (DL) Packet Access (HSDPA) and / or High Speed ​​Uplink (UL) Packet Access (HSUPA).

[0035] In one embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA) that may establish the air interface 116 using Long Term Evolution (LTE) and / or Advanced LTE (LTE-A) and / or Advanced LTE Pro (LTE-A Pro).

[0036] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology (such as NR radio access) that may establish the air interface 116 using NR.

[0037] In one embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement multiple radio access technologies. For example, the base station 114a and the WTRUs 102a, 102b, 102c may together implement LTE radio access and NR radio access, for example using dual connectivity (DC) principles. Thus, the air interface utilized by the WTRUs 102a, 102b, 102c may be characterized by multiple types of radio access technologies and / or transmissions transmitted to / from multiple types of base stations (e.g., eNBs and gNBs).

[0038] In other embodiments, the base station 114a and the WTRUs 102a, 102b, 102c may implement radio technologies such as IEEE 802.11 (i.e., Wireless Fidelity (WiFi)), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile Communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), GSM EDGE (GERAN), etc.

[0039] Figure 1A The base station 114b in the example may be, for example, a wireless router, a Home NodeB, a Home eNodeB, or an access point, and may utilize any suitable RAT to facilitate wireless connectivity in a local area, such as a business location, a home, a vehicle, a campus, an industrial facility, an air corridor (e.g., for use by drones), a road, and the like. In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.11 to establish a wireless local area network (WLAN). In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.15 to establish a wireless personal area network (WPAN). In another embodiment, the base station 114b and the WTRUs 102c, 102d may utilize a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.) to establish a micro-micro cell base station or a femto cell base station. As Figure 1A As shown, the base station 114b may have a direct connection to the Internet 110. Therefore, the base station 114b may not be required to access the Internet 110 via the CN 106.

[0040] The RAN 104 may be in communication with the CN 106, which may be any type of network configured to provide voice, data, applications, and / or Voice over Internet Protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. The data may have different quality of service (QoS) requirements, such as different throughput requirements, latency requirements, error tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, etc. The CN 106 may provide call control, billing services, mobile location-based services, prepaid calling, Internet connectivity, video distribution, etc., and / or perform high-level security functions, such as user authentication. Although not described in detail in the accompanying drawings, the CN 106 may be configured to provide voice, data, applications, and / or Voice over Internet Protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. Figure 1AAlthough not shown in the figure, it will be appreciated that the RAN 104 and / or the CN 106 may be in direct or indirect communication with other RANs that employ the same RAT as the RAN 104 or a different RAT. For example, in addition to being connected to the RAN 104, which may utilize NR radio technology, the CN 106 may also be in communication with another RAN (not shown) that employs GSM, UMTS, CDMA 2000, WiMAX, E-UTRA, or WiFi radio technology.

[0041] The CN 106 may also act as a gateway for the WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and / or other networks 112. The PSTN 108 may include a circuit-switched telephone network that provides plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices that use common communication protocols, such as the Transmission Control Protocol (TCP), the User Datagram Protocol (UDP), and / or the Internet Protocol (IP) in the TCP / IP Internet protocol suite. The networks 112 may include wired communication networks and / or wireless communication networks owned and / or operated by other service providers. For example, the networks 112 may include another CN connected to one or more RANs, which may employ the same RAT as the RAN 104 or a different RAT.

[0042] Some or all of the WTRUs 102a, 102b, 102c, 102d in the communication system 100 may include multi-mode capabilities (e.g., the WTRUs 102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks via different wireless links). Figure 1A The illustrated WTRU 102c may be configured to communicate with the base station 114a, which may employ a cellular-based radio technology, and with the base station 114b, which may employ an IEEE 802 radio technology.

[0043] Figure 1B is a system diagram illustrating an example WTRU 102. Figure 1B As shown, the WTRU 102 may include, among other things, a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and / or other peripherals 138. It will be appreciated that the WTRU 102 may include any sub-combination of the foregoing elements while remaining consistent with an embodiment.

[0044] The processor 118 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), any other type of integrated circuit (IC), a state machine, etc. The processor 118 may perform signal decoding, data processing, power control, input / output processing, and / or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit / receive element 122. Although Figure 1B The processor 118 and the transceiver 120 are depicted as separate components, but it is understood that the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip.

[0045] The send / receive element 122 may be configured to send a signal to a base station (e.g., base station 114a) or receive a signal from the base station via an air interface 116. For example, in one embodiment, the send / receive element 122 may be an antenna configured to send and / or receive an RF signal. In one embodiment, the send / receive element 122 may be a transmitter / detector configured to send and / or receive, for example, an IR, UV or visible light signal. In another embodiment, the send / receive element 122 may be configured to send and / or receive both an RF signal and an optical signal. It should be understood that the send / receive element 122 may be configured to send and / or receive any combination of wireless signals.

[0046] Although the transmit / receive element 122 Figure 1B Although depicted as a single element in the figure, the WTRU 102 may include any number of transmit / receive elements 122. More specifically, the WTRU 102 may employ MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 116.

[0047] The transceiver 120 may be configured to modulate signals to be transmitted by the transmit / receive element 122 and to demodulate signals received by the transmit / receive element 122. As noted above, the WTRU 102 may have multi-mode capabilities. For example, the transceiver 120 may include multiple transceivers for enabling the WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11.

[0048] The processor 118 of the WTRU 102 may be coupled to and may receive user input data from a speaker / microphone 124, a keypad 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or an organic light emitting diode (OLED) display unit). The processor 118 may also output user data to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128. In addition, the processor 118 may access information from and store data in any type of suitable memory, such as a non-removable memory 130 and / or a removable memory 132. The non-removable memory 130 may include a random access memory (RAM), a read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 118 may access information from and store data in a memory that is not physically located on the WTRU 102, such as on a server or a home computer (not shown).

[0049] The processor 118 may receive power from the power source 134, and may be configured to distribute and / or control power to the other components in the WTRU 102. The power source 134 may be any suitable device for powering the WTRU 102. For example, the power source 134 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel-metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0050] The processor 118 may also be coupled to the GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to or in lieu of the information from the GPS chipset 136, the WTRU 102 may receive location information from a base station (e.g., base stations 114a, 114b) over the air interface 116 and / or determine its location based on the timing of signals received from two or more nearby base stations. It will be appreciated that the WTRU 102 may acquire location information by any suitable location-determination method while remaining consistent with an embodiment.

[0051] The processor 118 may also be coupled to other peripherals 138, which may include one or more software modules and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an electronic compass, a satellite transceiver, a digital camera (for photos and / or video), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, module, FM radio unit, digital music player, media player, video game player module, Internet browser, virtual reality and / or augmented reality (VR / AR) device, activity tracker, etc. The peripheral device 138 may include one or more sensors. The sensor may be one or more of the following: a gyroscope, an accelerometer, a Hall effect sensor, a magnetometer, an orientation sensor, a proximity sensor, a temperature sensor, a time sensor; a geolocation sensor, an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric sensor, a humidity sensor, etc.

[0052] The WTRU 102 may include a full-duplex radio for which transmission and reception of some or all of the signals (e.g., associated with specific subframes for both UL (e.g., for transmission) and DL (e.g., for reception)) may be concurrent and / or simultaneous. The full-duplex radio may include an interference management unit for reducing and / or substantially eliminating self-interference via signal processing performed by hardware (e.g., a choke) or via a processor (e.g., a separate processor (not shown) or via the processor 118). In an embodiment, the WTRU 102 may include a half-duplex radio for which transmission and reception of some or all of the signals (e.g., associated with specific subframes for both UL (e.g., for transmission) or DL ​​(e.g., for reception)) may be concurrent and / or simultaneous.

[0053] exist Figure 1B In variations of the examples of , there may be an architecture of an XR processing engine that may be configured to implement the methods described herein. The WTRU 102 may be linked to other devices via a bus of the other devices and / or via an I / O interface. The WTRU 102 may include one or more of the following elements linked together by a data and address bus: a microprocessor (or CPU), which is, for example, a DSP (or digital signal processor); a ROM (or read-only memory); a RAM (or random access memory); a storage interface; an I / O interface for receiving data for transmission from an application; and a power source (e.g., a battery). In one case, the power source is external to the WTRU 102.

[0054] In each memory mentioned, the term "register" as used herein may correspond to an area of ​​small capacity (a few bits) or a very large area (e.g., an entire program or a large amount of received or decoded data). The ROM includes at least programs and parameters. The ROM may store algorithms and instructions for executing techniques according to the principles of the present invention. When turned on, the CPU uploads the program in the RAM and executes the corresponding instructions. The RAM includes programs in registers that are executed by the CPU and uploaded after the device is turned on, input data in registers, intermediate data in registers in different states of the method, and other variables used to execute the method in registers. The WTRU 102 may be linked to a set of sensors and a set of rendering devices, for example, via a bus. The sensor may be, for example, a camera, a microphone, a temperature sensor, an inertial measurement unit, a GPS, a hygrometric sensor, an IR or UV light sensor, or a wind sensor. The rendering device may be, for example, a display, a speaker, a vibrator, a thermal sensing device, a fan, etc.

[0055] Figure 1C 1 is a system diagram illustrating the RAN 104 and the CN 106 in accordance with an embodiment. As noted above, the RAN 104 may employ an E-UTRA radio technology to communicate with the WTRUs 102a, 102b, 102c over the air interface 116. The RAN 104 may also be in communication with the CN 106.

[0056] The RAN 104 may include evolved Node-Bs 160a, 160b, 160c, though it will be appreciated that the RAN 104 may include any number of evolved Node-Bs while remaining consistent with an embodiment. The evolved Node-Bs 160a, 160b, 160c may each include one or more transceivers for communicating with the WTRUs 102a, 102b, 102c over the air interface 116. In one embodiment, the evolved Node-Bs 160a, 160b, 160c may implement MIMO technology. Thus, the evolved Node-B 160a, for example, may use multiple antennas to transmit wireless signals to and / or receive wireless signals from the WTRU 102a.

[0057] Each of the evolved Node Bs 160a, 160b, 160c may be associated with a particular cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, scheduling of users in the UL and / or DL, etc. Figure 1C As shown, the eNode-Bs 160a, 160b, 160c may communicate with one another via an X2 interface.

[0058] Figure 1CThe illustrated CN 106 may include a mobility management entity (MME) 162, a serving gateway (SGW) 164, and a packet data network (PDN) gateway (PGW) 166. Although the foregoing elements are depicted as part of the CN 106, it should be understood that any of these elements may be owned and / or operated by an entity other than the CN operator.

[0059] The MME 162 may be connected to each of the evolved Node-Bs 162a, 162b, 162c in the RAN 104 via an S1 interface and may serve as a control node. For example, the MME 162 may be responsible for authenticating users of the WTRUs 102a, 102b, 102c, bearer activation / deactivation, selecting a particular serving gateway during an initial attach of the WTRUs 102a, 102b, 102c, etc. The MME 162 may provide a control plane function for switching between the RAN 104 and other RANs (not shown) that employ other radio technologies, such as GSM and / or WCDMA.

[0060] The SGW 164 may be connected to each of the evolved Node-Bs 160a, 160b, 160c in the RAN 104 via an S1 interface. The SGW 164 may generally route and forward user data packets to / from the WTRUs 102a, 102b, 102c. The SGW 164 may perform other functions, such as anchoring the user plane during an inter-evolved Node-B handover, triggering paging when DL data is available for the WTRUs 102a, 102b, 102c, managing and storing the context of the WTRUs 102a, 102b, 102c, and the like.

[0061] The SGW 164 may be connected to the PGW 166, which may provide the WTRUs 102a, 102b, 102c with access to packet-switched networks, such as the Internet 110, to facilitate communications between the WTRUs 102a, 102b, 102c and IP-enabled devices.

[0062] The CN 106 may facilitate communications with other networks. For example, the CN 106 may provide the WTRUs 102a, 102b, 102c with access to circuit-switched networks, such as the PSTN 108, to facilitate communications between the WTRUs 102a, 102b, 102c and traditional land-line communications devices. For example, the CN 106 may include, or may be in communication with, an IP gateway (e.g., an IP Multimedia Subsystem (IMS) server) that acts as an interface between the CN 106 and the PSTN 108. In addition, the CN 106 may provide the WTRUs 102a, 102b, 102c with access to other networks 112, which may include other wired networks and / or wireless networks owned and / or operated by other service providers.

[0063] Although the WTRU Figures 1A to 1D Although described as wireless terminals, it is contemplated that in certain representative embodiments, such terminals may (eg, temporarily or permanently) use a wired communications interface with a communications network.

[0064] In a representative embodiment, the other network 112 may be a WLAN.

[0065] A WLAN in infrastructure basic service set (BSS) mode may have an access point (AP) for the BSS and one or more stations (STAs) associated with the AP. The AP may have access or an interface to a distribution system (DS) or another type of wired / wireless network that carries traffic to and / or carries traffic away from the BSS. Traffic originating from outside the BSS and going to the STA may be reached by the AP and may be delivered to the STA. Traffic originating from the STA and going to a target outside the BSS may be transmitted to the AP to be delivered to the corresponding target. Traffic between STAs within the BSS may be transmitted by the AP, for example, wherein the source STA may transmit traffic to the AP, and the AP may deliver traffic to the target STA. Traffic between STAs within the BSS may be considered and / or referred to as point-to-point traffic. Direct link establishment (DLS) may be used to transmit point-to-point traffic between the source STA and the target STA (e.g., directly between them). In certain representative embodiments, the DLS may use 802.11e DLS or 802.11z tunnel DLS (TDLS). A WLAN using an independent BSS (IBSS) mode may not have an AP, and STAs within or using the IBSS (eg, all STAs in the STA) may communicate directly with each other. The IBSS communication mode may sometimes be referred to herein as an "ad hoc" communication mode.

[0066] When using the 802.11ac infrastructure operating mode or a similar operating mode, the AP may send beacons on a fixed channel, such as a primary channel. The primary channel may be a fixed width (e.g., a 20 MHz wide bandwidth) or a dynamically set width. The primary channel may be a working channel of the BSS and may be used by the STA to establish a connection with the AP. In certain representative embodiments, carrier sense multiple access / collision avoidance (CSMA / CA) may be implemented, for example, in an 802.11 system. For CSMA / CA, a STA (e.g., each STA) (including the AP) may listen to the primary channel. If the primary channel is listened / detected and / or determined to be busy by a particular STA, the particular STA may back off. One STA (e.g., only one station) may transmit at any given time in a given BSS.

[0067] High throughput (HT) STAs may communicate using a 40 MHz wide channel, for example, via a combination of a primary 20 MHz channel and adjacent or non-adjacent 20 MHz channels to form a 40 MHz wide channel.

[0068] Very high throughput (VHT) STA can support 20MHz, 40MHz, 80MHz and / or 160MHz wide channels. 40MHz channels and / or 80MHz channels can be formed by combining continuous 20MHz channels. 160MHz channels can be formed by combining 8 continuous 20MHz channels, or by combining two non-continuous 80MHz channels (this can be called 80+80 configuration). For 80+80 configuration, after channel coding, the data can pass through a segment parser that can divide the data into two streams. Each stream can be processed by inverse fast Fourier transform (IFFT) and time domain processing separately. These streams can be mapped to two 80MHz channels, and the data can be sent by sending STA. At the receiver of the receiving STA, the above-mentioned operation for 80+80 configuration can be reversed, and the combined data can be transmitted to the medium access control (MAC).

[0069] 802.11af and 802.11ah support operating modes below 1GHz. Channel operating bandwidths and carriers are reduced in 802.11af and 802.11ah relative to those used in 802.11n and 802.11ac. 802.11af supports 5MHz, 10MHz, and 20MHz bandwidths in the TV White Space (TVWS) spectrum, and 802.11ah supports 1MHz, 2MHz, 4MHz, 8MHz, and 16MHz bandwidths using non-TVWS spectrum. According to a representative embodiment, 802.11ah may support meter type control / machine type communication (MTC), such as MTC devices in macro coverage areas. MTC devices may have certain capabilities, such as limited capabilities, including support for (e.g., only support for) certain bandwidths and / or limited bandwidths. MTC devices may include batteries with battery life above a threshold (e.g., to maintain very long battery life).

[0070] WLAN systems that can support multiple channels and channel bandwidths (such as 802.11n, 802.11ac, 802.11af, and 802.11ah) include channels that can be designated as primary channels. The primary channel may have a bandwidth equal to the maximum common operating bandwidth supported by all STAs in the BSS. The bandwidth of the primary channel may be set and / or limited by a STA (which supports the minimum bandwidth operation mode) from all STAs operating in the BSS. In the example of 802.11ah, for STAs (e.g., MTC-type devices) that support (e.g., only support) a 1MHz mode, the primary channel may be 1MHz wide, even if the AP and other STAs in the BSS support 2MHz, 4MHz, 8MHz, 16MHz, and / or other channel bandwidth operation modes. Carrier sensing and / or network allocation vector (NAV) settings may depend on the state of the primary channel. If the primary channel is busy, for example, because a STA (which only supports a 1MHz operation mode) is transmitting to the AP, all available bands may be considered busy even if most of the available bands remain idle.

[0071] In the United States, the available frequency band for 802.11ah is 902MHz to 928MHz. In South Korea, the available frequency band is 917.5MHz to 923.5MHz. In Japan, the available frequency band is 916.5MHz to 927.5MHz. The total bandwidth available for 802.11ah is 6MHz to 26MHz, depending on the country code.

[0072] Figure 1D1 is a system diagram illustrating the RAN 104 and the CN 106 in accordance with an embodiment. As noted above, the RAN 104 may employ NR radio technology to communicate with the WTRUs 102a, 102b, 102c over the air interface 116. The RAN 104 may also be in communication with the CN 106.

[0073] The RAN 104 may include gNBs 180a, 180b, 180c, though it will be appreciated that the RAN 104 may include any number of gNBs while remaining consistent with an embodiment. The gNBs 180a, 180b, 180c may each include one or more transceivers for communicating with the WTRUs 102a, 102b, 102c over the air interface 116. In one embodiment, the gNBs 180a, 180b, 180c may implement MIMO technology. For example, the gNBs 180a, 180b may utilize beamforming to send signals to and / or receive signals from the gNBs 180a, 180b, 180c. Thus, the gNB 180a may, for example, use multiple antennas to send wireless signals to and / or receive wireless signals from the WTRU 102a. In one embodiment, the gNBs 180a, 180b, 180c may implement carrier aggregation technology. For example, the gNB 180a may transmit multiple component carriers to the WTRU 102a (not shown). A subset of these component carriers may be on an unlicensed spectrum, while the remaining component carriers may be on a licensed spectrum. In one embodiment, the gNBs 180a, 180b, 180c may implement coordinated multi-point (CoMP) technology. For example, the WTRU 102a may receive coordinated transmissions from the gNB 180a and gNB 180b (and / or gNB 180c).

[0074] The WTRUs 102a, 102b, 102c may communicate with the gNBs 180a, 180b, 180c using transmissions associated with a set of parameters that may be scalable. For example, OFDM symbol spacing and / or OFDM subcarrier spacing may vary for different transmissions, different cells, and / or different portions of the wireless transmission spectrum. The WTRUs 102a, 102b, 102c may communicate with the gNBs 180a, 180b, 180c using subframes or transmit time intervals (TTIs) of varying or scalable lengths (e.g., containing varying numbers of OFDM symbols and / or varying absolute time lengths over time).

[0075] The gNBs 180a, 180b, 180c may be configured to communicate with the WTRUs 102a, 102b, 102c in a standalone configuration and / or a non-standalone configuration. In a standalone configuration, the WTRUs 102a, 102b, 102c may communicate with the gNBs 180a, 180b, 180c while not accessing other RANs (e.g., such as the eNodeBs 160a, 160b, 160c). In a standalone configuration, the WTRUs 102a, 102b, 102c may use one or more of the gNBs 180a, 180b, 180c as mobility anchor points. In a standalone configuration, the WTRUs 102a, 102b, 102c may communicate with the gNBs 180a, 180b, 180c using signals in an unlicensed band. In a non-standalone configuration, the WTRUs 102a, 102b, 102c may communicate / connect with the gNB 180a, 180b, 180c while also communicating / connecting with another RAN, such as the eNodeB 160a, 160b, 160c. For example, the WTRUs 102a, 102b, 102c may implement the DC principle to communicate with one or more gNBs 180a, 180b, 180c and one or more eNodeBs 160a, 160b, 160c substantially simultaneously. In a non-standalone configuration, the eNodeB 160a, 160b, 160c may act as a mobility anchor for the WTRUs 102a, 102b, 102c, and the gNB 180a, 180b, 180c may provide additional coverage and / or throughput for serving the WTRUs 102a, 102b, 102c.

[0076] Each of the gNBs 180a, 180b, 180c may be associated with a specific cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, scheduling of users in UL and / or DL, support of network slicing, interworking between DC, NR and E-UTRA, routing of user plane data towards user plane functions (UPFs) 184a, 184b, routing of control plane information towards access and mobility management functions (AMFs) 182a, 182b, etc. Figure 1D As shown, gNBs 180a, 180b, and 180c may communicate with each other via an Xn interface.

[0077] Figure 1DThe illustrated CN 106 may include at least one AMF 182a, 182b, at least one UPF 184a, 184b, at least one session management function (SMF) 183a, 183b, and possible data networks (DNs) 185a, 185b. Although the aforementioned elements are depicted as part of the CN 106, it should be understood that any of these elements may be owned and / or operated by an entity other than the CN operator.

[0078] The AMF 182a, 182b may be connected to one or more of the gNBs 180a, 180b, 180c in the RAN 104 via the N2 interface and may act as a control node. For example, the AMF 182a, 182b may be responsible for authenticating users of the WTRU 102a, 102b, 102c, support of network slicing (e.g., handling of different protocol data unit (PDU) sessions with different requirements), selecting a specific SMF 183a, 183b, management of registration areas, termination of non-access stratum (NAS) signaling, mobility management, etc. The AMF 182a, 182b may use network slicing in order to customize CN support for the WTRU 102a, 102b, 102c based on the type of services utilized by the WTRU 102a, 102b, 102c. For example, different network slices may be established for different use cases, such as services relying on ultra-reliable low latency (URLLC) access, services relying on enhanced massive mobile broadband (eMBB) access, services for MTC access, etc. The AMF 182a, 182b may provide a control plane function for switching between the RAN 104 and other RANs (not shown) that employ other radio technologies, such as LTE, LTE-A, LTE-A Pro, and / or non-3GPP access technologies, such as WiFi.

[0079] The SMF 183a, 183b may be connected to the AMF 182a, 182b in the CN 106 via the N11 interface. The SMF 183a, 183b may also be connected to the UPF 184a, 184b in the CN 106 via the N4 interface. The SMF 183a, 183b may select and control the UPF 184a, 184b, and configure the traffic routing through the UPF 184a, 184b. The SMF 183a, 183b may perform other functions, such as managing and allocating UE IP addresses, managing PDU sessions, controlling policy implementation and QoS, providing DL data notification, etc. The PDU session type may be IP-based, non-IP-based, Ethernet-based, etc.

[0080] The UPF 184a, 184b may be connected to one or more of the gNBs 180a, 180b, 180c in the RAN 104 via the N3 interface, which may provide the WTRUs 102a, 102b, 102c with access to packet-switched networks, such as the Internet 110, to facilitate communications between the WTRUs 102a, 102b, 102c and IP-enabled devices. The UPF 184, 184b may perform other functions, such as routing and forwarding packets, enforcing user plane policies, supporting multi-homed PDU sessions, handling user plane QoS, buffering DL packets, providing mobility anchoring, etc.

[0081] The CN 106 may facilitate communications with other networks. For example, the CN 106 may include or may communicate with an IP gateway (e.g., an IP Multimedia Subsystem (IMS) server) that acts as an interface between the CN 106 and the PSTN 108. In addition, the CN 106 may provide the WTRUs 102a, 102b, 102c with access to other networks 112, which may include other wired networks and / or wireless networks owned and / or operated by other service providers. In one embodiment, the WTRUs 102a, 102b, 102c may be connected to the DNs 185a, 185b via the UPFs 184a, 184b via the N3 interfaces to the UPFs 184a, 184b and the N6 interfaces between the UPFs 184a, 184b and the local DNs 185a, 185b.

[0082] Given that Figures 1A to 1D as well as Figures 1A to 1D Corresponding to the description of the present invention, one or more or all of the functions described herein with reference to one or more of the following items may be performed by one or more simulation devices (not shown): WTRU102a to 102d, base station 114a to 114b, evolved Node B 160a to 160c, MME 162, SGW 164, PGW 166, gNB180a to 180c, AMF 182a to 182b, UPF 184a to 184b, SMF 183a to 183b, DN 185a to 185b and / or any other device described herein. The simulation device may be one or more devices configured to mimic one or more or all of the functions described herein. For example, the simulation device can be used to test other devices and / or simulate network and / or WTRU functions.

[0083] The simulation device may be designed to implement one or more tests of other devices in a laboratory environment and / or in an operator network environment. For example, one or more simulation devices may perform one or more functions or all functions while being fully or partially implemented and / or deployed as part of a wired and / or wireless communication network in order to test other devices within the communication network. One or more simulation devices may perform one or more functions or all functions while being temporarily implemented / deployed as part of a wired and / or wireless communication network. The simulation device may be directly coupled to another device for the purpose of testing and / or performing tests using over-the-air wireless communications.

[0084] One or more simulation devices can perform one or more (including all) functions without being implemented / deployed as part of a wired and / or wireless communication network. For example, the simulation device can be used in a test scenario in a test laboratory and / or a non-deployed (e.g., testing) wired and / or wireless communication network to implement testing of one or more components. One or more simulation devices can be test equipment. Direct RF coupling and / or wireless communication via an RF circuit system (e.g., which can include one or more antennas) can be used by the simulation device to send and / or receive data.

[0085] Extended reality (XR) such as augmented reality (AR), virtual reality (VR), mixed reality (MR), and related realities may be interchangeable as discussed herein. When implementing XR techniques, such as with an XR application running on a device (e.g., a WTRU having or connected to a head-mounted display), the scene description may be used as an instruction to the application, which may combine an explicit and easily parsable description of the scene structure with some binary representation of the media content.

[0086] Figure 1E An example hardware configuration for operating an XR application as described herein is illustrated. Figure 1E The WTRU 102 is intended to act as Figure 1B and / or Figure 1FAs shown, the WTRU 102 may (e.g., in addition to the elements described herein with respect to other figures) include an encoder 141, a storage device 142 (e.g., a memory and / or data storage medium, such as an SSD, NVME, PCIe storage device, etc.; for example, in some cases, the storage device 142 may be the same as other storage devices associated with the WTRU 102 described herein), and / or an input / output terminal (I / O) 145a (e.g., the I / O illustrates that the encoder can input and output data externally; for example, in some cases, the I / O is not required, and the data exchange can be internal to the device, such as to and from the storage device 142 by the processing of the XR application, or in some cases, the I / O is required and can be external to the device, such as received or accessed via the Internet or a remote source, or in some cases, the I / O is partially required to partially operate one or more of the foregoing situations). The I / O 145a may include any of the interface components described herein, such as a wireless transceiver, a wired transceiver, etc. The WTRU 102 may be able to encode a stream that may then be decoded for display.

[0087] Figure 1F An example hardware configuration for operating an XR application as described herein is illustrated. Figure 1F The WTRU 102 is intended to act as Figure 1B and / or Figure 1EVariations and / or enhancements of the WTRU 102 presented in . As shown, the WTRU 102 may include (e.g., in addition to the elements described herein with respect to other figures) a decoder 144, a storage device 143 (e.g., a memory and / or data storage medium, which in some cases may be the same as other storage devices associated with the WTRU 102 as described herein), an input / output terminal (I / O) 145b (e.g., the I / O illustrates that the decoder can input and output data externally; for example, in some cases, the I / O is not required and the data exchange may be internal to the device, such as to and from the storage device 143 by the processing of the XR application, or in some cases, the I / O is required and may be external to the device, such as received or accessed via the Internet or a remote source, or in some cases, the I / O is partially required to partially operate one or more of the foregoing), a head-mounted display (HMD) 146a as part of the WTRU 102, the HMD 146b connected to the WTRU 102 via a wire 147, and / or connected to the WTRU wirelessly (e.g., antenna / transceiver 148 to antenna / transceiver 149) 102. The I / O 145a may include any of the interface components described herein, such as a wireless transceiver, a wired transceiver, etc. The WTRU 102 may be capable of decoding a previously encoded stream for display to a user on the HMD 146.

[0088] In one scenario, a first WTRU may be responsible for encoding and may be separate from a second WTRU that performs decoding (e.g., remote over a wide area network or a local area network, e.g., using direct communication means as described herein, local but physically separate). In one scenario, one WTRU performs encoding and decoding.

[0089] In one scenario, instead of or in addition to the encoder / decoder arrangement, the WTRU 102 (e.g., Figure 1B , Figure 1E and / or Figure 1F The WTRU may use a GPU. The GPU may retrieve / receive scene information (e.g., locally or remotely) and process the scene information to generate display information. The display information may then be sent to a display (e.g., part of the WTRU, or external to the WTRU). For example, the WTRU may use a decoder to decode an encoded element (e.g., a scene description such as a JPG image, video, etc.) and then feed the decoded information into the GPU.

[0090] Figure 2Example process illustrating an XR application. The XR application may be implemented by one or more WTRUs, as described herein. Initially, at 201, scene data may be determined. In one example, the scene data may be retrieved / received locally or remotely (e.g., from a storage device or via the Internet). At 202, the scene data is encoded to generate a bitstream. At 203, the bitstream is decoded into display information. At 204, the display information is displayed. Details of the process are further described herein throughout this specification.

[0091] For example, a sequence of scenes (e.g., such as 3D scenes and / or XR scenes) may be provided to an encoder. The encoder may take as input a scene or a sequence of scenes and provide a bitstream representing the input. The bitstream may be stored in a memory and / or on an electronic data medium and may be sent over a network or locally on a display. The bitstream representing the sequence of scenes may be read from a memory and / or received from a network by a decoder. The decoder is input with the bitstream and provides a sequence of scenes, for example, in a point cloud format.

[0092] The encoder may include several circuits that implement several steps. In the first step, the encoder may project each scene (e.g., from scene data) onto at least one picture (e.g., a 2D picture). Scene projection is any method of mapping at least three-dimensional points to a smaller dimension such as a two-dimensional surface. For example, in a method for displaying graphic data on a two-dimensional medium based on a surface (e.g., pixel information from several bit planes), the use of this type of projection may be used in computer graphics, engineering, and / or mapping. The projection circuit may provide at least one two-dimensional frame for a sequence of scenes. The frame may include color information and depth information representing the scene projected onto the frame. In a variant, a combination of one or more of the following may be encoded in separate frames: color information, light information, and / or depth information.

[0093] The metadata may be used and updated by the projection circuitry.The metadata may include information about the projection operation (eg, projection parameters) and / or information about how other information is organized within a frame.

[0094] In some instances, metadata and video data (e.g., a bitstream from an encoding circuit that encodes a sequence of frames into a video, where pictures of a scene and / or sequences of pictures of a scene may be encoded in the stream by a video encoder) may be encapsulated in a data stream by a data encapsulation circuit.

[0095] In one case, the encoder may conform to an industry standard encoder, such as: JPEG, specification ISO / CEI 10918-1UIT-T recommendation T.81; AVC, also known as MPEG-4 AVC or h264. HEVC (T recommendation, H series, h265) specified in both UIT-T H.264 and ISO / CEI MPEG-4 Part 10 (ISO / CEI 14496-10); 3D-HEVC (extension of HVEC, T recommendation, H series, h265); VP9; and / or AV1 (AOMedia Video 1).

[0096] The data stream may be stored in a storage device (eg, a memory or other component as described herein) that can be accessed by a decoder, such as over a network.

[0097] The decoder may include different circuits to implement different steps of decoding. The decoder may take as input the data / bitstream generated by the encoder and provide a sequence of scenes (e.g., display information) to be rendered and displayed by a volumetric video display device (such as a WTRU having or connected to a head-mounted device (HMD)). The decoder may obtain the stream from a source. For example, the source may be one or more of the following: a local memory, such as a video memory or RAM (or random access memory), a flash memory, a ROM (or read-only memory), a hard disk; a storage interface, such as an interface with a mass storage device, RAM, flash memory, ROM, an optical disk, or a magnetic carrier; a communication interface, such as a wired interface (e.g., a bus interface, a wide area network interface, a local area network interface) or a wireless interface (such as an IEEE 802.11 interface or interface); and a user interface such as a graphical user interface, which enables a user to input data that is combined with other data to generate displayed information.

[0098] The decoder may include circuitry for extracting data encoded in the data / bitstream. The circuitry may take the data / bitstream as input and provide metadata corresponding to the metadata encoded in the stream and a two-dimensional video. The video is decoded by a video decoder that provides a sequence of frames. The decoded sequence of frames may include information (e.g., color and depth information, and / or any information disclosed herein). In one variant, the video decoder may provide two sequences of frames, one including color information and the other including depth information. In one variant, there may be as many sequences of frames as there is information. The circuitry may use metadata to back-project information from decoded frames to provide a sequence of scenes. The sequence of scenes may have a possible loss of precision associated with encoding into 2D video and video compression.

[0099] Hardware such as an HMD connected to the WTRU may display the output of the decoder so that the user may experience extended reality.

[0100] In general, XR is a technology that enables interactive experiences for users, where the real world environment and / or video content is enhanced by virtual content, which can be defined across multiple sensory modalities including vision, hearing, touch, etc. During the runtime of the XR application, virtual content (e.g., 3D content or audio / video files) can be rendered in real time in a manner consistent with the user's situation (e.g., environment, viewpoint, device, etc.). Scene graphs (e.g., such as scene graphs proposed by Khronos / glTF and its extensions defined in the MPEG scene description format, or scene graphs proposed by Apple / USDZ, for example) can be possible ways to represent the content to be rendered. They combine a declarative description of the scene structure that links real environment objects and virtual objects on the one hand, and a binary representation of virtual content on the other hand. The scene description framework ensures that timed media and corresponding related virtual content are available at any time during the rendering of the application. The scene description can also carry scene-level data that describes how the user can interact with the scene objects at runtime for an immersive XR experience.

[0101] Various XR applications can be applied to different contexts and real or virtual environments. For example, in an industrial XR application, when a camera mounted on a head-mounted display device detects a reference object (part B of an engine) in the real environment, a virtual 3D content item (e.g., part A of an engine) is displayed. The 3D content item is positioned in the real world at a defined position and scale relative to the detected reference object.

[0102] For example, in an XR application for interior design, when a given image from a catalog is detected in the input camera view, a 3D model of a piece of furniture is displayed. The 3D content is positioned in the real world at a position and scale defined relative to the detected reference image. In another application, when the user enters an area close to a church (e.g., real, or virtually rendered in an extended real environment), a certain audio file may start playing. In another example, when the user sees a given can of soda in the real environment, an ad jingle file may be played. In an outdoor gaming application, various virtual characters may appear depending on the semantics of the scenery observed by the user. For example, a bird character is suitable for a tree, so if the XR device's sensors detect a real object described by the semantic tag "tree", a bird flying around in the tree may be added. In a companion application implemented by smart glasses (e.g., a type of HMD), when a car is detected in the field of view of the user's camera, a car noise may be initiated in the user's headset to warn him of potential danger. In addition, the sound may be spatialized so that it arrives from the direction from which the car was detected.

[0103] XR applications can also augment video content instead of the real environment. The video can be displayed on a rendering device, and when a timed event is detected in the video, a virtual object described in the node tree can be overlaid. In this case, the node tree can only include virtual object descriptions.

[0104] In XR applications, scene descriptions can be used to combine an explicit and easily parsable description of the scene structure with some binary representation of the media content. In one example, there may be a grammar for a data stream that encodes an extended reality scene description. The structure of an XR scene may include a container that organizes the stream in independent elements of the grammar. The structure may include a header portion, which is a set of data common to each grammatical element of the stream. For example, the header portion may include some metadata about the grammatical elements, describing the properties and roles of each of them. The structure may also include a payload of one or more elements containing the grammar. The grammatical elements may include data representing media content items described in nodes related to virtual elements of the scene graph. Images, meshes, and / or other raw data may have been compressed according to a compression method. The grammatical elements may be part of the payload of the data stream and may include data encoding the described scene description according to the principles of the present invention as described herein.

[0105] Lighting can be an important element in an XR scene, and a description format is needed to describe different types of light with specific properties. Light can provide the scene with the ability to have shadows that AR experiences require.

[0106] Figure 3 Example illustrating the KHR_lights_punctual gltf extension. The Khronos gltf framework may provide such a scene description format featuring extensions specific to lights.

[0107] The KHR_lights_punctual extension defines three "punctual" light types: directional, point, and spot. Punctual lights can be defined as parameterized infinitesimal points that emit light in a well-defined direction and intensity. These lights can be indexed by a node and inherit the node's transformations.

[0108] KHR_lights_punctual can support area lights, because this light type can be an efficient way to integrate virtual objects in a real environment, can efficiently describe real lights, and can be incorporated into game engines such as Unity or UnrealEngine. As discussed in this article, the glTF format can include support for area lights.

[0109] The area light can be a square quad with the following parameters (e.g. added to each light parameter that already exists): 2D texture of the area, width and / or height. It inherits the parent transform like other light types; for example, light components describing its properties can be indexed by scene graph nodes that may feature transform elements (e.g. translation, rotation, scale) that can affect certain light spatial properties (e.g. the size of an area light or the pose of a spotlight).

[0110] Figure 4 Example illustrating the EXT_lights_image_based gltf extension.

[0111] The EXT_lights_image_based extension may define an array of image-based lights at the root of the glTF, and then index one per scene. Such image-based lights may provide ambient light that exists around the scene (e.g., not coming from a specific object). Each image-based light definition may include a single cubemap describing the specular radiance, l=2 spherical harmonic coefficients of the diffuse irradiance, and / or rotation and intensity values ​​for the scene. The specularImages array may contain indexes (e.g., indices) of images, one to be displayed on each interior face of the cube.

[0112] The MPEG-I scene description framework may define glTF extensions (eg, MPEG_animation_timing) that allow the state of a glTF animation to be controlled over time.

[0113] It may also define MPEG_accessors_timed that implements accessors dedicated to timed data (eg, audio, video, or any data that can change over time).

[0114] The KHR_lights_punctual and EXT_lights_image_based extensions define static properties of each light component, and in time evolving scenes, a mechanism may be needed / implemented to describe predefined time evolving light properties. For example, a lighting animation of a set of spots in a virtual scene that changes the color and intensity of the light. Or a light that moves in the scene.

[0115] As described herein, there may be devices, methods, and / or XR applications that utilize an extension that includes timing characteristics of an illumination source node. In some cases, the extension may be included in the glTF 2.0 framework.

[0116] The lighting sources described by the scene description document may be controlled by a mechanism that describes time-evolving light properties. The activation timing of the control event may be identified by the timing of samples in the lighting media source. Once a lighting temporal event is activated, the modifications to be applied to the lighting may be determined based on the lighting data in the scene description data and / or the information provided by the lighting samples.

[0117] This enhancement of the scene description may be defined at the scene level to describe a lighting animation that may involve several light nodes. Additionally / alternatively, it may be defined at the light node level to animate a single light node.

[0118] In some cases, the subject matter described herein may be read in the context of the MPEG-I Scene Description Framework using the Khronos glTF extension mechanism to support additional scene description features.

[0119] Figure 5 Example illustrating an MPEG_media element for time-evolving light properties.#

[0120] There may be a glTF extension MPEG_lighting_timing that identifies time-evolving light properties, as further disclosed herein.

[0121] Timing samples describing the evolution of light properties may be provided in a text JSON content file. The content may be described with MPEG_media extensions (e.g. such as defined in MPEG-I SD framework) with mime type equal to "application / json".

[0122] Figure 6 An example of a JSON file used to control a glTF light source.

[0123] Each sample in the JSON file may contain information for each light element, such as KHR_lights_punctual or EXT_lights_image_based types. For KHR_lights_punctual, there may be: an index into the glTF node array of the light node; a transformation element applied to the light node (e.g., a transformation matrix or a set of translation, rotation, and / or scale elements); and / or all nodes indexing the same light component are affected by the new light property. For both types, each sample may also include one or more light properties that override the initial static properties described in the light component from the glTF light array (e.g., static light properties may be defined in the glTF file, and the properties included in the JSON file may override them).

[0124] The timing accessor is preferably selected for high evolution rates (eg at the frame rate). Otherwise, the source index may be used to index a set of timing samples.

[0125] The MPEG_lighting_timing extension can be defined at another level in a glTF file, such as: at the scene level, for describing a lighting animation that may involve several light nodes, or for scene-unique EXT_lights_image_based lights; at the light component level, for EXT_lights_image_based lights, and also for KHR_lights_punctual lights, if the same light animation is to be applied to several nodes that index the same light component (e.g., there is no transform animation in this case); and / or at the light node level (e.g., for KHR_lights_punctual lights), for animating a single light node with modifications of transform elements.

[0126] Figure 7 An example of a scene-level MPEG_lighting_timing extension is illustrated. With respect to scene-level extensions, the MPEG_lighting_timing extension may specify an MPEG_media element containing light samples. At 701, a source file referenced in the MPEG_lighting_timing extension may include timing samples of a plurality of light sources.

[0127] Figure 8 An example of MPEG_lighting_timing extension at light component level is illustrated. With respect to the light component level extension, one or several light items in the light component array may be replaced by an MPEG_lighting_timing indexed by MPEG_accessor_timed, as shown at 801 and 802. In this case, all nodes indexing this light may be animated. In some cases, the timing accessor may point to a buffer containing light samples. In this case, the samples may contain only light component attributes and neither an index field nor a transform field. The buffer pointed to by the timing accessor may include time evolution parameters of the light items.

[0128] For image-based lights, if all you want is to animate the specular image, you may not need a light sample file: in the EXT_lights_image_based element, the image of the cubemap may be replaced by a video texture defined by MPEG_texture_video (e.g. in MPEG-I SD framework). For example, one image of a face of a cube (e.g., or any shape for that matter) may be replaced by a video texture for that face of the cube.

[0129] In one example, the MPEG_lighting_timing extension may also be applied to area lights if this type is included in the KHR_lights_punctual available types: the 2D parameters of the area lights may be replaced by the video texture.

[0130] Fig. 9 Example of a video texture illustrating EXT_lights_image_based lights. The specularImages parameter of image-based lights may reference a video texture instead of a static image, as shown at 901 .

[0131] Regarding the light node level extension, in order to animate only one node, the MPEG_lighting_timing extension may reference the MPEG_Media source.

[0132] In one example, the WTRU may obtain scene data and related lighting information. The lighting information may enhance / modify the lighting of the scene. The lighting information may be a video texture (e.g., MPEG_texture_video) of one or more faces of a cube map. If the lighting information includes a video, the video will contain both the lighting image and the timing information. The WTRU may render and display the scene based on the scene data and the lighting information. At a certain point in time (e.g., based on the video), there may be changes in the scene that cause the lighting of the scene to change. In some instances, the scene may be 3D, may include one or more objects or areas, and may be interactive with a user of the WTRU. In some instances, color parameters may also be included in the lighting information.

[0133] Fig.10 An example of the MPEG_lighting_timing extension at the light node level is illustrated. As shown at 1001, a source file indexed in the MPEG_lighting_timing extension may include a node entry and its indexed time evolution parameters of the light (eg, light parameters and / or transformation parameters).

[0134] Table 1 is an example of parameters that may be included in the MPEG_lighting_timing extension.

[0135]

[0136] Table 1

[0137] Each face may have an image texture or a video texture. In one example, all faces of a cubemap may be processed in the same way. In one example, each face may be processed differently. In one case, a cubemap may have six videos / images (e.g., one for each face of a cube). In one case, a cubemap may have one video / image for all faces. In one case, a cubemap may have less than six videos / images (e.g., where at least one file is reused for two or more faces; a subset of faces may share the same video / image). In one case, a cubemap may have one video / image that may be split into six sub-textures (e.g., one for each face of a cube). In one case, there may be one video / image for all faces of a cubemap.

[0138] Fig.11 An example of an XR application is illustrated. In one example, a WTRU may implement a method (e.g., an XR application may be executed by a processor of the WTRU and stored on a storage medium / memory of the WTRU). At 1101, the WTRU may obtain scene description data for a 3D scene. At 1102, the WTRU may obtain lighting information. The lighting information may include one or more parameters of a light source for a cube map, such as texture parameters, where the texture parameters may be a video. In one instance, the video may be applied to each face of the cube map. At 1103, the WTRU may render the 3D scene based on the lighting information and the scene description data, so that the lighting of the 3D scene changes over time according to the video. At 1104, the WTRU may display the rendered 3D scene (e.g., through an HMD that is part of the WTRU or external to the WTRU). In one example, the one or more parameters may also include intensity or color. In one example, the 3D scene may change over time (e.g., due to an event, as part of a video). In one example, the lighting information may apply to one or more light sources of the 3D scene. In one example, the 3D scene changes may be part of an animation of light sources involving the 3D scene over time. In one example, the scene is rendered continuously because the scene changes over time (e.g., due to a video file, which is a series of images linked to a time progression). In an alternative to the example shown, the scene description data may include lighting data, and / or there may be a receiving step that retrieves all necessary information for rendering the scene (e.g., one or more of the information / data / parameters disclosed herein).

[0139] In another example, a WTRU may implement a method (e.g., an XR application may be executed by a processor of the WTRU and stored on a storage medium / memory of the WTRU). At 1101, the WTRU may obtain scene description data for a 3D scene. At 1102, the WTRU may obtain lighting information and timing information. The lighting information may be associated with the timing information to enhance the lighting of the 3D scene. The lighting information may also include one or more parameters, such as texture parameters, where the texture parameters may include video textures. At 1103, the WTRU may display the 3D scene (e.g., through an HMD that is part of the WTRU or external to the WTRU). At 1104, the WTRU may process the lighting information. At 1105, the WTRU may determine that an event has occurred relative to the timing information. At 1106, the WTRU may enhance the 3D scene based on the event and the lighting information so that the lighting of the 3D scene changes over time according to the timing information and the lighting information. In one example, the 3D scene may include an object or an area. In one example, the one or more parameters may also include intensity or color.

[0140] In one example, the WTRU may include one or more processors, wherein at least one of the one or more processors may be a graphics processor. The graphics processor may generate an output (e.g., an extended reality) on an appropriate display (e.g., a headset, glasses, an immersive monitor, an LCD, an OLED, etc.). The output may be determined from one or more files including data about a scene or model (e.g., visual information, scene description data, etc.). The file may comply with one or more standards and may include one or more information and / or elements related to lighting (e.g., the aforementioned extensions, flags, accessors, etc.). For example, as disclosed herein, the file may contain one or more lighting-related elements that, when processed by the graphics processor, modify the output so that the lighting is presented differently. For example, if the output is of a scene or model (e.g., 3D or 2D), the one or more lighting-related elements may enable the output to display light in a specific area of ​​the scene, wherein the displayed light may have one or more properties that further control how precisely the light is displayed in the scene (e.g., intensity, power, color, lumens, etc.) and how aspects of objects in the scene are affected. The change may be based on the graphics processor calculating and / or processing one or more lighting-related elements, as discussed herein. In one example, one or more lighting-related elements may include timing aspects, as disclosed herein, including time-evolving light properties. There may be a lighting sample that provides a basis for applying and / or replicating lighting elements; the lighting sample may also include a timing element. After a predetermined event (e.g., as part of a timed event within a scene), a lighting event may be determined and activated, and the timeline of the lighting (e.g., the change of light with respect to time) may be determined by the lighting data in the scene description data. In addition, the information provided by the lighting sample may be applied. This change / enhancement of the scene description may be defined at the scene level to describe a lighting animation that may involve one or more light nodes. Additionally / alternatively, it may be defined at the light node level to animate a single light node.

[0141] In an example, there may be a method for timing lighting in an extended reality experience. A device may be configured to display a scene. The device may process lighting information, wherein the lighting information includes timing information related to enhancing the lighting of a scene, or an object in a scene, or an area in a scene. The device may determine that an event has occurred, such as a scheduled time event portion of a scene. The device may enhance the scene based on the event and the lighting information, so that the lighting of the scene, or an object in the scene, or an area in the scene may change over time based on the timing information. In some cases, the lighting information may apply to several light nodes of the scene. In some cases, the change in the lighting of the scene is part of an animation that describes the change of several lights of the scene over time.

[0142] Although the features and elements are described above in particular combinations, it will be understood by those of ordinary skill in the art that each feature or element may be used alone or in any combination with other features and elements. In addition, the methods described herein may be implemented in a computer program, software, or firmware incorporated in a computer-readable medium for execution by a computer or processor. Examples of computer-readable media include electronic signals (sent via a wired or wireless connection) and computer-readable storage media. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media (such as built-in hard disks and removable disks), magneto-optical media, and optical media (such as CD-ROM disks and digital versatile disks (DVDs)). A processor associated with software may be used to implement a radio frequency transceiver for a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. A method implemented by a wireless transmit receive unit (WTRU), the method comprising: Obtaining scene description data for a 3D scene; obtaining lighting information, wherein the lighting information comprises one or more parameters of a light source for a cubemap, wherein one of the one or more parameters comprises a texture to be applied to each face of the cubemap, wherein the texture comprises a video; and The 3D scene is rendered based on the lighting information and the scene description data such that lighting of the 3D scene changes over time according to the video of the texture. 2 . The method of claim 1 , wherein the 3D scene changes over time due to an event that is a scheduled time event that is part of the video. The method of claim 1 , wherein the lighting information applies to one or more light sources of the 3D scene. The method of claim 1 , wherein the 3D scene change is part of an animation of a number of lights involving the 3D scene over time. 5 . The method of claim 1 , wherein the one or more parameters further include intensity or color, and wherein the rendering of the 3D scene includes changing the intensity of light of the 3D scene or the color of light of the 3D scene.

6. The method of claim 1, wherein the rendered 3D scene is displayed via a head mounted display operatively connected to the WTRU or a portion of the WTRU.

7. A wireless transmit receive unit (WTRU), the WTRU comprising: A component for obtaining scene description data for a 3D scene; means for obtaining lighting information, wherein the lighting information comprises one or more parameters of a light source for a cubemap, wherein one of the one or more parameters comprises a texture to be applied to each face of the cubemap, wherein the texture comprises a video; and Means for rendering the 3D scene based on the lighting information and the scene description data such that lighting of the 3D scene changes over time in accordance with the video of the texture.

8. The WTRU of claim 7, wherein the 3D scene changes over time due to an event that is a scheduled time event that is part of the video.

9. The WTRU of claim 7, wherein the lighting information applies to one or more light sources of the 3D scene.

10. The WTRU of claim 7, wherein the 3D scene change is part of an animation of a number of lights involving the 3D scene over time.

11. The WTRU of claim 7, wherein the one or more parameters further include intensity or color, and wherein the rendering of the 3D scene includes changing the intensity of light of the 3D scene or the color of light of the 3D scene.

12. The WTRU of claim 7, wherein the rendered 3D scene is displayed via a head mounted display operatively connected to the WTRU or a portion of the WTRU.

Images