Mechanism for controlling real environment computing refresh rate for augmented reality (AR) experience

Abstract

Example embodiments provide a mechanism that enables a user equipment, UE, to adjust an environmental computing refresh rate to improve user augmented reality, XR, experience and limit resource usage. An example method includes obtaining sensor data describing a user environment; obtaining information indicative of at least one factor from among user movement information, environmental evolution information, network condition information, and battery life information; determining a candidate environment calculation refresh rate based on the at least one factor; selecting an environment calculation refresh rate as the minimum value of the candidate environment calculation refresh rate and the maximum environment calculation refresh rate; and updating the real environment calculation using the selected environment calculation refresh rate.

Description

[0001] Cross - reference to related applications

[0002] This application claims priority to European Patent Application EP22306886.7, filed on December 15, 2022, the entire content of which is incorporated herein by reference. Background Art

[0003] In an Augmented Reality (AR) experience, various devices such as optical see - through glasses or video see - through devices (e.g., smartphones, tablets, headsets) are used to insert computer - generated virtual elements into a user's real environment.

[0004] Therefore, information about the user's environment is crucial for the seamless spatial combination of virtual and real objects. This environmental information is used to implement features such as: the stable pose of inserted virtual objects depends on the localization and tracking of some natural features of the user's environment; proper collision handling (e.g., a virtual ball rolling on a real table and falling to the floor); and achieving coherent rendering by considering occlusion and lighting.

[0005] Information about the user's environment can be implemented using dedicated real - time computing modules such as Google's ARCore, Apple's ARKit environmental understanding module, or Microsoft's spatial mapping and scene understanding module. They rely on the real - time output of embedded device sensors such as depth or color cameras and inertial measurement units (IMUs) for user pose estimation. Summary of the Invention

[0006] Example embodiments provide a mechanism that enables a user equipment (UE) to adjust the environmental computing refresh rate to improve the user's XR experience and limit resource usage. In some embodiments, an XR content creator provides one or more of the following parameters to each user equipment sharing an AR experience at the start of a relevant XR session: the nominal (or maximum) and minimum values of the environmental computing refresh rate; a reference to an action (s) to be performed in case the minimum refresh rate cannot be achieved; a reference to a scan and / or semantic representation of the real environment; a parameter indicating whether the scan and / or semantic representation should be used only for localization (baseline), collision handling, and / or advanced rendering. Some embodiments include an adaptive runtime update of the environmental computing refresh rate managed by the UE based on a control mechanism.

[0007] A method according to some embodiments includes, at an augmented reality user device: obtaining sensor data describing a user environment; obtaining information indicating at least one factor from among: user movement information, environment evolution information, network condition information, and battery life information; determining a candidate environment calculation refresh rate based on the at least one factor; selecting the environment calculation refresh rate as the minimum of the candidate environment calculation refresh rate and a maximum environment calculation refresh rate; and using the selected environment calculation refresh rate to update a real environment calculation.

[0008] An apparatus according to some embodiments includes one or more processors configured to at least perform the following operations: obtaining sensor data describing a user environment; obtaining information indicating at least one factor from among: user movement information, environment evolution information, network condition information, and battery life information; determining a candidate environment calculation refresh rate based on the at least one factor; selecting the environment calculation refresh rate as the minimum of the candidate environment calculation refresh rate and a maximum environment calculation refresh rate; and using the selected environment calculation refresh rate to update a real environment calculation.

[0009] Some embodiments of the method or apparatus further include obtaining scene description data describing an extended reality scene and presenting the scene in the user environment.

[0010] Some embodiments of the method or apparatus further include receiving metadata indicating a maximum environment calculation refresh rate.

[0011] In some embodiments of the method or apparatus, the selection of the environment calculation refresh rate is based on a periodic basis.

[0012] In some embodiments of the method or apparatus, the candidate environment calculation refresh rate is determined using a weighted sum of parameters representing at least two of user movement, environment evolution, network conditions, or battery life.

[0013] Some embodiments of the method or apparatus further include receiving metadata indicating the weights used in the weighted sum.

[0014] In some embodiments of the method or apparatus, the candidate environment calculation refresh rate is determined based at least on user movement information and environment evolution information.

[0015] Some embodiments of the method or apparatus further include obtaining metadata indicating a maximum environment calculation refresh rate.

[0016] Some embodiments of the method or apparatus further include the maximum environment calculation refresh rate being determined at least in part based on network condition information.

[0017] In some embodiments of the method or apparatus, the candidate environment calculation refresh rate is determined based at least in part on at least one of user movement information, environmental evolution information, and network condition information.

[0018] Some embodiments of the method or apparatus further include: obtaining information indicating a threshold environment calculation refresh rate and a specified action; and performing the specified action in response to a determination that the environment calculation refresh rate is less than the threshold environment calculation refresh rate.

[0019] In some embodiments of the method or apparatus, the candidate environment calculation refresh rate is calculated based at least in part on metadata received in at least one of a scene description file and a manifest file.

[0020] A method according to some embodiments includes: providing scene description data for an extended reality experience, where the scene description data includes at least one of the following information: information indicating a threshold of the environment calculation refresh rate, information indicating a maximum value of the environment calculation update rate, and information indicating at least one specified action to be performed in response to the environment calculation refresh rate dropping below the threshold.

[0021] An apparatus according to some embodiments includes one or more processors configured to at least perform the following operations: providing scene description data for an extended reality experience, where the scene description data includes at least one of the following: information indicating a threshold of the environment calculation refresh rate, information indicating a maximum value of the environment calculation update rate, and information indicating at least one specified action to be performed in response to the environment calculation refresh rate dropping below the threshold.

[0022] In some embodiments of the method or apparatus, the scene description data further includes at least one weight value for calculating the environment calculation refresh rate.

[0023] In some embodiments of the method or apparatus, for the at least one weight value, the scene description data further includes information identifying a factor associated with the corresponding weight value.

[0024] In some embodiments of the method or apparatus, the factor includes at least one of user movement information, environmental evolution information, network condition information, and battery life information.

[0025] Example embodiments further include an apparatus that includes one or more processors configured to perform any of the methods described herein.

[0026] Example embodiments further include a computer-readable medium that includes instructions for causing one or more processors to perform any of the methods described herein. The computer-readable medium may be a non-transitory storage medium.

[0027] The example embodiment also includes a computer program product that includes instructions that, when executed by one or more processors, cause the one or more processors to perform any of the methods described herein.

[0028] A signal according to some embodiments includes scene description data for a 3D scene that includes elements as described above.

[0029] A computer-readable medium according to some embodiments includes scene description data for a 3D scene that includes elements as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1A is a cross-sectional schematic view of a waveguide display that can be used with an augmented reality application according to some embodiments.

[0031] Figure 1B - 1C is a cross-sectional schematic view of an alternative display type that can be used with an augmented reality application according to some embodiments.

[0032] Figure 1D is a functional block diagram of a system used in some embodiments described herein.

[0033] Figure 2 is a schematic diagram of an example of a spatial mapping and scene understanding computing module.

[0034] Figure 3 is a flowchart showing an initialization process according to some embodiments.

[0035] Figure 4 is a flowchart showing a refresh rate update process according to some embodiments.

[0036] Figure 5 is a flowchart showing a method for determining an updated refresh rate according to some embodiments.

[0037] Figure 6 is a flowchart showing a method for determining an updated refresh rate according to some embodiments.

[0038] Figure 7 is a flowchart showing a method for determining an updated refresh rate according to some embodiments.

[0039] Figure 8 is a flowchart showing a method for determining an updated refresh rate according to some embodiments. DETAILED DESCRIPTION

[0040] Figure 1A An example augmented reality (AR) display device is shown. In the present disclosure, augmented reality is also referred to as extended reality (XR). Figure 1AFIG. 0 is a schematic cross-sectional side view of a waveguide display device in operation. An image is projected by an image generator 102. The image generator 102 can project an image using one or more of a variety of techniques. For example, the image generator 102 can be a laser beam scanning (LBS) projector, a liquid crystal display (LCD), a light-emitting diode (LED) display (including an organic LED (OLED) or a micro LED (μLED) display), a digital light processor (DLP), a liquid crystal on silicon (LCoS) display, or other types of image generators or light engines.

[0041] Light representing the image 112 generated by the image generator 102 is coupled into the waveguide 104 through a diffractive inner coupler 106. The inner coupler 106 diffracts the light representing the image 112 into one or more diffraction orders. For example, a ray 108, which is one of the rays representing a part of the bottom of the image, is diffracted by the inner coupler 106, and one of the diffraction orders 110 (e.g., the second order) is at an angle capable of propagating through the waveguide 104 by total internal reflection. The image generator 102 displays an image according to the instructions of a control module 124, which is used to render image data, video data, point cloud data, or other displayable data.

[0042] At least a portion of the light 110 that has been coupled into the waveguide 104 through the diffractive inner coupler 106 is coupled out of the waveguide through a diffractive outer coupler 114. At least some of the light coupled out of the waveguide 104 replicates the incident angle of the light coupled into the waveguide. For example, in the illustration, the outer-coupled rays 116a, 116b, and 116c replicate the angle of the inner-coupled ray 108. Since the light leaving the outer coupler replicates the direction of the light entering the inner coupler, the waveguide substantially replicates the original image 112. The user's eye 118 can focus on the replicated image.

[0043] In Figure 1A the example of, the outer coupler 114 only couples a portion of the light with each reflection output, thus allowing a single input beam (such as beam 108) to generate multiple parallel output beams (such as beams 116a, 116b, and 116c). In this way, even if the eye is not perfectly aligned with the center of the outer coupler, at least some light from each part of the image may reach the user's eye. For example, if the eye 118 moves downward, even if the beams 116a and 116b do not enter the eye, the beam 116c may enter the eye, so the user can still perceive the bottom of the image 112 despite the shift in position. Therefore, the outer coupler 114 operates partially as an exit pupil expander in the vertical direction. The waveguide may also include one or more additional exit pupil expanders ( Figure 1A not shown in ) to expand the exit pupil in the horizontal direction.

[0044] In some embodiments, waveguide 104 is at least partially transparent to light originating outside the waveguide display. For example, at least some of the light 120 from a real-world object, such as object 122, passes through waveguide 104, allowing the user to see the real-world object while using the waveguide display. When the light 120 from the real-world object also passes through diffraction grating 114, there will be multiple diffraction orders, resulting in multiple images. To minimize the visibility of the multiple images, it is desirable for the zero-order diffraction (no deviation caused by grating 114) to have a high diffraction efficiency for light 120 and the zero-order, while the diffraction energy of the higher orders is low. Thus, in addition to expanding and out-coupling virtual images, out-coupler 114 is preferably configured to let the zero-order of the real image pass through. In such embodiments, the images displayed by the waveguide display may appear to be superimposed on the real world.

[0045] Figure 1B An alternative type of augmented reality head-mounted display that can be used in some embodiments is schematically illustrated. In augmented reality head-mounted display device 1250, control module 1254 controls display 1256 (which can be an LCD) to display an image. The head-mounted display includes a partially reflective surface 1258 that reflects (and in some embodiments, both reflects and focuses) the image displayed on the LCD to make the image visible to the user. The partially reflective surface 1258 also allows at least some external light to pass through, allowing the user to see their surroundings.

[0046] Figure 1C An alternative type of augmented reality head-mounted display that can be used in some embodiments is schematically illustrated. In head-mounted display device 1260, control module 1264 controls display 1266 (which can be an LCD) to display an image. The image is focused by one or more lenses of display optics 1268 to make the image visible to the user. In Figure 1C the example, the external light does not directly reach the user's eyes. However, in some such embodiments, external camera 1270 can be used to capture images of the external environment and display these images on display 1266 together with any virtual content that can also be displayed.

[0047] The embodiments described herein are not limited to any particular type or structure of AR display device.

[0048] The augmented reality display device and its control electronics can be implemented using a system such as Figure 1D shown. Figure 1DFIG. is a block diagram of an example of a system in which various aspects and embodiments are implemented. System 1000 may be embodied as a device including various components described below and is configured to perform one or more aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptops, smartphones, tablets, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected household appliances, and servers. The elements of system 1000 may be embodied in a single integrated circuit (IC), multiple ICs, and / or discrete components, either alone or in combination. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or discrete components. In various embodiments, system 1000 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more aspects described in this document.

[0049] System 1000 includes at least one processor 1010, which is configured to execute instructions loaded therein to implement various aspects described in this document, for example. Processor 1010 may include embedded memory, input / output interfaces, and various other circuits known in the art. System 1000 includes at least one memory 1020 (e.g., volatile memory devices and / or non-volatile storage devices). System 1000 includes a storage device 1040, which may include non-volatile memory and / or volatile storage, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, disk drives, and / or optical disk drives. As a non-limiting example, storage device 1040 may include internal storage devices, attached storage devices (including removable and non-removable storage devices), and / or network-accessible storage devices.

[0050] System 1000 includes an encoder / decoder module 1030, which is configured to process data, for example, to provide encoded video or decoded video, and encoder / decoder module 1030 may include its own processor and memory. Encoder / decoder module 1030 represents the module(s) that may be included in a device to perform encoding and / or decoding functions. As is well known, a device may include one or both of an encoding and a decoding module. Additionally, encoder / decoder module 1030 may be implemented as a separate element of system 1000 or may be incorporated into processor 1010 as a combination of hardware and software known to those skilled in the art.

[0051] The program code to be loaded onto the processor 1010 or the encoder / decoder 1030 to execute the various aspects described in this document can be stored in the storage device 1040 and subsequently loaded onto the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 can store one or more of the various items during the execution of the processes described in this document. Such stored items can include, but are not limited to, input video, decoded video or a portion of the decoded video, bitstreams, matrices, variables, and intermediate or final results from equations, formulas, operations, and operation logic processing.

[0052] In some embodiments, the memory internal to the processor 1010 and / or the encoder / decoder module 1030 is used to store instructions and provide working memory for the processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device can be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory can be the memory 1020 and / or the storage device 1040, such as, for example, dynamic volatile memory and / or non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store, for example, the operating system of a television. In at least one embodiment, a fast external dynamic volatile memory such as RAM is used as the working memory for video encoding and decoding operations, such as for MPEG-2 (MPEG refers to the Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, 13818-1 is also known as H.222, 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard being developed by the Joint Video Exploration Team JVET).

[0053] As shown in block 1130, input can be provided to the elements of the system 1000 through various input devices. Such input devices include, but are not limited to: (i) a radio frequency (RF) section that receives an RF signal, such as transmitted by a broadcaster over the air, (ii) component (COMP) input terminals (or a set of COMP input terminals), (iii) universal serial bus (USB) input terminals, and / or (iv) high-definition multimedia interface (HDMI) input terminals. Figure 1C Other examples not shown include composite video.

[0054] In various embodiments, the input device of block 1130 has corresponding input processing elements known in the art. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also known as selecting a signal, or restricting the signal band to a certain band), (ii) down-converting the selected signal, (iii) again restricting the band to a narrower band to select a signal band that may be referred to as a channel in some embodiments, (iv) demodulating the down-converted and band-restricted signal, (v) performing error correction, and (vi) de-multiplexing to select a desired data packet stream. The RF section of various embodiments includes one or more elements that perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, filters, a down-converter, a demodulator, an error corrector, and a de-multiplexer. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a near-baseband frequency) or to baseband. In one set-top box embodiment, the RF section and its associated input processing elements receive an RF signal transmitted through a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and filtering again to a desired frequency band. Various embodiments rearrange the order of the above (and other) elements, delete some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, such as inserting an amplifier and an analog-to-digital converter. In various embodiments, the RF section includes an antenna.

[0055] In addition, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 1000 to other electronic devices via a USB and / or HDMI connection. It should be understood that various aspects of input processing, such as Reed-Solomon error correction, may be implemented within a separate input processing IC or within the processor 1010 as needed. Similarly, aspects of USB or HDMI interface processing may be implemented within a separate interface IC or within the processor 1010 as needed. The streams of demodulation, error correction, and de-multiplexing are provided to various processing elements (including, for example, the processor 1010 and an encoder / decoder 1030 operating in conjunction with memory and storage elements) to process the data stream as needed for presentation on an output device.

[0056] The various elements of the system 1000 may be disposed within an integrated housing. Within the integrated housing, the various elements may be interconnected with each other and data may be transmitted therebetween using suitable connection means 1140, such as internal buses known in the art (including inter-integrated circuit (I2C) buses, wiring, and printed circuit boards).

[0057] System 1000 includes a communication interface 1050 that is capable of communicating with other devices via a communication channel 1060. The communication interface 1050 can include, but is not limited to, a transceiver that is configured to send and receive data over the communication channel 1060. The communication interface 1050 can include, but is not limited to, a modem or a network card, and the communication channel 1060 can be implemented, for example, within a wired and / or wireless medium.

[0058] In various embodiments, data is streamed or otherwise provided to system 1000 using a wireless network such as a Wi-Fi network (e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers)). The Wi-Fi signals of these embodiments are received via the communication channel 1060 and the communication interface 1050 that are adapted for Wi-Fi communication. The communication channel 1060 of these embodiments is typically connected to an access point or a router that provides access to an external network including the Internet to allow streaming applications and other top communication. Other embodiments use a set-top box to provide streamed data to system 1000, and the set-top box passes the data through an HDMI connection of the input box 1130. Still other embodiments use an RF connection of the input box 1130 to provide streamed data to system 1000. As described above, various embodiments provide data in a non-streaming manner. In addition, various embodiments use a wireless network other than Wi-Fi, such as a cellular network or a Bluetooth network.

[0059] System 1000 can provide output signals to various output devices, which include a display 1100, speakers 1110, and other peripheral devices 1120. The display 1100 of various embodiments includes, for example, one or more of a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. The display 1100 can be used for a television, a tablet, a laptop, a mobile phone (cellular phone), or other devices. The display 1100 can also be integrated with other components (e.g., as in a smartphone), or be separate (e.g., an external monitor for a laptop). In various examples of embodiments, the other peripheral devices 1120 include one or more of a standalone digital video disc (or digital versatile disc) (DVR, for both terms), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more of the peripheral devices 1120 that provide functions based on the output of system 1000. For example, a disc player performs the function of playing the output of system 1000.

[0060] In various embodiments, control signals are passed between system 1000 and display 1100, speaker 1110, or other peripheral devices 1120 using signaling such as AV.Link, consumer electronics control (CEC), or other communication protocols that enable control between devices with or without user intervention. The output devices may be communicatively coupled to system 1000 via dedicated connections through respective interfaces 1070, 1080, and 1090. Alternatively, the output devices may be connected to system 1000 using communication channel 1060 through communication interface 1050. Display 1100 and speaker 1110 may be integrated into a single unit with other components of system 1000 in an electronic device (such as a television). In various embodiments, display interface 1070 includes a display driver, such as a timing controller (T Con) chip.

[0061] Display 1100 and speaker 1110 may optionally be separate from one or more other components, for example, if the RF portion of input 1130 is part of a separate set-top box. In various embodiments where display 1100 and speaker 1110 are external components, the output signals may be provided through a dedicated output connection, which includes, for example, an HDMI port, a USB port, or a COMP output.

[0062] System 1000 may include one or more sensor devices 1095. Examples of sensor devices that may be used include one or more GPS sensors, gyroscope sensors, accelerometers, light sensors, cameras, depth cameras, microphones, and / or magnetometers. Such sensors may be used to determine information such as the location and orientation of a user. In cases where system 1000 is used as a control module (such as control modules 124, 1254) for an augmented reality display, the location and orientation of the user may be used to determine how to render image data so that the user perceives the correct part of a virtual object or virtual scene from the correct angle. In the case of a head-mounted display device, the location and orientation of the device itself may be used to determine the location and orientation of the user to present virtual content. In the case of other display devices, such as a phone, tablet, computer monitor, or television, other inputs may be used to determine the location and orientation of the user to present content. For example, a user may use a touch screen, keypad, or keyboard, trackball, joystick, or other input to select and / or adjust the desired viewpoint and / or viewing direction. In cases where the display device has sensors such as an accelerometer and / or gyroscope, the viewpoint and direction for presenting content may be selected and / or adjusted based on the movement of the display device.

[0063] Embodiments can be implemented by computer software implemented by a processor 1010, or by hardware, or by a combination of hardware and software. As a non-limiting example, embodiments can be implemented by one or more integrated circuits. The memory 1020 can be of any type suitable for the technical environment and can be implemented using any appropriate data storage technology, such as optical storage devices, magnetic storage devices, semiconductor-based storage devices, fixed memory, and removable memory, as non-limiting examples. The processor 1010 can be of any type suitable for the technical environment and can include one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture, as non-limiting examples.

[0064] Overview of the Scene Understanding Module

[0065] Figure 2 Schematically shows the relationship between sensor data, refresh rate, spatial mapping, and the scene understanding calculation module. Based on various sensor data collected by module 202, the real environment can be analyzed by the following two calculation modules. The spatial mapping calculation module 204 calculates a scanned representation of the real environment. The scanned representation data can include information such as a unique grid, which can be a set of connected primitives (triangles, quadrilaterals) and color textures related to the environmental geometry. The scene understanding calculation module 206 is used to segment the environment into separate semantic elements (e.g., labeled with "table", "laptop", "screen", etc.).

[0066] In some cases, depending on the AR experience to be presented, it is not necessary for both calculation modules to be present or active simultaneously. For example, if only a semantic environment representation consisting of a set of labels is used, spatial mapping calculation is not required. If only collision handling between real and virtual objects is used, scene understanding calculation is not required.

[0067] If the application is designed to process the geometry of a single real object (e.g., to remove an object to reduce the AR experience), spatial mapping and scene understanding calculations can be used.

[0068] Example embodiments relate to the adaptive selection and / or control of the environmental calculation refresh rate, which can be used by either or both of the spatial mapping calculation or the scene understanding calculation.

[0069] Traditional methods rely on real-time calculation of the real environment on each user's device sharing a common AR experience. However, even though this method provides effective knowledge of each user's environment, from the perspective of extended reality (XR) content creators, it lacks control over the environmental calculation refresh rate.

[0070] Overview of Example Embodiments

[0071] Example embodiments allow XR content creators to provide metadata based on context - and application - specific criteria, which enables user equipment (UE) to adjust the environmental computing refresh rate to improve the user experience and limit resource usage. During an AR / XR experience, the context and application - specific criteria may change. These criteria may include one or more of the following:

[0072] · The real environment, which can be static or dynamic (e.g., time - evolving).

[0073] · The user, who can be stationary or moving. As the user may explore the environment and may discover new parts of their real environment that were previously invisible, the stationary or moving classification of the user may change over time, and the amount of user movement may also vary.

[0074] · The user equipment state and / or evolution (e.g., battery life threshold, operation in different connection and power - saving modes such as idle, inactive, or connected mode, available CPU cycles, and / or available memory).

[0075] · In the case of delegating the computation of some parts (one or more parts) to an edge computing device, the network conditions and / or evolution (such as traffic load conditions in the uplink (UL) and / or downlink (DL), radio link conditions). Some thresholds for latency, round - trip time (RTT), and / or packet error rate can be defined.

[0076] · The use of the runtime user environment representation (e.g., positioning and tracking only, advanced rendering with occlusion and lighting, collision handling with physical simulation).

[0077] For example, for a static user environment, it may be more efficient for the XR content creator to provide a scan and / or semantic representation of the environment before the runtime AR experience to avoid unnecessary redundant scene understanding computations.

[0078] Even for a user environment that changes over time, the initial scan and / or semantic representation can be updated based on a predefined refresh rate / profile or based on runtime analysis that depends on certain prediction / change - detection algorithms.

[0079] XR metadata according to some embodiments can define the nominal (or maximum) value of the environmental computing refresh rate based on one or more of the following.

[0080] · The tolerable inaccuracies of the AR experience (e.g., the accuracy of the spatial positioning of virtual objects).

[0081] · The use of the runtime environment representation. For example, collision - only use will support a different refresh rate compared to rendering use.

[0082] The XR metadata provided in some embodiments can also define an additional minimum value of the refresh rate based on the environment in cases of poor user device capabilities and low battery power, allowing or preventing a degraded operation mode. In cases where part (one or more) of the real environment calculation is delegated to the network edge, other network-related conditions (such as congestion or a high packet error rate) may also switch to this degraded operation mode.

[0083] In some embodiments, the XR metadata can also define one or more actions for a user device to perform in cases where the supported environment calculation refresh rate is below the minimum refresh rate value. Examples of such actions include displaying a warning message, a gradual fade-out of virtual objects, and / or deactivating one or more predefined virtual objects (which can be identified in the configuration information) before stopping the rendering of virtual objects. In some embodiments, these actions provide a temporary and graceful transition from the XR experience to the stopping of virtual object rendering, such that the rendering of XR objects does not stop in a way that disorients the user.

[0084] Example embodiments provide a mechanism that enables a user equipment (UE) to adjust the environment calculation refresh rate to improve the user's XR experience and limit resource usage. This mechanism can use metadata provided by XR content creators related to context- and application-specific criteria.

[0085] In some embodiments, one or more of the following parameters are provided to a user device providing an AR experience to a user or to each user device providing a shared AR experience to multiple users in the metadata:

[0086] · The nominal (or maximum) and minimum values of the environment calculation refresh rate.

[0087] · A reference to the action(s) to be performed in cases where the minimum refresh rate cannot be reached before stopping the rendering of virtual objects.

[0088] · A reference to the scan and / or semantic representation of the real environment (if available at the start of the AR experience).

[0089] · A parameter indicating whether the scan and / or semantic representation is only used for localization (baseline), collision handling, and / or advanced rendering (such as any one or all of occlusion, coherent real / virtual lighting, object removal).

[0090] Some embodiments include an initialization where an application running on the UE retrieves the parameters provided by the XR content creator as metadata to configure the control mechanism for the environment calculation refresh rate. For example, the initialization can be performed at the start of an AR session.

[0091] Some embodiments include runtime updates to the ambient computing refresh rate managed by the UE based on the control mechanisms described herein. The runtime updates can be based on parameters and / or metadata received by the UE at the start of an AR session, based on updated parameters received by the UE during the session, or based on a combination of these parameters.

[0092] Initialization

[0093] In some embodiments, an XR scene description file is used to store control parameters for the ambient computing refresh rate mechanism. The scene description file can describe and provide references to all the assets that make up this AR experience, and can also provide other control parameters from the XR content creator, such as user navigation, interactivity, and AR anchoring. The XR scene description file can be used as an entry point for each client / user joining the XR session.

[0094] For the purposes of this description, initialization is detailed herein within the scope of the MPEG-I scene description framework, which uses the Khronos glTF extension mechanism to support additional scene description capabilities. However, the principles can alternatively be used with other existing or future descriptions of XR scenes.

[0095] Figure 3 An initialization process according to some embodiments is shown. At Figure 3 the example, an XR scene description is obtained at 302. Information indicating the nominal (or maximum) and minimum refresh rates for ambient computing is retrieved at 304. Such information can be retrieved from the XR scene description. At 306, information indicating which action(s) will be performed if the minimum refresh rate cannot be achieved is retrieved, for example, from the XR scene description. Not all embodiments must include using such information. At 308, an initial scan and / or semantic representation of the real environment is retrieved using sensor data, for example. At 310, the scan and / or semantic representation of the real environment is registered for handling rendering and / or collisions.

[0096] The control parameters obtained through initialization can be added to the scene description using extensions at the glTF scene or node level. Example embodiments are not limited to using a scene description file to provide metadata. For example, in some embodiments, the metadata can be provided in a manifest file (such as a DASH manifest file) or through other techniques.

[0097] In some embodiments, the MPEG_scene_real_environment glTF scene extension is defined with the parameters shown in Table 1 below.

[0098]

[0099]

[0100] Control parameters provided in the glTF scene extension in Table 1

[0101] In the example of Table 1, the nominal refresh rate parameter and the minimum refresh rate parameter are 60 frames per second and 20 frames per second, respectively. If the XR content creator does not want to define a fallback operation mode, the same value can be given to the nominal refresh rate and the minimum refresh rate, or the minimum refresh rate can be excluded.

[0102] If the minimum refresh rate cannot be achieved during runtime updates, parameters such as the above "actions" parameter can be used to provide references to one or more actions to be performed. If the XR content creator does not want to define these actions, it can be an empty array. In this example, the first two actions defined in the action array of the MPEG_scene_interactivity extension are referenced. The first action in this example is the ANIMATE action, which can be used to activate the display of a warning message. The second action in this example is the ANIMATE action, which can be used to animate the alpha / transparency channel of the color of a virtual object for fade-in and fade-out purposes.

[0103] Parameters such as the realNodes parameter in Table 1 can be used to reference one or more nodes of an array of glTF nodes that contain initial scan and / or semantic environment data (if available). In the example of Table 1, the environment data is provided in the first node of the glTF node array.

[0104] Parameters such as the envDataTargets parameter in Table 1 can be provided to indicate whether the environment data used for the pose of virtual objects should also be used additionally for collision handling and / or advanced rendering. In this example, the environment representation is used both for collision handling and for advanced rendering (e.g., for occlusion, coherent illumination).

[0105] In another example embodiment shown in Table 2 below, the control parameters are provided in the MPEG_node_real_environment glTF node extension. In this embodiment, the realNodes parameter is not used because the extension is added directly to the corresponding glTF node(s).

[0106]

[0107]

[0108] Control parameters provided in the glTF node extension in Table 2.

[0109] Refresh rate update

[0110] In applications that use a known static real environment, scans and / or semantic representations are already available and do not need to be computed at runtime. In this case, XR content creators can set the nominal refresh rate to zero and provide an initial (and therefore permanent) representation of the environment in the XR scene description file.

[0111] Figure 4 A method for providing a per-frame update of the ambient computation refresh rate according to some embodiments is shown. In some embodiments, the following may be performed before rendering each frame: Figure 4 The method shown. At 402, the latest representation of the real environment is obtained. At 404, the poses of virtual objects in the scene are updated. At 406, collisions in the scene are calculated. At 408, a new environment refresh rate is calculated. At 410, it is determined whether the new environment refresh rate is below a minimum threshold. If so, a defined action can be performed at 412, such as displaying a warning message. At 414, a frame is rendered.

[0112] Rendering of the 3D scene is performed by a rendering module, which can be located in the user equipment (UE) or implemented in an edge computing device in the case of a split rendering architecture.

[0113] Before rendering a frame, the rendering module obtains the latest scan and / or semantic representation of the real environment, which is calculated at a refresh rate determined during the previous frame.

[0114] Embedded sensor data is acquired by the user equipment (UE). The latest environment representation is then computed in the UE or using edge computing, depending on the UE's computing capabilities. The environment representation is then used to perform one or more of the following operations: update the pose of virtual objects in the real environment; compute collisions between virtual and real objects if the collision mode is defined as envDataTargets in the XR scene description; render frames using advanced rendering features (such as occlusion handling, coherent lighting between real and virtual objects, object removal, and other features that may facilitate realistic rendering and scene interaction) if the rendering mode is defined as envDataTargets in the XR scene description.

[0115] In different embodiments, the timing of determining a new value for the environment to calculate the refresh rate can be different. For example, it can be done on a per-frame basis, periodically every N frames, or on-demand / triggered by the application. The application can rely on previous refresh rate values to determine when to calculate a new value. For example, if the refresh rate value is very stable over a period of time, the application can determine to skip the calculation for that frame or calculate it once every N frames (where N can be a preset number). In the opposite case, when the refresh rate value changes rapidly, the application can determine to speed up the calculation to every frame.

[0116] In some embodiments, the calculation of the new value of the ambient computing refresh rate can be at least partially based on a prediction of the user's movement speed, which is extrapolated from a previous value. For example, the amount of user movement may change as the user may discover new parts of the real environment that were previously invisible. The refresh rate can be increased in response to determining an extrapolated faster user movement and can be decreased in response to determining an extrapolated slower user movement.

[0117] In some embodiments, the calculation of the new value of the ambient computing refresh rate can be at least partially based on a prediction of the real environment change speed, which is extrapolated from a previous value. The refresh rate can be increased in response to determining an extrapolated faster real environment change speed and can be decreased in response to determining an extrapolated slower real environment change speed.

[0118] In some embodiments, the calculation of the new value of the ambient computing refresh rate can be at least partially based on the evolution of the user device state, such as battery life. In response to determining that the battery life has dropped below a threshold (e.g., 10%), the refresh rate can be set to a defined minimum refresh rate, which can be an application-defined threshold.

[0119] In some embodiments, in the case of delegating some parts (one or more) of the calculation to edge computing, the calculation of the new value of the ambient computing refresh rate can be at least partially based on the evolution of network conditions, such as radio link conditions or traffic load conditions in the uplink (UL) and / or downlink (DL). The refresh rate can be increased in response to determining better network characteristics extrapolated from traffic measurements and can be decreased in response to determining worse network characteristics extrapolated from traffic measurements.

[0120] In some embodiments, the combined new refresh rate value can be produced by a weighted sum or other combination of the above two or more effects. In some embodiments, the battery life factor takes precedence over other conditions, such that a low battery life results in using the minimum refresh rate regardless of other conditions. In some embodiments, the nominal refresh rate is used as the maximum refresh rate. For example, if the combined new refresh rate value is higher than the nominal refresh rate, the user device uses the nominal refresh rate.

[0121] Figure 5 is a flowchart showing a method for determining an updated refresh rate according to some embodiments. In the following description, the current refresh rate is referred to as Rate(n), and the subsequent refresh rate is referred to as Rate(n + 1).

[0122] As Figure 5As shown, information about the battery level of the user device is obtained at 502. If the battery level is below a threshold (which can be a predetermined threshold), as determined at 504, then the subsequent refresh rate Rate(n+1) is set at 506 to be equal to the minimum refresh rate (which can be signaled to the user device using the parameter minimumRefreshRate). If the battery level is not below the threshold, then a new candidate refresh rate is obtained at 508. The candidate refresh rate can be obtained based on one or more environmental or other factors, such as user mobile data (obtained at 510), environmental evolution data (obtained at 512), or network condition data (obtained at 514).

[0123] In some embodiments, the candidate refresh rate value NewRate is calculated using a weighted sum, as follows.

[0124] NewRate = Rate(n) + (w1·Δ user + w2·Δ environment + w3·Δ network ) / (w1 + w2 + w3)

[0125] where ·Δ user is the change in the refresh rate with user movement (negative if the user moves slower), Δ environment is the change in the refresh rate with environmental evolution (negative if the environmental evolution slows down), Δ network is the change in the refresh rate depending on the evolution of network conditions (negative if the network condition evolution deteriorates). In this and other embodiments using weights such as w1, w2, and w3, some or all of the weights can be defined in a configuration file, signaled to the user device in the scenario description data or in other ways (such as in a manifest file), or these weights can be otherwise selected to control the impact of these different evolutions on the refresh rate calculation. For example, in a particular AR experience, user movement may be more or less critical. In some embodiments, some weights are provided to the user device (such as in the scenario description data), while other weights are determined locally at the user device. For example, weights more closely related to the AR experience (such as weights related to the impact of user movement or environmental evolution on the refresh rate) can be provided in the scenario description data, while weights related to the operation or state of the user device (such as weights related to network conditions, battery life, available CPU cycles, and / or available memory) can be selected locally. In some embodiments, the refresh rate at the start of the AR experience (which can be referred to as Rate(0)) is set to be equal to the nominal refresh rate, such as the parameter nominalRefreshRate. In other embodiments, a different value is selected for Rate(0).

[0126] In some embodiments, Δ user, Δ environment , Δ network Incremental values such as those represented by Δ indicate changes as a function of user movement, user environment, network conditions, or other relevant parameters. The incremental values may be proportional to the derivative of the change (e.g., proportional to acceleration). Extrapolation of previous measurements may be used to calculate incremental values of user and / or environmental change. For example, a user's previous pose may be used to determine the change and acceleration of user movement. The incremental value can then be proportional to that acceleration. For increments related to environmental change, the user equipment may rely on a similar approach. The user equipment may store the previous poses of each relevant / principal real object in the user environment to determine the change and acceleration of the real object.

[0127] In some embodiments, if the candidate refresh rate NewRate is greater than the nominal refresh rate (which may have been signaled to the user equipment using the parameter nominalRefreshRate), as determined at 516, then the nominal refresh rate is used as the subsequent refresh rate Rate(n+1) at 518. In this and similar embodiments, the nominal refresh rate serves as the maximum refresh rate. If the candidate refresh rate NewRate is less than the nominal refresh rate, then the candidate refresh rate NewRate is used as the subsequent refresh rate Rate(n+1) at 520. Specifically, the refresh rate Rate(n+1) can be selected as follows:

[0128] The minimum value in Rate(n+1) = {nominalRefreshRate, Rate(n)+(w1·Δuser+w2·Δenvironment+w3·Δnetwork) / (w1+w2+w3)}.

[0129] At 522, the new refresh rate is used to update the environmental calculation. The update of the environmental calculation may include an update of the spatial mapping of the environment, such as one or more of the following types of data representing the user environment: point cloud data, geometric mesh data, mesh segmentation information, texture data, and / or semantic labels, as well as other types of data representing the environment.

[0130] In some embodiments, for example Figure 6In an embodiment, the refresh rate can be updated as follows. The battery level is obtained at 602. If it is determined at 604 that the battery level is below a threshold (which can be a predetermined threshold or a threshold signaled in the scenario description data), then at 606 the subsequent refresh rate Rate(n+1) is set to be equal to the minimum refresh rate (which can be signaled to the user equipment using the parameter minimumRefreshRate). If the battery level is not below the threshold, then at 608 a new candidate refresh rate NewRate is obtained using some or all of the following: user movement data obtained at 610, environmental evolution data obtained at 612, and network condition data obtained at 614. In some embodiments, the new candidate refresh rate can be calculated using a weighted sum as follows.

[0131] NewRate = nominalRefreshRate+(w1·Δ user +w2·Δ environment +w3·Δ network ) / (w1+w2+w3)

[0132] In some such embodiments, even if NewRate is greater than nominalRefreshRate, at 606 the updated refresh rate Rate(n+1) can be set to be equal to NewRate.

[0133] At 618, the environmental calculations are updated using the new refresh rate. The update of the environmental calculations can include an update of the spatial mapping of the environment, such as one or more of the following types of data representing the user's environment: point cloud data, geometric mesh data, mesh segmentation information, texture data, and / or semantic labels, as well as other types of data representing the environment.

[0134] In some embodiments, such as Figure 7 the embodiment, a maximum refresh rate value R is imposed based on network bandwidth availability network . In such an embodiment, the refresh rate can be updated as follows. The current battery level is obtained at 702, and it is determined at 704 whether the battery level is below the threshold. If the battery level is below the threshold (which can be a predetermined threshold or a signaled threshold), then at 706 the subsequent refresh rate Rate(n+1) is set equal to the minimum refresh rate (which may have been signaled to the user equipment using the parameter minimumRefreshRate). If the battery level is not below the threshold, then at 708 a candidate refresh rate is obtained using some or all of the following: user movement data obtained at 710 and environmental evolution data obtained at 712. In some embodiments, the candidate refresh rate can be calculated as follows.

[0135] nominalRefreshRate+(w1·Δ user+w2·Δ environment ) / (w1 + w2)

[0136] The value of R used as the maximum refresh rate here network can be determined at 714 based on the network conditions obtained at 716. In one exemplary embodiment, better network conditions (e.g., high bandwidth, low packet error rate) result in a higher value of R network , while worse network conditions result in a lower value of R network . In response to determining at 716 that the candidate refresh rate is higher than the maximum refresh rate, the maximum refresh rate is used as the new refresh rate at 718. In response to determining at 716 that the candidate refresh rate is not higher than the maximum refresh rate, the candidate refresh rate is used as the new refresh rate at 720. In the embodiment as Figure 7 shown, the following equation can be used to calculate the new refresh rate.

[0137] NewRate = the minimum value in {R network , nominalRefreshRate+(w1·Δ user +w2·Δ environment ) / (w1 + w2)}.

[0138] At 722, the new refresh rate is used to update the environment calculation. The update of the environment calculation can include an update of the spatial mapping of the environment, such as one or more of the following types of data representing the user environment: point cloud data, geometric mesh data, mesh segmentation information, texture data, and / or semantic labels, as well as other types of data representing the environment.

[0139] In some embodiments, the refresh rate can be determined as follows. A maximum refresh rate value R network is imposed based on network bandwidth availability. If the battery level is below a threshold (which can be a predetermined threshold), the subsequent refresh rate Rate(n + 1) is set to be equal to the minimum refresh rate (which can be signaled to the user device using the parameter minimumRefreshRate). If the battery level is not below the threshold, a new candidate refresh rate NewRate is obtained using the following weighted sum.

[0140] NewRate = the minimum value in {R network , nominalRefreshRate+max{w1·Δ user , w2·Δ environment}}.

[0141] In this embodiment, instead of using a weighted average of user and environment factors, the refresh rate is updated based on which factor has the highest value.

[0142] In some embodiments, R networkThe value can be determined according to network conditions. Better network conditions (such as high bandwidth, low packet error rate) will result in a higher value of R network while worse network conditions will result in a lower value of R network .

[0143] In some embodiments, techniques other than the above examples are used to determine the refresh rate, such as techniques that do not use weighted sums. In some embodiments, determining the updated refresh rate includes adding a difference to a previous refresh rate value or a nominal refresh rate value. In some embodiments, the updated refresh rate is not necessarily determined based on a previous refresh rate value or a nominal refresh rate value. In some embodiments, the refresh rate can be calculated based on a function or algorithm that returns a relatively high value for greater user movement, greater environmental evolution, better network conditions, and / or higher battery life, and a relatively low value for less user movement, lower environmental evolution speed, worse network conditions, and / or lower battery life. In some embodiments, additional factors or other considerations representing the state of the user, user device, environment, network, AR content can be used to determine the refresh rate.

[0144] In some embodiments, a target refresh rate and an adaptive maximum refresh rate are used to determine the refresh rate. The target refresh rate can be a refresh rate that provides a desired level of AR presentation quality for a given level of movement of the user and / or the environment. As non-exclusive examples, one of the following formulas or different techniques can be used to determine the target refresh rate TargetRate(n + 1):

[0145] TargetRate(n + 1) = TargetRate(n) + (w1·Δ user + w2·Δ environment ) / (w1 + w2)

[0146] TargetRate(n + 1) = nominalRefreshRate + (w1·Δ user + w2·Δ environment ) / (w1 + w2)

[0147] TargetRate(n + 1) = nominalRefreshRate + max{w1·Δ user , w2·Δ environment}

[0148] In an example embodiment, a target refresh rate represents a refresh rate that is sufficient to provide a good user experience without being unnecessarily high (and thus unnecessarily consuming user device resources). For example, if the user is relatively stationary and their environment is relatively unchanging, a high refresh rate may not be needed to provide a good AR experience, and a lower target refresh rate may be desired to conserve resources. However, if the user and / or their environment is in motion, a higher target refresh rate may be needed to maintain the quality of the AR experience. In some embodiments, the battery level of the user device may affect the target refresh rate, with a longer battery life resulting in a higher target refresh rate and a shorter battery life resulting in a lower target update rate. In some embodiments, the target refresh rate is selected based, in part, on a predetermined (e.g., signaled in the scene description data) or adaptive minimum refresh rate. For example, one of the following non-limiting examples may be used to select the target refresh rate.

[0149] TargetRate(n+1) = {minimumRefreshRate,

[0150] TargetRate(n) + (w1·Δ user + w2·Δ environment ) / (w1 + w2)} where the maximum value

[0151] TargetRate(n+1) = {minimumRefreshRate,

[0152] nominalRefreshRate + (w1·Δ user + w2·Δ environment ) / (w1 + w2)} where the maximum value

[0153] TargetRate(n+1) = {minimumRefreshRate,

[0154] nominalRefreshRate + max{w1·Δ user , w2·Δ environment}} where the maximum value

[0155] For example, when both the user and their environment are static, the minimum refresh rate may be used as the target refresh rate.

[0156] The adaptive maximum refresh rate can represent the highest refresh rate that a user device can achieve. The adaptive maximum refresh rate can depend on one or more parameters of the user device. For example, a longer battery life, longer available CPU cycles, and / or larger available memory may result in a higher adaptive maximum refresh rate, while a shorter battery life, fewer available CPU cycles, and / or less available memory may result in a lower adaptive maximum refresh rate. In some cases, the calculation of the adaptive maximum refresh rate can be based on network conditions (such as bandwidth, latency, and / or network QoS), for example, for scenarios or environments that utilize edge or cloud computing resources for processing. Figure 8 The flowchart of Figure 8 illustrates an example method of selecting an adaptive refresh rate using a target refresh rate and an adaptive maximum refresh rate. In accordance with the principles disclosed herein, the target refresh rate and the adaptive maximum refresh rate can be determined using equations other than the equations specifically given above, or using techniques other than equations (such as a lookup table).

[0157] In Figure 8 the example of Figure 8 , a target refresh rate value is obtained at 802 based on some or all of the following data: user movement data obtained at 804, environmental evolution data obtained at 806, battery power data obtained at 808, and / or additional data. A maximum refresh rate value is obtained at 810 based on some or all of the following data: battery power data obtained at 808, network condition data obtained at 812, CPU data obtained at 814, and / or additional data. It is determined at 816 whether the target refresh rate is higher than the maximum refresh rate. In response to determining that the target refresh rate is higher than the maximum refresh rate, the maximum refresh rate is used as the new refresh rate at 818. In response to determining that the target refresh rate is higher than the maximum refresh rate, the maximum refresh rate is used as the new refresh rate at 818. In response to determining that the target refresh rate is not higher than the maximum refresh rate, the target refresh rate is used as the new refresh rate at 820.

[0158] In some embodiments, it is determined at 822 whether the selected new refresh rate is lower than a threshold (the threshold can be a threshold signaled in the scene description data or elsewhere, or can be a predefined threshold), and the user device performs the corresponding action 824 signaled in the scene description data, such as signaling a warning message, fading out virtual objects, and / or deactivating one or more virtual objects. In some embodiments, the minimum value used as the lower limit of the target refresh rate can be different from the threshold used to trigger the action signaled in the scene description data. In other embodiments, they can be the same.

[0159] At 826, the environmental computation is refreshed according to the selected new refresh rate. The update of the environmental computation may include an update of the spatial mapping of the environment, such as one or more of the following types of data representing the user's environment: point cloud data, geometric mesh data, mesh segmentation information, texture data, and / or semantic labels, as well as other types of data representing the environment.

[0160] In some embodiments, one or more of the mentioned factors (or other factors) may be used to set the maximum or minimum refresh rate. For example, greater user movement, a greater rate of environmental evolution, better network conditions, and / or higher battery life may cause the maximum or minimum refresh rate to be set relatively high, while less user movement, a lower rate of environmental evolution, worse network conditions, and / or lower battery life may cause the maximum or minimum refresh rate to be set relatively low. In some embodiments, a lookup table, a linear function, a piecewise linear function, or other similar techniques may be used to determine the refresh rate.

[0161] Although some of the above embodiments (e.g., with respect to Figures 5 - 7 ) set the refresh rate to the minimum value in response to low battery life, it should be noted that each of these embodiments can be implemented without such a feature. In some embodiments, battery life may have no effect on the refresh rate. In some embodiments, battery life may be one of the factors (possibly one of other factors) used to determine the refresh rate.

[0162] At the new refresh rate (determined according to Figure 5 or by another technique) is below the defined minimum refresh rate, the rendering engine may perform the defined action(s) before rendering the current frame.

[0163] Further embodiments

[0164] A method according to some embodiments includes: providing scene description data for an augmented reality experience, where the scene description data includes at least one of the following information: information indicating a minimum value of the environmental computation refresh rate, information indicating a nominal value of the environmental computation update rate, and information indicating at least one action to be performed if the environmental computation refresh rate drops below the minimum value.

[0165] In some embodiments, the scene description data further includes at least one of the following information: information indicating whether environmental data is used for collision handling, and information indicating whether environmental data is used for rendering.

[0166] In some embodiments, the scene description data is provided in a scene description file.

[0167] A method according to some embodiments includes, at an augmented reality user device: obtaining information indicating a minimum value of an environmental computing refresh rate; determining a battery life of the augmented reality user device; and in response to determining that the battery life is below a threshold, setting the environmental computing refresh rate to the minimum value.

[0168] In some embodiments, a method includes, at an augmented reality user device: obtaining information indicating at least one factor from among user movement, environmental evolution, network conditions, or battery life; and adaptively determining an environmental computing refresh rate based on the at least one factor.

[0169] Some embodiments further include receiving information indicating a nominal value of the environmental computing refresh rate, wherein the determined environmental computing refresh rate is at least partially based on the nominal value. In some such embodiments, the nominal value serves as a maximum value of the environmental computing refresh rate.

[0170] In some embodiments, the environmental computing refresh rate is determined based on periodicity.

[0171] In some embodiments, each environmental computing refresh rate is determined based on a previous environmental computing refresh rate.

[0172] Some embodiments further include using the determined environmental computing refresh rate to refresh real environmental computing.

[0173] An apparatus according to some embodiments includes at least one processor configured to perform: providing scene description data for an augmented reality experience, wherein the scene description data includes at least one of the following: information indicating a minimum value of an environmental computing refresh rate, information indicating a nominal value of the environmental computing refresh rate, and information indicating at least one action to be performed if the environmental computing refresh rate drops below the minimum value.

[0174] In some embodiments, the scene description data further includes at least one of the following information: information indicating whether environmental data is used for collision handling, and information indicating whether environmental data can be used for rendering.

[0175] In some embodiments, the scene description data is provided in a scene description file.

[0176] An augmented reality user device apparatus according to some embodiments includes at least one processor configured to perform: obtaining information indicating a minimum value of an environmental computing refresh rate; determining a battery life of the augmented reality user device; and in response to determining that the battery life is below a threshold, setting the environmental computing refresh rate to the minimum value.

[0177] An augmented reality user device apparatus according to some embodiments includes at least one processor configured to perform the following operations: obtaining information indicating at least one factor from user movement, environmental evolution, network conditions, or battery life; adaptively determining an environmental computing refresh rate based on the at least one factor.

[0178] Some embodiments further include receiving information indicating a nominal value of the environmental computing refresh rate, wherein the determined environmental computing refresh rate is at least partially based on the nominal value.

[0179] Some embodiments further include receiving information indicating a nominal value of the environmental computing refresh rate, wherein the nominal value serves as a maximum value of the environmental computing refresh rate.

[0180] In some embodiments, the environmental computing refresh rate is determined based on a period.

[0181] In some embodiments, each environmental computing refresh rate is determined based on a previous environmental computing refresh rate.

[0182] Some embodiments further include using the determined environmental computing refresh rate to refresh real environmental computing.

[0183] A method according to some embodiments includes, at an augmented reality user device: obtaining information indicating at least one factor from the following: user movement, environmental evolution, network conditions, battery level, available CPU cycles, or available memory; adaptively determining a target environmental computing refresh rate based on at least a first one of the factors; adaptively determining a maximum environmental computing refresh rate based on at least a second one of the factors; and setting the environmental computing refresh rate of the user device to the minimum of the target environmental computing refresh rate and the maximum refresh rate.

[0184] The present disclosure describes various aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are specifically described and are typically described in a way that may sound restrictive, at least to show individual characteristics. However, this is for clarity of description and does not limit the disclosure or scope of these aspects. In fact, all different aspects can be combined and interchanged to provide more aspects. Additionally, these aspects can also be combined and interchanged with aspects described in earlier applications.

[0185] Aspects described and contemplated in this disclosure can be implemented in many different forms. While some embodiments are specifically shown, other embodiments are also contemplated, and the discussion of specific embodiments does not limit the breadth of implementation. At least one aspect generally relates to scene encoding and decoding, and at least one other aspect generally relates to transmitting the generated or encoded bitstream. These and other aspects can be implemented as methods, apparatuses, computer-readable storage media having instructions stored thereon for encoding or decoding scene data according to any of the methods, and / or computer-readable recording media having a bitstream generated according to any of the methods stored thereon.

[0186] Various methods are described herein, each of which includes one or more steps or actions for implementing the method. Unless a specific order of steps or actions is required for the method to operate correctly, the order and / or use of specific steps and / or actions can be modified or combined. Additionally, in various embodiments, terms such as "first", "second", etc. can be used to modify elements, components, steps, operations, etc., such as "first decoding" and "second decoding". Unless specifically required, the use of such terms does not imply an ordering of the modified operations. Thus, in this example, the first decoding does not need to be performed before the second decoding, and can be performed, for example, before, during, or in a time period overlapping with the second decoding.

[0187] For example, various numerical values can be used in this disclosure. The specific values are for illustrative purposes only, and the described aspects are not limited to these specific values.

[0188] The embodiments described herein can be implemented by computer software or other hardware implemented by a processor, or by a combination of hardware and software. As a non-limiting example, an embodiment can be implemented by one or more integrated circuits. As a non-limiting example, the processor can be of any type suitable for the technical environment and can include one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0189] When a figure is presented in the form of a flowchart, it should be understood that it also provides a block diagram of the corresponding apparatus. Similarly, when a figure is presented in the form of a block diagram, it should be understood that it also provides a flowchart of the corresponding method / process.

[0190] The implementations and aspects described herein can be implemented in, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Even if discussed in the context of only a single form of implementation (e.g., only as a method), the implementation of the features discussed can be implemented in other forms (e.g., an apparatus or a program). For example, an apparatus can be implemented with appropriate hardware, software, and firmware. These methods can be implemented in, for example, a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. A processor also includes communication devices, such as a computer, a mobile phone, a portable / personal digital assistant (“PDA”), and other devices that facilitate information communication among end users.

[0191] Reference to “one embodiment” or “an embodiment”, “a mode of implementation” or “modes of implementation”, and other variants thereof means that the specific features, structures, characteristics, etc. related to the embodiment are included in at least one embodiment. Thus, the phrases “in one embodiment” or “in an embodiment” or “a mode of implementation” or “modes of implementation”, and any other variants, that appear in various places in this disclosure do not necessarily all refer to the same embodiment.

[0192] In addition, this disclosure may relate to “determining” various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, and retrieving information from a memory.

[0193] In addition, this disclosure may relate to “accessing” various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, and estimating information.

[0194] In addition, this disclosure may relate to “receiving” various information. Receiving, like “accessing”, is a broad term. Receiving information may include, for example, one or more of accessing information and retrieving information (e.g., from a memory). In addition, “receiving” is generally involved in some way during operations such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0195] It should be understood that the use of any one of the following, namely, " / " and " / or" (e.g., in the cases of "A / B", "A and / or B", and "at least one of A and B") is intended to include only the selection of the first-listed option (A), or only the selection of the second-listed option (B), or the selection of both options (A and B). As another example, in the cases of "A, B, and / or C" and "at least one of A, B, and C", such wording is intended to include only the selection of the first-listed option (A), or only the selection of the second-listed option (B), or only the selection of the third-listed option (C), or only the selection of the first and second-listed options (A and B), or only the selection of the first and third-listed options (A and C), or only the selection of the second and third-listed options (B and C). This can be extended to any number of items listed.

[0196] Furthermore, as used herein, the term "signal" among other things, refers to indicating certain information to a corresponding decoder. For example, in some embodiments, an encoder signals a particular one of a plurality of parameters for region-based filter parameter selection for artifact reduction filtering. Thus, in one embodiment the same parameters are used on both the encoder side and the decoder side. Accordingly, for example, an encoder may send (explicit signaling) a particular parameter to a decoder so that the decoder may use the same particular parameter. Conversely, if the decoder already has a particular parameter as well as other parameters, signaling (implicit signaling) may be used without transmission to simply allow the decoder to know and select the particular parameter. By avoiding the transmission of any actual functionality, bit savings are achieved in various embodiments. It should be understood that signaling may be implemented in various ways. For example, in various embodiments, one or more syntax elements, flags, etc. are used to signal information to a corresponding decoder. Although the foregoing refers to the verb form of the term "signal", "signaling" may also be used herein as a noun.

[0197] Implementations can generate various signals that are formatted to carry information that can be stored or transmitted. The information can include, for example, instructions for performing a method, or data generated by one of the described implementations. For example, a signal can be formatted to carry a bitstream of the described embodiment. For example, such a signal can be formatted as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or a baseband signal. Formatting can include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal can be, for example, analog or digital information. As is well known, signals can be transmitted over a variety of different wired or wireless links. Signals can be stored on a processor-readable medium.

[0198] We have described multiple embodiments. The features of these embodiments can be provided individually or in any combination across various claim categories and types. Additionally, the embodiments can include, individually or in any combination across various claim categories and types, one or more of the following features, devices, or aspects:

[0199] · A bitstream or signal that includes one or more of the grammar elements or their variants.

[0200] · A bitstream or signal that includes a grammar that conveys information generated according to any of the described embodiments.

[0201] · Creating and / or sending and / or receiving and / or decoding a bitstream or signal that includes one or more of the grammar elements or their variants.

[0202] · Creating and / or sending and / or receiving and / or decoding according to any of the described embodiments.

[0203] · A method, process, device, medium storing instructions, medium storing data or signals according to any of the described embodiments.

[0204] Note that the various hardware elements of one or more of the described embodiments can be referred to as "modules" that, in combination with the respective modules, perform (i.e., execute, implement, etc.) the various functions described herein. As used herein, a module includes hardware that is considered suitable for a given implementation (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more storage devices). Each described module can also include executable instructions for performing one or more functions described as being performed by the corresponding module, and it is noted that these instructions can take the form of hardware (i.e., hardwired) instructions, firmware instructions, software instructions, and / or similar instructions, or include hardware (i.e., hardwired) instructions, firmware instructions, software instructions, and / or similar instructions, and can be stored in any suitable non-transitory computer-readable medium, such as those commonly referred to as RAM, ROM, etc.

[0205] Although the features and elements have been described above in particular combinations, each feature or element can be used separately or in combination with other features and elements. Additionally, the methods described herein can be implemented in a computer program, software, or firmware contained in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memories, semiconductor storage devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and digital versatile disks (DVDs). A processor associated with the software can be used to implement a radio frequency transceiver for a WTRU, UE, terminal, base station, RNC, or any host.

Claims

1. A method, comprising: Obtaining sensor data describing a user environment; Obtaining information indicating at least one factor from among: user movement information, environmental evolution information, network condition information, and battery life information; Determining a candidate environmental calculation refresh rate based on the at least one factor; Selecting the environmental calculation refresh rate as the minimum of the candidate environmental calculation refresh rate and a maximum environmental calculation refresh rate; And Updating the real environmental calculation using the selected environmental calculation refresh rate.

2. An apparatus, comprising one or more processors configured to at least perform: Obtaining sensor data describing a user environment; Obtaining information indicating at least one factor from among: user movement information, environmental evolution information, network condition information, and battery life information; Determining a candidate environmental calculation refresh rate based on the at least one factor; Selecting the environmental calculation refresh rate as the minimum of the candidate environmental calculation refresh rate and a maximum environmental calculation refresh rate; And Updating the real environmental calculation using the selected environmental calculation refresh rate.

3. The method according to claim 1 or the apparatus according to claim 2 further comprises: Obtaining scene description data describing an extended reality scene and presenting the scene in the user environment.

4. The method according to claim 1 or claim 3 which depends on claim 1, or the apparatus according to claim 2 or claim 3 which depends on claim 2, further comprises: Receiving metadata indicating the maximum environmental calculation refresh rate.

5. The method according to claim 1 or any one of claims 3 - 4 depending on claim 1, or the apparatus according to claim 2 or any one of claims 3 - 4 depending on claim 2, wherein the selection of the environmental calculation refresh rate is based on a periodicity.

6. The method according to claim 1 or any one of claims 3 - 5 depending on claim 1, or the apparatus according to claim 2 or any one of claims 3 - 5 depending on claim 2, wherein a weighted sum of parameters representing at least two of the factors from among: user movement, environmental evolution, network condition, and battery life is used to determine the candidate environmental calculation refresh rate.

7. The method or apparatus according to claim 6, further comprising: Receiving metadata indicating the weights used in the weighted sum.

8. The method according to claim 1 or any one of claims 3 - 7 depending on claim 1, or the apparatus according to claim 2 or any one of claims 3 - 7 depending on claim 2, wherein the candidate environmental calculation refresh rate is determined based at least on the user movement information and the environmental evolution information.

9. The method according to claim 1 or any one of claims 3-8 dependent on claim 1, or the apparatus according to claim 2 or any one of claims 3-8 dependent on claim 2, further comprises: Obtaining metadata indicating the maximum environmental calculation refresh rate.

10. The method according to claim 1 or any one of claims 3-9 depending on claim 1, or the apparatus according to claim 2 or any one of claims 3-9 depending on claim 2, further comprises: Determining the maximum environmental calculation refresh rate based at least in part on the network condition information.

11. The method according to claim 1 or any one of claims 3 - 10 depending on claim 1, or the apparatus according to claim 2 or any one of claims 3 - 10 depending on claim 2, wherein the candidate environmental calculation refresh rate is determined based at least in part on at least one of the following: the user movement information, the environmental evolution information, and the network condition information.

12. The method according to claim 1 or any one of claims 3-11 dependent on claim 1, or the apparatus according to claim 2 or any one of claims 3-11 dependent on claim 2, further comprises: obtaining information indicating a threshold environmental calculation refresh rate and a specified action; and in response to determining that the selected environmental calculation refresh rate is less than the threshold environmental calculation refresh rate, performing the specified action.

13. The method according to claim 1 or any one of claims 3-12 dependent on claim 1, or the apparatus according to claim 2 or any one of claims 3-12 dependent on claim 2, wherein the candidate environmental calculation refresh rate is calculated based at least in part on metadata received in at least one of a scene description file and an inventory file.

14. A method comprising: providing scene description data for an extended reality experience, wherein the scene description data includes: information indicating a threshold of an environmental calculation refresh rate, information indicating a nominal value of the environmental calculation refresh rate, and information indicating at least one specified action to be performed in response to the environmental calculation refresh rate dropping below the threshold.

15. An apparatus comprising one or more processors configured to at least perform: providing scene description data for an extended reality experience, wherein the scene description data includes: information indicating a threshold of an environmental calculation refresh rate, information indicating a nominal value of the environmental calculation refresh rate, and information indicating at least one specified action to be performed in response to the environmental calculation refresh rate dropping below the threshold.

16. The method according to claim 14 or the apparatus according to claim 15, wherein the scene description data further includes at least one weight value for calculating the environmental calculation refresh rate.

17. The method or apparatus according to claim 16, wherein for the at least one weight value, the scene description data further includes information identifying a factor associated with the corresponding weight value.

18. The method or apparatus according to claim 17, wherein the factor includes at least one of the following: user movement information, environmental evolution information, network condition information, and battery life information.

19. A computer-readable medium comprising instructions for causing one or more processors to perform the method according to claim 1 or any one of claims 2-13 dependent on claim 1.

20. A computer-readable medium comprising instructions for causing one or more processors to perform the method according to claim 14 or any one of claims 16-18 dependent on claim 1.

21. The computer-readable medium according to claim 19 or 20, wherein the computer-readable medium is a non-transitory storage medium.

Images