SYSTEM AND METHOD FOR PASS-THROUGH EXTENDED REALITY (XR) CONTENT

Abstract

Extended reality (XR) management systems and techniques are described. In some examples, the XR management system receives sensor data from an XR interface device having at least one sensor. Based on receiving the sensor data from the XR interface device, the XR management system generates processing instructions for an XR processing device to process the sensor data to generate XR content. The XR management system sends the sensor data and the processing instructions to the XR processing device. The XR management system receives the XR content from the XR processing device. The XR management system generates layer content. The XR management system sends the XR content and layer content to the XR interface device to cause the XR interface device to output the XR content and layer content in a layered arrangement.

Description

[Technical Field]

[0001] This application relates to imaging, and more particularly to a system and method for split rendering and pass-through compressed media formats for extended reality (XR) systems. [Background technology]

[0002]

[0002] Many devices include one or more cameras. For example, a smartphone or tablet includes a front camera for capturing selfies and a rear camera for capturing images of a scene (such as a landscape or other scene of interest to the device user). The camera can capture images using the camera's image sensor, which can include an array of photodetectors. Some devices can analyze image data captured by the image sensor to detect objects within the image data. Sometimes, a camera can be used to capture images of a scene that includes one or more people. Summary of the Invention

[0003] Systems and techniques for extended reality (XR) management are described. In some examples, an XR management system receives sensor data from an XR interface device having at least one sensor. The sensor data may include, for example, image(s) and / or video(s) captured using a camera(s) of the XR interface device. Based on receiving the sensor data from the XR interface device, the XR management system generates processing instructions for an XR processing device to process the sensor data to generate XR content. The XR management system sends the sensor data and the processing instructions to the XR processing device. The XR management system receives the XR content from the XR processing device. In some examples, the processing instructions may instruct the XR processing device to limit the size or bitrate of the XR content based on (e.g., not to exceed) the bandwidth of a connection between the XR management system and the XR interface device, the quality of the connection, the resolution of a display screen of the XR interface device, and / or another limitation. In some examples, the processing instructions can specify a posture (e.g., location and / or orientation) of the XR interface device, which can dictate the location(s) and / or orientation(s) at which the XR processing device renders virtual content (e.g., video game content) within the context of the XR content. The XR management system generates the layer content. The XR management system sends the XR content and the layer content to the XR interface device to cause the XR interface device to output the XR content and the layer content in a layered arrangement. In some examples, the layer content can be overlaid on top of the XR content according to the layered arrangement. In some examples, the XR content can be overlaid on top of the layer content according to the layered arrangement. For example, the layer content can include communication(s), notification(s), alert(s), status indicator(s), or a combination thereof.In illustrative examples, the XR interface device is a headset (e.g., a head-mounted display (HMD) device, glasses, a wearable device), the XR management system is a mobile device (e.g., a mobile handset, a tablet, a laptop, a wearable device) that is locally coupled (e.g., wirelessly or using a wire(s)) to the XR interface device, and the XR processing device is a remote computing system (e.g., an edge node, a remote server). In some examples, the XR interface device is a mobile device (e.g., a mobile handset, a tablet, a laptop, a wearable device).

[0004] According to at least one example, a method for extended reality (XR) management is provided. The method includes receiving sensor data from an XR interface device having at least one sensor, generating processing instructions for an XR processing device to process the sensor data to generate XR content based on the reception of the sensor data from the XR interface device, sending the sensor data and the processing instructions to the XR processing device, receiving the XR content from the XR processing device, generating layer content, and sending the XR content and the layer content to the XR interface device, for the XR interface device to output the XR content and the layer content in a layered arrangement.

[0005]

[0005] In another example, an apparatus for XR management is provided, including at least one memory and at least one processor coupled to the at least one memory. The at least one processor is configured to receive sensor data from an XR interface device having at least one sensor, generate processing instructions for an XR processing device to process the sensor data to generate XR content based on the reception of the sensor data from the XR interface device, send the sensor data and the processing instructions to the XR processing device, receive the XR content from the XR processing device, generate layer content, and send the XR content and the layer content to the XR interface device, so that the XR interface device outputs the XR content and the layer content in a layered arrangement.

[0006]

[0006] In another example, a non-transitory computer-readable medium is provided having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to receive sensor data from an XR interface device having at least one sensor, generate processing instructions based on the receipt of the sensor data from the XR interface device for the XR processing device to process the sensor data to generate XR content, send the sensor data and the processing instructions to the XR processing device, receive XR content from the XR processing device, generate layer content, and send the XR content and layer content to the XR interface device for the XR interface device to output the XR content and layer content in a layered arrangement.

[0007]

[0007] In another example, an apparatus for XR management is provided, the apparatus including: means for receiving sensor data from an XR interface device having at least one sensor; means for generating, based on the reception of the sensor data from the XR interface device, processing instructions for an XR processing device to process the sensor data to generate XR content; means for sending the sensor data and the processing instructions to the XR processing device; means for receiving XR content from the XR processing device; means for generating layer content; and means for sending the XR content and the layer content to the XR interface device so that the XR interface device outputs the XR content and the layer content in a layered arrangement.

[0008] In some aspects, the device is part of and / or includes a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a head-mounted display (HMD) device, a wireless communication device, a mobile device (e.g., a mobile phone and / or mobile handset and / or a so-called "smartphone" or other mobile device), a camera, a personal computer, a laptop computer, a server computer, a vehicle or a computing device or component of a vehicle, another device, or a combination thereof. In some aspects, the device includes a camera or multiple cameras for capturing one or more images. In some aspects, the device further includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the devices described above may include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyroscopes, one or more gyrometers, one or more accelerometers, any combination thereof, and / or other sensors).

[0009]

[0009] This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used independently to determine the scope of the claimed subject matter, which subject matter should be understood by reference to the entire specification of this patent, any or all drawings, and appropriate portions of each claim.

[0010]

[0010] The above, together with other features and aspects, will become more apparent with reference to the following specification, claims, and accompanying drawings. [Brief explanation of the drawings]

[0011]

[0011] Illustrative aspects of the present application are described in detail below with reference to the following drawings: [Figure 1]

[0012] FIG. 1 is a block diagram illustrating an example architecture of an image capture and processing system, according to some examples. [Figure 2]

[0013] FIG. 1 is a block diagram illustrating an example architecture of an extended reality (XR) system, according to some examples. [Figure 3]

[0014] FIG. 3A is a perspective view illustrating a head-mounted display (HMD) used as part of an imaging system, according to some examples.

[0015] FIG. 3B is a perspective view illustrating the head-mounted display (HMD) of FIG. 3A being worn by a user, according to some examples. [Figure 4]

[0016] FIG. 4A is a perspective view illustrating the front of a mobile handset that includes a front-facing camera and can be used as part of an imaging system, according to some examples.

[0017] FIG. 4B is a perspective view illustrating the back of a mobile handset that includes a rear camera and can be used as part of an imaging system, according to some examples. [Figure 5]

[0018] FIG. 1 is a swimlane diagram illustrating a workflow for local processing, according to some examples. [Figure 6]

[0019] FIG. 1 is a swimlane diagram illustrating the lifecycle of an extended reality (XR) application, according to some examples. [Figure 7]

[0020] FIG. 1 is a block diagram illustrating an example architecture of an extended reality (XR) system including an extended reality (XR) interface device and an XR management system, according to some examples. [Figure 8]

[0021] 8 is a block diagram illustrating a split rendering process in the context of the architecture of the extended reality (XR) system of FIG. 7, according to some examples. [Figure 9]

[0022] 8 is a block diagram illustrating a split rendering process including encoding and decoding in the context of the architecture of the extended reality (XR) system of FIG. 7, according to some examples. [Figure 10]

[0023] FIG. 8 is a block diagram illustrating a process for using a pass-through encoding format and / or protocol for remote rendering in the context of the architecture of the extended reality (XR) system of FIG. 7, according to some examples. [Figure 11]

[0024] 7 is a block diagram illustrating an example architecture of an extended reality (XR) system spanning an extended reality (XR) interface device 702, an XR management system, and an XR processing device, according to some examples. [Figure 12]

[0025] 12 is a block diagram illustrating the use of a pass-through format for remote rendering in the context of the architecture of the extended reality (XR) system of FIG. 11, according to some examples. [Figure 13]

[0026] FIG. 1 is a block diagram illustrating an example architecture of an extended reality (XR) system that performs XR management to deliver compressed formats, according to some examples. [Figure 14]

[0027] FIG. 1 is a flow diagram illustrating a process for extended reality (XR) management, according to some examples. [Figure 15]

[0028] FIG. 1 illustrates an example of a computing system for implementing certain aspects described herein. DETAILED DESCRIPTION OF THE INVENTION

[0012]

[0029] Specific aspects of the present disclosure are provided below. As will be apparent to one skilled in the art, some of these aspects may be applied independently, and some of them may be applied in combination. In the following specification, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of the aspects of the present application. However, it will be apparent that various aspects may be practiced without these specific details. The figures and specification are not intended to be limiting.

[0013]

[0030] The following specification provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of exemplary embodiments will provide those skilled in the art with an enabling description for implementing the exemplary embodiments. It should be understood that various changes can be made in the function and arrangement of elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0014]

[0031] A camera is a device that uses an image sensor to receive light and capture image frames, such as still images or video frames. The terms “image,” “image frame,” and “frame” are used interchangeably herein. Cameras can be configured with various image capture and image processing settings. Different settings result in images with different appearances. Some camera settings, such as ISO, exposure time, aperture size, f / stop, shutter speed, focus, and gain, are determined and applied before or during the capture of one or more image frames. For example, settings or parameters can be applied to an image sensor to capture one or more image frames. Other camera settings, such as contrast, brightness, saturation, sharpness, levels, curves, or color modifications, can configure post-processing of one or more image frames. For example, settings or parameters can be applied to a processor (e.g., an image signal processor or ISP) to process one or more image frames captured by the image sensor.

[0015]

[0032] A device that includes a camera can analyze image data captured by an image sensor to detect, recognize, classify, and / or track objects within the image data. For example, by detecting and / or recognizing an object within multiple video frames of a video, the device can track the movement of the object over time.

[0016]

[0033] An extended reality (XR) management system and technique are described. The XR management system receives sensor data from an XR interface device having at least one sensor. The sensor data may include, for example, image(s) and / or video(s) captured using the camera(s) of the XR interface device, depth data captured using the depth sensor(s) of the XR interface device, posture data captured using sensor(s) for position and / or movement and / or orientation and / or acceleration, other sensor data from other types of sensors, or combinations thereof. Based on receiving the sensor data from the XR interface device, the XR management system generates processing instructions for an XR processing device to process the sensor data to generate XR content. The XR management system sends the sensor data and the processing instructions to the XR processing device. The XR management system receives the XR content from the XR processing device. In some examples, the processing instructions can instruct the XR processing device to limit the size or bitrate of the XR content based on (e.g., not to exceed) the bandwidth of the connection between the XR management system and the XR interface device, the quality of the connection, the resolution of the display screen of the XR interface device, and / or another limitation. In some examples, the processing instructions can specify the attitude (e.g., location and / or orientation) of the XR interface device, which can dictate the location(s) and / or orientation(s) at which the XR processing device renders virtual content (e.g., video game content) within the context of the XR content. The XR content can be, for example, a processed transformation of the sensor data, with the virtual content rendered and integrated into the environment represented in the sensor data.The XR management system generates layer content, which is content that is added by the XR management system, for example, on a layer separate from the XR content (e.g., overlaid on top of or underlaid below the XR content), without the need to directly modify (e.g., therefore decompress and / or decrypt and / or decrypt) the XR content in the XR management system. The XR management system sends the XR content and layer content to the XR interface device to cause the XR interface device to output the XR content and layer content in a layered arrangement. In some examples, the layer content can be overlaid on top of the XR content according to the layered arrangement. In some examples, the XR content can be overlaid on top of the layer content according to the layered arrangement. For example, the layer content can include communication(s), notification(s), alert(s), status indicator(s), or a combination thereof. In illustrative examples, the XR interface device is a headset (e.g., a head-mounted display (HMD) device, glasses, a wearable device), the XR management system is a mobile device (e.g., a mobile handset, a tablet, a laptop, a wearable device) that is locally coupled (e.g., wirelessly or using a wire(s)) to the XR interface device, and the XR processing device is a remote computing system (e.g., an edge node, a remote server). In some examples, the XR interface device is a mobile device (e.g., a mobile handset, a tablet, a laptop, a wearable device).

[0017]

[0034] The XR management systems and techniques disclosed herein provide several technical improvements that solve several technical problems in XR systems (and in other systems including sets of coupled devices). For example, XR systems are sensitive to latency and quality. In XR systems in which an XR management system (e.g., XR management system 1104) transcodes (e.g., decodes, encodes, decompresses, compresses, decrypts, and / or encrypts) content (e.g., sensor data and / or XR content) between an XR interface device (e.g., XR interface device 1102) and an XR processing device (e.g., XR processing device 1106), the transcoding (e.g., associated with decompression and recompression) results in latency and potential image quality loss, which can be technically problematic and can result in a user experience that is jittery, laggy, and / or characterized by poor image quality (e.g., with distracting and / or disorienting image artifacts). The XR management systems and techniques disclosed herein (see, e.g., FIGS. 12-14) can omit transcoding (e.g., decoding, encoding, decompressing, compressing, decrypting, and / or encrypting) of content (e.g., sensor data and / or XR content) in the XR management system (e.g., XR management system 1104), and instead pass the content in its encoded and / or compressed and / or encrypted form between the XR interface device (e.g., XR interface device 1102) and the XR processing device (e.g., XR processing device 1106). Thus, the XR management systems and techniques disclosed herein can reduce or eliminate latency and / or quality loss (e.g., image quality loss), improving the smoothness and image quality of the user experience.By eliminating transcoding of content in the XR management system, the XR management system and techniques disclosed herein can further reduce power consumption in the XR management system (e.g., by eliminating transcoding), can reduce heat generation (and thus the need for heat dissipation components) in the XR management system (e.g., by eliminating transcoding), can reduce computational resource usage in the XR management system (e.g., by eliminating transcoding), can improve security (e.g., by reducing potential points of failure by not decrypting content in the XR management system and / or by using the XR management system to negotiate security protocols between the XR interface device and the XR processing device), can improve reliability (e.g., by reducing potential points of failure), can improve bitrate (e.g., by using the XR management system to negotiate bitrate capabilities between the XR interface device and the XR processing device and reducing the need to transcode content for different bitrates), or a combination thereof.

[0018]

[0035] Various aspects of the present application are described with reference to the figures. FIG. 1 is a block diagram illustrating the architecture of an image capture and processing system 100. The image capture and processing system 100 includes various components used to capture and process images of one or more scenes (e.g., images of a scene 110). The image capture and processing system 100 can capture a single image (or photograph) and / or can capture a video including multiple images (or video frames) in a particular sequence. A lens 115 of the system 100 faces the scene 110 and receives light from the scene 110. The lens 115 bends the light toward an image sensor 130. The light received by the lens 115 passes through an aperture controlled by one or more control mechanisms 120 and is received by the image sensor 130. In some examples, the scene 110 is a scene within an environment. In some examples, the scene 110 is a scene of at least a portion of a user. For example, scene 110 may be a scene of one or both of a user's eyes and / or at least a portion of a user's face.

[0019]

[0036] The one or more controls 120 may control exposure, focus, and / or zoom based on information from image sensor 130 and / or based on information from image processor 150. The one or more controls 120 may include multiple mechanisms and components, for example, control 120 may include one or more exposure controls 125A, one or more focus controls 125B, and / or one or more zoom controls 125C. The one or more controls 120 may also include additional controls beyond those illustrated, such as controls for analog gain, flash, HDR, depth of field, and / or other image capture properties.

[0020]

[0037] The focus control mechanism 125B of the control mechanism 120 can obtain the focus setting. In some examples, the focus control mechanism 125B stores the focus setting in a memory register. Based on the focus setting, the focus control mechanism 125B can adjust the position of the lens 115 relative to the position of the image sensor 130. For example, based on the focus setting, the focus control mechanism 125B can move the lens 115 closer to or farther from the image sensor 130 by actuating a motor or servo, thereby adjusting the focus. In some cases, additional lenses, such as one or more microlenses above each photodiode of the image sensor 130, can be included in the system 100, each of which bends light received from the lens 115 toward a corresponding photodiode before the light reaches the photodiode. The focus setting can be determined via contrast detection autofocus (CDAF), phase detection autofocus (PDAF), or some combination thereof. The focus settings may be determined using the control mechanism 120, the image sensor 130, and / or the image processor 150. The focus settings may also be referred to as image capture settings and / or image processing settings.

[0021]

[0038] Exposure control 125A of control mechanism 120 can obtain an exposure setting. In some cases, exposure control 125A stores the exposure setting in a memory register. Based on the exposure setting, exposure control 125A can control the size of the aperture (e.g., aperture size or f / stop), the duration the aperture is open (e.g., exposure time or shutter speed), the sensitivity of image sensor 130 (e.g., ISO speed or film speed), the analog gain applied by image sensor 130, or any combination thereof. Exposure settings are sometimes referred to as image capture settings and / or image processing settings.

[0022]

[0039] The zoom control 125C of the control mechanism 120 can obtain a zoom setting. In some examples, the zoom control 125C stores the zoom setting in a memory register. Based on the zoom setting, the zoom control 125C can control the focal length of an assembly of lens elements (lens assembly) including the lens 115 and one or more additional lenses. For example, the zoom control 125C can control the focal length of the lens assembly by actuating one or more motors or servos to move one or more of the lenses relative to one another. The zoom setting may also be referred to as an image capture setting and / or an image processing setting. In some examples, the lens assembly can include a parfocal zoom lens or a variable-focus zoom lens. In some examples, the lens assembly can include a focusing lens (which may be the lens 115 in some cases) that first receives light from the scene 110, and the light then passes through an afocal zoom system between the focusing lens (e.g., the lens 115) and the image sensor 130 before reaching the image sensor 130. In some cases, an afocal zoom system may include two positive (e.g., converging, convex) lenses of equal or similar focal lengths (e.g., within a threshold difference), with a negative (e.g., diverging, concave) lens between them. In some cases, zoom control 125C moves one or more of the lenses in the afocal zoom system, such as the negative lens and one or both of the positive lenses.

[0023]

[0040] Image sensor 130 includes one or more arrays of photodiodes or other light-receiving elements. Each photodiode measures an amount of light that ultimately corresponds to a particular pixel in the image generated by image sensor 130. In some cases, different photodiodes may be covered with different color filters and therefore measure light that matches the color of the filter covering that photodiode. For example, a Bayer color filter includes red, blue, and green filters, and each pixel of the image is generated based on red light data from at least one photodiode covered by a red filter, blue light data from at least one photodiode covered by a blue filter, and green light data from at least one photodiode covered by a green filter. Other types of color filters may use yellow, magenta, and / or cyan (also referred to as "emerald") color filters instead of or in addition to red, blue, and / or green filters. Some image sensors may lack color filters entirely and instead use different photodiodes (possibly stacked vertically) throughout the pixel array. Different photodiodes across the pixel array may have different spectral sensitivity curves, and thus respond to different wavelengths of light. Monochrome image sensors may also lack color depth due to the lack of color filters.

[0024]

[0041] In some cases, image sensor 130 may alternatively or additionally include opaque and / or reflective masks, which may be used for phase-detection autofocus (PDAF), that block light from reaching specific photodiodes, or portions of specific photodiodes, at specific times and / or from specific angles. Image sensor 130 may also include analog gain amplifiers for amplifying analog signals output by the photodiodes, and / or analog-to-digital converters (ADCs) for converting analog signals output from the photodiodes (and / or amplified by the analog gain amplifiers) into digital signals. In some cases, certain components or functions discussed with respect to one or more of control mechanisms 120 may instead or additionally be included within image sensor 130. The image sensor 130 may be a charge-coupled device (CCD) sensor, an electron-multiplying CCD (EMCCD) sensor, an active-pixel sensor (APS), a complimentary metal-oxide semiconductor (CMOS), an N-type metal-oxide semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0025]

[0042] Image processor 150 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 154), one or more host processors (including host processor 152), and / or one or more of any other types of processors 1510 discussed with respect to computing system 1500. Host processor 152 may be a digital signal processor (DSP) and / or other types of processors. In some implementations, image processor 150 is a single integrated circuit or chip (e.g., referred to as a system-on-chip or SoC) that includes host processor 152 and ISP 154. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) ports 156), central processing units (CPUs), graphics processing units (GPUs), broadband modems (e.g., 3G, 4G, or LTE, 5G, etc.), memory, connectivity components (e.g., Bluetooth™, Global Positioning System (GPS), etc.), any combination thereof, and / or other components.I / O ports 156 may include any suitable input / output ports or interfaces according to one or more protocols or specifications, such as an Inter-Integrated Circuit 2 (I2C) interface, an Inter-Integrated Circuit 3 (I3C) interface, a Serial Peripheral Interface (SPI) interface, a serial General Purpose Input / Output (GPIO) interface, a Mobile Industry Processor Interface (MIPI) (MIPI CSI-2 physical (PHY) layer port or interface, an Advanced High-performance Bus (AHB) bus, any combination thereof, and / or other input / output ports. In one illustrative example, host processor 152 may communicate with image sensor 130 using an I2C port, and ISP 154 may communicate with image sensor 130 using a MIPI port.

[0026]

[0043] Image processor 150 may perform several tasks, such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging image frames to form HDR images, image recognition, object recognition, feature recognition, accepting input, managing output, managing memory, or some combination thereof. Image processor 150 may store image frames and / or processed images in random access memory (RAM) 140 and / or 1520, read-only memory (ROM) 145 and / or 1525, a cache, a memory unit, another storage device, or some combination thereof.

[0027]

[0044] Various input / output (I / O) devices 160 may be connected to image processor 150. I / O device 160 may include a display screen, a keyboard, a keypad, a touch screen, a trackpad, a touch-sensitive surface, a printer, any other output device 1535, any other input device 1545, or some combination thereof. In some cases, captions may be entered into image processing device 105B through a physical keyboard or keypad of I / O device 160 or through a virtual keyboard or keypad of a touch screen of I / O device 160. I / O 160 may include one or more ports, jacks, or other connectors that enable a wired connection between system 100 and one or more peripheral devices, through which system 100 may receive data from and / or send data to one or more peripheral devices. I / O 160 may include one or more wireless transceivers that enable a wireless connection between system 100 and one or more peripheral devices, through which system 100 may receive data from and / or transmit data to one or more peripheral devices. A peripheral device may include any of the types of I / O devices 160 discussed above, and may itself be considered an I / O device 160 when coupled to a port, jack, wireless transceiver, or other wired and / or wireless connector.

[0028]

[0045] In some cases, image capture and processing system 100 may be a single device. In some cases, image capture and processing system 100 may be two or more separate devices, including image capture device 105A (e.g., a camera) and image processing device 105B (e.g., a computing device coupled to a camera). In some implementations, image capture device 105A and image processing device 105B may be coupled together, for example, via one or more wires, cables, or other electrical connectors, and / or wirelessly via one or more wireless transceivers. In some implementations, image capture device 105A and image processing device 105B may be decoupled from each other.

[0029]

[0046] 1, a vertical dashed line divides image capture and processing system 100 of FIG. 1 into two portions representing image capture device 105A and image processing device 105B, respectively. Image capture device 105A includes lens 115, and control mechanism 120, image sensor 130. Image processing device 105B includes image processor 150 (including ISP 154 and host processor 152), RAM 140, ROM 145, and I / O 160. In some cases, certain components shown in image capture device 105A, such as ISP 154 and / or host processor 152, may be included within image capture device 105A.

[0030]

[0047] Image capture and processing system 100 may include an electronic device such as a mobile or fixed telephone handset (e.g., a smartphone, a mobile phone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video gaming console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, image capture and processing system 100 may include one or more wireless transceivers for wireless communication, such as cellular network communication, 1502.11 Wi-Fi communication, a wireless local area network (WLAN) communication, or some combination thereof. In some implementations, image capture device 105A and image processing device 105B may be different devices. For example, image capture device 105A may include a camera device, and image processing device 105B may include a computing device, such as a mobile handset, a desktop computer, or other computing device.

[0031]

[0048] While image capture and processing system 100 is shown to include certain components, those skilled in the art will understand that image capture and processing system 100 may include many more components than those shown in FIG. 1 . The components of image capture and processing system 100 may include software, hardware, or one or more combinations of software and hardware. For example, in some implementations, the components of image capture and processing system 100 may include and / or be implemented using electronic circuitry or other electronic hardware, including one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof, for performing various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device implementing image capture and processing system 100.

[0032]

[0049] 2 is a block diagram illustrating an example architecture of an extended reality (XR) system 200. In some examples, the XR system 200 of FIG. 2 is or includes an XR system having a Media Capabilities for Augmented Reality (MeCAR) architecture, an Edge-dependent Augmented Reality (EDGAR) architecture, another architecture discussed herein, or a combination thereof. For example, in some examples, the XR system 200 can include and / or be an optical see-through XR device, a video pass-through XR device, or a combination thereof.

[0033]

[0050] The XR system 200 includes an XR runtime subsystem 212 that interfaces with input device(s) and / or output device(s). For example, the XR runtime subsystem 212 receives sensor data from sensor(s) 208, such as camera(s) 210, microphone(s) 226, and / or other sensor(s). The camera(s) 210 may include image sensor(s) (e.g., image sensor(s) 130) that capture, for example, image(s) and / or video(s) of a scene (or environment) in front of and / or around a user of the XR system 200, image(s) and / or video(s) of a user of the XR system 200 (e.g., of the user's eyes for eye tracking, the user's hands for hand tracking, the user's face for facial expression tracking, the user's body for body posture tracking, or a combination thereof). Similarly, the microphone(s) 226 may capture audio of the scene (e.g., of other people or objects in the scene), audio of the user (e.g., the user's voice), or a combination thereof. In some examples, the other sensor(s) (of sensor(s) 208) may include accelerometer(s), gyroscope(s), gyrometer(s), inertial measurement unit(s) (IMU(s)), depth sensor(s), altimeter(s), barometer(s), positioning receiver(s), or combinations thereof.The positioning receiver(s) may include a GNSS receiver or transceiver used to determine the location of the XR system 200, such as a Global Positioning System (GPS), a Global Navigation Satellite System (GLONASS), a BeiDou Navigation Satellite System (BDS), the European-based Galileo Global Navigation Satellite System (GNSS), another GNSS, or a combination thereof. The depth sensor(s) (sometimes referred to as range sensors or distance sensors) may include a light detection and ranging (LIDAR) sensor, a radio detection and ranging (RADAR) sensor, a sound detection and ranging (SODAR) sensor, a sound navigation and ranging (SONAR) sensor, a time of flight (ToF) sensor, a structured light sensor, a set of cameras performing stereoscopic depth sensing, or a combination thereof.

[0034]

[0051] The XR runtime subsystem 212 of the XR system 200 performs various runtime functions 216, such as pose tracking (e.g., of the pose of the XR system 200 and / or a user), eye tracking (e.g., of the user's eyes), hand tracking (e.g., of the user's hands), body tracking (e.g., of the user's body), feature tracking (e.g., of features in a scene and / or a user), object tracking (e.g., of objects and / or parts of a user in a scene), face tracking (e.g., of the user's face and / or the faces of other people in a scene), simultaneous localization and mapping (SLAM), or a combination thereof. Pose refers to location (e.g., longitude, latitude, altitude), orientation (e.g., pitch, yaw, roll), or a combination thereof. Pose can be tracked over 3 degrees of freedom (3DoF), 6 degrees of freedom (6DoF), or another range. The XR runtime subsystem 212 of the XR system 200 includes a visual synthesis subsystem 214 that couples to an eye buffer display 220, which may be a buffer for one or more display(s) (e.g., display(s) 340, display 440) that are directed toward the user's eye(s). The XR runtime subsystem 212 of the XR system 200 includes an audio subsystem 218 that receives audio from microphone(s) 226 and / or outputs audio to speaker(s) 224. The speaker(s) 224 may include loudspeakers, headphones, earphones, other audio output devices, or a combination thereof.

[0035]

[0052] The XR system 200 includes an XR application 246 that can receive user input(s) 248 via an input interface of the XR system 200. The input(s) 248 can be passed to an XR runtime application programming interface (API) 202, an XR source management subsystem 244, a scene manager 238 and / or a presentation engine 242, a media access function (MAF) API 254, and / or a network system 278. In some examples, the XR application 246 is or is associated with a video game, and the user input(s) 248 include, for example, input(s) to the video game (e.g., controller input) that can affect what virtual content is rendered within the XR content to be shown to the user (e.g., via the eye buffer display 220).

[0036]

[0053] The XR system 200 is configured for split rendering, such that at least a subset of processing tasks will be performed by the XR management system and / or an external XR processing device (e.g., edge node, remote server) that is coupled to the XR system 200 via and / or is part of the network system 278 (e.g., as in the case of an edge node). For example, sensor data, user input(s) 248, and / or associated metadata captured by the XR system (e.g., as discussed above) are collected by the XR system 200. The XR runtime API 202 and XR source management subsystem 244 collect this as XR media and metadata 252 (e.g., including image(s), input(s), pose, etc.) and communicate this to the media access function (MAF) subsystem 276. The MAF subsystem 276 encodes, compresses, and / or encrypts the XR media and metadata 252 (e.g., using a metadata codec 290, a video codec 292, and / or an audio codec 294) to form uplink compressed media 272 that is sent to an external XR processing device (e.g., an edge node, a remote server). The XR processing device decrypts, decompresses, decodes, and / or processes the uplink compressed media 272 to generate XR content. For example, in some examples, the XR processing device adds virtual content to generate the XR content. The XR processing device encodes, compresses, and / or encrypts the resulting XR content, which is received via the network system 278 as downlink compressed media 274. The MAF subsystem 276 decrypts, decompresses, and / or decodes the downlink compressed media 274 to extract pre-rendered media 258 (e.g., 2D media, 2.5D media, and / or pose-dependent media), scene description 256, or a combination thereof.

[0037]

[0054] The pre-rendered media 258, scene description 256, and / or user input(s) 248 are passed to the scene manager 238 and / or presentation engine 242 and / or the XR runtime API 202 to the XR runtime subsystem 212. The scene manager 238 and / or presentation engine 242 and / or the XR runtime API 202 determine what image(s) the visual synthesis subsystem 214 of the XR runtime subsystem 212 outputs to the eye buffer display 220 to be displayed to the user.

[0038]

[0055] In some examples, the XR system 200 is a single device, such as a headset (e.g., a head-mounted display (HMD) 310, glasses, eyewear, wearable device(s), or a combination thereof), a mobile device (e.g., a mobile handset, a tablet, a laptop, a wearable device), or a combination thereof. In some examples, the XR system 200 includes multiple devices, such as an XR interface device and an XR management system. In some examples, the XR interface device includes sensor(s) (e.g., sensor 208, camera 210, microphone 226), display(s) (e.g., eye-buffer display 220), user interface(s) (e.g., through which user input(s) 248 are received), and / or specific elements that control these. In illustrative examples, the XR interface device is a headset (e.g., head-mounted display (HMD) 310, glasses, eyewear, wearable device(s), or a combination thereof), a mobile device (e.g., a mobile handset, a tablet, a laptop, a wearable device), or a combination thereof. In some examples, the XR management system can include other elements (e.g., other than sensors and / or a display) shown in XR system 200 and can interface between the XR interface device and the XR processing device. In some examples, XR system 200 can include XR application 246. In some examples, the XR management system is a mobile device (e.g., a mobile handset, a tablet, a laptop, a wearable device). In some examples, the XR management system is locally coupled to the XR interface device (e.g., wirelessly or using wire(s)). In some examples, the XR processing device is a remote computing system (e.g., an edge node, a remote server) coupled to the XR management system via a network communication interface, such as a cellular network (e.g., 5G) interface.In some examples, the XR interface device may lack a connection to the Internet, but the XR management system has a connection to the Internet (e.g., via network system 278).

[0039]

[0056] In some examples, split rendering across a link may have limitations regarding the formats that can be used, as well as the connectivity and associated bitrates that are supported. Knowledge of the capabilities of the tethered XR interface device, accessible through the XR runtime API 202 on the XR management system, can support operation of the XR management system over a network (e.g., a 5G cell network) regarding the required bitrates and / or exemplary formats.

[0040]

[0057] Technical issues that may arise in split rendering may include limited raw formats. For example, in some cases, the formats usable by the XR interface device may be limited or restricted, e.g., with respect to resolution, based on the capabilities of the XR interface device and / or the capabilities (e.g., bandwidth, bit rate, baud rate) of the connection(s) (e.g., between the XR interface device and the XR management system, between the XR management system and the XR processing device, or both). The capabilities (e.g., bandwidth, bit rate, baud rate) of the connection(s) may be limited and, in some cases, can change dynamically (e.g., to reduce power usage or temperature, to allow other content to be transferred, such as firmware updates). In some cases, the security frameworks (e.g., encryption, decryption, etc.) between the XR interface device and the XR management system should be aligned, and inconsistencies may cause problems.

[0041]

[0058] Technical issues that may arise in split rendering may include power usage and added latency from transcoding. For example, in some instances, the XR interface device provides compressed data to the XR management device, which decompresses the data, then recompresses the data and sends it to the XR processing device, which decompresses the data again, processes the data, recompresses the data and sends it back to the XR management device, which decompresses the data again, processes the data, and recompresses the data and sends it back to the XR interface device. These multiple decompression and compression steps may result in visual artifacts, distortion, and / or other losses of quality, use more power, and represent a potential gap in security while the data is not compressed or encrypted.

[0042]

[0059] Solutions in which sensor data is compressed at the XR interface device and remains compressed through the XR management device to the XR processing device, and solutions in which processed XR data is compressed at the XR processing device and remains compressed through the XR management device to the XR interface device, solve the technical problems discussed above. Examples of such solutions are shown in Figures 10, 12, 13, and 14.

[0043]

[0060] Compression and decompression as discussed herein may refer to spatial compression (e.g., of individual video frames of a video), temporal compression (e.g., across multiple video frames of a video), predictive motion compensation, or a combination thereof. In some examples, compression and decompression as discussed herein may refer to compression and / or decompression using video coding and / or video compression formats such as H.264, H.265, Versatile Video Coding (VVC), AOMedia Video 1 (AV1), differential pulse-code modulation (DPCM), discrete cosine transform (DCT), discrete wavelet transform (DWT), or a combination thereof. In some examples, compression and / or decompression as discussed herein is lossy. In some examples, compression and / or decompression as discussed herein is lossless.

[0044]

[0061] In some examples, the XR system 200 can perform visual processing according to the MeCAR architecture. The XR system 200 can be adapted to an XR runtime API 202 and / or may be aligned with OpenXR processing. In some examples, to present images to a user, the XR runtime subsystem 212 can provide images organized in a swap chain for applications to render (e.g., images captured by the camera(s) 210 and / or as XR content provided to the eye buffer display 220). The swap chain images can be 2D images or 2D arrays (e.g., a 2.5D image set). The array allows for extracting a subset of the 2D image for rendering.

[0045]

[0062] In some examples, the XR runtime subsystem 212 may support different swap chain image formats, and the supported image formats may be provided to applications through the XR runtime API 202. In an illustrative example, the XR runtime subsystem 212 supports at least R8G8B8A8 and R8G8B8A8 sRGB formats. Details may depend on the graphics API specified in xrCreateSession. Options include DirectX and / or open graphics library (OpenGL). In an illustrative example, the XR system may support OpenGL for embedded systems (OpenGL ES), for example, using functionality provided in XR_KHR_opengl_es_enable.

[0046]

[0063] In some examples, the XR application 246 and / or the scene manager 238 and / or the presentation engine 242 can offload the compositing of XR content (to be displayed to the user via the eye buffer display 220) to, for example, an XR runtime-supplied compositor of the XR processing device. Offloading the compositing of XR content, for example, offloading functions such as frame rate interpolation and distortion correction from the XR interface device (e.g., the XR runtime subsystem 212) to the XR processing device reduces rendering complexity.

[0047]

[0064] In some examples, the XR runtime subsystem 212 (e.g., visual compositing subsystem 214) of the XR system 200 supports functionality such as OpenXR compositing, projective compositing, quad compositing, cube compositing, cylinder compositing, equirectangular compositing, or combinations thereof.

[0048]

[0065] The XR system 200 of FIG. 2 can perform audio processing according to the MeCAR PD architecture. In some examples, the audio processing subsystem 218 can use OpenXR and / or Open Sound Library (OpenSL) ES. In some examples, the audio processing subsystem 218 can perform immersive audio rendering as discussed herein. In some examples, the interface to the XR runtime subsystem 212 passes raw audio buffers to determine how the XR application 246 and scene manager 238 access the audio capabilities of the XR interface device. For example, OpenSL ES supports both file-based and in-memory data sources, as well as buffer queues for efficient streaming of audio data from memory to the audio system. A buffer queue in OpenSL can be seen as equivalent to a visual swap chain. OpenSL ES can be seen as a complement to a 3D graphics API, such as OpenGL ES. The 3D graphics engine renders the 3D graphics scene to a two-dimensional display device (e.g., eye buffer display 220), and the OpenSL ES implementation renders the 3D audio scene to an audio output device (e.g., speaker 224).

[0049]

[0066] In addition to functionality from such buffer queues, different types of audio signals may be provided, and additional or alternative processing steps may be performed. The audio signals (i.e., combinations of metadata and buffer queues) may be: (a) non-immersive, also known as non-diegetic, i.e., they are not rendered according to pose; (b) immersive, describing the full 6DoF experience in the reference space of the XR session, in which case the XR runtime creates a signal rendered according to the latest pose; (c) immersive, pre-rendered for a specific rendering pose (in which case the signal may be prepared so that the runtime can use the audio signal and the associated rendering pose and supplementary data for pose correction to the latest pose); (d) a mix of such signals presented together; (e) signals may originate from different sources, e.g., some may be generated locally and others may be part of a pre-rendered or complete scene created within the network; or (f) a combination thereof.

[0050]

[0067] 3A is a perspective view 300 illustrating a head-mounted display (HMD) 310 used as part of a sensor data processing system. The HMD 310 may be, for example, an augmented reality (AR) headset, a virtual reality (VR) headset, a mixed reality (MR) headset, an extended reality (XR) headset, or some combination thereof. The HMD 310 includes a first camera 330A and a second camera 330B along a front portion of the HMD 310. The HMD 310 includes a third camera 330C and a fourth camera 330D that face the user's eye(s) when the user's eye(s) face the display(s) 340. In some examples, the HMD 310 may have only a single camera with a single image sensor. In some examples, the HMD 310 may include one or more additional cameras in addition to the first camera 330A, the second camera 330B, the third camera 330C, and the fourth camera 330D. In some examples, the HMD 310 may include one or more additional sensors in addition to the first camera 330A, the second camera 330B, the third camera 330C, and the fourth camera 330D. In some examples, the first camera 330A, the second camera 330B, the third camera 330C, and / or the fourth camera 330D may be examples of the image capture and processing system 100, the image capture device 105A, the image processing device 105B, or a combination thereof.

[0051]

[0068] The HMD 310 may include one or more displays 340 visible to the user 320 wearing the HMD 310 on their head. In some examples, the HMD 310 may include one display 340 and two viewfinders. The two viewfinders may include a left viewfinder for the user's 320 left eye and a right viewfinder for the user's 320 right eye. The left viewfinder may be oriented so that the user's 320 left eye views the left side of the display. The right viewfinder may be oriented so that the user's 320 right eye views the right side of the display. In some examples, the HMD 310 may include two displays 340, including a left display that displays content to the user's 320 left eye and a right display that displays content to the user's 320 right eye. The one or more displays 340 of the HMD 310 may be digital "pass-through" displays or optical "see-through" displays.

[0052]

[0069] The HMD 310 may include one or more earpieces 335 that may function as speakers and / or headphones to output audio to one or more ears of a user of the HMD 310. While one earpiece 335 is shown in Figures 3A and 3B, it should be understood that the HMD 310 may include two earpieces, one for each ear (left and right) of the user. In some examples, the HMD 310 may also include one or more microphones (not shown). In some examples, the audio output by the HMD 310 to the user through the one or more earpieces 335 may include or be based on audio recorded using one or more microphones.

[0053]

[0070] FIG. 3B is a perspective view 350 showing the head-mounted display (HMD) of FIG. 3A being worn by a user 320. The user 320 wears the HMD 310 on the user's head over the user's eyes. The HMD 310 can capture images using a first camera 330A and a second camera 330B. In some examples, the HMD 310 displays one or more output images to the user's eyes using a display(s) 340. In some examples, the output images can include virtual content and / or processed content as discussed herein. The output images can be based, for example, on images (e.g., image(s) and / or other sensor data) captured by the first camera 330A and the second camera 330B overlaid with processed content (e.g., virtual content and / or processed content as discussed herein). The output image may provide a stereoscopic view of the environment, possibly with processed content overlaid and / or other modifications. For example, the HMD 310 may display a first display image to the right eye of the user 320 based on an image captured by the first camera 330A. The HMD 310 may display a second display image to the left eye of the user 320 based on an image captured by the second camera 330B. For example, the HMD 310 may provide overlaid processed content within the display image overlaid on the images captured by the first camera 330A and the second camera 330B. The third camera 330C and the fourth camera 330D may capture images of the eyes before, during, and / or after the user views the display image displayed by the display(s) 340. In this manner, sensor data from the third camera 330C and / or the fourth camera 330D can capture the reaction of the user's eyes (and / or other parts of the user) to the processed content. The earpieces 335 of the HMD 310 are shown in the ears of the user 320.The HMD 310 may output audio to the user 320 through the earpiece 335 and / or through another earpiece (not shown) of the HMD 310 in the other ear (not shown) of the user 320.

[0054]

[0071] 4A is a perspective view 400 showing the front of a mobile handset 410 that includes a front-facing camera and can be used as part of a sensor data processing system. The mobile handset 410 can be, for example, a mobile phone, a satellite phone, a portable gaming console, a music player, a health tracking device, a wearable device, a wireless communication device, a laptop, a mobile device, any other type of computing device or computing system discussed herein, or a combination thereof.

[0055]

[0072] The front surface 420 of the mobile handset 410 includes a display 440. The front surface 420 of the mobile handset 410 includes a first camera 430A and a second camera 430B. The first camera 430A and the second camera 430B can face the user, including the user's eye(s), while content (e.g., virtual content and / or processed content as discussed herein) is displayed on the display 440.

[0056]

[0073] The first camera 430A and the second camera 430B are shown within a bezel around the display 440 on the front face 420 of the mobile handset 410. In some examples, the first camera 430A and the second camera 430B can be positioned in a notch or cutout cut out of the display 440 on the front face 420 of the mobile handset 410. In some examples, the first camera 430A and the second camera 430B can be under-display cameras positioned between the display 440 and the rest of the mobile handset 410, so that light passes through a portion of the display 440 before reaching the first camera 430A and the second camera 430B. The first camera 430A and the second camera 430B in the perspective view 400 are front-facing cameras. The first camera 430A and the second camera 430B face in a direction perpendicular to the plane of the front face 420 of the mobile handset 410. The first camera 430A and the second camera 430B may be two of one or more cameras of the mobile handset 410. In some examples, the front face 420 of the mobile handset 410 may have only a single camera.

[0057]

[0074] In some examples, the display 440 of the mobile handset 410 displays one or more output images to a user using the mobile handset 410. In some examples, the output images can include virtual content and / or processed content as discussed herein. The output images can be based, for example, on images (e.g., image(s) and / or other sensor data) captured by the first camera 430A, the second camera 430B, the third camera 430C, and / or the fourth camera 430D overlaid with processed content (e.g., virtual content and / or processed content as discussed herein).

[0058]

[0075] In some examples, the front surface 420 of the mobile handset 410 may include one or more additional cameras in addition to the first camera 430A and the second camera 430B. In some examples, the front surface 420 of the mobile handset 410 may include one or more additional sensors in addition to the first camera 430A and the second camera 430B. In some cases, the front surface 420 of the mobile handset 410 includes two or more displays 440. For example, the one or more displays 440 may include one or more touchscreen displays.

[0059]

[0076] The mobile handset 410 may include one or more speakers 435A and / or other audio output devices (e.g., earphones or headphones or connectors thereto) that can output audio to one or more ears of a user of the mobile handset 410. While one speaker 435A is shown in FIG. 4A , it should be understood that the mobile handset 410 can include more than one speaker and / or other audio device. In some examples, the mobile handset 410 also includes one or more microphones (not shown). In some examples, the mobile handset 410 can include one or more microphones along and / or adjacent to the front surface 420 of the mobile handset 410. In some examples, audio output by the mobile handset 410 to the user through the one or more speakers 435A and / or other audio output devices may include or be based on audio recorded using the one or more microphones.

[0060]

[0077] 4B is a perspective view 450 showing the back surface 460 of a mobile handset that includes a rear-facing camera and can be used as part of a sensor data processing system. The mobile handset 410 includes a third camera 430C and a fourth camera 430D on the back surface 460 of the mobile handset 410. The third camera 430C and the fourth camera 430D in the perspective view 450 are at the back surface. The third camera 430C and the fourth camera 430D are oriented perpendicular to the plane of the back surface 460 of the mobile handset 410.

[0061]

[0078] The third camera 430C and the fourth camera 430D may be two of one or more cameras of the mobile handset 410. In some examples, the back surface 460 of the mobile handset 410 may have only a single camera. In some examples, the back surface 460 of the mobile handset 410 may include one or more additional cameras in addition to the third camera 430C and the fourth camera 430D. In some examples, the back surface 460 of the mobile handset 410 may include one or more additional sensors in addition to the third camera 430C and the fourth camera 430D. In some examples, the first camera 430A, the second camera 430B, the third camera 430C, and / or the fourth camera 430D may be examples of the image capture and processing system 100, the image capture device 105A, the image processing device 105B, or a combination thereof.

[0062]

[0079] The mobile handset 410 may include one or more speakers 435B and / or other audio output devices (e.g., earphones or headphones or connectors thereto) that can output audio to one or more ears of a user of the mobile handset 410. While one speaker 435B is shown in FIG. 4B, it should be understood that the mobile handset 410 can include more than one speaker and / or other audio device. In some examples, the mobile handset 410 also includes one or more microphones (not shown). In some examples, the mobile handset 410 can include one or more microphones along and / or adjacent to the back surface 460 of the mobile handset 410. In some examples, audio output by the mobile handset 410 to the user through the one or more speakers 435B and / or other audio output devices may include or be based on audio recorded using the one or more microphones.

[0063]

[0080] The mobile handset 410 may use the display 440 on the front face 420 as a pass-through display. For example, the display 440 may display an output image, such as the virtual content and / or processed content discussed herein. The output image may be based on an image (e.g., image(s) and / or other sensor data) captured by the third camera 430C and / or the fourth camera 430D, overlaid with the processed content (e.g., the virtual content and / or processed content discussed herein). The first camera 430A and / or the second camera 430B may capture images of the user's eyes (and / or other parts of the user) before, during, and / or after the output image with the processed content is displayed on the display 440. In this manner, the sensor data from the first camera 430A and / or the second camera 430B may capture a response by the user's eyes (and / or other parts of the user) to the processed content.

[0064]

[0081] 5 is a swimlane diagram illustrating a process 500 for XR processing. The process 500 is divided between an XR system 505 (e.g., including an XR interface device and / or an XR management system) and an XR processing device (e.g., an edge node and / or a remote server). The XR system 505 can include an XR application, an XR runtime, a scene manager, and a media access function subsystem. The XR processing device 510 includes a media delivery function, a scene manager, a scene provider, and, in some examples, an XR application provider.

[0065]

[0082] The illustrated process 500 represents a media delivery pipeline 590 for extended reality (XR). Process 500 includes an operation 515 in which a scene manager initializes an XR delivery session. Process 500 includes an operation 520 in which a media access function (MAF) establishes an XR delivery session with a media delivery function. Process 500 includes an operation 525 in which the MAF may receive an update to the scene description from a scene provider. Process 500 includes an operation 530 in which the MAF passes the scene update to the scene manager of the XR system 505. Process 500 includes an operation 535 in which the scene manager of the XR system 505 updates the scene. Process 500 includes an operation 540 in which the scene manager of the XR system 505 obtains the latest pose information and user actions. Process 500 includes an operation 545 in which the scene manager of the XR system 505, its information, is with the scene manager in the XR processing device 510.

[0066]

[0083] The media rendering loop 595 of the media delivery pipeline 590 includes three loops: operation 550, operations 555A-555C, operation 560, operations 565A-565B, operation 570, operations 575A-575B, and operation 580. At operation 550, a session is created for each new object in the scene. For each new object in the scene, at operation 555A, the scene manager of the XR system 505 triggers the MAF to fetch the associated media, at operation 555B, the MAF creates a dedicated media pipeline to process the input, and at operation 555C, the MAF establishes a transport session for each component of the media object. At operation 560, the transport session is processed. For each transport session, in operation 565A, the media pipeline fetches media data (e.g., the media data can be static, segmented, and / or real-time media streams), and in operation 565B, the media pipeline processes the media and makes the processed media available in a buffer. Rendering is performed in operation 570. For each object to be rendered, in operation 575A, the scene manager of the XR system 505 obtains the processed media data from the media pipeline buffer, and in operation 575B, the scene manager of the XR system 505 reconstructs and renders the object. In operation 580, the scene manager of the XR system 505 passes the rendered frames to the XR runtime for display on the XR interface device (e.g., the user's headset, HMD 310, eyewear, wearable device, or mobile handset 410).

[0067]

[0084] In some examples, an additional operation can be added before operation 515. In this additional operation, the capabilities of the XR runtime can be queried. In some examples, operation 580 can adjust the render frame format to be more specific (e.g., the render frame format is a swap chain format). In the systems and methods described herein, the flow in process 500 can be applied, for example, to XR processing in a MeCAR architecture.

[0068]

[0085] 6 is a swimlane diagram showing an extended reality (XR) application lifecycle 600. In an illustrative example, the XR application lifecycle 600 of FIG. 6 may represent an OpenXR application lifecycle. In some examples, certain actions within the XR application lifecycle 600 are synchronized with each other and / or with respect to each other. The operations of the XR application lifecycle 600 are performed by an XR system 605, which may include an XR interface device, an XR management system, an XR processing device, or a combination thereof.

[0069]

[0086] At operation 610, the XR application retrieves the action state of the XR runtime. At operation 615, the XR application waits for a frame to be provided by the XR runtime. At operation 620, the xrWaitFrame frame state (for the frame(s) from the XR runtime) includes the predictedDisplayTime. At operation 625, the XR application and / or XR runtime obtains the predicted pose corresponding to the predictedDisplayTime. At operation 630, the XR application begins the rendering process for the XR runtime via xrBeginFrame. At operation 635, the XR application and / or XR runtime locates the view via xrLocateViews. At operation 640, the swap chain image is available to the XR application (e.g., from the XR runtime).

[0070]

[0087] The XR application lifecycle 600 includes a rendering loop 670. For all views, the graphics framework starts the rendering loop at operation 645, and at operation 650, respectively, the graphics framework renders the view (operation 650), and at operation 655, the graphics framework writes the view to a swap chain associated with the XR runtime.

[0071]

[0088] Once the rendering loop 670 is complete, the graphics framework indicates to the XR application that rendering is complete in operation 660. In operation 665, the XR application sends information (e.g., time, mode, and / or layer) for display via xrEndFrame.

[0072]

[0089] In some examples, the XR application lifecycle 600 of FIG. 6 may represent an OpenXR application lifecycle. In an illustrative example, after creating an OpenXR session, the application starts a frame loop. The frame loop is executed for each frame. The frame loop consists of the following operations: The first frame loop operation synchronizes actions, including, for example, retrieving action state (e.g., the state of user input controller buttons, the pose of the XR interface device, and / or the location(s) of different trackable object(s)) and / or sending haptic feedback. The second frame loop operation starts a new frame and begins by waiting for a frame to be provided by the XR runtime, as appropriate, for example, to synchronize application frame submission with the display. In some examples, the xrWaitFrame function returns the frame state of the requested frame, including the predictedDisplayTime, which is a prediction of when the corresponding composited frame will be displayed. This information is used by the application to request a predicted pose at display time. Upon completion of the xrWaitFrame function, the application calls xrBeginFrame to signal the start of the rendering process. The third frame loop operation retrieves rendering resources. For example, the application begins by locating views in space and time by calling the xrLocateViews function, provided with the expected display time and XR space. The application obtains swap chain images associated with all views in the compositing layer and waits for the swap chain images to become available so that the application can write to them. The fourth frame loop operation relates to rendering. The application performs its rendering work, for example, by iterating through the scene graph nodes and rendering each object to a view.In some examples, rendering uses a graphics framework such as Vulkan, OpenGL, and / or Direct3D to perform graphics operations. A fifth frame loop operation relates to releasing resources. Once rendering is done for a view, the application releases the corresponding swap chain image. Once all views have been rendered, the application sends the views for display (e.g., on the display(s) of the XR interface device) by calling the xrEndFrame function.

[0073]

[0090] For audio media, a similar process as for video typically applies. In some examples, OpenXR and / or Open Sound Library (OpenSL) ES are used for audio and / or video processing. In some examples, immersive audio rendering can include operations as discussed herein. Interfaces to the XR runtime are available for passing raw audio buffers to determine how XR applications and scene managers access the device's audio capabilities. In an illustrative example, some systems (e.g., OpenSL ES) support file-based data sources, in-memory data sources, and / or buffer queues for efficient streaming of audio data from memory to the audio system. A buffer queue in OpenSL can be seen as equivalent to a visual swap chain. OpenSL ES can be seen as an adjunct to a 3D graphics API such as OpenGL ES. A 3D graphics engine renders a 3D graphics scene to a two-dimensional display device, and an OpenSL ES implementation renders a 3D audio scene to an audio output device. In current implementations, in addition to functionality from such buffer queues, different types of audio signals can be provided and additional / alternative processing steps can be performed. The audio signal (i.e., the combination of the metadata and the buffer queue) can be:(a) non-immersive and / or non-diegetic (e.g., not rendered according to pose), (b) immersive and describing a full 6DoF experience in the reference space of the XR session (e.g., the XR runtime creates rendered signals according to the latest pose), (c) immersive and / or pre-rendered for a specific rendering pose (e.g., the signals are prepared so that the XR runtime can use the audio signal and / or associated rendering pose and / or supplemental data for pose correction to the latest pose), (d) a mix of such signals presented together, (e) signals originating from different source(s) (e.g., some signals may be generated locally, while others may be part of a pre-rendering or a complete scene created within a network), or (f) a combination thereof. In some examples, the audio data may be compressed or uncompressed.

[0074]

[0091] 7 is a block diagram illustrating an example architecture of an extended reality (XR) system 700 including an extended reality (XR) interface device 702 and an XR management system 704. The XR system 700 may be an example of the XR system 200. In an illustrative example, the XR interface device 702 is a headset (e.g., a head-mounted display (HMD) 310, glasses, eyewear, wearable device(s), or a combination thereof), a mobile device (e.g., a mobile handset 410, a tablet, a laptop, a wearable device), or a combination thereof. The XR management system 704 is coupled to the XR interface device 702 and, in some cases, may be coupled to an XR processing device through a cellular network subsystem 776. In some examples, the XR management system 704 is a mobile device (e.g., a mobile handset 410, a tablet, a laptop, a wearable device). In some examples, the XR management system 704 is locally coupled to the XR interface device 702 (e.g., wirelessly or using wire(s)). In some examples, the XR processing device is a remote computing system (e.g., edge node, remote server) that is coupled to the XR management system via a network communication interface, such as a cellular network (e.g., 5G) interface. In some examples, the XR interface device may lack a connection to the internet, but the XR management system has a connection to the internet (e.g., via network system 278).

[0075]

[0092] The XR interface device 702 of the XR system 700 includes an XR runtime subsystem 712 that interfaces with input device(s) and / or output device(s). For example, the XR runtime subsystem 712 receives sensor data from sensor(s) 708, such as camera(s) 710, microphone(s) 726, and / or other sensor(s). The camera(s) 710 may include image sensor(s) (e.g., image sensor(s) 130) that capture, for example, image(s) and / or video(s) of a scene (or environment) in front of and / or around a user of the XR system 700, image(s) and / or video(s) of a user of the XR system 700 (e.g., of the user's eyes for eye tracking, the user's hands for hand tracking, the user's face for facial expression tracking, the user's body for body posture tracking, or a combination thereof). Similarly, the microphone(s) 726 may capture audio of the scene, audio of the user, or a combination thereof. In some examples, the other sensor(s) (of the sensor(s) 708) may include any of the sensors discussed with respect to the sensor(s) 208.

[0076]

[0093] The XR runtime subsystem 712 of the XR system 700 performs various runtime functions such as pose tracking (e.g., of the XR system 700 and / or the user's pose), eye tracking (e.g., of the user's eyes), hand tracking (e.g., of the user's hands), body tracking (e.g., of the user's body), feature tracking (e.g., of features in the scene and / or the user), object tracking (e.g., of objects and / or parts of the user in the scene), face tracking (e.g., of the user's face and / or the faces of other people in the scene), SLAM, or a combination thereof.

[0077]

[0094] The XR interface device 702 is coupled to the XR management system 704 through a tethering communication interface 732, which may also be referred to as a connection, tethering, link, coupling, or interface. The tethering communication interface 732 may be a local coupling or interface, which refers to, for example, a wired coupling or connection, a short-range wireless coupling or connection, or a combination thereof. On one end of the tethering communication interface 732 is a communication interface 730 of the XR interface device 702, which is managed using the XR link function 728 of the XR interface device 702. On the other end of the tethering communication interface 732 is a communication interface 734 of the XR management system 704, which is managed using the XR link function 736 of the XR management system 704. In some examples, the tethering communication interface 732 coupling the XR interface device 702 and the XR management system 704 is a wireless communication interface, in which case the communication interface 730 of the XR interface device 702 and the communication interface 734 of the XR management system 704 may include a wireless communication interface, such as a wireless communication transceiver(s) and / or a wireless communication antenna(s). If the tethering communication interface 732 is a wireless communication interface, the tethering communication interface 732 and associated components may use a wireless technology, such as Wi-Fi, Bluetooth, wireless local area network(s) (WLAN(s)), personal area network(s) (PAN(s)), cellular network connection, other wireless technologies discussed herein, or a combination thereof.In some examples, the tethering communication interface 732 may use the Real-time Transport Protocol (RTP), the Secure Real-time Transport Protocol (SRTP), another transport protocol, or a combination thereof.

[0078]

[0095] In some examples, the tethering communication interface 732 coupling the XR interface device 702 and the XR management system 704 is a wired communication interface, in which case the communication interface 730 and the communication interface 734 can include a port, a plug, a jack, a wire connector, or a combination thereof. If the tethering communication interface 732 is a wired communication interface, the tethering communication interface 732 and associated components can use a wired technology such as a universal serial bus (USB), Apple® Lightning®, Thunderbolt®, an Ethernet connector, a serial connector, an I / O connector, a local area network(s) (LAN(s)), other wired connection technologies discussed herein, or a combination thereof. In some examples, the XR link functions 728 and the XR link functions 736 can manage encoding, decoding, security (e.g., encryption and / or decryption), compression, decompression, wireless protocol management, transmission error correction, or a combination thereof.

[0079]

[0096] The XR management system 704 includes an XR application 746, which can receive user input(s) 748 via an input interface (e.g., a touchscreen, trackpad, button(s), controller(s), keypad(s), knobs, switches, or combinations thereof) of the XR system 700 (e.g., of the XR management system 704, of an input device coupled to the XR management system 704, of the XR interface device 702, of an input device coupled to the XR interface device 702, or a combination thereof). The input(s) 748 can be passed to an XR runtime application programming interface (API) 740, an XR scene API 750, an XR scene manager 742, a media access function (MAF) API 754, a MAF subsystem 760, and / or a cellular network subsystem 776, which can be coupled to a cellular network. In some examples, the XR application 746 is a video game. In some examples, the XR runtime API 740 may be an OpenXR application programming interface (API).

[0080]

[0097] The XR system 700 is configured for split rendering, such that at least a subset of processing tasks will be performed by the XR management system 704 and / or an external XR processing device (e.g., edge node, remote server) that is coupled to the XR system 700 via and / or is part of (e.g., edge node) the cellular network subsystem 776 of the XR management system 704. For example, sensor data and / or associated metadata captured by the XR interface device 702 (e.g., as discussed above) is sent from the XR interface device 702 to the XR management system 704, e.g., using the XR runtime API 740. The XR runtime API 740 also receives user input(s) 748 and / or associated metadata. These inputs to the XR runtime API 740 are collected and / or combined and sent as media and / or sensor data 752 to the media access function subsystem 760 of the XR management system 704. The Media Access Function subsystem 760 encodes, compresses, and / or encrypts the media and / or sensor data 752 to generate uplink media 772, which is sent through the Cellular Network Subsystem 776 to an external XR processing device for further processing. The XR processing device decrypts, decompresses, decodes, and / or processes the uplink media 772 to generate XR content. For example, in some examples, the XR processing device adds virtual content to generate the XR content. The XR processing device encodes, compresses, and / or encrypts the resulting XR content, which is received via the Cellular Network Subsystem 776 as downlink compressed media 774. The MAF subsystem 760 decrypts, decompresses, and / or decodes the downlink compressed media 774 to extract primitive buffers 758 (e.g., XR content), scene descriptions 756, or a combination thereof.

[0081]

[0098] The primitive buffer 758, scene description 756, and / or user input(s) 748 are passed to the XR scene manager 742, the XR scene API 750, and / or the XR runtime API 740, which are passed to the XR runtime subsystem 712. The XR scene API 750, the XR scene manager 742, the XR runtime API 740, and / or the XR runtime subsystem 712 determine what image(s) the visual composition subsystem 714 of the XR runtime subsystem 712 outputs to the eye buffer display 720 to be displayed to the user.

[0082]

[0099] In some examples, split rendering across a link may have limitations and / or technical issues, such as limited formats, limited bandwidth, protocol inconsistencies, security inconsistencies, transcoding latency, transcoding power usage, and other issues discussed herein. Some of these technical issues may be related to insufficient communication between the XR interface device 702, the XR management system 704, and / or the XR processing device, which can be resolved through communication of important contextual information about the XR interface device 702 (and / or the tethering communication interface 732) (e.g., as processing instructions) from the XR management system 704 to the XR processing device, as discussed herein (e.g., as in FIGS. 12, 13, and / or 14). Some of these technical issues may be related to transcoding in the XR management system 704, for example, which can be resolved by avoiding or minimizing transcoding, as discussed herein (e.g., as in FIGS. 10, 12, 13, and / or 14).

[0083]

[0100] FIG. 8 is a block diagram illustrating a split rendering process 800 in the context of the architecture of the extended reality (XR) system 700 of FIG. 7. In the split rendering process 800, the XR runtime includes and / or performs split rendering operations 810, and the XR runtime subsystem 712 communicates with the XR scene manager and / or XR applications via the XR runtime API 740 in a raw format 820. The raw format 820 may refer to a format that is not compressed, encoded, and / or encrypted. In some examples, sensor data from the XR interface device 702 may be sent to the XR management system 704 (e.g., via the tethering communication interface 732) using the raw format 820. In some examples, processed XR content generated in the XR processing device and / or the XR management system 704 may be sent from the XR management system 704 to the XR interface device 702 (e.g., via the tethering communication interface 732) using the raw format 820.

[0084]

[0101] In the split rendering process 800, the swap chain image is provided to the XR interface device 702 by the XR scene manager 742 of the XR management system 704. In some examples, action and / or pose information is provided to the XR application 746 via user input 748, sensor data from the XR interface device 702, and / or other input to the XR application 746 (e.g., from other subsystems of the XR management system 704).

[0085]

[0102] Figure 9 is a block diagram illustrating a split rendering process 900, including encoding and decoding, in the context of the architecture of the extended reality (XR) system 700 of Figure 7. The split rendering process 900 is an example of the split rendering process 800, with additional detail shown around the tethering communication interface 732. In the split rendering process 900, the XR interface device 702 and the XR management system 704 each encode (and / or compress and / or encrypt) data before sending it and decode (and / or decompress and / or decrypt) the data received, so that the raw format 820 can be used at either end (e.g., before encoding and / or compression and / or encryption and after decoding and / or decompression and / or decryption).

[0086]

[0103] For example, going from the XR interface device 702 to the XR management system 704, the action and / or pose encoder 905 of the XR interface device 702 encodes, compresses, and / or encrypts the action and / or pose data (e.g., possibly including or based on sensor data such as images and / or audio) from the XR interface device 702 based on, for example, the bit rate 925, protocol 930, and / or security 935 associated with the tethering communication interface 732. The XR management system 704 receives this encoded and / or compressed and / or encrypted data and uses the action and / or pose decoder 940 to decrypt, decode, and / or decompress the data.

[0087]

[0104] Once passing from the XR management system 704 to the XR interface device 702, the XR management system 704 processes the XR content generated by the XR management system 704 and / or the XR processing device, using an encoder 945 to encode, compress, and / or encrypt the XR content to conform to the bitrate 925, protocol 930, and / or security 935 associated with the tethering communication interface 732. The XR interface device 702 includes a decoder 910 that decrypts, decodes, and / or decompresses the XR content. The format 950 on the XR management system 704 and the projection format 915 used on the eye buffer display 720 of the XR interface device 702 may not match by default in some cases, which may cause problems and / or require the XR management system 704 and / or XR interface device 702 to modify the XR content to get it from the format 950 to the projection format 915. In some examples, the XR management system 704 and the XR interface device 702 communicate before transferring XR content (or any other data) to ensure that the security 955 (e.g., encryption scheme) used by the XR management system 704, the security 920 (e.g., encryption scheme) used by the XR interface device 702, and the security 935 (e.g., encryption scheme) used or available to the tethering communication interface 732 are consistent.

[0088]

[0105] Finally, the encoding, compression, encryption, decryption, decompression, and / or decryption operations in both the XR interface device 702 and the XR management system 704 under the split rendering process 900 may be numerous and cause latency, power usage, battery drain, heat generation, and increased use of computational resources in both the XR interface device 702 and the XR management system 704.

[0089]

[0106] 10 is a block diagram illustrating a process 1000 for using a pass-through encoding format 1015 for remote rendering in the context of the architecture of the extended reality (XR) system of FIG. 7. Under process 1000, the XR management system 704 receives XR content from the XR processing device through the cellular network subsystem 776 in a format that has already been encoded, compressed, and / or encrypted. Thus, the XR management system 704 can pass the XR content through the XR management system 704 to the XR interface device 702 without decrypting, decompressing, and / or decrypting the XR content after receipt at the XR management system 704, and without re-encoding, compressing, and / or encrypting the XR content before transmission of the XR content from the XR management system 704 to the XR interface device 702. Thus, the XR content maintains its encoded (and / or compressed and / or encrypted) format throughout the XR management system 704, and thus the XR content is in pass-through encoding format 1015 under process 1000. The raw format 820 of split rendering process 800 and split rendering process 900 is replaced with an encoded format 1020 (e.g., compressed and / or encrypted). Rather than having to use multiple encoders and decoders at both ends of the tethering communication interface 732 under split rendering process 900, in process 1000 the XR interface device 702 instead receives the pass-through encoded format 1015 and uses a decoder 1025 to decode, decompress, and / or decrypt the XR content encoded therein. The projected format 1030 and security 1035 (e.g., encryption method) remain the primary priority from a format and security perspective; the format and security using any format or security method on the XR management system 704 are deprioritized and no longer require matching the projected format 1030 and security 1035 of the XR interface device 702.In some examples, under process 1000, the XR runtime subsystem 712 communicates in a compressed format (e.g., in pass-through encoding format 1015 and / or encoding format 1020) via the XR runtime API 740, the XR scene manager 742, and / or the XR application 746.

[0090]

[0107] Compared to the split rendering process 900, the process 1000 involves fewer operations (e.g., fewer encoding and / or decoding), reduced complexity (e.g., fewer potential points of failure), and may result in reduced latency, power usage, battery drain, heat generation, and use of computational resources in both the XR interface device 702 and the XR management system 704.

[0091]

[0108] FIG. 11 is a block diagram illustrating an example architecture of an extended reality (XR) system 1100 spanning an extended reality (XR) interface device 1102, an XR management system 1104, and an XR processing device 1106. In an illustrative example, the XR interface device 1102 is a headset (e.g., a head-mounted display (HMD) 310, glasses, eyewear, wearable device(s), or a combination thereof), a mobile device (e.g., a mobile handset 410, a tablet, a laptop, a wearable device), or a combination thereof. The XR management system 1104 couples to the XR interface device 1102 and, in some cases, can couple to the XR processing device through a cellular network subsystem 1176. In some examples, the XR management system 1104 is a mobile device (e.g., a mobile handset, a tablet, a laptop, a wearable device). In some examples, the XR management system 1104 is locally coupled to the XR interface device 1102 (e.g., wirelessly or using wire(s)). In some examples, the XR processing device 1106 is a remote computing system (e.g., edge node, remote server) that is coupled to the XR management system via a network communication interface, such as a cellular network (e.g., 5G) interface. In some examples, the XR interface device may lack a connection to the internet, but the XR management system has a connection to the internet (e.g., via network system 278).

[0092]

[0109] In the XR system 1100 of FIG. 11 , the XR processing device 1106 includes a media access function subsystem 1196, an instance (or backend) of an XR application 1146, a scene manager 1184, a presentation engine 1182, and an XR shim layer 1198. The XR management system 1104 communicates with the XR processing device 1106 via a cellular subsystem 1176 and / or a cellular network 1178. The XR management system 1104 includes a media access function subsystem 1160, an uplink media management subsystem 1144, an XR application that receives user input 1148 (e.g., via an input interface), a thin presentation engine 1142, an XR runtime API 1140, and / or a cellular subsystem 1176 (e.g., 5G). The phone and the AR glasses device are coupled via an XR runtime tethering communication interface 1132. The architecture of the XR system 1100 shown in the block diagram of FIG. 11 is sometimes referred to as the SmarTAR architecture.

[0093]

[0110] The XR interface device 1102 of the XR system 1100 includes an XR runtime subsystem 1112 that interfaces with input device(s) and / or output device(s). For example, the XR runtime subsystem 1112 receives sensor data from sensor(s) 1108, such as camera(s) 1110, microphone(s) 1126, and / or other sensor(s). The camera(s) 1110 can include image sensor(s) (e.g., image sensor(s) 130) that capture image(s) and / or video(s) of a scene (or environment) in front of and / or around a user of the XR system 1100, the user of the XR system 1100, or a combination thereof. Similarly, the microphone(s) 1126 can capture audio of the scene, audio of the user, or a combination thereof. In some examples, the other sensor(s) (of sensor(s) 1108) can include any of the sensors discussed with respect to sensor(s) 208. In some examples, XR runtime subsystem 1112 includes a visual synthesis subsystem 1114 that controls the output of visual XR content to eye buffer display 1120, a haptic subsystem 1115 that controls the output of haptic feedback XR content to one or more haptic actuators 1122, and / or an audio synthesis subsystem 1116 that controls the reception of audio from microphone(s) 1126 and / or the output of audio XR content through speaker(s) 1124.

[0094]

[0111] In some examples, the XR runtime subsystem 1112 of the XR system 1100 performs various runtime functions such as those discussed with respect to the XR runtime subsystem 212 and / or the XR runtime subsystem 712, such as pose tracking, eye tracking, hand tracking, body tracking, feature tracking, object tracking, face tracking, SLAM, or a combination thereof.

[0095]

[0112] The XR interface device 1102 is coupled to the XR management system 1104 through a tethering communication interface 1132, similar to the coupling of the XR interface device 702 to the XR management system 704 through the tethering communication interface 732. The tethering communication interface 1132 can be a local coupling or interface, which refers to, for example, a wired coupling or connection, a short-range wireless coupling or connection, or a combination thereof. On one end of the tethering communication interface 1132 is the communication interface 1130 of the XR interface device 1102, which is managed using the XR link function 1128 of the XR interface device 1102. On the other end of the tethering communication interface 1132 is the communication interface 1134 of the XR management system 1104, which is managed using the XR link function 1136 of the XR management system 1104. The tethering communication interface 1132 may include, for example, a wireless communication interface, a wired communication interface, or a combination thereof, as discussed with respect to the tethering communication interface 732 and associated components and examples.

[0096]

[0113] The XR management system 1104 includes an XR application 1146, which can receive user input(s) 1148 via an input interface (e.g., a touchscreen, a trackpad, a button(s), a controller(s), a keypad(s), a knob, a switch, or a combination thereof) of the XR system 1100 (e.g., of the XR management system 1104, of an input device coupled to the XR management system 1104, of the XR interface device 1102, of an input device coupled to the XR interface device 1102, or a combination thereof). The input(s) 1148 can be passed to an XR runtime application programming interface (API) 1140, an uplink media management subsystem 1144, an XR presentation API 1150, a thin presentation engine 1142, a media access function (MAF) API 1154, a MAF subsystem 1160, and / or a cellular network subsystem 1176, which can be coupled to a cellular network 1178. In some examples, the XR application 1146 is a video game. In some examples, the XR runtime API 1140 can be an OpenXR API.

[0097]

[0114] The XR system 1100 is configured for split rendering, such that at least a subset of processing tasks will be performed by the XR management system 1104 and / or the XR processing device 1106 (e.g., edge node, remote server), which is coupled to the XR system 1100 via a cellular network subsystem 1176 and / or a cellular network 1178 of the XR management system 1104 and / or is part of the cellular network 1178 (e.g., edge node). For example, sensor data and / or associated metadata captured by the XR interface device 1102 (e.g., as discussed above) is sent from the XR interface device 1102 to the XR management system 1104, e.g., using the XR runtime API 1140. The XR runtime API 1140 also receives user input(s) 1148 and / or associated metadata. These inputs to the XR runtime API 1140 are collected and / or combined (e.g., via the uplink media management subsystem 1144) and sent as XR metadata 1152 (and / or sensor data) to the media access functions subsystem 1160 of the XR management system 1104. The media access functions subsystem 1160 encodes, compresses, and / or encrypts the XR metadata 1152 (and / or sensor data), for example, via codecs (e.g., the metadata codec 1162, the haptic codec 1164, the video codec 1166, and / or the audio codec 1168), to generate uplink compressed media 1172 that is sent to the XR processing device 1106 through the cellular network subsystem 1176 and / or the cellular network 1178 for further processing.

[0098]

[0115] The XR processing device 1106 decrypts, decompresses, decodes, and / or processes the uplink compressed media 1172 via a media access function subsystem 1196 of the XR processing device 1106 (e.g., via a codec such as a codec of the media access function subsystem 1160) to extract XR metadata 1152 (and / or sensor data). The XR processing device 1106 may include an instance of an XR application 1146, which may also provide input to a media access function API 1155 of the media access function subsystem 1196 of the XR processing device 1106. Similarly, the media access function subsystem 1196 of the XR processing device 1106 may provide input to the XR application 1146 via the media access function API 1155. The XR processing device 1106 provides XR metadata 1152 (and / or sensor data) from a media access functionality subsystem 1196 to a set of subsystems that ultimately render XR content (e.g., virtual content), which is returned from the set of subsystems to the media access functionality subsystem 1196 as pre-rendered media 1158 (e.g., 2D media, 2.5D media, and possibly posture-dependent media that depends on the posture of the XR interface device 1102). The set of subsystems includes an XR shim layer 1198, an XR runtime API 1180 (e.g., which may be associated with the XR runtime API 1140), a presentation engine 1182 with a corresponding XR presentation API 1186 that interfaces with the XR application 1146, a scene manager 1184 with a corresponding XR scene API 1188 that interfaces with the XR application 1146, and / or the XR application 1146 itself. The presentation engine 1182 includes a visual renderer 1190 that renders visual content, such as 3D models, textures, 2D images, sets of images (e.g., 2.5D visual content), videos, or combinations thereof, which may be part of the pre-rendered media 1158.The presentation engine 1182 includes a haptic renderer 1192 that renders haptic content, e.g., indicating haptic feedback to be output using haptic actuator(s) 1122 of the XR interface device 1102 and / or haptic actuator(s) of associated device(s) (e.g., a controller, the XR management system 1104, or another device). This haptic feedback may also be part of the pre-rendered media 1158. The presentation engine 1182 includes an audio renderer 1194 that renders audio content, e.g., music, voice, sound effects, and / or other sound clips or audio cues based on the real environment, the virtual environment, the augmented environment, the composite environment, and / or the pose of the XR interface device 1102. The audio content may also be part of the pre-rendered media 1158.

[0099]

[0116] The media access function subsystem 1196 encodes, compresses, and / or encrypts the pre-rendered media 1158, for example, via a codec (e.g., a metadata codec, a haptic codec, a video codec, and / or an audio codec) to generate downlink compressed media 1174, which is sent through a cellular network 1178 and / or a cellular network subsystem 1176 to the XR management system 1104 and back towards the XR interface device 1102. In some examples, the XR processing device 1106 also includes its own cellular network subsystem similar to the cellular network subsystem 1176 of the XR management system 1104. In some examples, the media access function subsystem 1160 decodes, decompresses, and / or decrypts the downlink compressed media 1174, for example, via codecs (e.g., metadata codec 1162, haptic codec 1164, video codec 1166, and / or audio codec 1168), to extract pre-rendered media 1158 and / or scene description 1156, which it passes to a thin presentation engine 1142 (e.g., managing presentation and / or compositing), a corresponding XR presentation API 1150, and / or an XR runtime API 1140. The pre-rendered media 1158 and / or scene description 1156 can be returned to the XR interface device 1102 through a tethering communication interface 1132, where components of the pre-rendered media 1158 may be output. For example, visual XR content from pre-rendered media 1158 is output using visual synthesis subsystem 1114 and eye buffer display 1120, haptic feedback XR content from pre-rendered media 1158 is output using haptic subsystem 1116 and haptic actuator(s) 1122, and audio XR content from pre-rendered media 1158 is output using audio synthesis subsystem 1118 and speaker(s) 1124.

[0100]

[0117] The split rendering process of the XR system 1100 shown in FIG. 11 can add another connection, this time between the XR management system 1104 and the XR processing device 1106, which can include additional stages of encoding, compression, encryption, decryption, decompression, decryption, and / or transcoding. In some examples, this reflects and adds to the technical issues of the split rendering process 900, including, for example, limited formats, limited bandwidth, protocol inconsistencies, security inconsistencies, transcoding latency, transcoding power usage, and other issues discussed herein. Some of these technical issues may be related to insufficient communication between the XR interface device 1102, the XR management system 1104, and / or the XR processing device 1106, which can be resolved through communication of important contextual information about the XR interface device 1102 (and / or the tethering communication interface 1132) (e.g., as processing instructions) from the XR management system 1104 to the XR processing device, as discussed herein (e.g., as in FIGS. 12, 13, and / or 14). Some of these technical issues may be related to, for example, transcoding in the XR management system 1104, which can be resolved by avoiding or minimizing transcoding as discussed herein (e.g., as in Figures 10, 12, 13, and / or 14).

[0101]

[0118] Figure 12 is a block diagram illustrating a process 1200 for using pass-through encoding formats and / or protocols (e.g., pass-through encoding format and / or protocol 1210, pass-through encoding format and / or protocol 1215) for remote rendering in the context of the architecture of the extended reality (XR) system of Figure 11. Under process 1200, the XR interface device 1102 uses an encoder 1205 to encode, compress, and / or decrypt its sensor data and / or metadata into the pass-through encoding format and / or protocol 1210, which the XR management system 1104 receives and passes to the XR processing device 1106 without decoding, decompressing, or decrypting.

[0102]

[0119] In some examples, the XR management system 1104 separately obtains metadata regarding the capabilities of the XR interface device 1102 and / or the tethering communication interface 1132 (and / or the XR processing device 1106) and generates processing instructions 1240 for the XR processing device 1106 along with instructions for the XR processing device 1106 on how to process the XR interface device 1102. In some examples, the processing instructions 1240 can identify or be based on characteristic(s) of the XR interface device 1102, such as image capture resolution, image capture size, image capture color space, image display resolution, image display size, image display color space, projection format 1230, attitude, location, orientation, bandwidth, bit rate, baud rate, transfer speed, transfer protocol, encryption scheme, security scheme (e.g., security 1235), temperature, battery level (or battery life), or a combination thereof. In some examples, the processing instructions 1240 may identify or be based on characteristic(s) of the tethering communication interface 1132 (e.g., a local communication interface), such as a connection type (e.g., wireless or wired), bandwidth, bit rate, baud rate, transfer speed, transfer protocol, encryption scheme, security scheme, connection quality, or a combination thereof. In some examples, the processing instructions 1240 may identify or be based on characteristic(s), capabilities, capacity, processing power, bandwidth, bit rate, baud rate, transfer speed, transfer protocol, encryption scheme, or security scheme of the XR processing device 1106.

[0103]

[0120] In an illustrative example, if the display(s) of the XR interface device 1102 (e.g., the eye buffer display 1120) outputs at 1080P resolution, the XR management system 1104 can generate processing instructions 1240 to the XR processing device 1106 indicating not to render content at a resolution higher than 1080P (e.g., not to render content at 2K or 4K resolution) and can send those processing instructions 1240 to the XR processing device 1106 to follow in order to avoid the XR processing device 1106 using unnecessary computational resources, bandwidth, power, heat generation, latency, etc. Similarly, if the tethering communication interface 1132 has a maximum bitrate of 30 megabits / second, the XR management system 1104 can generate processing instructions 1240 to the XR processing device 1106 indicating not to render content that would require a bitrate higher than 30 megabits / second (e.g., 100 megabits / second) to be transferred in a timely manner to be output by the XR interface device 1102 with a latency exceeding a threshold amount, and can send those processing instructions 1240 to the XR processing device 1106 to follow in order to avoid the XR processing device 1106 using unnecessary computational resources, bandwidth, power, heat generation, etc., and failing to send rendered XR content to the XR processing device 1106 in a timely manner. Similarly, if the tethering communication interface 1132 has poor connection quality (e.g., of a wireless connection), the XR management system 1104 can generate processing instructions 1240 to the XR processing device 1106 requesting that the XR processing device 1106 render abbreviated content that will use less than a threshold bit rate, and can send those processing instructions 1240 to the XR processing device 1106 to follow in order to improve the probability that the entire XR content arrives at the XR interface device 1102, and to make additional room (e.g., in terms of bandwidth and / or bit rate) for error correction data to compensate for the poor connection quality.If the XR interface device 1102 has a low battery (e.g., below a threshold) or is at a high temperature (e.g., above a threshold), similar processing instructions 1240 may be generated and sent to attempt to conserve battery power in the XR interface device 1102 and / or to attempt to reduce or maintain the temperature of the XR interface device 1102 (or at least slow down the temperature rise of the XR interface device 1102).

[0104]

[0121] Under process 1200, the XR processing device 1106 uses the media access function subsystem 1196 (e.g., decoders and / or codecs therein) to decode, decompress, and / or decrypt the pass-through encoding format and / or protocol 1210. The XR processing device 1106 may also receive processing instructions 1240 from the XR management system 1104. The processing device 1106 may process the sensor data and / or metadata decoded from the pass-through encoding format and / or protocol 1210 in accordance with the processing instructions 1240 to generate XR content (e.g., pre-rendered media 1158). The XR processing device 1106 uses the media access function subsystem 1196 (e.g., encoders and / or codecs therein) to encode, compress, and / or encrypt the XR content (e.g., pre-rendered media 1158) to generate a pass-through encoding format and / or protocol 1215, which the XR management system 1104 receives and passes to the XR interface device 1102 without decoding, decompressing, or decrypting. In some examples, the processing instructions 1240 may indicate details of data reception at the XR interface device 1102, such as protocol 1220, decoder 1225, projection format 1230, and / or security 1235, such that the XR processing device 1106 generates the pass-through encoding format and / or protocol 1215 using a format and / or protocol and / or security scheme that is consistent with a format and / or protocol and / or security scheme usable by the XR interface device 1102.

[0105]

[0122] In some examples, the XR management system 1104 can generate layer content 1245 and send this layer content 1245 to the XR interface device 1102. The XR management system 1104 has limited ability to directly modify the XR content without decoding, decompressing, and / or decrypting the pass-through encoding format and / or protocol 1215. However, the XR management system 1104 can generate layer content 1245 that can be overlaid above or below at least a portion(s) of the XR content according to a layering arrangement, and send the layer content 1245 to the XR interface device for output without decrypting, decompressing, and / or decrypting the pass-through encoding format and / or protocol 1215. In some illustrative examples, the layer content may include content from the XR application 1146, such as a counter indicating the time remaining in a game match (e.g., of a video game or sports match), a counter indicating wins or losses or score(s) or other statistics by a player(s) in a game (e.g., of a video game or sports match), a message(s) received by the user from other users (e.g., other players), a message(s) sent by the user to other users (e.g., other players), user interface elements, elements that may be rendered by the XR management system 1104 separately from the XR content rendered by the XR processing device 1106, or combinations thereof.

[0106]

[0123] 11 , process 1200 involves fewer operations (e.g., fewer encoding and / or decoding), reduced complexity (e.g., fewer potential points of failure), and may result in reduced latency, power usage, battery drain, heat generation, and use of computational resources in the XR interface device 1102, the XR management system 1104, and the XR processing device 1106. In some examples, the architecture shown in FIG. 12 (e.g., the XR system 1100 using process 1200) may be referred to as a SmarTAR architecture.

[0107]

[0124] In some examples, under process 1200, the XR system 1100 may perform a capability exchange, for example, via processing instructions 1240. In some examples, the XR management system 1104 uses at least one capability query to check with the XR interface device 1102 about what is supported by the XR interface device 1102. Examples of such checks include, for example, format resolution, quality degradation incurred by the link and / or coding, the security framework and capabilities of the XR interface device 1102, dynamic metrics and / or information of the signal quality and / or signal strength of the tethering communication interface 1132, audio and / or video and / or haptic feedback decoding capabilities of the XR interface device 1102, security capabilities of the XR interface device 1102, static and dynamic updates to the bitrate and / or latency of the tethering communication interface 1132, or combinations thereof. In some examples, based on this information, the XR management system 1104 can decide to use process 1200 of FIG. 12 or the process shown in FIG. 11, to use a compressed or uncompressed format, to adapt the bitrate and / or resolution associated with the rendered XR content, or a combination thereof.

[0108]

[0125] Process 1200 may be used for various communication services between the XR management system 1104 and the XR processing device 1106, such as: (1) in the case of a media streaming client application, dynamically selecting an appropriate content format, bit rate, and quality based on dynamic connection metrics and / or statistics, (2) dynamically negotiating between the XR interface device 1102, the XR management system 1104, and / or the XR processing device 1106 to dynamically support an appropriate content format, bit rate, and quality for communication, (3) performing split rendering, or (4) a combination thereof.

[0109]

[0126] Some advantages of process 1200 and using pass-through compression formats and protocols between the XR interface device 1102 and the XR processing device 1106 (through the XR management system 1104) include lower latency (no transcoding required), improved (e.g., maximized) quality (no transcoding), lower power consumption, improved reliability (reliability is addressed end-to-end), improved bitrate (bitrate is addressed end-to-end), improved security (security is addressed end-to-end), other improvements discussed herein, or combinations thereof. Even with the use of pass-through compression formats and protocols between the XR interface device 1102 and the XR processing device 1106 (through the XR management system 1104) under process 1200, local data can still be added by the XR management system 1104 (e.g., as processing instructions 1240 and layer content 1245).

[0110]

[0127] Certain aspects of the use of pass-through compression formats and protocols between the XR interface device 1102 and the XR processing device 1106 (through the XR management system 1104) can be handled at the API level. In some examples, an XR API (e.g., the OpenXR API or extension) can be used with the architectures shown in the block diagrams of FIGS. 11 and 12, with or without the use of pass-through compression formats and protocols of process 1200 between the XR interface device 1102 and the XR processing device 1106 (through the XR management system 1104). In some examples, one or more of the devices (XR interface device 1102, the device, the XR management system 1104, and / or the XR processing device 1106) can communicate according to a predetermined schedule and / or a modifiable schedule to handle the timing of the communications (e.g., of the pass-through compression formats and protocols).

[0111]

[0128] In some examples, the architecture shown in XR system 1100 and / or process 1200 may include specific features or extensions. For example, in some examples, split rendering operations may be defined by the endpoint capabilities of XR interface device 1102. For a portion of a submitted swap chain buffer, one layer may be sent in compressed form and a locally generated layer may be sent in raw form (e.g., layer content 1245).

[0112]

[0129] In some examples, capabilities and configurations can be checked statically (e.g., available projection formats, security, decoders, and / or protocols). In some examples, capabilities and configurations can be checked dynamically (e.g., bitrate measurements). In some examples, metrics and observations are also performed across pass-through compression formats and protocols (e.g., audio, video, and / or attitude).

[0113]

[0130] In some examples, the feature or extension may include support for runtime queries for supported format additional information (including, statically and dynamically, the audio and video decoding capabilities of the XR interface device 1102, the security capabilities of the XR interface device 1102, the security framework and capabilities of the XR interface device 1102, and / or the bitrate and latency of the tethering communication interface 1132). The feature or extension may include support for using the provided static and dynamic information, for example, for the communication client to negotiate with the XR processing device 1106 to support appropriate content formats, codecs, bitrates, and qualities, possibly in a dynamic manner, in order to select appropriate content formats, bitrates, codecs, and qualities in the XR management system 1104, possibly in a dynamic manner, and / or in split rendering where formats are provided, as in process 1200. The feature or extension may include advertising appropriate information from the XR processing device 1106 to the XR management system 1104 (or vice versa) so that such selections can be made. In some examples, details of these features or extensions may be added to the streaming manifest, the session description protocol, or the scene description.

[0114]

[0131] In some examples, media formats are passed from the cellular network 1178 over the tethering communication interface 1132 using the XR runtime API 1140 (e.g., the OpenXR API or another XR API). Use of this API allows the XR management system 1104 to also add local data as layer content 1245, which may be added, for example, on a different layer than the XR content generated by the XR processing device 1106. Passing through may include passing encoded and / or compressed media data, but may also include and / or formatted according to a selected security framework and / or protocol, which may depend on the capabilities and / or settings of the XR interface device 1102.

[0115]

[0132] Figure 13 is a block diagram illustrating an example architecture of an extended reality (XR) system 1300 that implements XR management to pass compressed formats. The XR system 1300 is illustrated in Figure 13 as using an approach in which compressed formats are exchanged via an XR runtime API 1320 (e.g., the XR runtime API 1140). The XR system of Figure 13 passes pose and other data 1330 in compressed format (e.g., data using a pass-through encoding format and / or protocol 1210) from tracking system(s) 1305 (e.g., of the XR runtime subsystem 1112) and corresponding encoder 1310 (e.g., of the XR runtime subsystem 1112 and / or the XR link functionality 1128) through a link 1315 (e.g., the tethering communication interface 1132) and the XR runtime API 1320 to a scene manager and game engine 1335 (e.g., the scene manager 1184, the presentation engine 1182, and / or the XR application 1146). 13 passes scene / game / virtual content (e.g., pre-rendered media 1158) from a scene manager and game engine and corresponding encoder 1340 in a pass-through compression format and protocol 1345 (e.g., pass-through encoding format and / or protocol 1215) through a link 1350 (e.g., a tethering communication interface 1132) and an XR runtime API 1320 to a decoder 1355 (e.g., of the XR runtime subsystem 1112 and / or the XR link functionality 1128) and subsystems 1360 associated with content compositing and warping (e.g., the XR runtime subsystem 1112, the visual synthesis subsystem 1114, the haptic subsystem 1116, and / or the audio synthesis subsystem 1116). An application (e.g., an AR, MR, VR, and / or XR application) (e.g., the XR application 1146) can be associated with the XR runtime API 1320.

[0116]

[0133] In some examples, the XR system 1300 may include and / or use various features or extensions, such as extensions to swapchain image management associated with OpenXR. For example, in some examples, xrEnumerateSwapchainFormats lists the texture formats supported by the current session. The type of format returned depends on the graphics API specified in xrCreateSession.

[0117]

[0134] In some examples, a particular graphics API, such as Vulkan, allows for compressed image formats. In some examples, the compressed image format used may depend on a compressed data format that conforms to a particular format, protocol, and / or standard, such as those associated with Khronos.

[0118]

[0135] In some examples, the XR runtime API 1320 references the use of a video compression format for each of the swap chain images. In some examples, the XR systems and methods of Figures 10 and 12-14 enable formats that add compressed video formats as swap chain images where a timestamp (e.g., an RTP timestamp) is included to indicate a target display time, where the compressed format includes a rendering pose, and / or where compressed texture formats are passed.

[0119]

[0136] In some examples, swap chain image management using compressed data is applied to only a portion of the submitted swap chain buffer. For example, in an illustrative example, one layer may be sent in compressed form (e.g., data using pass-through encoding format and / or protocol 1215), while a locally generated layer may be sent in raw form (e.g., layer content 1245). Synchronization of these layers can be performed by the XR management system 1104, for example, at runtime. This can apply to video, audio, haptics, and other output types, with compressed bitstreams passed by the XR management system 1104 from the XR processing device 1106 to the XR interface device 1102.

[0120]

[0137] In the case of uplink media data from the XR interface device 1102, action and / or posture data may be passed to the XR processing device 1106 in compressed form as received at the XR interface device 1102. The XR runtime API may even provide such data in compressed form (for sending to the XR processing device 1106) and in raw form (for processing on the XR management system 1104).

[0121]

[0138] To properly support tethered tethering communication interface 1132 and XR interface device 1102-based endpoints, the XR system framework can provide adaptability to the capabilities of the XR interface device 1102 to maximize end-to-end quality in terms of signal quality, latency, power consumption, etc. To support this, the XR system can support additional information in runtime queries for supported formats, including, for example, the format's resolution, quality degradation incurred by the combination of tethering communication interface 1132 and coding, the device's security framework and capabilities, dynamically providing metrics and information on signal quality over the tethering communication interface 1132, the audio and video decoding capabilities of the XR interface device 1102, the security capabilities of the XR interface device 1102, the device's security framework and capabilities, and statically and dynamically the bit rate and latency of the tethering communication interface 1132. The XR system can support the use of provided static and dynamic information, for example, in the XR management system 1104, to select, possibly in a dynamic manner, an appropriate content format, bit rate, and quality, and negotiate with the XR processing device 1106, possibly in a dynamic manner, to support an appropriate content format, bit rate, and quality with split rendering, where the format is provided from rendering by the XR processing device 1106. The XR system can advertise appropriate information from the XR interface device 1102 and / or the XR processing device 1106 to the XR management system 1104 to enable the XR management system 1104 to make such selections of format, protocol, security, etc.In some examples, the protocol may include a stage 3 protocol such as Dynamic Adaptive Streaming over HTTP (DASH), ISO base media file format (ISO BMFF), Session Description Protocol (SDP), Real-time Transport Protocol (RTP), Secure RTP (SRTP), RTP Control Protocol (RTCP), or a combination thereof.

[0122]

[0139] 14 is a flow diagram illustrating a process 1400 for extended reality (XR) management. Process 1400 may be performed by an XR management system. In some examples, the XR management system may include, for example, image capture and processing system 100, image capture device 105A, image processing device 105B, image processor 150, ISP 154, host processor 152, XR system 200, HMD 310, mobile handset 410, XR system 505, XR processing device 510, XR system 605, XR system 700, XR interface device 702, XR management system 704, XR system 1100, XR interface device 1102, XR management system 1104, XR processing device 1106, XR system 1300, computing system 1500, processor 1510, an apparatus, a system, a non-transitory computer-readable medium coupled to a processor, or a combination thereof.

[0123]

[0140] At operation 1405, the XR management system (or at least one component thereof) is configured to and capable of receiving sensor data from an XR interface device (e.g., XR interface device 702, XR interface device 1102) having at least one sensor.

[0124]

[0141] Examples of sensors include image sensor 130, first camera 330A, second camera 330B, third camera 330C, fourth camera 330D, first camera 430A, second camera 430B, third camera 430C, fourth camera 430D, input device 1545, another image sensor described herein, another sensor described herein, or a combination thereof. Examples of images include images captured by any of the example image sensors listed above.

[0125]

[0142] In some examples, the XR interface device is a headset (e.g., HMD 310, glasses) coupled to the XR management system using a local communication interface (e.g., tethering communication interface 732, tethering communication interface 1132, link 1315, link 1350), such as a local wireless communication interface, a wired communication interface, or a combination thereof. In some examples, the XR interface device is a mobile device (e.g., mobile handset 410, phone, tablet, watch, wearable device) coupled to the XR management system using a local communication interface, such as a local wireless communication interface, a wired communication interface, or a combination thereof. In some examples, the XR management system is a mobile device (e.g., mobile handset 410, phone, tablet, watch, wearable device) coupled to the XR interface device using a local communication interface, such as a local wireless communication interface, a wired communication interface, or a combination thereof. The local wireless communication interface may include, for example, Bluetooth® connection(s), personal area network (PAN) connection(s), Wi-Fi connection(s), wireless local area network (WLAN) connection(s), or combinations thereof.

[0126]

[0143] At operation 1410, the XR management system (or at least one component thereof) is configured to, and can generate, based on receipt of the sensor data from the XR interface device, processing instructions (e.g., processing instructions 1240) for an XR processing device (e.g., XR processing device 1106) to process the sensor data to generate XR content. At operation 1415, the XR management system (or at least one component thereof) is configured to, and can send the sensor data and the processing instructions to the XR processing device.

[0127]

[0144] In some examples, the sensor data received from the XR interface device and sent to the XR processing device includes (and / or is) compressed sensor data.

[0128]

[0145] In some examples, at least a portion of the sensor data is uncompressed when received from the XR interface device, and the XR management system (or at least one component thereof) is configured to and can compress at least a portion of the sensor data before sending the sensor data to the XR processing device. For example, the compression can be performed using a codec in the media access function 1160 of the XR management system 1104. In some examples, a second portion of the sensor data is compressed when received from the XR interface device, and the XR management system (or at least one component thereof) is configured to and can keep the second portion of the sensor data compressed (e.g., not decompress it) before sending the sensor data to the XR processing device.

[0129]

[0146] In some examples, generating the processing instructions based on receiving the sensor data from the XR interface device (as in act 1410) includes generating at least a subset of the processing instructions based on an uncompressed portion of the sensor data that was uncompressed when received from the XR interface device. In some examples, the compressed portion of the sensor data was compressed when received from the XR interface device. In some examples, for example, the image data (of the sensor data) was compressed when received from the XR interface device, but the metadata (of the sensor data) was uncompressed when received from the XR interface device. The XR management system (or at least one component thereof) is configured to and can generate the processing instructions based on the metadata, which may include sufficient information about the image (e.g., resolution, storage size, compression format, etc.) to enable the XR management system (or at least one component thereof) to generate processing instructions for processing the image without decompressing the image itself.

[0130]

[0147] In some examples, generating the processing instructions based on receiving the sensor data from the XR interface device (as in act 1410) includes generating at least a subset of the processing instructions based on at least one characteristic of a local communication interface between the apparatus and the XR interface device (e.g., tethering communication interface 732, tethering communication interface 1132, link 1315, link 1350, wired communication interface, local wireless communication interface) through which the sensor data is received from the XR interface device. In some examples, the at least one characteristic includes at least one of a bandwidth, a bit rate, a baud rate, a transfer speed, a transfer protocol, an encryption scheme, a connection quality, a security scheme, or a combination thereof. For example, the processing instructions may instruct the XR processing device not to generate an amount of data greater than the local communication interface is capable of transferring to the XR interface device.

[0131]

[0148] In some examples, generating the processing instructions based on receiving the sensor data from the XR interface device (as in act 1410) includes generating at least a subset of the processing instructions based on at least one characteristic of the sensor data received from the XR interface device, e.g., obtained via metadata associated with the sensor data. In some examples, the at least one characteristic includes at least one of a file size, an image size, an image resolution, a frame rate, an amount of channels (e.g., color channels, depth channels), an encoding scheme, an encryption scheme, or a combination thereof. For example, the processing instructions may instruct the XR processing device to render the virtual content at a size that is appropriate for the image size and / or image resolution of the sensor data, such that the XR processing device does not waste computational resources rendering the virtual content at a size larger than is available in the XR content given the image size and / or image resolution of the sensor data.

[0132]

[0149] In some examples, generating the processing instructions based on receiving sensor data from the XR interface device (as in Act 1410) includes generating at least a subset of the processing instructions based on at least one characteristic of the XR interface device. In some examples, the at least one characteristic includes at least one of an image capture resolution, an image capture size, an image capture color space, an image display resolution, an image display size, an image display color space, a posture, a location, an orientation, a bandwidth, a bit rate, a baud rate, a transfer speed, a transfer protocol, an encryption scheme, a security scheme, a temperature, a battery level, or a combination thereof. For example, the processing instructions may instruct the XR processing device to render the virtual content at a size that is appropriate for the image display resolution and / or the image display size of the display of the XR interface device, such that the XR processing device does not waste computational resources rendering the virtual content at a size larger than is available for the XR content given the image display resolution and / or the image display size of the display of the XR interface device. Similarly, the processing instructions may instruct the XR processing device to render virtual content using colors that work well on the XR interface device's display, given the image display color space of the XR interface device's display. Similarly, the processing instructions may instruct the XR processing device to adjust the viewpoint(s) of particular element(s) in the XR content (e.g., virtual content) based on posture, location, orientation, etc. Similarly, the processing instructions may instruct the XR processing device to limit how much XR content the XR processing device is generating (e.g., in terms of image resolution, image size, frame rate, or a combination thereof) based on bandwidth, bit rate, baud rate, transfer speed, transfer protocol, encryption scheme, security scheme, temperature, battery level, or a combination thereof.

[0133]

[0150] In some examples, generating the processing instructions based on receiving the sensor data from the XR interface device (as in Act 1410) includes generating at least a subset of the processing instructions based on at least one characteristic of the XR processing device. In some examples, the at least one characteristic includes at least one of capabilities, capacity, processing power, bandwidth, bit rate, baud rate, transfer speed, transfer protocol, encryption scheme, security scheme, or a combination thereof. For example, the processing instructions may be generated to instruct the XR processing device to generate XR content within the capabilities of the XR processing device. The XR processing instructions should take into account the capabilities of the XR processing device to ensure that it is not impossible or unlikely for the XR processing device to complete the XR processing instructions on time.

[0134]

[0151] In some examples, the processing instructions indicate at least a pose of virtual content to be rendered as part of the XR content. In some examples, the virtual content is associated with a video game. For example, the pose of the virtual content to be rendered as part of the XR content can be based on the pose of the XR interface device (e.g., whether the user is looking up or down), based on the video game (e.g., whether an object or character represented by the virtual content is tilted, rotated, or moved within the video game's environment), or a combination thereof.

[0135]

[0152] In some examples, the XR management system (or at least one component thereof) is configured to and capable of managing an application programming interface (API). In some examples, receiving sensor data from the XR interface device (of act 1405) is associated with a first API call to the API. In some examples, receiving XR content from the XR processing device (of act 1420) is associated with a second API call to the API.

[0136]

[0153] At operation 1420, the XR management system (or at least one component thereof) is configured to and can receive XR content from the XR processing device. The XR content can include, for example, downlink media 774, primitive buffers 758, scene descriptions 756, data using pass-through encoding formats and / or protocols 1015, scene descriptions 1156, pre-rendered media 1158, downlink compressed media 1174, data using pass-through encoding formats and / or protocols 1215, data using pass-through compression formats and / or protocols 1345, or combinations thereof.

[0137]

[0154] In some examples, the XR processing device includes an edge node, such as an edge node in a cellular network (e.g., a 5G cellular network). In some examples, the XR processing device includes a server.

[0138]

[0155] The XR content can be a processed transformation of the sensor data. For example, in some examples, the sensor data includes a representation of the environment around the XR interface device (e.g., around the user), and the XR content includes a processed transformation of the representation of the environment, e.g., adding rendered objects, textures, colors, shadows, lighting effects, or a combination thereof. In an illustrative example, the sensor data includes image(s) and / or depth data of a room in which a user of the XR interface device is located, and the XR content can visually modify the appearance of the room to appear like the interior of a spaceship, a combat arena, a pirate ship, an outdoor space, or some other type of environment with a particular theme and / or style and / or appearance, depending on the XR application 246 (e.g., depending on the video game), while preserving the aspects of the room (e.g., dimensions, location of walls, location of particular objects within the room). In some examples, objects in the real-world environment (e.g., as tracked in the sensor data) may be maintained in the XR content (in terms of position and / or orientation and / or dimensions) but may be treated to appear differently (e.g., to look like a different object), for example, a sofa may look like a set of crates or boxes to better match the theme and / or style and / or appearance of the XR application 246 (e.g., of a video game). In some examples, virtual objects and / or characters that do not exist in the real-world environment (e.g., as tracked in the sensor data) may be added to the XR content, such as other characters representing other people (e.g., other players in a multiplayer game), other characters that do not represent other people (e.g., non-player character(s) (NPC(s)), items, decorations, or combinations thereof.

[0139]

[0156] At operation 1425, the XR management system (or at least one component thereof) is configured to and can generate layer content (e.g., layer content 1245). At operation 1430, the XR management system (or at least one component thereof) is configured to and can send the XR content and layer content to the XR interface device to cause the XR interface device to output the XR content and layer content in a layered arrangement.

[0140]

[0157] In some examples, the layer content includes content that can be added (e.g., overlaid on top of or underlaid below the XR content) by the XR management system, e.g., on a layer separate from the XR content, without having to directly modify (e.g., therefore decompress and / or decode) the XR content in the XR management system. For example, in some examples, the layer content can include a clock, a timer, a frame rate counter (e.g., in frames per second (FPS)), a mission objective in a video game, a score in a video game, statistics associated with a player in a video game, a heads-up display associated with a player and / or the video game, statistics associated with a team in a video game, statistics associated with a match or tournament in a video game, an outline or overlay that highlights a particular element in the XR content in a particular color (e.g., a highlight or outline in red, orange, yellow, green, blue, purple, or another color), the name of a particular item or character, or a combination thereof.In an illustrative example, the sensor data includes image(s) and / or depth data of a real-world environment in which a user of the XR interface device is located, the XR content is a processed transformation of the sensor data that transforms the real-world environment into a stylized environment (e.g., themed to look like a spaceship or other fictional or non-fictional area), virtual content (e.g., virtual characters and / or items) is incorporated into the stylized environment while maintaining certain aspects (e.g., dimensions) of the real-world environment, and the layer content includes a heads-up display of the user's player character to be overlaid on the XR content, names of other characters in the XR content to be overlaid on the XR content, colored outlines or colored highlights highlighting goals (e.g., a ball or other object, an exit from a room, etc.) in the stylized environment to be overlaid on the XR content, video game scores to be overlaid on the XR content, or combinations thereof.

[0141]

[0158] In some examples, a portion of the layer content may be configured to be and may be overlaid on at least a portion of the XR content in a layered arrangement. For example, in some examples, the layer content may be overlaid in front of the XR content (or one or more layer(s) thereof) from the user's perspective (and / or from the perspective shown on the display(s) of the XR interface device). In some examples, at least a portion of the XR content may be configured to be and may be overlaid on at least a portion of the layer content in a layered arrangement. For example, in some examples, the XR content may include gaps or may itself include layers, and the layer content may be overlaid behind one or more layer(s) of the XR content from the user's perspective (and / or from the perspective shown on the display(s) of the XR interface device).

[0142]

[0159] In some examples, the XR interface device performs further processing on the XR content and / or layer content before the XR interface device displays the XR content and / or layer content on its display(s) (e.g., display(s) 340, display 440). For example, in some examples, the XR interface device may apply a filter to the XR content and / or layer content (e.g., as overlaid according to a layered arrangement). In some examples, the XR interface device may resize, rescale, warp, skew, rotate, reorient, translate, or modify the XR content and / or layer content (e.g., as overlaid according to a layered arrangement), e.g., to better fit the XR content and / or layer content to the dimensions, orientation, and / or angle of the display(s) relative to the viewpoint(s) of the user's eye(s).

[0143]

[0160] In some examples, the processes described herein (e.g., the processes of Figures 1, 2, 5, 6, 7, 8, 9, 10, 11, 12, 13, process 1400 of Figure 14, and / or other processes described herein) may be performed by a computing device or apparatus. In some examples, the processes described herein may be performed by image capture and processing system 100, image capture device 105A, image processing device 105B, image processor 150, ISP 154, host processor 152, XR system 200, HMD 310, mobile handset 410, XR system 505, XR processing device 510, XR system 605, XR system 700, XR interface device 702, XR management system 704, XR system 1100, XR interface device 1102, XR management system 1104, XR processing device 1106, XR system 1300, computing system 1500, processor 1510, an apparatus, a system, a non-transitory computer-readable medium coupled to a processor, or a combination thereof.

[0144]

[0161] The computing device may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a network-connected wristwatch or smartwatch, or other wearable device), a server computer, an autonomous vehicle or computing device for an autonomous vehicle, a robotic device, a television, and / or any other computing device having the resource capabilities to perform the processes described herein. In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other component(s) configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other component(s). The network interface may be configured to communicate and / or receive Internet Protocol (IP)-based data or other types of data.

[0145]

[0162] Components of a computing device may be implemented in circuit configurations. For example, components may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuitry), and / or may include and / or be implemented using computer software, firmware, or any combination thereof, to perform various operations described herein.

[0146]

[0163] The processes described herein are shown as logical flow diagrams, block diagrams, or conceptual diagrams, where the operations represent sequences of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the described operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and / or in parallel to implement a process.

[0147]

[0164] Additionally, the processes described herein may be executed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors, by hardware, or a combination thereof. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program that includes instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0148]

[0165] Figure 15 illustrates an example of a system for implementing certain aspects of the present technology. In particular, Figure 15 illustrates an example of a computing system 1500, which may be, for example, an internal computing system, a remote computing system, a camera, or any computing device comprising any components whose components communicate with each other using a connection 1505. The connection 1505 may be a physical connection using a bus or a direct connection to a processor 1510, such as in a chipset architecture. The connection 1505 may also be a virtual connection, a networked connection, or a logical connection.

[0149]

[0166] In some aspects, computing system 1500 is a distributed system in which the functionality described in this disclosure may be distributed across a data center, multiple data centers, a peer network, etc. In some aspects, one or more of the system components described represent many such components, each performing some or all of the functionality described. In some aspects, the components may be physical or virtual devices.

[0150]

[0167] The exemplary system 1500 includes at least one processing unit (CPU or processor) 1510 and connections 1505 that couple various system components to the processor 1510, including system memory 1515 such as read-only memory (ROM) 1520 and random access memory (RAM) 1525. The computing system 1500 may include a cache 1512 of high-speed memory that is directly connected to the processor 1510, connected in close proximity to the processor 1510, or integrated as part of the processor 1510.

[0151]

[0168] Processor 1510 can include any general-purpose processor, hardware or software services, such as services 1532, 1534, and 1536 stored in storage device 1530, configured to control processor 1510, and special-purpose processors where software instructions are incorporated into the actual processor design. Processor 1510 can essentially be a completely self-contained computing system incorporating multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors can be symmetric or asymmetric.

[0152]

[0169] To enable user interaction, computing system 1500 includes input devices 1545, which can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, speech, etc. Computing system 1500 can also include output devices 1535, which can be one or more of several output mechanisms. In some cases, a multimodal system can allow a user to provide multiple types of input / output for communicating with computing system 1500. Computing system 1500 can include a communication interface 1540, which can generally govern and manage user input and system output.The communications interface may be an audio jack / plug, a microphone jack / plug, a Universal Serial Bus (USB) port / plug, an Apple® Lightning® port / plug, an Ethernet port / plug, an optical fiber port / plug, a proprietary wired port / plug, a BLUETOOTH® wireless signal transmission, a BLUETOOTH® low energy (BLE) wireless signal transmission, an IBEACON® wireless signal transmission, a radio-frequency identification (RFID) wireless signal transmission, a near-field communications (NFC) wireless signal transmission, a dedicated short range communication (DSRC) wireless signal transmission, a 1502.11 Wi-Fi wireless signal transmission, a wireless local area network (WLAN) signal transmission, a Visible Light Communication (VLC), a Worldwide Interoperability for Microwave Access (WiMAX), an Infrared (IR) wireless signal transmission, a Public Switched Telephone Network (PSTN), a The reception and / or transmission of wired or wireless communications may be performed or facilitated using wired and / or wireless transceivers, including those that use Public Service Telephone Network (PSTN) signal transfer, Integrated Services Digital Network (ISDN) signal transfer, 3G / 4G / 5G / LTE cellular data network wireless signal transfer, ad hoc network signal transfer, radio wave signal transfer, microwave signal transfer, infrared signal transfer, visible light signal transfer, ultraviolet light signal transfer, wireless signal transfer along the electromagnetic spectrum, or any combination thereof.Communications interface 1540 may also include one or more GNSS receivers or transceivers used to determine the location of computing system 1500 based on reception of one or more signals from one or more satellites associated with one or more Global Navigation Satellite System (GNSS) systems, including, but not limited to, the U.S.-based Global Positioning System (GPS), the Russian-based Global Navigation Satellite System (GLONASS), the Chinese-based BeiDou Navigation Satellite System (BDS), and the European-based Galileo GNSS. There is no constraint to operating on any particular hardware configuration, and therefore, the basic features herein may be easily replaced with improved hardware or firmware arrangements as they are developed.

[0153]

[0170] The storage device 1530 can be a non-volatile and / or non-transitory and / or computer readable memory device, such as a magnetic cassette, a flash memory card, a solid state memory device, a digital versatile disk, a cartridge, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic strip / stripe, any other magnetic storage medium, a flash memory, a memristor memory, any other solid state memory, a compact disc read only memory (CD-ROM) optical disk, a rewritable compact disc (CD) optical disk, a digital video disk (DVD) optical disk, a blu-ray disc (BDD) optical disk, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a memory stick card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM, RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash EPROM (FLASHEPROM), cache memory (L1 / L2 / L3 / L4 / L5 / L#), resistive RAM (RRAM / ReRAM), phase change memory (PCM), spin transfer torqueIt may be a hard disk or other type of computer-readable medium capable of storing data that is accessible by a computer, such as RAM, STT-RAM), another memory chip or cartridge, and / or a combination thereof.

[0154]

[0171] The storage devices 1530 may include software services, servers, services, etc., where code defining such software, when executed by the processor 1510, causes the system to perform functions. In some aspects, a hardware service that performs a particular function may include software components stored on a computer-readable medium in association with the hardware components, such as the processor 1510, connections 1505, output devices 1535, etc., necessary to perform that function.

[0155]

[0172] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, storing, or transporting instruction(s) and / or data. Computer-readable media may include non-transitory media that can store data and do not include carrier waves and / or transitory electronic signals propagated wirelessly or via wired connections. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memory, memory, or memory devices. Code and / or machine-executable instructions may be stored on a computer-readable medium, which may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. can be passed, forwarded, or transmitted using any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0156]

[0173] In some aspects, computer-readable storage devices, media, and memories may include cables or wireless signals containing bitstreams, etc. However, when referred to, non-transitory computer-readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0157]

[0174] Specific details are provided in the above specification to provide a thorough understanding of the aspects and examples provided herein. However, those skilled in the art will understand that aspects may be practiced without these specific details. For ease of explanation, in some instances, the technology may be presented as including individual functional blocks, including devices, device components, method steps or routines embodied in software, or functional blocks comprising a combination of hardware and software. Additional components other than those shown in the figures and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the aspects in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail so as to avoid obscuring the aspects.

[0158]

[0175] Individual aspects may be described above as a process or method that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. While a flowchart may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. A process terminates when its operations are completed, but may have additional steps not included in the figures. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or the main function.

[0159]

[0176] The processes and methods according to the above examples can be implemented using computer-executable instructions stored on or available from a computer-readable medium. Such instructions can include, for example, instructions and data that cause a general-purpose computer, special-purpose computer, or processing device to perform a particular function or group of functions, or configure a general-purpose computer, special-purpose computer, or processing device to perform a particular function or group of functions. Portions of the computer resources used can be accessible over a network. The computer-executable instructions can be, for example, binary or intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer-readable media that can be used to store instructions, information used, and / or information created during methods according to the described examples include magnetic or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0160]

[0177] Devices implementing processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program product) to perform the necessary tasks may be stored in a computer-readable or machine-readable medium. A processor or processors may perform the necessary tasks. Typical example form factors include laptops, smartphones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, standalone devices, etc. The functionality described herein may also be embodied in a peripheral device or add-in card. Such functionality may also be implemented across different chips on a circuit board or across different processes running within a single device, as further examples.

[0161]

[0178] The instructions, media for communicating such instructions, computing resources for executing those instructions, and other structures for supporting such computing resources are exemplary means for providing the functionality described in this disclosure.

[0162]

[0179] While aspects of the present application have been described above with reference to specific aspects thereof, those skilled in the art will recognize that the present application is not limited thereto. Accordingly, while illustrative aspects of the present application have been described in detail herein, it should be understood that the concepts of the present application may be embodied and employed in various other ways, and that the appended claims are intended to be construed to include such variations except insofar as limited by the prior art. The various features and aspects of the present application described above may be used individually or in combination. Furthermore, aspects may be utilized in any number of environments and applications other than those described herein without departing from the broader spirit and scope of the present specification. Accordingly, the specification and drawings should be regarded as illustrative and not restrictive. For illustrative purposes, methods have been described in a particular order. It should be understood that in alternative embodiments, methods may be performed in an order different from that described.

[0163]

[0180] Those skilled in the art will understand that the less than ("<") and greater than (">") symbols or terms used herein can be replaced with the less than or equal to ("≦") and greater than or equal to ("≧") symbols, respectively, without departing from the scope of this description.

[0164]

[0181] Where a component is described as being "configured to" perform a particular operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the operation, by programming a programmable electronic circuit (e.g., a microprocessor or other suitable electronic circuitry) to perform the operation, or any combination thereof.

[0165]

[0182] The phrase "coupled to" refers to any component that is physically connected, either directly or indirectly, to another component and / or that is in communication, either directly or indirectly, with another component (e.g., connected to the other component via a wired or wireless connection and / or other suitable communication interface).

[0166]

[0183] Claim language or other language referring to "at least one of" a set and / or "one or more" of a set indicates that one member of the set, or multiple members of the set (in any combination), satisfies the claim. For example, a claim language referring to "at least one of A and B" means A, B, or A and B. As another example, a claim language referring to "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A, B, and C. The language referring to "at least one of" a set and / or "one or more" of a set does not limit the set to the items listed in the set. For example, a claim language referring to "at least one of A and B" can mean A, B, or A and B, and can additionally include items not listed in the set of A and B.

[0167]

[0184] The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the aspects disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in varying ways for each particular application, and such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0168]

[0185] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general-purpose computer, a wireless communication device handset, or an integrated circuit device having multiple uses, including applications in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may comprise memory or data storage media, such as random access memory (RAM), such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), FLASH memory, magnetic or optical data storage media, etc. These techniques may additionally, or alternatively, be realized at least in part by a computer-readable communications medium, such as a propagated signal or wave, carrying or communicating program code in the form of instructions or data structures that can be accessed, read, and / or executed by a computer.

[0169]

[0186] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Accordingly, the term “processor,” as used herein, may refer to any of the above structures, any combination of the above structures, or any other structure or apparatus suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided within dedicated software or hardware modules configured for encoding and decoding, or may be incorporated within a combined video encoder-decoder (CODEC).

[0170]

[0187] Illustrative aspects of the present disclosure include the following.

[0171]

[0188] Aspect 1. An apparatus for extended reality (XR) management, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory, wherein the at least one processor is configured to: receive sensor data from an XR interface device having at least one sensor; generate processing instructions for the XR processing device to process the sensor data to generate XR content based on the receipt of the sensor data from the XR interface device; send the sensor data and the processing instructions to the XR processing device; receive XR content from the XR processing device; generate layer content; and send the XR content and the layer content to the XR interface device, so that the XR interface device outputs the XR content and the layer content in a layered arrangement.

[0172]

[0189] Embodiment 2. The apparatus of embodiment 1, wherein the XR interface device is a headset that is coupled to the apparatus using a local communication interface.

[0173]

[0190] Aspect 3. The apparatus of any one of aspects 1 or 2, wherein the XR interface device is a mobile device that is coupled to the apparatus using a local communication interface.

[0174]

[0191] Embodiment 4. The apparatus of any one of embodiments 1 to 3, wherein the apparatus is a mobile device coupled to an XR interface device using a local communication interface.

[0175]

[0192] Embodiment 5. The apparatus of any one of embodiments 1 to 4, wherein the XR processing device includes a server.

[0176]

[0193] Embodiment 6. The apparatus of any one of embodiments 1 to 5, wherein the XR processing device includes an edge node.

[0177]

[0194] Embodiment 7. The apparatus of any one of embodiments 1-6, wherein the sensor data received from the XR interface device and sent to the XR processing device comprises compressed sensor data.

[0178]

[0195] Embodiment 8. The apparatus of any one of embodiments 1-7, wherein the at least one processor is configured to compress at least a portion of the sensor data before sending the sensor data to the XR processing device, and wherein at least a portion of the sensor data is uncompressed when received from the XR interface device.

[0179]

[0196] Aspect 9. The apparatus of any one of Aspects 1-8, wherein, to generate the processing instructions based on receipt of the sensor data from the XR interface device, the at least one processor is configured to generate at least a subset of the processing instructions based on an uncompressed portion of the sensor data that was uncompressed when received from the XR interface device, and the compressed portion of the sensor data that was compressed when received from the XR interface device.

[0180]

[0197] Aspect 10. The apparatus of any one of Aspects 1-9, wherein, to generate the processing instructions based on receipt of the sensor data from the XR interface device, the at least one processor is configured to generate at least a subset of the processing instructions based on at least one characteristic of a local communication interface between the apparatus and the XR interface device, through which the sensor data is received from the XR interface device, the at least one characteristic including at least one of a bandwidth, a bit rate, a baud rate, a transfer speed, a transfer protocol, an encryption scheme, a connection quality, or a security scheme.

[0181]

[0198] Aspect 11. The apparatus of any one of Aspects 1-10, wherein, to generate the processing instructions based on receipt of the sensor data from the XR interface device, the at least one processor is configured to generate at least a subset of the processing instructions based on at least one characteristic of the sensor data received from the XR interface device, the at least one characteristic including at least one of a file size, an image size, an image resolution, a frame rate, an amount of channels, an encoding scheme, or an encryption scheme.

[0182]

[0199] Aspect 12. The apparatus of any one of Aspects 1-11, wherein to generate processing instructions based on receipt of sensor data from the XR interface device, at least one processor is configured to generate at least a subset of the processing instructions based on at least one characteristic of the XR interface device, the at least one characteristic including at least one of image capture resolution, image capture size, image capture color space, image display resolution, image display size, image display color space, posture, location, orientation, bandwidth, bit rate, baud rate, transfer speed, transfer protocol, encryption scheme, security scheme, temperature, or battery level.

[0183]

[0200] Aspect 13. The apparatus of any one of Aspects 1-12, wherein to generate processing instructions based on receipt of sensor data from the XR interface device, the at least one processor is configured to generate at least a subset of the processing instructions based on at least one characteristic of the XR processing device, the at least one characteristic including at least one of capability, capacity, processing power, bandwidth, bit rate, baud rate, transfer speed, transfer protocol, encryption scheme, or security scheme.

[0184]

[0201] Aspect 14. The device of any one of aspects 1 to 13, wherein the processing instructions indicate at least a pose of virtual content to be rendered as part of the XR content, and the virtual content is associated with a video game.

[0185]

[0202] Aspect 15. The device of any one of aspects 1 to 14, wherein a portion of the layer content is configured to be overlaid on top of at least a portion of the XR content in a layered arrangement.

[0186]

[0203] Embodiment 16. The device of any one of embodiments 1 to 15, wherein at least a portion of the XR content is configured to be overlaid on top of at least a portion of the layer content in a layered arrangement.

[0187]

[0204] Aspect 17. The apparatus of any one of Aspects 1-16, wherein at least one processor is configured to manage an application programming interface (API), wherein receiving sensor data from the XR interface device is associated with a first API call to the API, and receiving XR content from the XR processing device is associated with a second API call to the API.

[0188]

[0205] Aspect 18. A method for extended reality (XR) management, the method comprising: receiving sensor data from an XR interface device having at least one sensor; generating, based on the receipt of the sensor data from the XR interface device, processing instructions for an XR processing device to process the sensor data to generate XR content; sending the sensor data and the processing instructions to the XR processing device; receiving the XR content from the XR processing device; generating layer content; and sending the XR content and layer content to the XR interface device, so that the XR interface device outputs the XR content and layer content in a layered arrangement.

[0189]

[0206] Aspect 19. The method of aspect 18, wherein the XR interface device is a headset that is coupled to the apparatus using a local communication interface.

[0190]

[0207] Aspect 20. The method of any one of aspects 18 or 19, wherein the XR interface device is a mobile device coupled to the apparatus using a local communication interface.

[0191]

[0208] Embodiment 21. The method of any one of embodiments 18 to 20, wherein the method is performed by a mobile device that is coupled to the XR interface device using a local communication interface.

[0192]

[0209] Embodiment 22. The method of any one of embodiments 18 to 21, wherein the XR processing device includes a server.

[0193]

[0210] Embodiment 23. The method of any one of embodiments 18 to 22, wherein the XR processing device includes an edge node.

[0194]

[0211] Embodiment 24. The method of any one of embodiments 18-23, wherein the sensor data received from the XR interface device and sent to the XR processing device comprises compressed sensor data.

[0195]

[0212] Embodiment 25. The method of any one of embodiments 18-24, further comprising compressing at least a portion of the sensor data before sending the sensor data to the XR processing device, wherein at least a portion of the sensor data is uncompressed when received from the XR interface device.

[0196]

[0213] Aspect 26. The method of any one of aspects 18-25, wherein generating the processing instructions includes generating at least a subset of the processing instructions based on an uncompressed portion of the sensor data that is uncompressed when received from the XR interface device, and the compressed portion of the sensor data is compressed when received from the XR interface device.

[0197]

[0214] Aspect 27. The method of any one of Aspects 18-26, wherein generating the processing instructions based on receiving the sensor data from the XR interface device includes generating at least a subset of the processing instructions based on at least one characteristic of a local communication interface between the apparatus and the XR interface device, through which the sensor data is received from the XR interface device, wherein the at least one characteristic includes at least one of a bandwidth, a bit rate, a baud rate, a transfer speed, a transfer protocol, an encryption method, a connection quality, or a security method.

[0198]

[0215] Aspect 28. The method of any one of aspects 18 to 27, wherein generating the processing instructions based on receiving the sensor data from the XR interface device includes generating at least a subset of the processing instructions based on at least one characteristic of the sensor data received from the XR interface device, wherein the at least one characteristic includes at least one of a file size, an image size, an image resolution, a frame rate, an amount of channels, an encoding scheme, or an encryption scheme.

[0199]

[0216] Aspect 29. The method of any one of aspects 18-29, wherein generating the processing instructions based on receiving sensor data from the XR interface device includes generating at least a subset of the processing instructions based on at least one characteristic of the XR interface device, wherein the at least one characteristic includes at least one of image capture resolution, image capture size, image capture color space, image display resolution, image display size, image display color space, posture, location, orientation, bandwidth, bit rate, baud rate, transfer speed, transfer protocol, encryption scheme, security scheme, temperature, or battery level.

[0200]

[0217] Aspect 30. The method of any one of aspects 18 to 29, wherein generating the processing instructions based on receiving the sensor data from the XR interface device includes generating at least a subset of the processing instructions based on at least one characteristic of the XR processing device, wherein the at least one characteristic includes at least one of capability, capacity, processing power, bandwidth, bit rate, baud rate, transfer speed, transfer protocol, encryption scheme, or security scheme.

[0201]

[0218] Aspect 31. The method of any one of aspects 18 to 30, wherein the processing instructions indicate at least a pose of virtual content to be rendered as part of the XR content, and the virtual content is associated with a video game.

[0202]

[0219] Embodiment 32. The method of any one of embodiments 18 to 31, wherein a portion of the layer content is configured to be overlaid on top of at least a portion of the XR content in a layered arrangement.

[0203]

[0220] Embodiment 33. The method of any one of embodiments 18 to 32, wherein at least a portion of the XR content is configured to be overlaid on top of at least a portion of the layer content in a layered arrangement.

[0204]

[0221] Aspect 34. The method of any one of aspects 18 to 33, further comprising managing an application programming interface (API), wherein receiving sensor data from the XR interface device is associated with a first API call to the API, and receiving XR content from the XR processing device is associated with a second API call to the API.

[0205]

[0222] Aspect 35. The method of any one of aspects 18 or 19, wherein receiving sensor data from the XR interface device includes receiving sensor data from the XR interface device via a local communication interface, sending XR content and layer content to the XR interface device includes sending XR content and layer content to the XR interface device via the local communication interface, and generating processing instructions includes generating at least a subset of the processing instructions based on at least one characteristic of the local communication interface, wherein the at least one characteristic includes at least one of bandwidth, bit rate, baud rate, transfer speed, transfer protocol, encryption method, connection quality, or security method.

[0206]

[0223] Aspect 36. A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform operations according to any of aspects 18-35.

[0207]

[0224] Aspect 37. An apparatus for extended reality (XR), comprising one or more means for performing the operations according to any of aspects 18 to 35.

Claims

1. 1. An apparatus for extended reality (XR) management, comprising: at least one memory; at least one processor coupled to the at least one memory; wherein the at least one processor: receiving sensor data from an XR interface device having at least one sensor; generating, based on the reception of the sensor data from the XR interface device, processing instructions for an XR processing device to process the sensor data to generate XR content; sending the sensor data and the processing instructions to the XR processing device; receiving the XR content from the XR processing device; Generate the layer content, 10. An apparatus configured to send the XR content and the layer content to the XR interface device so that the XR interface device outputs the XR content and the layer content in a layered arrangement.

2. The apparatus of claim 1 , wherein the XR interface device is a headset coupled to the apparatus using a local communication interface.

3. The apparatus of claim 1 , wherein the XR interface device is a mobile device coupled to the apparatus using a local communication interface.

4. The apparatus of claim 1 , wherein the apparatus is a mobile device coupled to the XR interface device using a local communication interface.

5. The apparatus of claim 1 , wherein the XR processing device comprises a server.

6. The apparatus of claim 1 , wherein the XR processing device comprises an edge node.

7. 10. The apparatus of claim 1, wherein the sensor data received from the XR interface device and sent to the XR processing device comprises compressed sensor data.

8. the at least one processor:

10. The apparatus of claim 1, configured to compress at least a portion of the sensor data before sending the sensor data to the XR processing device, and wherein at least the portion of the sensor data is uncompressed when received from the XR interface device.

9. to generate the processing instructions based on the reception of the sensor data from the XR interface device, the at least one processor:

10. The apparatus of claim 1, wherein the apparatus is configured to generate at least a subset of the processing instructions based on an uncompressed portion of the sensor data that is uncompressed when received from the XR interface device, the compressed portion of the sensor data being compressed when received from the XR interface device.

10. to generate the processing instructions based on the reception of the sensor data from the XR interface device, the at least one processor:

2. The apparatus of claim 1, wherein the apparatus is configured to generate at least a subset of the processing instructions based on at least one characteristic of a local communication interface between the apparatus and the XR interface device through which the sensor data is received from the XR interface device, the at least one characteristic including at least one of a bandwidth, a bit rate, a baud rate, a transfer speed, a transfer protocol, an encryption scheme, a connection quality, or a security scheme.

11. to generate the processing instructions based on the reception of the sensor data from the XR interface device, the at least one processor:

10. The apparatus of claim 1, configured to generate at least a subset of the processing instructions based on at least one characteristic of the sensor data received from the XR interface device, the at least one characteristic including at least one of a file size, an image size, an image resolution, a frame rate, an amount of channels, an encoding scheme, or an encryption scheme.

12. to generate the processing instructions based on the reception of the sensor data from the XR interface device, the at least one processor:

10. The apparatus of claim 1, wherein the apparatus is configured to generate at least a subset of the processing instructions based on at least one characteristic of the XR interface device, the at least one characteristic including at least one of an image capture resolution, an image capture size, an image capture color space, an image display resolution, an image display size, an image display color space, a posture, a location, an orientation, a bandwidth, a bit rate, a baud rate, a transfer speed, a transfer protocol, an encryption scheme, a security scheme, a temperature, or a battery level.

13. to generate the processing instructions based on the reception of the sensor data from the XR interface device, the at least one processor:

10. The apparatus of claim 1, configured to generate at least a subset of the processing instructions based on at least one characteristic of the XR processing device, the at least one characteristic including at least one of capability, capacity, processing power, bandwidth, bit rate, baud rate, transfer speed, transfer protocol, encryption scheme, or security scheme.

14. 10. The device of claim 1, wherein the processing instructions indicate at least a pose of virtual content to be rendered as part of the XR content, and the virtual content is associated with a video game.

15. 10. The device of claim 1, wherein a portion of the layer content is configured to be overlaid on at least a portion of the XR content in the layered arrangement.

16. 10. The device of claim 1, wherein at least a portion of the XR content is configured to be overlaid on at least a portion of the layer content in the layered arrangement.

17. the at least one processor:

10. The apparatus of claim 1, configured to manage an application programming interface (API), wherein the receiving of the sensor data from the XR interface device is associated with a first API call to the API, and the receiving of the XR content from the XR processing device is associated with a second API call to the API.

18. 1. A method for extended reality (XR) management, comprising: receiving sensor data from an XR interface device having at least one sensor; generating, based on the reception of the sensor data from the XR interface device, processing instructions for an XR processing device to process the sensor data to generate XR content; sending the sensor data and the processing instructions to the XR processing device; receiving the XR content from the XR processing device; generating layer content; sending the XR content and the layer content to the XR interface device so that the XR interface device outputs the XR content and the layer content in a layered arrangement.

19. 20. The method of claim 18, wherein generating the processing instructions comprises generating at least a subset of the processing instructions based on an uncompressed portion of the sensor data that was uncompressed when received from the XR interface device, and wherein the compressed portion of the sensor data was compressed when received from the XR interface device.

20. 20. The method of claim 18, wherein receiving the sensor data from the XR interface device comprises receiving the sensor data from the XR interface device via a local communication interface; sending the XR content and the layer content to the XR interface device comprises sending the XR content and the layer content to the XR interface device via the local communication interface; and generating the processing instructions comprises generating at least a subset of the processing instructions based on at least one characteristic of the local communication interface, the at least one characteristic comprising at least one of a bandwidth, a bit rate, a baud rate, a transfer speed, a transfer protocol, an encryption scheme, a connection quality, or a security scheme.

Images