Method and apparatus for standardizing api for decoupled rendering

Abstract

This disclosure relates to methods and apparatus for graphics processing. Various aspects of this disclosure can define at least one discrete API for a user device and a server. Furthermore, aspects of this disclosure can establish a communication interface between the user device and the server based on the discrete API. Additionally, aspects of this disclosure can enable communication between the user device and the server based on the communication interface and the discrete API. In some aspects, this disclosure can implement at least one discrete API on at least one of the user device or the server. Various aspects of this disclosure can also partition at least one application workload between the user device and the server based on at least one discrete API. Furthermore, aspects of this disclosure can encode or decode application information at the user device or the server.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of Indian Provisional Application No. 201941002809, filed January 23, 2019, entitled “METHODS AND APPARATUS FOR STANDARDIZED APIS FOR SPLITRENDERING”, and U.S. Patent Application No. 16 / 741,554, filed January 13, 2020, entitled “METHODS AND APPARATUS FOR STANDARDIZED APIS FOR SPLITRENDERING”, the entire contents of which are expressly incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to processing systems, and more specifically, to one or more techniques for graphics processing. Background Technology

[0004] Computing devices frequently utilize graphics processing units (GPUs) to accelerate the rendering of graphics data used for display. Examples of such devices include computer workstations, mobile phones such as so-called smartphones, embedded systems, personal computers, tablets, and video game consoles. A GPU executes a graphics processing pipeline comprising one or more processing stages that work together to execute graphics processing commands and output frames. A central processing unit (CPU) controls the operation of the GPU by issuing one or more graphics processing commands to it. Modern CPUs are typically capable of executing multiple applications simultaneously, each potentially requiring the GPU to utilize its resources during execution. Devices that provide content for visual presentation on a display typically include GPUs.

[0005] Typically, a device's GPU is configured to execute processes in the graphics processing pipeline. However, with the advent of wireless communication and smaller handheld devices, there is a growing need for improved graphics processing. Summary of the Invention

[0006] The following is a simplified overview of one or more aspects to provide a basic understanding of them. This overview is not a comprehensive overview of all anticipated aspects, and is neither intended to identify key elements of all aspects nor to depict the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed descriptions that follow.

[0007] In one aspect of this disclosure, a method, computer-readable medium, and apparatus are provided. The apparatus may be a user device, a server, a central processing unit (CPU), a graphics processing unit (GPU), or any device capable of performing graphics processing. The apparatus may define at least one split application program interference (API) for the user device and the server. Furthermore, the apparatus may establish a communication interface between the user device and the server based on at least one split API. In some aspects, the apparatus may implement at least one split API on at least one of the user device or the server. The apparatus may also partition at least one application workload between the user device and the server based on at least one split API. Additionally, the apparatus may encode or decode application information at the user device or the server. Furthermore, the apparatus may communicate between the user device and the server based on the communication interface and at least one split API.

[0008] Details of one or more examples of this disclosure are set forth in the accompanying drawings and the following description. Other features, objects, and advantages of this disclosure will become apparent from the specification, drawings, and claims. Attached Figure Description

[0009] Figure 1 This is a block diagram illustrating an example content generation system based on one or more techniques according to this disclosure.

[0010] Figure 2 An example GPU according to one or more technologies disclosed herein is shown.

[0011] Figure 3 An example flowchart including the application interface is shown.

[0012] Figure 4 An example architecture of one or more technologies according to this disclosure is shown.

[0013] Figure 5 An example architecture of one or more technologies according to this disclosure is shown.

[0014] Figure 6 An example architecture of one or more technologies according to this disclosure is shown.

[0015] Figure 7 An example architecture of one or more technologies according to this disclosure is shown.

[0016] Figure 8 An example architecture of one or more technologies according to this disclosure is shown.

[0017] Figure 9An example flowchart of an example method according to one or more techniques of this disclosure is shown. Detailed Implementation

[0018] In some cases, open extended reality (XR) can be utilized by different XR devices and applications to communicate with each other via a standardized application programming interface (API). However, some processes in open XR may assume that the application and the XR runtime are co-located, i.e., on the same device. In the context of discrete XR where the rendering load may be completely or partially offset to a remote server, a standardized API may not exist. For example, it may be necessary to design an application for a single device because applications within discrete XR may not be able to interoperate with any type of device. Aspects of this disclosure include systems and methods for standardizing APIs for discrete rendering. By doing so, aspects of the present invention can introduce interoperability between applications and devices in XR, augmented reality (AR), or virtual reality (VR) settings. Thus, based on the API architecture of this disclosure, different types of discrete rendering devices or applications can interoperate with other discrete rendering devices or applications.

[0019] The various aspects of the systems, apparatus, computer program products, and methods will be described more fully below with reference to the accompanying drawings. However, this disclosure may be embodied in many different forms and should not be construed as limited to any particular structure or function presented throughout this disclosure. Rather, these aspects are provided to make this disclosure exhaustive and complete, and to fully convey the scope of this disclosure to those skilled in the art. Based on the teachings herein, those skilled in the art should understand that the scope of this disclosure is intended to cover any aspect of the systems, apparatus, computer program products, and methods disclosed herein, whether implemented independently of or in combination with other aspects of this disclosure. For example, any number of aspects set forth herein may be used to implement an apparatus or practice method. Furthermore, the scope of this disclosure is intended to cover an apparatus or method practiced using a structure, function, or structure and function other than or different from the aspects of this disclosure set forth herein. Any aspect disclosed herein may be embodied by one or more elements of the claims.

[0020] While various aspects have been described herein, numerous variations and substitutions of these aspects fall within the scope of this disclosure. Although some potential benefits and advantages of the aspects of this disclosure have been mentioned, the scope of this disclosure is not intended to be limited to a particular benefit, use, or objective. Rather, the aspects of this disclosure are intended to be broadly applicable to different wireless technologies, system configurations, networks, and transport protocols, some of which are illustrated by way of example in the accompanying drawings and the following description. The detailed description and accompanying drawings are merely illustrative of this disclosure and are not intended to limit its scope, which is defined by the appended claims and their equivalents.

[0021] Several aspects are presented with reference to various apparatuses and methods. These apparatuses and methods are described in the following specific embodiments and are illustrated in the accompanying drawings by various blocks, components, circuits, processes, algorithms, etc. (collectively, "elements"). These elements can be implemented using electronic hardware, computer software, or any combination thereof. Whether these elements are implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system.

[0022] For example, an element, or any part of an element, or any combination of elements, can be implemented as a “processing system” including one or more processors (also referred to as processing units). Examples of processors include microprocessors, microcontrollers, graphics processing units (GPUs), general-purpose GPUs (GPGPUs), central processing units (CPUs), application processors, digital signal processors (DSPs), reduced instruction set computing (RISC) processors, system-on-a-chip (SoCs), baseband processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gating logic, discrete hardware circuits, and other suitable hardware configured to perform the various functions described throughout this disclosure. One or more processors in a processing system can execute software. Whether referring to software, firmware, middleware, microcode, hardware description languages, or others, software can be broadly understood as instructions, instruction sets, code, code segments, program code, programs, subroutines, software components, applications, software applications, software packages, routines, subroutines, objects, executable programs, threads of execution, procedures, functions, etc. The term application can refer to software. As described herein, one or more technologies can refer to an application, that is, software configured to perform one or more functions. In such examples, the application may be stored on memory (e.g., the processor's on-chip memory, system memory, or any other memory). The hardware described herein (e.g., a processor) may be configured to execute the application. For example, the application may be described as including code that, when executed by the hardware, causes the hardware to perform one or more techniques described herein. As an example, the hardware may access and execute code accessed from memory to perform one or more techniques described herein. In some examples, components are identified in this disclosure. In such examples, a component may be hardware, software, or a combination thereof. These components may be individual components or subcomponents of a single component.

[0023] Therefore, in one or more examples described herein, the described functionality can be implemented in hardware, software, or any combination thereof. If implemented in software, these functions can be stored on a computer-readable medium or encoded as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media. Storage media can be any available medium accessible to a computer. By way of example and not limitation, such computer-readable media can include random access memory (RAM), read-only memory (ROM), electrically erasable programmable ROM (EEPROM), optical disk storage, magnetic disk storage, other magnetic storage devices, combinations of computer-readable media of the types described above, or any other medium that can be used to store computer-executable code in the form of computer-accessible instructions or data structures.

[0024] In general, this disclosure describes techniques for having a graphics processing pipeline in a single device or multiple devices, improving the rendering of graphics content, and / or reducing the load on processing units (i.e., any processing unit, such as a GPU, configured to perform one or more of the techniques described herein). For example, this disclosure describes techniques for utilizing graphics processing in any device. Other exemplary benefits are described throughout this disclosure.

[0025] As used herein, instances of the term "content" can refer to "graphic content," "image," or vice versa. This is true regardless of whether these terms are used as adjectives, nouns, or other parts of speech. In some examples, as used herein, the term "graphic content" can refer to content produced by one or more processes in the graphics processing pipeline. In some examples, as used herein, the term "graphic content" can refer to content produced by a processing unit configured to perform graphics processing. In some examples, as used herein, the term "graphic content" can refer to content produced by a graphics processing unit.

[0026] In some examples, as used herein, the term "display content" can refer to content generated by a processing unit configured to perform display processing. Graphical content can be processed into display content. For example, a graphics processing unit can output graphical content (e.g., frames) to a buffer (which may be referred to as a frame buffer). A display processing unit can read graphical content (e.g., one or more frames) from the buffer and perform one or more display processing techniques on it to generate display content. For example, a display processing unit can be configured to perform compositing on one or more rendering layers to generate frames. As another example, a display processing unit can be configured to composite, blend, or otherwise combine two or more layers into a single frame. A display processing unit can be configured to perform scaling, such as zooming in or out of a frame. In some examples, a frame can refer to a layer. In other examples, a frame can refer to two or more layers that have been blended together to form the frame; that is, a frame comprises two or more layers, and frames comprising two or more layers can be subsequently blended.

[0027] Figure 1This is a block diagram illustrating an example content generation system 100 configured to implement one or more technologies of the present disclosure. The content generation system 100 includes a device 104. Device 104 may include one or more components or circuitry for performing the various functions described herein. In some examples, one or more components of device 104 may be components of a System-on-a-Chip (SOC). Device 104 may include one or more components configured to perform one or more technologies of the present disclosure. In the illustrated example, device 104 may include a processing unit 120, a content encoder / decoder 122, and system memory 124. In some aspects, device 104 may include a plurality of optional components, such as a communication interface 126, a transceiver 132, a receiver 128, a transmitter 130, a display processor 127, and one or more displays 131. Reference to display 131 may refer to one or more displays 131. For example, display 131 may include a single display or multiple displays. Display 131 may include a first display and a second display. The first display may be a left-eye display, and the second display may be a right-eye display. In some examples, the first and second displays may receive different frames for presentation thereon. In other examples, the first and second displays may receive the same frames used for rendering on them. In yet another example, the results of graphics processing may not be displayed on the device; for example, the first and second displays may not receive any frames used for rendering on them. Instead, the frames or graphics processing results may be transmitted to another device. In some respects, this can be referred to as split rendering.

[0028] Processing unit 120 may include internal memory 121. Processing unit 120 may be configured (e.g., in graphics processing pipeline 107) to perform graphics processing. Content encoder / decoder 122 may include internal memory 123. In some examples, device 104 may include a display processor, such as display processor 127, to perform one or more display processing techniques on one or more frames generated by processing unit 120 before being rendered by one or more displays 131. Display processor 127 may be configured to perform display processing. For example, display processor 127 may be configured to perform one or more display processing techniques on one or more frames generated by processing unit 120. One or more displays 131 may be configured to display or otherwise render the frames processed by display processor 127. In some examples, one or more displays 131 may include one or more of the following: liquid crystal display (LCD), plasma display, organic light-emitting diode (OLED) display, projection display device, augmented reality display device, virtual reality display device, head-mounted display, or any other type of display device.

[0029] External memory (e.g., system memory 124) can be accessed by processing unit 120 and content encoder / decoder 122. For example, processing unit 120 and content encoder / decoder 122 can be configured to read from and / or write to external memory (e.g., system memory 124). Processing unit 120 and content encoder / decoder 122 can be communicatively coupled to system memory 124 via a bus. In some examples, processing unit 120 and content encoder / decoder 122 can be communicatively coupled to each other via a bus or different connections.

[0030] Content encoder / decoder 122 can be configured to receive graphical content from any source (e.g., system memory 124 and / or communication interface 126). System memory 124 can be configured to store received encoded or decoded graphical content. Content encoder / decoder 122 can be configured to receive encoded or decoded graphical content in the form of encoded pixel data (e.g., from system memory 124 and / or communication interface 126). Content encoder / decoder 122 can be configured to encode or decode any graphical content.

[0031] Internal memory 121 or system memory 124 may include one or more volatile or non-volatile memories or storage devices. In some examples, internal memory 121 or system memory 124 may include RAM, SRAM, DRAM, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic data media or optical storage media or any other type of memory.

[0032] According to some examples, internal memory 121 or system memory 124 may be a non-transitory storage medium. The term "non-transitory" may indicate that the storage medium is not embodied in a carrier wave or a propagating signal. However, the term "non-transitory" should not be construed as meaning that internal memory 121 or system memory 124 is not movable, or that its contents are static. As an example, system memory 124 may be removed from device 104 and moved to another device. As another example, system memory 124 may be not removable from device 104.

[0033] Processing unit 120 may be a central processing unit (CPU), a graphics processing unit (GPU), a general-purpose GPU (GPGPU), or any other processing unit that can be configured to perform graphics processing. In some examples, processing unit 120 may be integrated into the motherboard of device 104. In some examples, processing unit 120 may reside on a graphics card mounted in a port on the motherboard of device 104, or may otherwise be incorporated into a peripheral device configured to interoperate with device 104. Processing unit 120 may include one or more processors, such as one or more microprocessors, GPUs, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), arithmetic logic units (ALUs), digital signal processors (DSPs), discrete logic, software, hardware, firmware, other equivalent integrated or discrete logic circuits, or any combination thereof. If these techniques are implemented in part in software, processing unit 120 may store software instructions in a suitable non-transitory computer-readable storage medium (e.g., internal memory 121) and may execute the instructions in hardware using one or more processors to perform the techniques of this disclosure. Any of the foregoing, including hardware, software, and combinations of hardware and software, can be considered as one or more processors.

[0034] The content encoder / decoder 122 can be any processing unit configured to perform content decoding. In some examples, the content encoder / decoder 122 can be integrated into the motherboard of device 104. The content encoder / decoder 122 may include one or more processors, such as one or more microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), arithmetic logic units (ALUs), digital signal processors (DSPs), video processors, discrete logic, software, hardware, firmware, other equivalent integrated or discrete logic circuits, or any combination thereof. If these technologies are partially implemented in software, the content encoder / decoder 122 may store software instructions in a suitable non-transitory computer-readable storage medium (e.g., internal memory 123), and may use one or more processors to execute the instructions in hardware to perform the technologies of this disclosure. Any of the foregoing, including hardware, software, combinations of hardware and software, etc., can be considered as one or more processors.

[0035] In some aspects, the content generation system 100 may include an optional communication interface 126. Communication interface 126 may include a receiver 128 and a transmitter 130. Receiver 128 may be configured to perform any of the receiving functions described herein with respect to device 104. Furthermore, receiver 128 may be configured to receive information from another device, such as eye or head position information, rendering commands, or location information. Transmitter 130 may be configured to perform any of the transmitting functions described herein with respect to device 104. For example, transmitter 130 may be configured to transmit information to another device, which may include a request for content. Receiver 128 and transmitter 130 may be combined into transceiver 132. In such an example, transceiver 132 may be configured to perform any of the receiving and / or transmitting functions described herein with respect to device 104.

[0036] Refer again Figure 1 In some aspects, the graphics processing pipeline 107 may include a determining component 198 configured to determine at least one discrete application programming interface (API) between the user device and the server. The determining component 198 may also be configured to establish a communication interface between the user device and the server based on at least one discrete API. The determining component 198 may also be configured to implement at least one discrete API on at least one of the user device or the server. The determining component 198 may also be configured to partition at least one application workload between the user device and the server based on at least one discrete API. The determining component 198 may also be configured to encode or decode application information at the user device or the server. The determining component 198 may also be configured to communicate between the user device and the server based on the communication interface and at least one discrete API.

[0037] As described herein, a device such as device 104 can refer to any device, apparatus, or system configured to perform one or more of the technologies described herein. For example, a device can be a server, base station, user equipment, client device, station, access point, computer (e.g., personal computer, desktop computer, laptop computer, tablet computer, computer workstation, or mainframe computer) and terminal products, apparatus, telephone, smartphone, server, video game platform or console, handheld device (e.g., portable video game device or personal digital assistant (PDA)), wearable computing device (smartwatch, augmented reality device, or virtual reality device), non-wearable device, display or display device, television, set-top box, intermediate network device, digital media player, video streaming device, content streaming device, in-vehicle computer, any mobile device, any device configured to generate graphical content, or any device configured to perform one or more of the technologies described herein. The processes described herein may be described as being performed by specific components (e.g., GPU) conforming to the disclosed embodiments, but in other embodiments, other components (e.g., CPU) conforming to the disclosed embodiments may be used.

[0038] A GPU can process various types of data or data packets in the GPU pipeline. For example, in some aspects, a GPU can process two types of data or data packets, such as context register packets and draw call data. A context register packet can be a set of global state information (e.g., information about global registers, shaders, or constant data) that controls how the graphics context is processed. For example, a context register packet may include information about color formats. In some aspects of a context register packet, there may be a bit indicating which workload belongs to the context register. Furthermore, there can be multiple functions or programs that run simultaneously and / or in parallel. For example, a function or program may describe an operation, such as a color mode or color format. Therefore, context registers can define multiple states of the GPU.

[0039] Context states can be used to determine how individual processing units (e.g., vertex fetcher (VFD), vertex shader (VS), shader processor, or geometry processor) operate, and / or in what mode the processing unit operates. For this purpose, the GPU can use context registers and programming data. In some aspects, the GPU can generate workloads, such as vertex or pixel workloads in a pipeline defined by context registers based on modes or states. Certain processing units (e.g., VFDs) can use these states to determine certain functions, such as how vertices are assembled. Because these modes or states can change, the GPU may need to modify the corresponding context. Furthermore, workloads corresponding to modes or states may follow changing modes or states.

[0040] Figure 2 An example GPU 200 according to one or more technologies disclosed herein is shown. Figure 2 As shown, GPU 200 includes a command processor (CP) 210, a drawing call package 212, a VFD 220, a VS 222, a vertex cache (VPC) 224, a triangle setup engine (TSE) 226, a rasterizer (RAS) 228, a Z-process engine (ZPE) 230, a pixel interpolator (PI) 232, a fragment shader (FS) 234, a render backend (RB) 236, an L2 cache (UCHE) 238, and system memory 240. Although Figure 2 The GPU 200 is shown to include processing units 220-238, but the GPU 200 may include multiple additional processing units. Furthermore, processing units 220-238 are merely an example, and any combination or order of processing units can be used by the GPU according to this disclosure. The GPU 200 also includes a command buffer 250, a context register packet 260, and a context state 261.

[0041] like Figure 2As shown, the GPU can use a CP (e.g., CP 210) or a hardware accelerator to parse the command buffer into context register packets (e.g., context register packet 260) and / or draw call packets (e.g., draw call packet 212). CP 210 can then send the context register packet 260 or the draw call packet 212 to a processing unit or block in the GPU via a separate path. Furthermore, the command buffer 250 can alternate between context registers and draw calls of different states. For example, the command buffer can be constructed as follows: context register of context N, multiple draw calls of context N, context register of context N+1, and multiple draw calls of context N+1.

[0042] GPUs can render images in various ways. In some cases, GPUs can use either tiled rendering or regular rendering to render images. In tiled rendering GPUs, an image can be divided or separated into different parts or tiles. After the image is divided, each part or tile can be rendered separately. Tiled rendering GPUs can divide computer graphics images into a mesh format, so that each part of the mesh (i.e., a tile) is rendered individually. In some aspects, during the binning pass, an image can be divided into different bins or tiles. In some aspects, during the binning pass, a visibility flow can be constructed if visible primitives or draw calls can be identified.

[0043] In some respects, the rendering of an image or frame can be performed in multiple locations and / or multiple devices, for example, to divide the rendering workload among different devices. For instance, the rendering workload can be separated between the server and the user device. In some respects, this process can be referred to as "split rendering." In some cases, split rendering can be a method for bringing content to a user device or head-mounted display (HMD), where a portion of the computer or graphics processing can be performed outside the user device or HMD (e.g., on a server).

[0044] Split rendering can be performed for many different types of applications, such as virtual reality (VR) applications, augmented reality (AR) applications, augmented virtual (AV) and / or extended reality (XR) applications. In VR applications, the content displayed on the user device can correspond to rendered or animated content (e.g., content rendered on a server or user device). In AR or XR applications, a portion of the content displayed on the user device can correspond to real-world content (e.g., real-world objects), and a portion of the content can be rendered or animated content.

[0045] Furthermore, rendered or animated content and real-world content can be displayed in optical or video perspective devices, allowing users to view real-world objects and rendered content simultaneously. In some respects, XR content can refer to both VR and AR content. XR, VR, and AR applications can all involve rendered content viewed by users through head-mounted displays.

[0046] Split rendering can deliver a high-quality user experience in XR, AR, or VR applications by separating the computational workload between the client or user device (e.g., HMD or head-mounted display) and the server. In some aspects, the user device and server can be connected to each other via low-latency communication links (e.g., 5G or Wi-Fi). Furthermore, the server can be located near the low-latency communication link to reduce latency. In some ways, the server can be referred to as an edge server or a cloud server.

[0047] In some XR, VR, and AR applications, the user device or head-mounted display may include a GPU or graphics processing unit (GPU) that performs the necessary computations or graphics processing for rendering the content. Decoupled rendering allows most of the computations or graphics processing to be offloaded to a server. In these cases, the server can be any device capable of performing some of the computations or graphics processing offloaded from the user device. For example, the server could be a cloud server, edge server, personal computer, smartphone, or any suitable device.

[0048] OpenXR, or OpenXR, is a method for different XR, AR, AV, and / or VR devices and applications to communicate with each other. For example, in the context of XR, AR, AV, and / or VR applications, OpenXR allows different devices and applications to communicate. This can allow application developers and device manufacturers to be largely unaware of each other. For instance, game developers can develop OpenXR-compatible VR games that can work with different VR headsets or VR devices that, in turn, have OpenXR-compatible runtimes.

[0049] Figure 3 Example flowchart 300 is shown. Figure 3 As shown, Figure 300 is a diagram of an open XR interface. More specifically, Figure 300 includes one or more steps or applications: portable VR and AR application and engine step 302, open XR application interface step 304, VR and AR vendor runtime system distortion correction and display output coordinate system unification and prediction step 306, open XR device plug-in extension step 308, device driver provided by device vendor step 310, and portable VR and AR device step 312.

[0050] Figure 3The current device status is displayed, which can flow from step 312 (portable VR and AR devices) to step 302 (portable VR and AR applications and engines). For example... Figure 3 The current device state, as further shown, includes the controller or peripheral device state or the raw pose. Additionally, the current device state includes the normalized predicted pose or input events. Figure 3 It also shows that outgoing requests can flow from portable VR and AR applications and engines (step 302) to portable VR and AR devices (step 312). Outgoing requests can include pre-distortion images to display, haptics, or post-distortion images to display.

[0051] As mentioned above, OpenXR can be a method for different XR devices and applications to communicate with each other through a standardized application programming interface (API). The API can run on both the server and the device. However, some processes in OpenXR may assume that the application and the XR runtime are co-located, i.e., on the same device. In the context of SplitXR, where rendering loads may be entirely or partially off-target to a remote server, a standardized API may not exist. For example, it may be necessary to design an application for a single device, as applications within SplitXR may not interoperate with any type of device. Therefore, there is currently a need for SplitXR devices to interoperate with different applications via an API architecture.

[0052] Various aspects of this disclosure include systems and methods for standardizing APIs used for discrete rendering. By doing so, various aspects of the invention can introduce interoperability between applications and devices in XR, AR, or VR settings. Therefore, applications can be designed for any device that supports discrete XR, not just for a single device. Thus, based on the API architecture of this disclosure, different types of devices can interoperate with other devices or applications. For example, this disclosure may include an architecture for standardizing APIs (e.g., for discrete rendering).

[0053] This disclosure may propose one or more API architectures that allow game developers to create discrete XR experiences, even though the game developers are agnostic to the end device or end device consumer of the application. For example, it may not be necessary to design an application for a specific device because interoperability with different devices exists. Furthermore, this disclosure may propose API architectures that allow discrete XR applications to be agnostic to the user device or head-mounted display (HMD) device specifications. For example, this disclosure includes methods for including open XR-compatible applications and devices as specializations of the proposed architecture.

[0054] Figure 4 An architecture 400 according to one or more technologies disclosed herein is shown. For example... Figure 4 As shown, Architecture 400 is an API architecture proposed for discrete XR. For example... Figure 4 As further shown, architecture 400 includes device 450 and mobile edge compute (MEC) server 410. Figure 4 The standard protocol for split XR traffic communication between device 450 and MEC server 410 is also shown.

[0055] MEC server 410 may include a MEC-based XR application and engine 412, a discrete XR API 414, a discrete XR-compatible MEC vendor middleware 416, a discrete XR communication interface 430, and / or a MEC-to-cloud interface 420. Device 450 may include an XR runtime 452, a discrete XR API 454, a discrete XR-compatible device middleware 456, a discrete XR communication interface 460, and / or a device-to-cloud interface 470. For example... Figure 4 As shown, a standardized protocol for split XR traffic can communicate between split XR communication interface 430 and split XR communication interface 460.

[0056] In one respect, the game or application can be deployed on an edge server (e.g., MEC server 410). The game can submit frames to the device runtime (e.g., XR runtime 452) via a separate XR communication interface 430 on the MEC server 410. In another respect, the underlying middleware at the edge server (e.g., middleware 416) can ensure the implementation of a standardized communication stack to the device 450, and the orchestration of game instances. Middleware 416 can encode frames or information into, for example, a codec, and transmit that information to middleware 456 on the device 450. Middleware 456 can decode the information from middleware 416. Furthermore, MEC-based XR applications and engines 412 can communicate with middleware 416 via a separate XR API 414. Figure 4 As shown, middleware 416 can be a discrete XR compatible component.

[0057] Furthermore, XR device manufacturers can similarly implement middleware (e.g., middleware 456) that receives and sends discrete XR traffic via a standardized communication interface (e.g., communication interface 460). In some cases, this middleware can be provided by the chip manufacturer. Additionally, XR runtime 452 can receive information and send it to server 410 via discrete XR APIs (e.g., discrete XR APIs 414 and 454). For example, XR runtime 452 can send a stream of camera frames to an edge server to perform object recognition.

[0058] In some cases, middleware 416 on server 410 can communicate with middleware 456 on device 450, for example, via split XR communication interface 430 and split XR communication interface 460. Furthermore, XR runtime 452 can communicate with middleware 456 via split XR API 454. In this way, middleware 456 can provide information to runtime 452. Middleware 456 can also be a proxy for an application or game (e.g., on server 410).

[0059] Furthermore, middleware 456 helps ensure that runtime 452 is unaware of whether an application or game is running on a server. Therefore, runtime 452 may not know whether the application is running on device 450 or server 410. In some respects, XR runtime 452 can expose the decoupled XR API 454 to middleware 456. Middleware 456 can also extract pose information from runtime 452 and communicate this information to middleware 416. Therefore, both middleware 416 on server 410 and middleware 456 on device 450 can encode or decode information or data.

[0060] In some respects, a portion of an application or game may run on server 410, and a portion of an application or game may run on device 450. When certain information is generated on server 410, runtime 452 on device 450 can obtain this information. Furthermore, this information may be processed on server 410 before it can be obtained by runtime 452. Additionally, certain information generated on device 450 may be obtained by server 410 before or after it is processed on device 450. For example, camera information or other information generated on device 450 may be sent to an application running on server 410. In some respects, runtime 452 may be an application or game running on device 450. Furthermore, server 410 may generate information that will be utilized by runtime 452.

[0061] In some respects, middleware 416 can encode or decode information (e.g., information from middleware 456). For example, middleware 416 can encode or decode camera information, application or game information, clip information, and / or point cloud information from device 450. Similarly, middleware 456 can encode or decode information (e.g., information from middleware 416). For example, middleware 456 can encode or decode camera information, application or game information, clip information, and / or point cloud information from server 410.

[0062] Furthermore, APIs 414 and 454 can be standardized to enable interoperability with any device. Communication interfaces 430 and 460 can also be standardized. This allows the same application to run on device middleware (e.g., middleware 456), regardless of the middleware type. In some cases, middleware 416 and 456 may not need to be standardized; for example, they may come from different vendors. Based on the above, architecture 400 helps ensure interoperability between different types of devices.

[0063] In some respects, applications that are compatible with the discrete XR API can run on servers with discrete XR API middleware and / or communicate with discrete XR API-compatible HMDs or devices. Similarly, devices that are compatible with the discrete XR API can communicate with servers that are compatible with the discrete XR API and access applications that are also compatible with the discrete XR API and are running on that server.

[0064] Figure 5 An architecture 500 according to one or more technologies disclosed herein is shown. For example... Figure 5 As shown, Architecture 500 is another API architecture for discrete XR. (As...) Figure 5 As further shown, architecture 500 includes device 550 and MEC server 510. Figure 5 The standardized protocol for split XR traffic communication between device 550 and MEC server 510 is also shown.

[0065] MEC server 510 may include MEC-based XR applications and engines 512, a discrete XR API 514, discrete XR-compatible MEC vendor middleware 516, a discrete XR communication interface 530, and / or a MEC-to-cloud interface 520. Devices may include XR runtime 552, a discrete XR API 554, XR applications 562, a discrete XR API 564, discrete XR-compatible device middleware 556, a discrete XR communication interface 560, and / or a device-to-cloud interface 570.

[0066] In some respects, applications or games can be separated between the MEC server 510 and the device 550. Applications or games at the edge server can submit textures to the runtime 552 via the separate XR communication interface 530. Applications on the device 550 can consume these textures, render complementary parts, and / or submit frames to the XR runtime 552 via the separate XR API 554. Furthermore, the runtime 552 can transmit information directly to the server 510, or this traffic can be intercepted by applications on the device. In some respects, the XR runtime 552 can expose the separate XR API 564 to the middleware 556.

[0067] Figure 6 An architecture 600 according to one or more technologies disclosed herein is shown. Figure 6 As shown, Architecture 600 integrates open XR within a discrete XR architecture. For example... Figure 6 As further shown, architecture 600 includes device 650 and MEC server 610. Figure 6 The standardized protocol for split XR traffic communication between device 650 and MEC server 610 is also shown.

[0068] MEC server 610 may include MEC-based XR applications and engines 612, a discrete XR API 614, discrete XR-compatible MEC vendor middleware 616, a discrete XR communication interface 630, and / or a MEC-to-cloud interface 620. Device 650 may include an XR runtime 652, an open XR API 654, discrete XR-compatible device middleware 656, a discrete XR communication interface 660, and / or a device-to-cloud interface 670.

[0069] In some aspects, middleware 656 on device 650 can receive frames from the game engine on server 610 via a discrete XR communication interface 660. Middleware 656 can then submit these frames to runtime 652 via open XR API 654. For example, open XR API 654 on device 650 can be a specialization of discrete XR communication interface 660. By utilizing open XR API 654 on device 650, aspects of this disclosure can enable discrete XR applications that can be written for open XR. In some aspects, XR runtime 652 can expose open XR API 654 to middleware 656.

[0070] Furthermore, the middleware can poll or monitor poses via the Open XR API 654 and / or stream them to the server 610. Additionally, the game engine can be compatible with the Decoupled XR API 614. The game engine can also be agnostic to whether the user device is Open XR compatible. For example, the game engine can respond to pose streams to render frames.

[0071] Figure 7 An architecture 700 according to one or more technologies disclosed herein is shown. Similar to architecture 600 discussed above, architecture 700 can integrate open XR within a discrete XR. Also as Figure 7 As shown, architecture 700 includes device 750 and MEC server 710. Figure 7 The standardized protocol for split XR traffic communication between device 750 and MEC server 710 is also shown.

[0072] like Figure 7 As shown, the MEC server 710 may include a MEC-based XR application and engine 712, an open XR API 714, a discrete XR-compatible MEC vendor middleware 716, a discrete XR communication interface 730, and / or a MEC-to-cloud interface 720. The device 750 may include an XR runtime 752, an open XR API 754, a discrete XR-compatible device middleware 756, a discrete XR communication interface 760, and / or a device-to-cloud interface 770.

[0073] In some respects, applications or games on server 710 can be open XR compatible. For example, a game or application on server 710 may not know whether it is to be deployed on server 710 or device 750. Furthermore, games or applications on server 710 can poll or monitor gesture and controller information via an open XR interface. The game or application can then submit frames to the MEC vendor middleware 716.

[0074] In some cases, the open XR API 714 on server 710 can be a type of discrete XR API. By utilizing the open XR API 714 on server 710 and / or the open XR API 754 on device 750, aspects of this disclosure can enable discrete XR applications that can be written for open XR. For example, open XR API 714 and open XR API 754 can communicate via predefined interfaces (e.g., interface 730 and interface 760). In some aspects, XR runtime 752 can expose open XR API 754 to middleware 756. Furthermore, MEC vendor middleware 716 can ensure that the game engine is agnostic to device 750. MEC vendor middleware 716 can also ensure that the game engine is agnostic to what is deployed on server 710.

[0075] Figure 8 An architecture 800 according to one or more technologies disclosed herein is shown. Similar to architectures 600 and 700 discussed above, architecture 800 can integrate open XR into discrete XR. Also as Figure 8 As shown, architecture 800 includes device 850 and MEC server 810. Figure 8 The standardized protocol for split XR traffic communication between device 850 and MEC server 810 is also shown.

[0076] like Figure 8 As shown, the MEC server 810 may include a MEC-based XR application and engine 812, a discrete XR API 814, a discrete XR-compatible MEC vendor middleware 816, a discrete XR communication interface 830, and / or a MEC-to-cloud interface 820. The device 850 may include an XR runtime 852, an open XR API 854, an XR application 862, a discrete XR API 864, a discrete XR-compatible device middleware 856, a discrete XR communication interface 860, and / or a device-to-cloud interface 870.

[0077] In some respects, the game or application can be separated between server 810 and device 850. The application at device 850 can be open XR compatible when submitting frames to runtime 852 and / or polling gesture information or controller events from device 850. In some instances, XR runtime 852 can expose open XR API 854 to middleware 856. Furthermore, the application can communicate with its counterpart on MEC server 810 via a separate XR API (e.g., separate XR API 814 and / or separate XR API 864).

[0078] Figure 6 , Figure 7 and Figure 8 An example of the aforementioned process for standardizing the API used for split rendering is shown. Figure 6 , Figure 7 and Figure 8 As shown, various aspects of this disclosure (e.g., the server and user equipment described herein) can perform numerous different steps or processes to standardize the API used for split rendering. For example, the server and user equipment described herein can determine at least one split API (e.g., API 814 and API 864) for the user equipment (e.g., device 850) and the server (e.g., server 810). In some aspects, the at least one split API (e.g., API 814 and API 864) can be at least one split extended reality (XR) API (e.g., split XR API 814 and split XR API 864).

[0079] Furthermore, the server and user equipment described herein can establish communication interfaces (e.g., interfaces 830 and 860) between the user equipment (e.g., device 850) and the server (e.g., server 810) based on at least one discrete API (e.g., API 814 and API 864). The apparatus can also implement at least one discrete API (e.g., API 714 and API 764) on at least one of the user equipment (e.g., device 750) or the server (e.g., server 710). In some aspects, the at least one discrete API implemented on at least one of the user equipment (e.g., device 750) or the server (e.g., server 710) can be at least one open XR API (e.g., open XR API 714 and open XR API 764). The apparatus can also partition at least one application workload between the user equipment (e.g., device 750) and the server (e.g., server 710) based on at least one discrete API (e.g., open XR API 714 and open XR API 764).

[0080] Furthermore, the device can encode or decode application information at the user equipment or server via middleware 816 or middleware 856. In some aspects, the encoded or decoded application information can allow the user equipment and server to communicate with each other via interfaces 830 and 860. Similarly, application information can be encoded or decoded at the user equipment or server based on middleware (e.g., middleware 816 or middleware 856).

[0081] In some cases, the user equipment may include device middleware (e.g., device middleware 856), and the server may include server middleware (e.g., MEC vendor middleware 816). Furthermore, the user equipment may include an XR runtime application (e.g., XR runtime 852) based on device middleware (e.g., device middleware 856) and a separate API (e.g., separate XR API 864). In some aspects, application information may include at least one of camera information, game information, fragment information, or point cloud information, such as application information encoded at middleware 856 or middleware 816. Additionally, the server may include at least one server application based on reference information (e.g., application and engine 812).

[0082] Furthermore, the device can communicate between the user equipment and the server based on a communication interface and at least one separate API; for example, interfaces 830 and 860 can communicate information. In some aspects, the user equipment may include a device communication interface (e.g., communication interface 860), and the server may include a server communication interface (e.g., communication interface 830). Additionally, the user equipment and the server can communicate via the device communication interface (e.g., communication interface 860) and the server communication interface (e.g., communication interface 860).

[0083] Figure 9 A flowchart 900 illustrating an example method according to one or more techniques of this disclosure is shown. This method can be performed by a user device, server, CPU, GPU, or any device capable of performing graphics processing. Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 and Figure 8 As illustrated in the example, in 902, the device can determine at least one separate API for the user equipment and the server. In some aspects, combined with Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 and Figure 8 As illustrated in the example, at least one discrete API can be at least one discrete Extended Reality (XR) API.

[0084] Combination Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 and Figure 8 As illustrated in the example, in 904, the device can establish a communication interface between the user equipment and the server based on at least one discrete API. Combined with... Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 and Figure 8 As illustrated in the example, in 906, the device may also implement at least one separate API on at least one of the user equipment or the server.

[0085] In some aspects, combined Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 and Figure 8 As illustrated in the example, at least one discrete API implemented on at least one user device or server can be at least one open XR API. Combined Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 and Figure 8 As illustrated in the example, in 908, the device can partition at least one application workload between the user device and the server based on at least one separate API.

[0086] Combination Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 and Figure 8 As illustrated in the example, in 910, the device can encode or decode application information at a user device or server. In some aspects, combined with Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 and Figure 8 As illustrated in the examples, encoded or decoded application information allows user devices and servers to communicate with each other. Furthermore, in conjunction with... Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 and Figure 8 As illustrated in the example, application information can be encoded or decoded at the user device or server based on middleware.

[0087] In some aspects, combined Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 and Figure 8 As illustrated in the example, the user equipment may include device middleware, and the server may include server middleware. Furthermore, in conjunction with... Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 and Figure 8 As illustrated in the examples, user devices may include XR runtime applications based on device middleware and decoupled APIs. In some aspects, combining Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 and Figure 8 As illustrated in the example, application information may include at least one of camera information, game information, clip information, or point cloud information. Furthermore, in conjunction with... Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 and Figure 8 As illustrated in the example, the server may include at least one server application based on application information.

[0088] Combination Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 and Figure 8 As illustrated in the example, in 912, the device can communicate between a user equipment and a server based on a communication interface and at least one separate API. In some aspects, combined with Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 and Figure 8 As illustrated in the example, the user equipment may include a device communication interface, and the server may include a server communication interface. Furthermore, in conjunction with... Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 and Figure 8 As illustrated in the example, the user equipment and the server can communicate via the device communication interface and the server communication interface.

[0089] In one configuration, a method or apparatus for graphics processing is provided. The apparatus may be a user device, a server, a CPU, a GPU, or some other processor capable of performing graphics processing. In one aspect, the apparatus may be a processing unit 120 within device 104, or some other hardware within device 104 or another device. The apparatus may include components for determining at least one discrete application programming interface (API) for the user device and the server. The apparatus may also include components for establishing a communication interface between the user device and the server based on at least one discrete API. The apparatus may further include components for communicating between the user device and the server based on the communication interface and at least one discrete API. The apparatus may further include components for implementing at least one discrete API on at least one of the user device or the server. The apparatus may further include components for partitioning at least one application workload between the user device and the server based on at least one discrete API. The apparatus may also include components for encoding or decoding application information at the user device or the server.

[0090] The subjects described herein can be implemented to achieve one or more benefits or advantages. For example, the described graphics processing techniques can be used by user devices, servers, GPUs, CPUs, or some other processors capable of performing graphics processing to implement the standardized API techniques described herein. This can also be accomplished at a lower cost compared to other graphics processing techniques. Furthermore, the graphics processing techniques of this invention can improve or accelerate data processing or execution. Additionally, the graphics processing techniques of this invention can improve resource or data utilization and / or resource efficiency. Moreover, aspects of this disclosure can leverage standardized APIs to improve performance during discrete rendering processes.

[0091] According to this disclosure, unless otherwise specified in the context, the term "or" can be interpreted as "and / or". Furthermore, while phrases such as "one or more" or "at least one" may be used for some features disclosed herein but not others, features not using such language may be interpreted as having this implied meaning unless otherwise specified in the context.

[0092] In one or more examples, the functions described herein may be implemented in hardware, software, firmware, or any combination thereof. For example, although the term "processing unit" has been used throughout this disclosure, such a processing unit may be implemented in hardware, software, firmware, or any combination thereof. If any function, processing unit, technique, or other module described herein is implemented in software, then the function, processing unit, technique, or other module described herein may be stored on or transmitted through a computer-readable medium as one or more instructions or code. A computer-readable medium may include a computer data storage medium or a communication medium, including any medium that facilitates the transfer of a computer program from one place to another. In this way, a computer-readable medium may generally correspond to (1) a tangible computer-readable storage medium that is non-transitory, or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures that implement the techniques described herein. By way of example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices. The disks and optical discs used in this document include compact optical discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs reproduce data optically using lasers. Combinations of these should also be included within the scope of computer-readable media. Computer program products may include computer-readable media.

[0093] The code can be implemented by one or more processors (e.g., one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), arithmetic logic units (ALUs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits). Therefore, the term "processor" as used herein can refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Furthermore, these techniques can be implemented entirely within one or more circuit or logic elements.

[0094] The techniques disclosed herein can be implemented in a wide variety of devices or apparatuses, including wireless handheld devices, integrated circuits (ICs), or IC sets (chipsets). Various components, modules, or units are described in this disclosure to emphasize functional aspects of a device configured to perform the disclosed techniques, but they do not necessarily need to be implemented through different hardware units. Rather, as described above, various units can be combined in any hardware unit, or provided by a collection of interoperable hardware units including one or more processors as described above, combined with suitable software and / or firmware.

[0095] Various examples have been described. These and other examples are within the scope of the appended claims.

Claims

1. A method for image processing, comprising: For user devices and servers, at least one discrete rendering application programming interface (API) is defined, wherein the at least one discrete rendering API is at least one discrete rendering extended reality (XR) API. A communication interface is established between the user device and the server based on at least one of the separate rendering APIs; Implement the at least one separate rendering API on the user equipment and the server; Based on the at least one split rendering API, at least one application workload is partitioned between the user device and the server, wherein the at least one application workload includes a rendering workload; and Communication is conducted between the user device and the server based on the communication interface and the at least one separate rendering API, wherein the user device includes device middleware and the server includes server middleware, wherein application information is decoded at the user device based on the device middleware, wherein pose information is extracted at the device middleware, and wherein the pose information is transmitted from the device middleware to the server middleware.

2. The method according to claim 1, further comprising: The application information is encoded at the server.

3. The method according to claim 2, wherein, Encoded application information allows the user equipment and the server to communicate with each other.

4. The method according to claim 2, wherein, The application information is encoded at the server based on the server middleware.

5. The method according to claim 1, wherein, The user equipment includes an XR runtime application based on the device middleware and the at least one discrete rendering API.

6. The method according to claim 2, wherein, The application information includes at least one of camera information, game information, fragment information, or point cloud information.

7. The method according to claim 2, wherein, The server includes at least one server application based on the application information.

8. The method according to claim 1, wherein, The user equipment includes a device communication interface, and the server includes a server communication interface.

9. The method according to claim 8, wherein, The user equipment and the server communicate via the device communication interface and the server communication interface.

10. An apparatus for graphics processing, comprising: Memory; as well as At least one processor coupled to the memory is configured to: For user devices and servers, at least one discrete rendering application programming interface (API) is defined, wherein the at least one discrete rendering API is at least one discrete rendering extended reality (XR) API. A communication interface is established between the user device and the server based on at least one of the separate rendering APIs; Implement the at least one separate rendering API on the user equipment and the server; Based on the at least one split rendering API, at least one application workload is partitioned between the user device and the server, wherein the at least one application workload includes a rendering workload; and Communication is conducted between the user device and the server based on the communication interface and the at least one separate rendering API, wherein the user device includes device middleware and the server includes server middleware, wherein application information is decoded at the user device based on the device middleware, wherein pose information is extracted at the device middleware, and wherein the pose information is transmitted from the device middleware to the server middleware.

11. The apparatus according to claim 10, wherein, The at least one processor is further configured to: The application information is encoded at the server.

12. The apparatus according to claim 11, wherein, Encoded application information allows the user equipment and the server to communicate with each other.

13. The apparatus according to claim 11, wherein, The application information is encoded at the server based on the server middleware.

14. The apparatus according to claim 10, wherein, The user equipment includes an XR runtime application based on the device middleware and the at least one discrete rendering API.

15. The apparatus according to claim 11, wherein, The application information includes at least one of camera information, game information, fragment information, or point cloud information.

16. The apparatus according to claim 11, wherein, The server includes at least one server application based on the application information.

17. The apparatus according to claim 10, wherein, The user equipment includes a device communication interface, and the server includes a server communication interface.

18. The apparatus according to claim 17, wherein, The user equipment and the server communicate via the device communication interface and the server communication interface.

19. An apparatus for graphics processing, comprising: A component for determining at least one split-rendering application programming interface (API) for user devices and servers, wherein the at least one split-rendering API is at least one split-rendering extended reality (XR) API. A component for establishing a communication interface between the user device and the server based on at least one separate rendering API; A component for implementing the at least one separate rendering API on the user device and the server; A component for partitioning at least one application workload between the user device and the server based on the at least one split rendering API, wherein the at least one application workload includes a rendering workload; and Components for communicating between the user device and the server based on the communication interface and the at least one discrete rendering API, wherein the user device includes device middleware and the server includes server middleware, wherein application information is decoded at the user device based on the device middleware, wherein pose information is extracted at the device middleware, and wherein the pose information is transmitted from the device middleware to the server middleware.

20. The apparatus of claim 19, further comprising: Components used to encode the application information at the server.

21. The apparatus according to claim 20, wherein, Encoded application information allows the user equipment and the server to communicate with each other.

22. The apparatus according to claim 20, wherein, The application information is encoded at the server based on the server middleware.

23. The apparatus according to claim 19, wherein, The user equipment includes an XR runtime application based on the device middleware and the at least one discrete rendering API.

24. The apparatus according to claim 20, wherein, The application information includes at least one of camera information, game information, fragment information, or point cloud information.

25. The apparatus according to claim 20, wherein, The server includes at least one server application based on the application information.

26. The apparatus according to claim 19, wherein, The user equipment includes a device communication interface, and the server includes a server communication interface.

27. The apparatus according to claim 26, wherein, The user equipment and the server communicate via the device communication interface and the server communication interface.

28. A computer-readable medium having instructions stored thereon, which, when executed by a processor, cause the processor to perform the method according to any one of claims 1-9.

29. A computer program product comprising computer instructions, which, when executed by a processor, cause the processor to perform the method according to any one of claims 1-9.

Images