Sending PVS over UDP for segmentation rendering

Abstract

This disclosure provides systems, devices, apparatus, and methods, including computer programs encoded on a storage medium, for transmitting PVS through UDP for split rendering. A graphics processor may obtain a PVS and a set of video packets. If a data loss associated with the PVS or the set of video packets is detected, the graphics processor may determine one or more first visible primitives of the set of visible primitives that are currently affected by the data loss associated with the PVS or the set of video packets. The graphics processor may identify hidden information for the one or more first visible primitives in the set of visible primitives based on the one or more first visible primitives in the previous frame. The graphics processor may update information associated with the one or more first visible primitives based on the hidden information for the one or more first visible primitives.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of and priority to U.S. Provisional Application Serial No. 63 / 377,001, filed on September 23, 2022, entitled “PVS OVER UDP FOR SPLITRENDERING,” and U.S. Non-Provisional Patent Application Serial No. 18 / 168,532, filed on February 13, 2023, entitled “PVS OVER UDP FOR SPLIT RENDERING,” the entire texts of which are expressly incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to processing systems and, more particularly, to one or more techniques for graphics processing. Background Art

[0004] Computing devices typically perform graphics and / or display processing (e.g., using a graphics processing unit (GPU), a central processing unit (CPU), a display processor, etc.) to render and display visual content. Such computing devices may include, for example, computer workstations, mobile phones (such as smart phones), embedded systems, personal computers, tablet computers, and video game consoles. The GPU is configured to execute a graphics processing pipeline, which includes one or more processing stages that operate together to execute graphics processing commands and output frames. The central processing unit (CPU) can control the operation of the GPU by issuing one or more graphics processing commands to the GPU. Modern CPUs are typically capable of executing multiple applications concurrently, and each application may need to utilize a GPU during execution. The display processor may be configured to convert digital information received from the CPU into analog values, and may issue commands to a display panel to display visual content. A device that provides content for visual presentation on a display may utilize a CPU, a GPU, and / or a display processor.

[0005] Current techniques may not be able to address the delay and packet loss associated with segmenting potentially visible sets (PVS) in a rendering environment. Improved delay and packet loss handling techniques may be needed. Summary of the invention

[0006] The following presents a simplified summary of one or more aspects in order to provide a basic understanding of these aspects. This summary is not an extensive overview of all contemplated aspects, and is neither intended to identify key or important elements of all aspects, nor to describe the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to a more detailed description presented later.

[0007] In one aspect of the present disclosure, a method, a computer-readable medium, and an apparatus are provided. The apparatus may obtain a PVS and a set of video packets. The PVS may include PVS information associated with a set of visible primitives that are potentially visible in a first frame. If data loss associated with the PVS or the set of video packets is detected, the apparatus may determine one or more first visible primitives in the set of visible primitives that are currently affected by the data loss associated with the PVS or the set of video packets. The apparatus may identify hidden information for one or more first visible primitives in the set of visible primitives based on the one or more first visible primitives in a previous frame. The apparatus may update information associated with the one or more first visible primitives based on the hidden information for the one or more first visible primitives.

[0008] To achieve the foregoing and related ends, one or more aspects include the features fully described below and particularly pointed out in the claims. The following description and the accompanying drawings set forth in detail some illustrative features of one or more aspects. However, these features are merely indicative of some of the various ways in which the principles of the various aspects may be employed, and this specification is intended to include all such aspects and their equivalents. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a block diagram illustrating an example content generation system in accordance with one or more techniques of this disclosure.

[0010] Figure 2 is a block diagram illustrating an example environment in which aspects of the disclosure may be practiced in accordance with one or more aspects.

[0011] Figure 3 is a block diagram illustrating example operations at a server in a split rendering application in accordance with one or more aspects.

[0012] Figure 4 is a diagram illustrating an example meshlet bitstream according to one or more aspects.

[0013] Figure 5 is a flow diagram illustrating example operations performed at a client device in accordance with one or more aspects.

[0014] Figure 6is a diagram illustrating an example meshlet according to one or more aspects.

[0015] Figure 7 is a diagram illustrating an example meshlet error in accordance with one or more aspects.

[0016] Figure 8 is a call flow diagram illustrating example communications between a client device and a server in accordance with one or more techniques of this disclosure.

[0017] Fig. 9 is a flowchart of an example method of graphics processing in accordance with one or more techniques of this disclosure.

[0018] Fig.10 is a flowchart of an example method of graphics processing in accordance with one or more techniques of this disclosure. DETAILED DESCRIPTION

[0019] Various aspects of the system, device, computer program product and method will be described more fully below with reference to the accompanying drawings. However, the present disclosure can be embodied in many different forms and should not be interpreted as being limited to any specific structure or function presented throughout the present disclosure. On the contrary, these aspects are provided so that the present disclosure will be thorough and complete, and the scope of the present disclosure will be fully conveyed to those skilled in the art. Based on the teachings of this article, it should be understood by those skilled in the art that the scope of the present disclosure is intended to cover any aspect of the system, device, computer program product and method disclosed herein, whether it is implemented independently of other aspects of the present disclosure or implemented in combination with other aspects of the present disclosure. For example, any number of aspects set forth herein can be used to implement a device or practice method. In addition, the scope of the present disclosure is intended to cover such devices or methods implemented using other structures, functionality, or structures and functionality other than the various aspects of the disclosure set forth herein or different from the various aspects of the disclosure set forth herein. Any aspect disclosed herein can be embodied by one or more elements of the claims.

[0020] Although various aspects are described herein, many variations and permutations of these aspects fall within the scope of the present disclosure. Although some potential benefits and advantages of various aspects of the present disclosure are mentioned, the scope of the present disclosure is not intended to be limited to specific benefits, uses or objectives. On the contrary, various aspects of the present disclosure are intended to be widely applicable to different wireless technologies, system configurations, processing systems, networks and transmission protocols, some of which are illustrated by way of example in the drawings and the description below. The specific embodiments and drawings merely illustrate the present disclosure and do not limit the present disclosure, and the scope of the present disclosure is defined by the attached claims and their equivalent technical solutions.

[0021] Several aspects are presented with reference to various devices and methods. These devices and methods are described in the following detailed description and illustrated in the accompanying drawings by various blocks, components, circuits, processes, algorithms, etc. (collectively referred to as "elements"). These elements can be implemented using electronic hardware, computer software, or any combination thereof. Whether these elements are implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system.

[0022] For example, an element or any part of an element or any combination of elements may be implemented as a "processing system" including one or more processors (which may also be referred to as processing units). Examples of processors include microprocessors, microcontrollers, graphics processing units (GPUs), general-purpose GPUs (GPGPUs), central processing units (CPUs), application processors, digital signal processors (DSPs), reduced instruction set computing (RISC) processors, systems on chips (SOCs), baseband processors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gated logic, discrete hardware circuits, and other suitable hardware configured to perform the various functionalities described in this disclosure. One or more processors in a processing system may execute software. Whether referred to as software, firmware, middleware, microcode, hardware description language, or other names, software may be broadly understood to mean instructions, instruction sets, codes, code segments, program codes, programs, subroutines, software components, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, processes, functions, etc.

[0023] The term "application" may refer to software. As described herein, one or more technologies may refer to an application (e.g., software) configured to perform one or more functions. In such examples, the application may be stored in a memory (e.g., an on-chip memory of a processor, a system memory, or any other memory). The hardware described herein (such as a processor) may be configured to execute an application. For example, an application may be described as including code that, when executed by hardware, causes the hardware to perform one or more technologies described herein. As an example, the hardware may access code from a memory and execute the code accessed from the memory to perform one or more technologies described herein. In some examples, components are identified in the present disclosure. In such examples, components may be hardware, software, or a combination thereof. Each component may be a separate component or a subcomponent of a single component.

[0024] In one or more examples described herein, the functions described can be implemented in hardware, software, or any combination thereof. If implemented in software, the functions can be stored or encoded on a computer-readable medium as one or more instructions or codes. Computer-readable media include computer storage media. Storage media can be any available media that can be accessed by a computer. By way of example and not limitation, such computer-readable media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable ROM (EEPROM), optical disk storage, magnetic disk storage, other magnetic storage devices, a combination of computer-readable media of the above types, or any other medium that can be used to store computer executable code in the form of instructions or data structures that can be accessed by a computer.

[0025] As used herein, instances of the term "content" may refer to "graphic content," "image," and the like, regardless of whether the term is used as an adjective, noun, or other part of speech. In some examples, as used herein, the term "graphic content" may refer to content produced by one or more processes of a graphics processing pipeline. In other examples, as used herein, the term "graphic content" may refer to content produced by a processing unit configured to perform graphics processing. In yet other examples, as used herein, the term "graphic content" may refer to content produced by a graphics processing unit.

[0026] When delays and / or packet loss occur in the PVS channel, visual artifacts may be produced in the image displayed at the client device. For example, if the client device cannot obtain the correct encoded UV texture coordinates, the surface in the displayed image may have incorrect shading. Therefore, techniques for handling delays and / or packet loss in the PVS channel may be needed so that visual artifacts can be reduced, minimized and / or eliminated.

[0027] According to one or more aspects, a client device may obtain a PVS and a set of video packets. The PVS may include PVS information associated with a set of visible primitives that are potentially visible in a first frame. If data loss associated with the PVS or the set of video packets is detected, the client device may determine one or more first visible primitives in the set of visible primitives that are currently affected by the data loss associated with the PVS or the set of video packets. The client device may identify hidden information for one or more first visible primitives in the set of visible primitives based on one or more second visible primitives. The one or more second visible primitives may correspond to the one or more first visible primitives and may be associated with a frame prior to the first frame. The client device may update information associated with the one or more first visible primitives based on the hidden information for the one or more first visible primitives.

[0028] Figure 11 is a block diagram illustrating an example content generation system 100 configured to implement one or more techniques of the present disclosure. The content generation system 100 includes a device 104. The device 104 may include one or more components or circuits for performing various functions described herein. In some examples, one or more components of the device 104 may be components of a SOC. The device 104 may include one or more components configured to perform one or more techniques of the present disclosure. In the example shown, the device 104 may include a processing unit 120, a content encoder / decoder 122, and a system memory 124. In some aspects, the device 104 may include multiple components (e.g., a communication interface 126, a transceiver 132, a receiver 128, a transmitter 130, a display processor 127, and one or more displays 131). The display 131 may refer to one or more displays 131. For example, the display 131 may include a single display or multiple displays, which may include a first display and a second display. The first display may be a left-eye display, and the second display may be a right-eye display. In some examples, the first display and the second display may receive different frames for presentation thereon. In other examples, the first display and the second display may receive the same frame for presentation thereon. In another example, the result of the graphics processing may not be displayed on the device, for example, the first display and the second display may not receive any frame for presentation thereon. Instead, the frame or graphics processing result may be passed to another device. In some aspects, this situation is referred to as split rendering.

[0029] The processing unit 120 may include an internal memory 121. The processing unit 120 may be configured to perform graphics processing using the graphics processing pipeline 107. The content encoder / decoder 122 may include an internal memory 123. In some examples, the device 104 may include a processor that may be configured to perform one or more display processing techniques on one or more frames generated by the processing unit 120 and then display the frames through one or more displays 131. Although the processor in the example content generation system 100 is configured as a display processor 127, it should be understood that the display processor 127 is an example of a processor and other types of processors, controllers, etc. may be used in place of the display processor 127. The display processor 127 may be configured to perform display processing. For example, the display processor 127 may be configured to perform one or more display processing techniques on one or more frames generated by the processing unit 120. The one or more displays 131 may be configured to display or otherwise present the frames processed by the display processor 127. In some examples, one or more displays 131 may include one or more of the following: a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, a projection display device, an augmented reality display device, a virtual reality display device, a head-mounted display, or any other type of display device.

[0030] Memory external to processing unit 120 and content encoder / decoder 122, such as system memory 124, may be accessible to processing unit 120 and content encoder / decoder 122. For example, processing unit 120 and content encoder / decoder 122 may be configured to read from and / or write to external memory, such as system memory 124. Processing unit 120 may be communicatively coupled to system memory 124 via a bus. In some examples, processing unit 120 and content encoder / decoder 122 may be communicatively coupled to internal memory 121 via the bus or via a different connection.

[0031] The content encoder / decoder 122 may be configured to receive graphics content from any source, such as the system memory 124 and / or the communication interface 126. The system memory 124 may be configured to store the received encoded or decoded graphics content. The content encoder / decoder 122 may be configured to receive the encoded or decoded graphics content in the form of encoded pixel data, for example, from the system memory 124 and / or the communication interface 126. The content encoder / decoder 122 may be configured to encode or decode any graphics content.

[0032] The internal memory 121 or the system memory 124 may include one or more volatile or non-volatile memories or storage devices. In some examples, the internal memory 121 or the system memory 124 may include RAM, static random access memory (SRAM), dynamic random access memory (DRAM), erasable programmable ROM (EPROM), EEPROM, flash memory, magnetic data medium or optical storage medium or any other type of memory. According to some examples, the internal memory 121 or the system memory 124 may be a non-transitory storage medium. The term "non-transitory" may indicate that the storage medium is not embodied in a carrier or propagation signal. However, the term "non-transitory" should not be interpreted as meaning that the internal memory 121 or the system memory 124 is non-removable or that its contents are static. For example, the system memory 124 may be removed from the device 104 and moved to another device. As another example, the system memory 124 may not be removable from the device 104.

[0033] The processing unit 120 may be a CPU, a GPU, a GPGPU, or any other processing unit that may be configured to perform graphics processing. In some examples, the processing unit 120 may be integrated into the motherboard of the device 104. In another example, the processing unit 120 may be present on a graphics card installed in a port of the motherboard of the device 104, or may be otherwise incorporated into a peripheral device configured to interoperate with the device 104. The processing unit 120 may include one or more processors, such as one or more microprocessors, GPUs, ASICs, FPGAs, arithmetic logic units (ALUs), DSPs, discrete logic pieces, software, hardware, firmware, other equivalent integrated or discrete logic circuits, or any combination thereof. If the technology is partially implemented in software, the processing unit 120 may store instructions for the software in a suitable non-transitory computer-readable storage medium (e.g., internal memory 121), and may use one or more processors to execute instructions in hardware to perform the technology of the present disclosure. Any of the above (including hardware, software, a combination of hardware and software, etc.) may be considered to be one or more processors.

[0034] The content encoder / decoder 122 may be any processing unit configured to perform content decoding. In some examples, the content encoder / decoder 122 may be integrated into the mainboard of the device 104. The content encoder / decoder 122 may include one or more processors, such as one or more microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), arithmetic logic units (ALUs), digital signal processors (DSPs), video processors, discrete logic, software, hardware, firmware, other equivalent integrated or discrete logic circuits, or any combination thereof. If the technology is partially implemented in software, the content encoder / decoder 122 may store instructions for the software in a suitable non-transitory computer-readable storage medium (e.g., internal memory 123), and may use one or more processors to execute instructions in hardware to perform the technology of the present disclosure. Any of the above (including hardware, software, a combination of hardware and software, etc.) may be considered to be one or more processors.

[0035] In some aspects, the content generation system 100 may include a communication interface 126. The communication interface 126 may include a receiver 128 and a transmitter 130. The receiver 128 may be configured to perform any receiving functions described herein with respect to the device 104. In addition, the receiver 128 may be configured to receive information from another device, such as eye or head position information, rendering commands, and / or position information. The transmitter 130 may be configured to perform any sending functions described herein with respect to the device 104. For example, the transmitter 130 may be configured to send information to another device, which may include a request for content. The receiver 128 and the transmitter 130 may be combined into a transceiver 132. In such examples, the transceiver 132 may be configured to perform any receiving functions and / or sending functions described herein with respect to the device 104.

[0036] Reference again Figure 1In some aspects, the processing unit 120 may include a PVS processor 198 configured to obtain a PVS and a set of video packets. The PVS may include PVS information associated with a set of visible primitives that are potentially visible in a first frame. The PVS processor 198 may be configured to: if data loss associated with the PVS or the set of video packets is detected, determine one or more first visible primitives in the set of visible primitives that are currently affected by the data loss associated with the PVS or the set of video packets. The PVS processor 198 may be configured to identify hidden information for one or more first visible primitives in the set of visible primitives based on one or more second visible primitives. The one or more second visible primitives may correspond to the one or more first visible primitives and may be associated with a frame prior to the first frame. The PVS processor 198 may be configured to update information associated with the one or more first visible primitives based on the hidden information for the one or more first visible primitives. Although the following description may focus on graphics processing, the concepts described herein may be applicable to other similar processing techniques.

[0037] Devices such as device 104 may refer to any device, apparatus or system configured to perform one or more technologies described herein. For example, a device may be a server, a base station, a user equipment, a client device, a station, an access point, a computer (such as a personal computer, a desktop computer, a laptop computer, a tablet computer, a computer workstation or a mainframe computer), a final product, an apparatus, a phone, a smart phone, a server, a video game platform or console, a handheld device (such as a portable video game device or a personal digital assistant (PDA)), a wearable computing device (such as a smart watch, an augmented reality device or a virtual reality device), a non-wearable device, a display or display device, a television, a television set-top box, an intermediate network device, a digital media player, a video streaming device, a content streaming device, a vehicle-mounted computer, any mobile device, any device configured to generate graphics content, or any device configured to perform one or more technologies described herein. The process herein may be described as being performed by a specific component (e.g., a GPU), but in other embodiments, other components (e.g., a CPU) that conform to the disclosed embodiments may be used to perform.

[0038] Figure 2200 in which aspects of the present disclosure may be practiced according to one or more aspects. Split rendering may be implemented between a client device 204 (e.g., a display, a head mounted display (HMD), etc.) and a server 202 (e.g., for applications such as games). In general, the client device 204 may receive a rendered image from the server 202 in a downlink 208. In some configurations, the client device 204 may combine the received image with a locally generated and rendered image to form a single image presented to the user. In some other configurations, the server 202 may send one or more of the encoded geometry information, visibility information, or shading information to the client device 204 in a downlink 208. The client device 204 may receive control inputs (e.g., via a controller), which may then be communicated to the server 202 in an uplink 206 (e.g., after potential pre-processing). In some configurations, the control input may also include a pose or predicted pose (eg, a six degrees of freedom (6DOF) pose) associated with the client device 204 (eg, a (predicted) pose of an HMD).

[0039] In segmented rendering, the server 202 may determine a PVS based on a predicted pose associated with a (thin) client device 204. The PVS may include one or more of the following: 1) a surface that the client device 204 can render, and 2) coded UV texture coordinates (which may also be referred to as UV mapping information) for sampling a texture atlas (a texture atlas may refer to an image containing different textures for texture mapping) for the surface. The server 202 may communicate the PVS to the client device 204 in a downlink 208 for frame rate upsampling. Thus, segmented rendering may decouple shading (e.g., lighting calculations) from rasterization (e.g., converting primitives to raster images). When latency and / or packet loss occur in the PVS channel, visual artifacts may be generated in the image displayed at the client device 204. For example, if the client device 204 cannot obtain the correct coded UV texture coordinates, the surface in the displayed image may have incorrect shading. Therefore, techniques for handling latency and / or packet loss in the PVS channel may be needed so that visual artifacts can be reduced, minimized, and / or eliminated.

[0040] Vector streaming may be a split rendering architecture for offloading computationally intensive rendering to a cloud server (e.g., server 202) and streaming the results to a thin client (e.g., client device 204). On an uplink (e.g., uplink 206), a client (e.g., client device 204) may transmit a predicted pose to a server (e.g., server 202). For each frame, the server (e.g., server 202) may determine / identify visible triangles (i.e., visible primitives) using the most recent predicted pose transmitted by the client (e.g., client device 204). The server (e.g., server 202) may shade the visible triangles and may transmit to the client (e.g., client device 204) via a downlink (e.g., downlink 208): a) a texture atlas containing the shading (or lighting) of the triangles (e.g., using standard H.26x compression), and b) texture coordinates associated with the visible triangles, so that the visible triangles may be associated with their corresponding shading in the texture atlas based on the texture coordinates.

[0041] In some configurations, the message containing the texture coordinates may also indicate the triangles for which the texture coordinates are intended. Thus, by processing the message including the texture coordinates, the client (e.g., client device 204) may also know which triangles may be (potentially) visible, and thus the client (e.g., client device 204) may render these triangles. This set of potentially visible primitives (e.g., potentially visible triangles) may be referred to as PVS hereinafter. In some configurations, the client (e.g., client device 204) may render a new view using the PVS until a new server frame arrives at the client (e.g., client device 204) from the server (e.g., server 202).

[0042] Embodiments of the present disclosure may be used in various environments. One example environment may include a segmented extended reality (XR) scene. For example, XR processing may be segmented between a computer / edge server and an HMD. Another example environment may include a segmented augmented reality (AR) scene. Occlusion rendering may be used in a segmented AR scene. In addition, having a geometric representation at the client device can help hide latency. Specifically, when the scene geometry is known to the client, latency hiding techniques (also known as new view synthesis or generalized reprojection) can produce geometrically accurate renderings from the latest viewpoint, regardless of the latency in question. Yet another example environment may include a cloud rendering service that may respond (answer) with different latency, depending on the application specification. For example, in automotive / architectural design applications, a tolerable latency may be approximately 50ms-100ms, while in e-sports applications, a tolerable latency may be approximately 10ms. Yet another example environment may include cloud-connected games (e.g., personal computer (PC) games or mobile games). Specifically, due to the limited physical memory and computing power of client devices (e.g., mobile devices), scenes in the game may not be fully rendered locally, and rendered frames may be streamed to the client device in a real-time manner as needed. Features used with cloud-connected games may include, for example, hybrid rendering (mixing local rendering and remote rendering (i.e., rendering on a server / cloud)) (which can be used, for example, for cloud avatar rendering), large geospatial datasets (which can be used, for example, for flight simulators), or virtualized geometry systems.

[0043] In one or more aspects of the present disclosure, messages including PVS packets (e.g., messages containing texture coordinates) may be sent via a User Datagram Protocol (UDP). UDP may be preferred over a Transmission Control Protocol (TCP) because, although TCP may be more reliable and may deliver packets in order, TCP may introduce longer delays when packet loss occurs, which may increase motion-to-rendering-to-photon (M2R2P) latency in streaming virtual / augmented / extended reality use cases, thereby degrading the user experience.

[0044] If a subset of video packets for a given frame (e.g., a message containing a texture atlas) fails to arrive at the client (e.g., within a specified time period), the client can detect the area of ​​the frame that is affected by the lost video group. The client can then conceal the affected area. In other words, the client device may receive an incomplete frame and still continue to operate without stopping the user experience at the client device (assuming the error is not severe enough to cause error concealment to exceed its limits).

[0045] In some configurations, for real-time vector streaming in the presence of network errors, PVS packets may have similar behavior to video packets: when a PVS packet is dropped (i.e., when a PVS packet loss occurs at a client device), the client device may detect triangles in the PVS that are affected by the lost PVS packet. The client device may then conceal the error. Thus, the client device may receive an incomplete set of PVS packets without disrupting the user experience at the client device or causing unrecoverable errors. Aspects of the present disclosure may relate to protocols, error detection, and error concealment for PVS packets.

[0046] Figure 3 is a block diagram 300 illustrating example operations at a server in a split rendering application in accordance with one or more aspects. Figure 3 The operations in can be used in e.g. Figure 2 202. In some configurations, the PVS may include meshlets. As shown, the server may generate visible meshlets 302. Specifically, the server may divide the geometry associated with the frame into meshlets 302. Meshlets 302 may be used because meshlets may be more compressible than other representations, and meshlets may be more consistent with graphics processing unit (GPU) driven rendering. In general, if a meshlet is visible, then the polygons that make it up may also be visible. In addition, meshlets may encode local surface attributes (e.g., visibility, position, texture coordinates (e.g., texcoords, UV texture coordinates), etc.). With the assistance of the memory management block 304, the meshlet compression and packaging block 306 may generate (compressed) meshlet packets, which may be sent to the client device via the transmitter 308.

[0047] Figure 4is a diagram illustrating an example meshlet bitstream 400 according to one or more aspects. The meshlet bitstream 400 may correspond to a datagram and may represent a meshlet transported in a communication channel when a server sends a meshlet to a client device. As shown, a meshlet header portion 402 may include a bit indicating whether the meshlet corresponds to an alpha channel (bit 31: "IsAlpha"), bits indicating the size of the meshlet (bits 23 to 30: "MeshletSize"), and bits indicating an identifier (ID) of the meshlet (bits 0 to 22: "MeshletID"). A visible portion 404 may indicate whether the meshlet is visible. An invisible portion 406 may indicate whether the meshlet is invisible. The (texture) atlas information portion 408 may include a bit indicating whether a flip operation is to be performed when mapping the texture (bit 29: “Flip”), bits indicating the position in the (texture) atlas (bits 8 to 28: “Position”), bits indicating the width of the meshlet (bits 4 to 7: “Width Log2”), and bits indicating the height of the meshlet (bits 0 to 3: “Height Log2”). In addition, the single color portion 410 may indicate the color (bits 14 to 29: “R5G6B5”). The visible portion 404, the invisible portion 406, the atlas information portion 408, and the single color portion 410 may contribute to the size of the meshlet. In some configurations, the meshlet size may be no higher than 128 bits.

[0048] If the meshlet corresponds to an alpha channel (i.e., the "IsAlpha" bit is equal to 1), the alpha atlas information portion 412 may include a bit indicating whether a flip operation is to be performed when mapping the texture (bit 30: "Flip"), and bits indicating the position in the alpha atlas and the width and height of the meshlet (bits 1 to 29: "Position", "WidthLog2", and "HeightLog2"). In addition, the single alpha portion 414 may indicate an alpha value (bits 23 to 30: "Alpha").

[0049] In some configurations, a server (e.g., server 202) may bias the byte statistics to improve compression. Specifically, many shading units within a datagram may have the same superblock. Because the same superblock may appear frequently, the same level may also appear frequently. In addition, some bits in the meshlet bitstream (e.g., meshlet bitstream 400) may be compressed (e.g., using a zlib library). For example, Figure 4As shown, the bits corresponding to the position, width, and height in the atlas information portion 408, the bits corresponding to the color in the single color portion 410, the bits corresponding to the position, width, and height in the alpha atlas information portion 412, and the bits corresponding to the alpha value in the single alpha portion 414 may be compressed.

[0050] In some aspects, because most meshlets may not change level between frames, a heartbeat may be used to reduce the amount of data representing a meshlet sent from a server to a client device. If a meshlet does not change level from one frame to another, the server may transmit a heartbeat message for that meshlet for the following frame. Thus, a heartbeat may be a (small size) message that may indicate (i) that the meshlet in question may still be (potentially) visible, and (ii) that the position of the meshlet in question in the atlas may not have changed.

[0051] Specifically, in some configurations, for each frame, the server may send a message to the client device. The message may include the set of (potentially) visible primitives (e.g., triangles), and the texture coordinates (i.e., PVS) of the (potentially) visible primitives. In some configurations, the data layout of the PVS message may adopt a compressed form customized for vector streaming applications. For example, the PVS message may include a set of structures as shown in the following example:

[0052]

[0053] Where triangle_id may be a globally unique identifier for the triangle in the scene, and texture_coordinates may be the position of the triangle in the texture atlas.

[0054] Meshlet signaling and triangle signaling can be two different implementations of PVS signaling. In some configurations, a meshlet may include triangles. In some configurations, if visible triangles are signaled separately, the corresponding meshlet may not be signaled. The aspects of error concealment described herein may be used with meshlet signaling or triangle signaling. In some configurations, even if triangles are signaled separately, the concept of meshlet can still be used as a unit of visibility at the server, i.e., the server can determine which meshlets, and by association, which triangles are potentially visible in each frame. Therefore, in some cases herein, the terms "visible / invisible meshlet" and "visible / invisible triangle" may be used interchangeably.

[0055] Since most triangles from the previous frame are still visible in the current frame, and the positions of triangles may not change in the texture atlas, the server can use heartbeat messages to encode temporal redundancy. An example heartbeat message may be as follows:

[0056] struct HeartbeatMessage{

[0057] uint32_t triangle_id;

[0058] };

[0059] The heartbeat message may signal to the client device that the triangle is still visible and that the position of the triangle may not have changed in the texture atlas. The use of heartbeat messages may result in significant bitrate savings, since information about the position in the texture atlas (i.e., atlas information) may affect the PVS bitrate. After sending N (e.g., 10, 15, etc.) consecutive heartbeats for a triangle, the server may resend the full message (i.e., VisibleTriangleMessage). The resending of the full message may represent a tradeoff between bitrate and error recovery speed.

[0060] Figure 5 5 is a flowchart 500 illustrating example operations performed at a client device. For example, Figure 5 The operations in may be performed at the client device 204. The receiver 502 may receive (compressed) packaged meshlets (meshlet packets) and video packets (e.g., corresponding to (texture) atlases) for a frame (e.g., from the server 202). These meshlets may correspond to a PVS. Specifically, a meshlet message may constitute or describe a PVS. In other words, a meshlet message may provide (complete) information about (potentially) visible primitives. The client device may attempt to decode the PVS at the PVS decoder thread 504. In some configurations, the client device may decode the meshlet packets into a GPU-friendly memory representation. At 506, the client device may determine whether all meshlets have been received (e.g., based on a "visible meshlet count" in a datagram header). If all meshlets have been received or if a timeout has occurred (e.g., a "meshlet message timeout"), then at 508, the client device may signal to the rendering thread 516 that the PVS is ready to render the frame. In some configurations, if a packet arrives late (eg, because a newer frame has been decoded, or because the packet timed out), the client device may ignore the packet.

[0061] In parallel, the client device may attempt to decode the video packets at the video decoder thread 510. At 512, the client device may determine whether all video packets have been received. If all video packets have been received or if a timeout has occurred, at 514, the client device may signal to the rendering thread 516 that the (texture) atlas is ready to render the frame.

[0062] At 518, the rendering thread 516 may determine whether the (texture) atlas and PVS are ready for rendering. If so, the client device may perform an error detection operation 520. Two example types of errors may include a dropped meshlet packet error and a dropped video slice error. The error detection operation 520 may be explained in further detail below. Next, the client device may perform an error concealment operation 522 and a drawing command 524. The error concealment operation 522 may be explained in further detail below. Thereafter, the client device may perform a rendering operation 526.

[0063] In some configurations, if there is no new data for the current frame, the process may proceed directly from 518 to rendering operation 526 .

[0064] In some configurations, meshlet error detection operations may be performed (e.g., in Figure 5 520 in the error detection operation 520). For example, a meshlet error may occur when there is packet loss. If the meshlet was visible in the previous frame and the client device did not receive a message for the meshlet for the current frame (e.g., a heartbeat message or a message including the retransmission level of the meshlet), the meshlet may be in error. In another example, if the meshlet was in error for the previous frame and the client device did not receive a message for the meshlet for the current frame, the meshlet may still be in error.

[0065] Another type of meshlet error may be associated with a heartbeat failure. Heartbeats can be used to reduce the transmit bit rate. However, heartbeats can introduce complexity to error detection. For example, a heartbeat failure may occur when a client device has not received a message for the meshlet for the previous frame (i.e., a dropped / lost (PVS) packet) and has received a heartbeat message for the meshlet for the current frame. Because the dropped packets for the previous frame may have included a new level that is different from the level from the earlier frame, the dropped packets may have invalidated the older level. However, because the packets are dropped, the client device may have no way of knowing the correct level even for the current frame (for which only the heartbeat message was received).

[0066] In rare cases, an entire frame may be lost in transmission from a server to a client device. For example, a client device may encounter a series of messages that include L(X)H in this order, where L(X)H may correspond to the state of a particular meshlet across three consecutive frames. Specifically, L may include a level for a frame, X may indicate that no updates have been received for a particular meshlet, and H may be a heartbeat message for another frame. The level of the meshlet may have changed in frame X, but based on H, the client device may continue to use the level from frame L for frame H. As a result, the level that the client device uses for frame H may be incorrect.

[0067] As another example, a heartbeat failure may occur when the meshlet previously experienced a heartbeat failure and the client device has not yet received a message including a new level frame for the meshlet.

[0068] In other words, in some configurations, when a client device detects a lost PVS packet, the client device may detect triangles in the PVS that are affected by the packet loss. To detect the affected triangles (i.e., erroneous triangles), the client device may use a heuristic: if a triangle was visible in the previous frame and the client device has not received a message for the triangle in the current frame, then the triangle may be erroneous. The heuristic may be based on the assumption that triangles that were visible in the previous frame are still visible in the current frame. This assumption may not always hold true: some triangles that were previously visible may become invisible in the current frame (e.g., when a meshlet becomes visible / invisible, all triangles that make it up may change their states accordingly, according to the association). However, the client device may not know which triangles may become invisible in the current frame, because the PVS only encodes visible triangles. These triangles (i.e., triangles that have become invisible in the current frame but are detected as erroneous by the client device) may be referred to as "false positive" triangles.

[0069] In other examples, a triangle may also be erroneous when its texture coordinates correspond to (overlap) a lost region of an incomplete video frame (eg, due to lost video packets).

[0070] Figure 6 600 is a diagram illustrating example meshlets according to one or more aspects. As shown, a frame 602 of a scene may include a first meshlet 604a, a second meshlet 604b, and a third meshlet 604c.

[0071] Figure 7700 is a diagram illustrating an example meshlet error according to one or more aspects. As shown, a client device may receive a message indicating a first meshlet 604a, a second meshlet 604b, and a third meshlet 604c for frame N. For frame N, the client device may receive level updates for all three meshlets (e.g., when new or different locations in the atlas are allocated to describe corresponding primitives). The client device may add the three meshlets to a meshlet visibility buffer.

[0072] For frame N+1 (ie, the frame immediately following frame N), the client device may receive a heartbeat message from the first meshlet 604a. Therefore, the client device may reuse the level of the first meshlet 604a from frame N for frame N+1.

[0073] The second meshlet 604b and the third meshlet 604c are visible in frame N. However, the client device may not receive the message (i.e., the dropped packet) for the second meshlet 604b or the third meshlet 604c for frame N+1. Therefore, the client device may notice that the packet is lost. As a result, for frame N+1, the second meshlet 604b and the third meshlet 604c may be in error.

[0074] For frame N+2 (i.e., the frame immediately following frame N+1), the client device may receive a heartbeat message from the first meshlet 604a. Therefore, the client device may reuse the level of the first meshlet 604a from frame N (and frame N+1) for frame N+2.

[0075] Additionally, the client device may receive a message including the level of the second meshlet 604b for frame N+2. Thus, for frame N+2, the second meshlet 604b may no longer be faulty.

[0076] However, the third meshlet 604c may still be in error for frame N+2 because the dropped packets of the third meshlet 604c for frame N+1 may have included the level change when the client device has received the heartbeat message of the third meshlet 604c for frame N+2. In other words, because the packets of the third meshlet 604c for frame N+1 are lost, the client device may not know the correct level of the third meshlet 604c for frame N+2. Therefore, the third meshlet 604c may still be in error for frame N+2.

[0077] In some configurations, the client device may perform error concealment operations (e.g., Figure 5 ). In one configuration, to perform error concealment, the client device may use a screen space cache (e.g., reprojection). However, even if the correct depth and motion vectors are available, geometric errors may still persist due to lack of occlusion. In another configuration, to perform error concealment, the client device may use a texture space cache. Specifically, the client device may use texture coordinates associated with the previous frame to look up the shading of the previous frame. The client device may then use the shading again for the current frame. This technique may be geometrically accurate. In other configurations, when vector streaming is used, the client device may use a texture space cache to hide errors.

[0078] In other words, in some configurations, after error detection, the client device may have two sets of visible triangles for the current frame, namely, triangles with errors and triangles without errors.

[0079] In some configurations, when the set of visible triangles is drawn to the screen, the error concealment technique may use any of the following mechanisms to determine the final pixel color of the erroneous triangle: a) texture space cache, b) screen space cache, and c) image repair. Specifically, for the erroneous triangle, the client device may first perform a cache lookup in the texture space cache, which may include the last successfully received texture value for the triangle. If there is a cache hit, the client device may use the texture value stored in the texture space cache for the triangle. If a texture space cache miss occurs, the client device may fall back to the screen space cache, which may use temporal reprojection to sample the position of the triangle (in screen space) in the previous frame. If a screen space cache miss occurs (e.g., because the triangle is off-screen or depth discontinuous), the client device may use image repair to render the current pixel of the current triangle.

[0080] In some configurations, when all PVS groups are received correctly, the client device can mark the "false positive" triangle as invisible by checking whether the "false positive" triangle corresponds to a triangle in the VisibleTriangleMessage collection of the current frame. If the triangle does not exist in the group, the client device can mark the "false positive" triangle as invisible.

[0081] In some configurations, a texture cache may include 5 channels: RGBA, and an additional ReprojectionMask channel. The ReprojectionMask channel may be a binary value that indicates whether the cache contains a valid texture value at that texel.

[0082] In some configurations, the client device may assign texture IDs from a list of available texture IDs for triangles that just became corrupted in the current frame, where the available texture IDs may correspond to locations in the texture cache. The client device may then look up texture values ​​for these triangles from the decoded texture atlas of the previous frame, and if found, the client device may store these texture values ​​in the RGBA channels of the texture cache at the locations specified by the texture IDs, and may set ReprojectionMask to 0. If the client device cannot locate a valid texture value for a triangle from the previous frame (which may occur, for example, when a triangle has just become visible in the current frame but the texture content of the triangle corresponds to a lost area of ​​a (overlapping) video frame), the client device may set ReprojectionMask to 1, indicating that any pixels in a rasterized triangle that sample that portion of the texture cache may not use the contents of the texture cache; instead, a fallback to the screen space cache may be used.

[0083] In some configurations, when the triangle i) becomes invisible or ii) is no longer in error, the client device may deallocate the texture ID of the triangle, returning the texture ID to the list of available texture IDs.

[0084] Figure 8 8 is a call flow diagram 800 illustrating example communications between a client device 802 and a server 804 in accordance with one or more techniques of this disclosure. Client device 802 may correspond to client device 204. Additionally, server 804 may correspond to server 202. At 806, client device 802 may send pose information associated with client device 802 to server 804.

[0085] At 808, the client device 802 may obtain a PVS and a set of video packets. The PVS may include PVS information associated with a set of visible primitives that are potentially visible in the first frame. In this document, a primitive may refer to a geometric primitive, i.e., a basic unit of graphics information for rendering purposes. For example, a triangle or a meshlet may be a primitive. In addition, the PVS may refer to information describing a mapping of a primitive to a location (place) in a texture atlas (texture space cache), where the current shading of the primitive is described. In addition, the PVS may also implicitly encode the visibility of the primitive. In other words, if a primitive is indicated (mentioned) for the first time in the PVS stream, the primitive may be considered visible. In this way, based on the received PVS and video packets, the client device 802 may derive a shading texture for the corresponding primitive.

[0086] The PVS may be based on a predicted pose of pose information associated with the client device 802 (the pose prediction may be performed at the server 804 or the client device 802). Specifically, at 808a, the client device 802 may receive the PVS and the set of video packets from the server 804.

[0087] In one configuration, the PVS and the set of video packets may be received from the server 804 via one or more UDP packets.

[0088] In one configuration, the set of visible primitives may correspond to a set of meshlets. Each meshlet in the set of meshlets may include at least one visible primitive in the set of visible primitives.

[0089] In one configuration, each first visible primitive of the one or more first visible primitives may be affected by at least one of a video grouping error, a meshlet visibility error, a meshlet atlas information error, or a meshlet level change error.

[0090] In one configuration, a meshlet visibility error or a meshlet level change error may be associated with a heartbeat failure.

[0091] In one configuration, the PVS may also include UV mapping information.

[0092] At 810 , if data loss associated with the PVS or the set of video packets is detected, the client device 802 may determine one or more first visible primitives in the set of visible primitives that are currently affected by the data loss associated with the PVS or the set of video packets.

[0093] In one configuration, the loss of data associated with the PVS or the set of video packets may correspond to obtaining less than all video packets of the set of video packets or less than all PVS data packets (all PVS information) associated with all of the set of visible primitives. For example, video packets and / or PVS data packets associated with one or more first visible primitives may not be successfully received and / or decoded by the client device 802.

[0094] In one configuration, a loss of data associated with the PVS or the set of video packets may be detected by the client device 802 within a first period of time (eg, within a timeout period).

[0095] For example, each primitive that was visible (e.g., signaled) in a previous server frame may also be signaled / updated in the current frame. If the client device 802 detects that a primitive has not received an update when an update is expected, the client device 802 may infer that the corresponding PVS packet is lost. In another example, a video decoder at the client device 802 may detect video packet drops (e.g., real-time transport protocol (RTP) packet drops) based on inconsistencies in a Moving Picture Experts Group (MPEG) video stream.

[0096] At 812 , the client device 802 may identify hidden information for one or more first visible primitives in the set of visible primitives based on the one or more first visible primitives in the previous frame.

[0097] In some configurations, the one or more first visible primitives in the previous frame may not be affected by any previous data loss.The one or more first visible primitives in the previous frame may be associated with one or more textures used in the first frame.

[0098] In some configurations, the hidden information may be associated with the derivation of a hidden texture representing the one or more first visible primitives in the first frame.

[0099] In some configurations, hidden information may be determined / identified based on image inpainting (or screen-space cache lookup) if a corresponding visible primitive associated with a previous frame is not available (e.g., the primitive was not visible in the previous frame but is now visible in the current frame and is affected by data loss; or the primitive is always affected by data loss and therefore the corresponding hidden information may not be derived).

[0100] At 814 , the client device 802 may update information associated with the one or more first visible primitives based on the hidden information for the one or more first visible primitives.

[0101] At 816, the client device 802 may render the content of the first frame based on the updated information associated with the one or more first visible primitives (e.g., replacing the lost texture with a hidden texture identified based on the hidden information), the hidden information, and / or the one or more second visible primitives in the set of visible primitives. The one or more second visible primitives may not currently be affected by the data loss. In other words, the data grouping and PVS information associated with the one or more second visible primitives are currently available.

[0102] In one configuration, video information for the one or more second visible primitives may be associated with a texture space cache (eg, a texture atlas) or image inpainting.

[0103] At 818, the client device 802 may derive first texture information associated with the one or more second visible primitives based on the set of video packets. The association between the first texture information and the one or more second visible primitives may be based on the derivation of PVS information (e.g., associated with the one or more second visible primitives).

[0104] At 820 , the client device 802 may store first texture information associated with the one or more second visible primitives in a texture space cache.

[0105] Fig. 9 900 is a flowchart of an example method of graphics processing according to one or more techniques of the present disclosure. The method may be performed by a first device such as: an apparatus for graphics processing, a GPU, a CPU, a wireless communication device, etc. (such as a combination of Figures 1 to 8 used in various aspects).

[0106] At 902, a first device may obtain a PVS and a set of video packets. The PVS may include PVS information associated with a set of visible primitives that are potentially visible in a first frame. Figure 8 At 808, the first device (e.g., client device 802) may obtain the PVS and a set of video packets. 902 may also be performed by Figure 1 The processing unit 120 executes.

[0107] At 904, if data loss associated with the PVS or the group of video packets is detected, the first device may determine one or more first visible primitives in the group of visible primitives that are currently affected by the data loss associated with the PVS or the group of video packets. Figure 8 At 810, if data loss associated with the PVS or the group of video packets is detected, the first device (e.g., client device 802) may determine one or more first visible primitives in the group of visible primitives that are currently affected by the data loss associated with the PVS or the group of video packets. Figure 1 The processing unit 120 executes.

[0108] At 906, the first device may identify hidden information for one or more first visible primitives in the set of visible primitives based on the one or more first visible primitives in the previous frame. Figure 8 At 812, the first device (eg, client device 802) may identify hidden information for one or more first visible primitives in the set of visible primitives based on the one or more first visible primitives in the previous frame. Figure 1 The processing unit 120 executes.

[0109] At 908, the first device may update information associated with the one or more first visible primitives based on the hidden information for the one or more first visible primitives. Figure 8 At 814, the first device (eg, client device 802) may update information associated with the one or more first visible primitives based on the hidden information for the one or more first visible primitives. Figure 1 The processing unit 120 executes.

[0110] Fig.10 1000 is a flowchart of an example method of graphics processing according to one or more techniques of the present disclosure. The method may be performed by a first device such as: an apparatus for graphics processing, a GPU, a CPU, a wireless communication device, etc. (such as a combination of Figures 1 to 8 used in various aspects).

[0111] At 1004, the first device may obtain a PVS and a set of video packets. The PVS may include PVS information associated with a set of visible primitives that are potentially visible in the first frame. Figure 8 At 808, the first device (eg, client device 802) may obtain the PVS and a set of video packets. 1004 may also be performed by Figure 1 The processing unit 120 executes.

[0112] At 1006, if data loss associated with the PVS or the group of video packets is detected, the first device may determine one or more first visible primitives in the group of visible primitives that are currently affected by the data loss associated with the PVS or the group of video packets. Figure 8 At 810, if data loss associated with the PVS or the group of video packets is detected, the first device (e.g., client device 802) may determine one or more first visible primitives in the group of visible primitives that are currently affected by the data loss associated with the PVS or the group of video packets. Figure 1 The processing unit 120 executes.

[0113] At 1008, the first device may identify hidden information for one or more first visible primitives in the set of visible primitives based on the one or more first visible primitives in the previous frame. Figure 8 At 812, the first device (eg, client device 802) may identify hidden information for one or more first visible primitives in the set of visible primitives based on the one or more first visible primitives in the previous frame. Figure 1 The processing unit 120 executes.

[0114] At 1010, the first device may update information associated with one or more first visible primitives based on hidden information for the one or more first visible primitives. Figure 8 At 814, the first device (eg, client device 802) may update information associated with the one or more first visible primitives based on the hidden information for the one or more first visible primitives. Figure 1 The processing unit 120 executes.

[0115] In one configuration, one or more second visible primitives in the set of visible primitives may not currently be affected by the data loss. The one or more second visible primitives may be associated with one or more textures used in the first frame. The one or more textures may be based on the set of video packets. The association between the one or more textures and the one or more second visible primitives may be based on a derivation of PVS information.

[0116] In one configuration, the hidden information may be associated with the derivation of a hidden texture representing one or more first visible primitives in the first frame. Specifically, the hidden texture may store an atlas spatial rendition of a meshlet affected by the data loss. Thus, the hidden texture may be used during client rendering to replace the lost latest meshlet representation.

[0117] In one configuration, at 1012, the first device may render the content of the first frame based on the update information associated with the one or more first visible primitives, the hidden information, and / or the one or more second visible primitives in the set of visible primitives. The one or more second visible primitives may not currently be affected by the data loss. For example, referring to Figure 8 At 816, the first device (eg, client device 802) may render the content of the first frame based on the update information associated with the one or more first visible primitives, the hidden information, and the one or more second visible primitives in the set of visible primitives. Figure 1 The processing unit 120 executes.

[0118] In one configuration, data loss associated with the PVS or the set of video packets may correspond to receiving less than all of the video packets of the set of video packets or less than all of a set of PVS data packets associated with one or more first visible primitives.

[0119] In one configuration, at 1014, the first device may derive first texture information associated with one or more second visible primitives based on the set of video packets. The association between the first texture information and the one or more second visible primitives may be based on the derivation of PVS information. Figure 8At 818, the first device (eg, client device 802) may derive first texture information associated with one or more second visible primitives based on the set of video packets. Figure 1 The processing unit 120 executes.

[0120] At 1016, the first device may store the first texture information in a texture space cache. Figure 8 At 820, the first device (eg, client device 802) may store the first texture information in a texture space cache. 1016 may also be performed by Figure 1 The processing unit 120 executes.

[0121] In one configuration, if there is a hit on the texture space cache, then at least one hidden texture may be associated with a texture cache lookup. Specifically, the texture space cache may contain the last successfully received texture value for each geometric primitive (e.g., meshlet or triangle). If a given primitive (particularly its texture information) has never been correctly received before, the cache for the primitive may be empty and the corresponding lookup may result in a cache miss. Otherwise, a reference texture may be retrieved from the texture space cache based on the index of the primitive. If there is a miss on the texture space cache, then the at least one hidden texture may be associated with a screen space cache lookup or an image inpainting process.

[0122] In one configuration, to obtain the PVS and the set of video packets at 1004, at 1004a, the first device may receive the PVS and the set of video packets from the server. Figure 8 At 808a, the first device (eg, client device 802) may receive the PVS and the set of video packets from the server 804. 1004a may also be performed by Figure 1 The processing unit 120 executes.

[0123] In one configuration, reference Figure 8 At 808a, the PVS and the set of video packets may be received from the server 804 via one or more UDP packets.

[0124] In one configuration, the set of visible primitives may correspond to a set of meshlets. Each meshlet in the set of meshlets may include at least one visible primitive in the set of visible primitives.

[0125] In one configuration, each first visible primitive of the one or more first visible primitives may be affected by at least one of a video grouping error, a meshlet visibility error, a meshlet atlas information error, or a meshlet level change error.

[0126] In one configuration, a meshlet visibility error or a meshlet level change error may be associated with a heartbeat failure.

[0127] In one configuration, at 1002, the first device may send pose information associated with the first device to the second device. The PVS may predict a pose based on the pose information associated with the first device. For example, referring to Figure 8 At 806, the first device (eg, client device 802) may send pose information associated with the first device (eg, client device 802) to the second device (eg, server 804). 1002 may also be performed by Figure 1 The processing unit 120 executes.

[0128] In one configuration, loss of data associated with the PVS or the set of video packets may be associated with expiration of the first time period.

[0129] In one configuration, one or more first visible primitives in a previous frame may not be affected by any previous data loss.

[0130] In one configuration, the PVS may also include UV mapping information.

[0131] In a configuration, a method or apparatus for graphics processing is provided. The apparatus may be a GPU, a CPU, or some other processor that can perform graphics processing. In various aspects, the apparatus may be a processing unit 120 within the device 104, or may be some other hardware within the device 104 or another device. The apparatus may include components for obtaining a PVS and a set of video packets. The PVS may include PVS information associated with a set of visible primitives that are potentially visible in a first frame. The apparatus may include components for: if data loss associated with the PVS or the set of video packets is detected, then determining one or more first visible primitives in the set of visible primitives that are currently affected by the data loss associated with the PVS or the set of video packets. The apparatus may include components for: identifying hidden information for one or more first visible primitives in the set of visible primitives based on one or more first visible primitives in a previous frame. The apparatus may include components for updating information associated with one or more first visible primitives based on the hidden information for the one or more first visible primitives.

[0132] In one configuration, the means for updating information associated with one or more first visible primitives based on hidden information for the one or more first visible primitives may be further configured to hide the one or more first visible primitives based on the hidden information. In one configuration, the hidden information may be associated with the derivation of a hidden texture representing the one or more first visible primitives in the first frame. The apparatus may include means for rendering the content of the first frame based on the update information associated with the one or more first visible primitives, the hidden information, and / or one or more second visible primitives in the set of visible primitives. The one or more second visible primitives may not currently be affected by data loss. In one configuration, data loss associated with the PVS or the set of video packets may correspond to obtaining less than all video packets of the set of video packets or less than all PVS data packets of a set of PVS data packets associated with the one or more first visible primitives. In one configuration, the apparatus may include means for deriving first texture information associated with the one or more second visible primitives based on the set of video packets. The association between the first texture information and the one or more second visible primitives may be based on the derivation of the PVS information. The apparatus may include means for storing the first texture information in a texture space cache. In one configuration, if there is a hit on the texture space cache, then the at least one hidden texture may be associated with a texture cache lookup. If there is a miss on the texture space cache, then the at least one hidden texture may be associated with a screen space cache lookup or an image inpainting process. In one configuration, the component for obtaining the PVS and the set of video packets may be further configured to receive the PVS and the set of video packets from a server. In one configuration, the PVS and the set of video packets may be received from the server via one or more UDP packets. In one configuration, the set of visible primitives may correspond to a set of meshlets. Each meshlet in the set of meshlets may include at least one visible primitive in the set of visible primitives. In one configuration, each first visible primitive of the one or more first visible primitives may be affected by at least one of a video packet error, a meshlet visibility error, a meshlet atlas information error, or a meshlet level change error. In one configuration, the meshlet visibility error or the meshlet level change error may be associated with a heartbeat failure. In one configuration, the apparatus may include means for sending pose information associated with the first device to the second device. The PVS may predict a pose based on the pose information associated with the first device. The first device may be a client device. The second device may be a server. In one configuration, a data loss associated with the PVS or the group of video packets may be detected within a first time period. In one configuration, one or more first visible primitives in a previous frame may not be affected by any previous data loss. In one configuration, the PVS may also include UV mapping information.

[0133] It should be understood that the specific order or hierarchy of the blocks / steps in the processes, flow charts, and / or call flow charts disclosed herein is merely an illustration of an exemplary method. It should be understood that the specific order or hierarchy of the blocks / steps in these processes, flow charts, and / or call flow charts may be rearranged based on design preferences. In addition, some blocks / steps may be combined or omitted. Other blocks / steps may also be added. The attached method claims provide the elements of various blocks / steps in a sample order, but are not meant to be limited to the specific order or hierarchy provided.

[0134] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the various aspects shown herein, but should be given the full scope consistent with the language of the claims, wherein unless otherwise specified, reference to an element in the singular is not intended to mean "one and only one", but rather "one or more". The word "exemplary" is used herein to mean "used as an example, instance, or illustration". Any aspect described herein as "exemplary" is not necessarily to be interpreted as being preferred or having advantages over other aspects.

[0135] Unless specifically stated otherwise, the term “some” refers to one or more, and the term “or” may be interpreted as “and / or” if the context does not dictate otherwise. Combinations such as “at least one of A, B, or C,” “one or more of A, B, or C,” “at least one of A, B and C,” “one or more of A, B, and C,” and “A, B, C, or any combination thereof” include any combination of A, B, and / or C, which may include multiple A, multiple B, or multiple C. Specifically, combinations such as “at least one of A, B, or C,” “one or more of A, B, or C,” “at least one of A, B, and C,” “one or more of A, B, and C,” and “A, B, C, or any combination thereof” may be only A, only B, only C, A and B, A and C, B and C, or A and B and C, where any such combination may include one or more members of A, B, or C. All structural and functional equivalents of the elements of various aspects described throughout this disclosure that are known or will later become known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be covered by the claims. In addition, nothing disclosed herein is intended to be dedicated to the public, regardless of whether such disclosure is explicitly stated in the claims. Words such as "module," "mechanism," "element," "device," etc. cannot replace the word "component." Therefore, no claim element will be understood to be a part-plus-function unless the element is explicitly stated using the phrase "component for..."

[0136] In one or more examples, the functions described herein may be implemented in hardware, software, firmware, or any combination thereof. For example, although the term "processing unit" is used throughout this disclosure, such processing unit may be implemented in hardware, software, firmware, or any combination thereof. If any function, processing unit, technique, or other module described herein is implemented in software, the function, processing unit, technique, or other module described herein may be stored or sent as one or more instructions or codes on a computer-readable medium.

[0137] Computer-readable media may include computer data storage media and communication media, including any media that facilitate the transfer of computer programs from one place to another. In this manner, computer-readable media may generally correspond to: (1) a non-transitory tangible computer-readable storage medium; or (2) a communication medium, such as a signal or carrier wave. Data storage media may be any available medium that can be accessed by one or more computers or one or more processors to extract instructions, codes, and / or data structures for implementing the techniques described in this disclosure. As an example and not limitation, such computer-readable media may include RAM, ROM, EEPROM, compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, or other magnetic storage devices. As used herein, magnetic disks and optical disks include compact discs (CDs), laser discs, optical disks, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where magnetic disks typically reproduce data magnetically, while optical disks reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media. A computer program product may include computer-readable media.

[0138] The technology of the present disclosure can be implemented in a variety of devices or apparatuses, including wireless mobile phones, integrated circuits (ICs) or IC groups (e.g., chipsets). Various components, modules, or units are described in the present disclosure to emphasize the functional aspects of devices configured to perform the disclosed technology, but they do not necessarily need to be implemented by different hardware units. On the contrary, as described above, the various units can be combined in any hardware unit, or provided by a collection of interoperable hardware units (including one or more processors as described above) in combination with appropriate software and / or firmware. Therefore, the term "processor" as used herein can refer to any of the above structures or any other structure suitable for implementing the technology described herein. Similarly, these technologies can be fully implemented in one or more circuits or logic elements.

[0139] The following aspects are merely illustrative and may be combined with other aspects or teachings described herein without limitation.

[0140] Aspect 1 is a method of performing graphics processing at a first device, the method comprising: obtaining a PVS and a set of video packets, wherein the PVS includes PVS information associated with a set of visible primitives that are potentially visible in a first frame; if data loss associated with the PVS or the set of video packets is detected, determining one or more first visible primitives in the set of visible primitives that are currently affected by the data loss associated with the PVS or the set of video packets; identifying hidden information for the one or more first visible primitives in the set of visible primitives based on the one or more first visible primitives in a previous frame; and updating information associated with the one or more first visible primitives based on the hidden information for the one or more first visible primitives.

[0141] Aspect 2 can be combined with aspect 1 and includes: one or more second visible primitives in the set of visible primitives are currently not affected by the data loss, the one or more second visible primitives are associated with one or more textures used in the first frame, the one or more textures are based on the set of video packets, and the association between the one or more textures and the one or more second visible primitives is based on the derivation of the PVS information.

[0142] Aspect 3 may be combined with any one of aspects 1 to 2, and comprises that the hidden information is associated with a derivation of a hidden texture representing the one or more first visible primitives in the first frame.

[0143] Aspect 4 can be combined with any one of Aspects 1 to 3, and also includes: rendering the content of the first frame based on update information associated with the one or more first visible primitives, the hidden information, and one or more second visible primitives in the set of visible primitives, wherein the one or more second visible primitives are not currently affected by the data loss.

[0144] Aspect 5 may be combined with Aspect 4 and includes: the data loss associated with the PVS or the set of video packets corresponds to receiving less than all video packets of the set of video packets or less than all PVS data packets of a set of PVS data packets associated with the one or more first visible primitives.

[0145] Aspect 6 can be combined with any one of Aspects 4 to 5, and also includes: deriving first texture information associated with the one or more second visible primitives based on the set of video packets, the association between the first texture information and the one or more second visible primitives being based on the derivation of the PVS information; and storing the first texture information in a texture space cache.

[0146] Aspect 7 can be combined with Aspect 6 and includes: in the event of a hit on the texture space cache, at least one hidden texture is associated with a texture cache lookup, and wherein in the event of a miss on the texture space cache, the at least one hidden texture is associated with a screen space cache lookup or an image inpainting process.

[0147] Aspect 8 may be combined with any one of aspects 1 to 7, and includes: obtaining the PVS and the set of video packets further includes: receiving the PVS and the set of video packets from a server.

[0148] Aspect 9 may be combined with aspect 8, and comprises: receiving the PVS and the set of video packets from the server via one or more UDP packets.

[0149] Aspect 10 may be combined with any one of aspects 1 to 9, and includes: the set of visible primitives corresponds to a set of meshlets, and each meshlet in the set of meshlets includes at least one visible primitive in the set of visible primitives.

[0150] Aspect 11 may be combined with aspect 10 and comprises: each first visible picture element of the one or more first visible picture elements is affected by at least one of a video grouping error, a meshlet visibility error, a meshlet atlas information error or a meshlet level change error.

[0151] Aspect 12 may be combined with Aspect 11 and includes: the meshlet visibility error or the meshlet level change error is associated with a heartbeat failure.

[0152] Aspect 13 can be combined with any one of Aspects 1 to 12, and also includes: sending posture information associated with the first device to a second device, wherein the PVS predicts a posture based on the posture information associated with the first device, wherein the first device is a client device and the second device is a server.

[0153] Aspect 14 may be combined with any of aspects 1 to 13, and comprises that said loss of data associated with said PVS or said set of video packets is associated with expiry of a first time period.

[0154] Aspect 15 may be combined with any one of aspects 1 to 14, and comprises: the one or more first visible primitives in the previous frame are not affected by any previous data loss.

[0155] Aspect 16 may be combined with any one of aspects 1 to 15, and includes: the PVS further including UV mapping information.

[0156] Aspect 17 is an apparatus for performing graphics processing at a first device, the apparatus comprising: at least one processor, the at least one processor being coupled to a memory, and the at least one processor being configured to implement a method according to any one of Aspects 1 to 16 based at least in part on information stored in the memory.

[0157] Aspect 18 may be combined with aspect 17, and further comprises: a transceiver coupled to the at least one processor, wherein the apparatus is a wireless communication device.

[0158] Aspect 19 is a device for graphics processing, the device comprising means for implementing the method according to any one of aspects 1 to 16.

[0159] Aspect 20 is a computer-readable medium (eg, a non-transitory computer-readable medium) storing computer-executable code, which, when executed by at least one processor, causes the at least one processor to implement the method according to any one of aspects 1 to 16.

[0160] Various aspects have been described herein. These and other aspects are within the scope of the following claims.

Claims

1. An apparatus for performing graphics processing at a first device, the apparatus comprising: Memory; and at least one processor coupled to the memory and based at least in part on the information stored in the memory, the at least one processor configured to: Obtaining a potentially visible set (PVS) and a set of video packets, wherein the PVS includes PVS information associated with a set of visible primitives that are potentially visible in a first frame; upon detecting a data loss associated with the PVS or the set of video packets, determining one or more first visible picture elements in the set of visible picture elements currently affected by the data loss associated with the PVS or the set of video packets; identifying hidden information for the one or more first visible primitives in the set of visible primitives based on the one or more first visible primitives in a previous frame; as well as Information associated with the one or more first visible primitives is updated based on the hidden information for the one or more first visible primitives.

2. The apparatus of claim 1 , wherein one or more second visible primitives in the set of visible primitives are currently not affected by the data loss, the one or more second visible primitives are associated with one or more textures used in the first frame, the one or more textures are based on the set of video packets, and the association between the one or more textures and the one or more second visible primitives is based on a derivation of the PVS information. 3 . The apparatus of claim 1 , wherein the hidden information is associated with a derivation of a hidden texture representing the one or more first visible primitives in the first frame.

4. The apparatus of claim 1 , wherein the at least one processor is further configured to: Content of the first frame is rendered based on update information associated with the one or more first visible primitives, the hidden information, and one or more second visible primitives in the set of visible primitives, wherein the one or more second visible primitives are not currently affected by the data loss.

5. The apparatus of claim 4, wherein the data loss associated with the PVS or the set of video packets corresponds to receiving less than all of the set of video packets or less than all of the PVS data packets of a set of PVS data packets associated with the one or more first visible primitives.

6. The apparatus of claim 4, wherein the at least one processor is further configured to: deriving first texture information associated with the one or more second visible primitives based on the set of video packets, the association between the first texture information and the one or more second visible primitives being based on the derivation of the PVS information; and The first texture information is stored in a texture space cache.

7. The apparatus of claim 6 , wherein in case there is a hit on the texture space cache, at least one hidden texture is associated with a texture cache lookup, and wherein in case there is a miss on the texture space cache, the at least one hidden texture is associated with a screen space cache lookup or an inpainting process.

8. The apparatus of claim 1 , wherein to obtain the PVS and the set of video packets, the at least one processor is configured to: The PVS and the set of video packets are received from a server.

9. The apparatus of claim 8, wherein to receive the PVS and the set of video packets from the server, the at least one processor is configured to receive the PVS and the set of video packets from the server via one or more User Datagram Protocol (UDP) packets.

10. The apparatus of claim 1, wherein the set of visible primitives corresponds to a set of meshlets, and each meshlet in the set of meshlets includes at least one visible primitive in the set of visible primitives.

11. The apparatus of claim 10, wherein each of the one or more first visible primitives is affected by at least one of a video grouping error, a meshlet visibility error, a meshlet atlas information error, or a meshlet level change error. 12 . The apparatus of claim 11 , wherein the meshlet visibility error or the meshlet level change error is associated with a heartbeat failure.

13. The apparatus of claim 1 , wherein the at least one processor is further configured to: Sending pose information associated with the first device to a second device, wherein the PVS is based on a predicted pose of the pose information associated with the first device, wherein the first device is a client device and the second device is a server.

14. The apparatus of claim 1, wherein the data loss associated with the PVS or the set of video packets is associated with expiration of a first time period.

15. The apparatus of claim 1, wherein the one or more first visible primitives in the previous frame are not affected by any previous data loss.

16. The apparatus of claim 1, wherein the PVS further comprises UV mapping information.

17. The apparatus of claim 1, further comprising a transceiver coupled to the at least one processor, wherein the apparatus is a wireless communication device, wherein to obtain the PVS and the set of video packets, the at least one processor is configured to obtain the PVS and the set of video packets via the transceiver.

18. A method for performing graphics processing at a first device, the method comprising: Obtaining a potentially visible set (PVS) and a set of video packets, wherein the PVS includes PVS information associated with a set of visible primitives that are potentially visible in a first frame; upon detecting a data loss associated with the PVS or the set of video packets, determining one or more first visible picture elements in the set of visible picture elements currently affected by the data loss associated with the PVS or the set of video packets; identifying hidden information for the one or more first visible primitives in the set of visible primitives based on the one or more first visible primitives in a previous frame; as well as Information associated with the one or more first visible primitives is updated based on the hidden information for the one or more first visible primitives.

19. The method of claim 18, wherein one or more second visible primitives in the set of visible primitives are currently not affected by the data loss, the one or more second visible primitives are associated with one or more textures used in the first frame, the one or more textures are based on the set of video packets, and the association between the one or more textures and the one or more second visible primitives is based on the derivation of the PVS information.

20. The method of claim 19, wherein the hidden information is associated with derivation of a hidden texture representing the one or more first visible primitives in the first frame.

21. The method according to claim 18, further comprising: Content of the first frame is rendered based on update information associated with the one or more first visible primitives, the hidden information, and one or more second visible primitives in the set of visible primitives, wherein the one or more second visible primitives are not currently affected by the data loss.

22. The method of claim 21, wherein the data loss associated with the PVS or the set of video packets corresponds to receiving less than all of the set of video packets or less than all of the PVS data packets of a set of PVS data packets associated with the one or more first visible primitives.

23. The method according to claim 21, further comprising: deriving first texture information associated with the one or more second visible primitives based on the set of video packets, the association between the first texture information and the one or more second visible primitives being based on the derivation of the PVS information; and The first texture information is stored in a texture space cache.

24. The method of claim 23, wherein in the event of a hit on the texture space cache, at least one hidden texture is associated with a texture cache lookup, and wherein in the event of a miss on the texture space cache, the at least one hidden texture is associated with a screen space cache lookup or an inpainting process.

25. The method of claim 18, wherein obtaining the PVS and the set of video packets further comprises: The PVS and the set of video packets are received from a server.

26. The method of claim 25, wherein the PVS and the set of video packets are received from the server via one or more User Datagram Protocol (UDP) packets.

27. The method of claim 18, wherein the set of visible primitives corresponds to a set of meshlets, and each meshlet in the set of meshlets includes at least one visible primitive in the set of visible primitives.

28. The method of claim 27, wherein each of the one or more first visible primitives is affected by at least one of a video grouping error, a meshlet visibility error, a meshlet atlas information error, or a meshlet level change error.

29. The method of claim 28, wherein the meshlet visibility error or the meshlet level change error is associated with a heartbeat failure.

30. A computer readable medium storing computer executable code which, when executed by at least one processor, causes the at least one processor to: Obtaining a potentially visible set (PVS) and a set of video packets, wherein the PVS includes PVS information associated with a set of visible primitives that are potentially visible in a first frame; upon detecting a data loss associated with the PVS or the set of video packets, determining one or more first visible picture elements in the set of visible picture elements currently affected by the data loss associated with the PVS or the set of video packets; identifying hidden information for the one or more first visible primitives in the set of visible primitives based on the one or more first visible primitives in a previous frame; as well as Information associated with the one or more first visible primitives is updated based on the hidden information for the one or more first visible primitives.

Images