Selective patch packing for immersive video

Abstract

Embodiments generally relate to selective patch packing for immersive video. An embodiment of a processing system includes: one or more processor cores; and a memory for storing data of the immersive video, the data comprising a plurality of patches for a plurality of projection directions. The system is used to select patches for packaging, the selection of the patches based at least in part on which of the plurality of projection directions is associated with each of the patches. The system is used to encode a patch into one or more encoded pictures according to a selection of the patch.

Description

This application is a divisional application of the invention patent application with application date of June 26, 2019, priority date of July 31, 2018, application number 201910561436.0, and titled “Selective patch packaging for immersive video”. Related Applications

[0001] This application is related to commonly assigned U.S. patent application serial number 16 / 050,153, entitled “REDUCED RENDERING OF SIX-DEGREE OF FREEDOM VIDEO,” filed by Jill Boyce on July 31, 2018, the entire contents of which are incorporated herein by reference. Technical Field

[0002] Embodiments described herein relate generally to the field of electronic devices, and more particularly to selective patch packing for immersive video. Background Art

[0003] Six degrees of freedom (6DoF) video is an emerging immersive video use case that provides viewers with an immersive media experience in which they control the viewpoint of the scene. Simpler three degrees of freedom (3DoF) video (such as 360-degree or panoramic video) allows the viewer to change the orientation around the X, Y, and Z axes (described as yaw, pitch, and roll) from a fixed position. 6DoF video enables the viewer to change position by translational movement along the X, Y, and Z axes.

[0004] Point clouds can be used to represent 6DoF videos. However, rendering of point cloud data is computationally expensive, making it difficult to render point cloud videos containing a large number of points at high frame rates. In addition, the point cloud data rate is large, requiring a large capacity for storage or transmission.

[0005] When encoding 6DoF video captured from multiple cameras, patches are formed by the projection of a single camera or at the virtual camera position of the content that is most visible to a particular camera / virtual camera. The content from a given camera or virtual camera can be referred to as the projection direction.

[0006] Texture and depth patches from multiple projection directions are packed into a single picture and encoded with a normal video codec such as HEVC (High Efficiency Video Coding) or AVC (Advanced Video Coding). MPEG Point Cloud Coding (PCC) uses projections from six projection directions corresponding to the faces of a cuboid. However, the client then needs to decode the entire encoded picture, which is a high-resolution image because it contains patches from multiple projection directions. Decoding the entire encoded picture will put a large burden on the decoder and will require high network bandwidth for transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The embodiments described herein are illustrated by way of example and not limitation in the accompanying drawings in which like reference numerals refer to similar elements.

[0008] Figure 1 is a block diagram of a processing system according to some embodiments;

[0009] Figure 2 is a block diagram of an embodiment of a processor having one or more processor cores, an integrated memory controller, and an integrated graphics processor;

[0010] Figure 3 is a block diagram of a graphics processor according to some embodiments;

[0011] Figure 4 is a block diagram of a graphics processing engine of a graphics processor according to some embodiments;

[0012] Figure 5 is a block diagram of the hardware logic of a graphics processor core according to some embodiments;

[0013] FIG. 6A to FIG. 6B Thread execution logic is shown according to some embodiments, the thread execution logic comprising an array of processing elements employed in a graphics processor core;

[0014] Figure 7 A block diagram illustrating a graphics processor instruction format according to some embodiments;

[0015] Figure 8 is a block diagram of another embodiment of a graphics processor;

[0016] Fig. 9A is a block diagram illustrating a graphics processor command format according to some embodiments;

[0017] Fig. 9B is a block diagram illustrating a graphics processor command sequence according to an embodiment;

[0018] Fig.10An exemplary graphics software architecture for a data processing system according to some embodiments is presented;

[0019] Fig.11A is a block diagram illustrating an IP core development system that may be used to manufacture an integrated circuit for performing operations according to an embodiment;

[0020] Fig. 11B illustrates a cross-sectional side view of an integrated circuit package assembly according to some embodiments;

[0021] Fig.12 is a block diagram illustrating an exemplary system-on-chip integrated circuit that may be fabricated using one or more IP cores according to an embodiment;

[0022] Fig.13A An exemplary graphics processor of a system-on-chip integrated circuit that may be manufactured using one or more IP cores according to an embodiment is shown;

[0023] Fig. 13B Additional exemplary graphics processors of system-on-chip integrated circuits that may be fabricated using one or more IP cores in accordance with embodiments are shown;

[0024] Fig.14A A graphics core that may be included within a graphics processor according to some embodiments is shown;

[0025] Fig. 14B A highly parallel general purpose graphics processing unit suitable for deployment on a multi-chip module according to some embodiments is presented;

[0026] Fig.15A Various forms of immersive video were demonstrated;

[0027] Fig. 15B Demonstrated image projection and texture planes for immersive video;

[0028] Fig.16 A client-server system for generating 1 and consuming immersive video is presented;

[0029] FIG. 17A to FIG. 17B Demonstrated a system for encoding and decoding 3DoF+ (3DoF Plus) content;

[0030] FIG. 18A to FIG. 18B A system for encoding and decoding 6DoF texture geometry data is presented;

[0031] FIG. 19A to FIG. 19B Demonstrated a system for encoding and decoding 6DoF point cloud data;

[0032] Fig. 20Ais an illustration of a processing system for providing selective patch packing according to some embodiments;

[0033] Fig. 20B is an illustration of a client system for decoding one or more encoded pictures including a selected patch according to some embodiments;

[0034] Fig.21 is an illustration of an image of an immersive video utilizing selective patch packing according to some embodiments;

[0035] Fig. 22 is a simplified illustration of patches generated for an exemplary set of images;

[0036] Fig.23 is an illustration of an exemplary view group of a system including selective patch packing according to some embodiments;

[0037] Fig.24 is an illustration of selective patch packing for providing coded pictures for each viewpoint according to some embodiments;

[0038] FIG. 25A to FIG. 25D showing multiple sets of viewpoints for selective patch packing according to some embodiments; Fig.26 is an illustration of selective patch packing for providing coded pictures for each set of views according to some embodiments;

[0039] Fig.27A is an illustration of selective packing of patches into tiles according to projection direction according to some embodiments;

[0040] Fig.27B is an illustration of selective packing of patches into tiles according to multiple sets of projection directions according to some embodiments;

[0041] Fig.28A is a flow chart illustrating a process for selective patch packing according to some embodiments; and

[0042] Fig.28B is a flow chart illustrating a process for decoding and depacketizing selectively packed patches according to some embodiments. DETAILED DESCRIPTION

[0043] Embodiments described herein generally relate to selective patch packing for immersive video.

[0044] When encoding 6DoF (six degrees of freedom) video captured from multiple cameras, patches are formed by the projection of a single camera or virtual camera position of the content that is most visible to a particular camera / virtual camera. The content from a given camera or virtual camera can be referred to as the projection direction.

[0045] Texture and depth patches from multiple projection directions are packed into a single picture and encoded with a normal video codec such as HEVC (High Efficiency Video Coding) or AVC (Advanced Video Coding). MPEG Point Cloud Coding (PCC) uses projections from six projection directions corresponding to the faces of a cuboid. However, the client then needs to decode the entire encoded picture, which is a high-resolution image because it contains patches from multiple projection directions. Decoding the entire encoded picture will put a large burden on the decoder and will require high network bandwidth for transmission.

[0046] In some embodiments, an apparatus, system, or process provides selective patch packing for immersive video.In some embodiments, patches are selected based on which of a plurality of projection directions is associated with each of the patches.

[0047] System Overview

[0048] Figure 1 1 is a block diagram of a processing system 100 according to an embodiment. In various embodiments, system 100 includes one or more processors 102 and one or more graphics processors 108, and may be a single processor desktop system, a multi-processor workstation system, or a server system having a large number of processors 102 or processor cores 107. In one embodiment, system 100 is a processing platform incorporated into a system-on-chip (SoC) integrated circuit for use in a mobile device, handheld device, or embedded device.

[0049] In one embodiment, system 100 may include a server-based game platform, a game console, or be incorporated into a server-based game platform, a game console, and the game console includes a game and media console, a mobile game console, a handheld game console, or an online game console. In some embodiments, system 100 is a mobile phone, a smart phone, a tablet computing device, or a mobile Internet device. Processing system 100 may also include a wearable device, be coupled with a wearable device, or be integrated in a wearable device, such as a smart watch wearable device, a smart glasses device, an augmented reality device, or a virtual reality device. In some embodiments, processing system 100 is a television or a set-top box device, and the television or the set-top box device has one or more processors 102 and a graphical interface generated by one or more graphics processors 108.

[0050] In some embodiments, one or more processors 102 each include one or more processor cores 107 for processing instructions, which when executed perform operations of the system and user software. In some embodiments, each processor core in the one or more processor cores 107 is configured to process a specific instruction set 109. In some embodiments, the instruction set 109 can facilitate complex instruction set computing (CISC), reduced instruction set computing (RISC), or computing via very long instruction words (VLIW). Multiple processor cores 107 can each process different instruction sets 109, which may include instructions for facilitating emulation of other instruction sets. The processor core 107 may also include other processing devices, such as a digital signal processor (DSP).

[0051] In some embodiments, the processor 102 includes a cache memory 104. Depending on the architecture, the processor 102 may have a single internal cache or multiple levels of internal cache. In some embodiments, the cache memory is shared between the components of the processor 102. In some embodiments, the processor 102 also uses an external cache (e.g., a level 3 (L3) cache or a last level cache (LLC)) (not shown), which may be shared between the processor cores 107 using known cache coherence techniques. In addition, a register file 106 is included in the processor 102, which may include different types of registers (e.g., integer registers, floating point registers, status registers, and instruction pointer registers) for storing different types of data. Some registers may be general purpose registers, while other registers may be specific to the design of the processor 102.

[0052] In some embodiments, one or more processors 102 are coupled to one or more interface buses 110 to transmit communication signals such as address, data, or control signals between the processors 102 and other components in the system 100. In one embodiment, the interface bus 110 may be a processor bus, such as a version of a direct media interface (DMI) bus. However, the processor bus is not limited to a DMI bus and may include one or more peripheral component interconnect buses (e.g., PCI, PCI Express), memory buses, or other types of interface buses. In one embodiment, the processor(s) 102 include an integrated memory controller 160 and a platform controller hub 130. The memory controller 116 facilitates communication between memory devices and other components of the system 100, while the platform controller hub (PCH) 130 provides connections to I / O devices via a local I / O bus.

[0053] The memory device 120 may be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, a flash memory device, a phase change memory device, or some other memory device having suitable properties to act as a process memory. In one embodiment, the memory device 120 may operate as a system memory of the system 100 to store data 122 and instructions 121 for use when one or more processors 102 execute applications or processes. The memory controller 116 is also coupled to an optional external graphics processor 112, which may communicate with one or more graphics processors 108 in the processor 102 to perform graphics and media operations. In some embodiments, a display device 111 may be connected to the processor(s) 102. The display device 111 may be one or more of the following: an internal display device, such as in a mobile electronic device or laptop device; or an external display device attached via a display interface (e.g., a display port (DisplayPort) or the like). In one embodiment, the display device 111 may be a head mounted display (HMD), such as a stereoscopic display device for use in virtual reality (VR) applications or augmented reality (AR) applications.

[0054] In some embodiments, the platform controller hub 130 enables peripheral devices to connect to the memory device 120 and the processor 102 via a high-speed I / O bus. The I / O peripherals include, but are not limited to, an audio controller 146, a network controller 134, a firmware interface 128, a wireless transceiver 126, a touch sensor 125, a data storage device 124 (e.g., a hard drive, flash memory, etc.). The data storage device 124 can be connected via a storage interface (e.g., SATA) or via a peripheral bus such as a peripheral component interconnect bus (e.g., PCI, PCI Express). The touch sensor 125 may include a touch screen sensor, a pressure sensor, or a fingerprint sensor. The wireless transceiver 126 may be a Wi-Fi transceiver, a Bluetooth transceiver, or a mobile network transceiver such as a 3G, 4G or long term evolution (LTE) transceiver. The firmware interface 128 enables communication with the system firmware and may be, for example, a unified extensible firmware interface (UEFI). The network controller 134 may enable network connection to a wired network. In some embodiments, a high-performance network controller (not shown) is coupled to the interface bus 110. In one embodiment, the audio controller 146 is a multi-channel high definition audio controller. In one embodiment, the system 100 includes an optional legacy I / O controller 140 for coupling legacy (e.g., Personal System 2 (PS / 2)) devices to the system. The platform controller hub 130 may also be connected to one or more Universal Serial Bus (USB) controllers 142 to connect input devices such as a keyboard and mouse 143 combination, a camera 144, or other USB input devices.

[0055] It will be appreciated that the illustrated system 100 is exemplary and not limiting, as other types of data processing systems configured in different manners may also be used. For example, instances of the memory controller 116 and the platform controller hub 130 may be integrated into a discrete external graphics processor such as the external graphics processor 112. In one embodiment, the platform controller hub 130 and / or the memory controller 116 may be external to one or more processors 102. For example, the system 100 may include an external memory controller 116 and a platform controller hub 130 that may be configured as a memory controller hub and a peripheral controller hub within a system chipset that communicates with the processor(s) 102.

[0056] Figure 2 is a block diagram of an embodiment of a processor 200 having one or more processor cores 202A- 202N, an integrated memory controller 214 , and an integrated graphics processor 208 . Figure 2Those elements having the same reference numerals (or names) as elements of any other figure herein may operate or function in any manner similar to that described elsewhere herein, but are not limited thereto. Processor 200 may include additional cores, up to and including additional core 202N represented by a dashed box. Each of processor cores 202A-202N includes one or more internal cache units 204A-204N. In some embodiments, each processor core may also access one or more shared cache units 206.

[0057] Internal cache units 204A-204N and shared cache unit 206 represent a cache memory hierarchy within processor 200. The cache memory hierarchy may include at least one level of instruction and data cache within each processor core, and one or more levels of shared mid-level cache, such as level 2 (L2), level 3 (L3), level 4 (L4), or other levels of cache, where the highest level of cache before external memory is classified as LLC. In some embodiments, cache coherence logic maintains coherence between each cache unit 206 and 204A-204N.

[0058] In some embodiments, the processor 200 may further include a set of one or more bus controller units 216 and a system agent core 210. The one or more bus controller units 216 manage a set of peripheral buses, such as one or more PCI or PCI Express buses. The system agent core 210 provides management functions for various processor components. In some embodiments, the system agent core 210 includes one or more integrated memory controllers 214 to manage access to various external memory devices (not shown).

[0059] In some embodiments, one or more of the processor cores 202A-202N include support for simultaneous multithreading. In such embodiments, the system agent core 210 includes components for coordinating and operating the cores 202A-202N during multithreaded processing. The system agent core 210 may additionally include a power control unit (PCU) that includes logic and components for regulating the power state of the processor cores 202A-202N and the graphics processor 208.

[0060] In some embodiments, the processor 200 additionally includes a graphics processor 208 for performing graphics processing operations. In some embodiments, the graphics processor 208 is coupled to a set of shared cache units 206 and a system agent core 210, which includes one or more integrated memory controllers 214. In some embodiments, the system agent core 210 also includes a display controller 211 to drive the graphics processor output to one or more coupled displays. In some embodiments, the display controller 211 can also be a separate module coupled to the graphics processor via at least one interconnect, or can be integrated within the graphics processor 208.

[0061] In some embodiments, a ring-based interconnect unit 212 is used to couple the internal components of the processor 200. However, alternative interconnect units may be used, such as point-to-point interconnects, switched interconnects, or other technologies, including those known in the art. In some embodiments, the graphics processor 208 is coupled to the ring interconnect 212 via an I / O link 213.

[0062] Exemplary I / O link 213 represents at least one of a variety of I / O interconnects, including an on-package I / O interconnect that facilitates communication between various processor components and a high-performance embedded memory module 218 (such as an eDRAM module). In some embodiments, each of the processor cores 202A-202N and the graphics processor 208 use the embedded memory module 218 as a shared last-level cache.

[0063] In some embodiments, the processor cores 202A-202N are homogeneous cores that execute the same instruction set architecture. In another embodiment, the processor cores 202A-202N are heterogeneous in terms of instruction set architecture (ISA), wherein one or more of the processor cores 202A-202N execute a first instruction set, while at least one of the other cores executes a subset of the first instruction set or a different instruction set. In one embodiment, the processor cores 202A-202N are homogeneous in terms of microarchitecture, wherein one or more cores with relatively high power consumption are coupled with one or more power cores with lower power consumption. In addition, the processor 200 may be implemented on one or more chips or as a SoC integrated circuit having the illustrated components in addition to other components.

[0064] Figure 33 is a block diagram of a graphics processor 300, which may be a discrete graphics processing unit or may be a graphics processor integrated with multiple processing cores. In some embodiments, the graphics processor communicates via a memory-mapped I / O interface to registers on the graphics processor and using commands placed in processor memory. In some embodiments, the graphics processor 300 includes a memory interface 314 for accessing memory. The memory interface 314 may be an interface to local memory, one or more internal caches, one or more shared external caches, and / or to system memory.

[0065] In some embodiments, the graphics processor 300 also includes a display controller 302 for driving display output data to a display device 320. The display controller 302 includes hardware for a combination of one or more overlapping planes of a display and multiple layers of video or user interface elements. The display device 320 can be an internal or external display device. In one embodiment, the display device 320 is a head-mounted display device, such as a virtual reality (VR) display device or an augmented reality (AR) display device. In some embodiments, the graphics processor 300 includes a video codec engine 306 to encode, decode or transcode media to one or more media encoding formats, encode, decode or transcode media from one or more media encoding formats, or encode, decode or transcode media between one or more media encoding formats, including but not limited to: Moving Picture Experts Group (MPEG) format (such as MPEG-2), Advanced Video Coding (AVC) format (such as H.264 / MPEG-4AVC), and Society of Motion Picture & Television Engineers (SMPTE) 421M / VC-1, and Joint Photographic Experts Group (JPEG) format (such as JPEG, and Motion JPEG (MJPEG) format).

[0066] In some embodiments, graphics processor 300 includes a block image transfer (BLIT) engine 304 for performing two-dimensional (2D) rasterizer operations including, for example, bit-boundary block transfers. However, in one embodiment, 2D graphics operations are performed using one or more components of a graphics processing engine (GPE) 310. In some embodiments, GPE 310 is a compute engine for performing graphics operations, including three-dimensional (3D) graphics operations and media operations.

[0067] In some embodiments, GPE 310 includes a 3D pipeline 312 for performing 3D operations, such as rendering three-dimensional images and scenes using processing functions that operate on 3D primitive shapes (e.g., rectangles, triangles, etc.). 3D pipeline 312 includes programmable fixed-function elements that perform various tasks within the element and / or spawn execution threads to 3D / media subsystem 315. While 3D pipeline 312 can be used to perform media operations, embodiments of GPE 310 also include a media pipeline 316 specifically for performing media operations such as video post-processing and image enhancement.

[0068] In some embodiments, the media pipeline 316 includes fixed-function or programmable logic units to perform one or more specialized media operations, such as video decoding acceleration, video deinterlacing, and video encoding acceleration, instead of, or on behalf of the video codec engine 306. In some embodiments, the media pipeline 316 additionally includes a thread generation unit for generating threads for execution on the 3D / media subsystem 315. The generated threads perform calculations for media operations on one or more graphics execution units included in the 3D / media subsystem 315.

[0069] In some embodiments, the 3D / media subsystem 315 includes logic for executing threads generated by the 3D pipeline 312 and the media pipeline 316. In one embodiment, the pipeline sends thread execution requests to the 3D / media subsystem 315, which includes thread dispatch logic for arbitrating and dispatching each request to available thread execution resources. The execution resources include an array of graphics execution units for processing 3D and media threads. In some embodiments, the 3D / media subsystem 315 includes one or more internal caches for thread instructions and data. In some embodiments, the subsystem also includes shared memory, including registers and addressable memory, for sharing data between threads and for storing output data.

[0070] Graphics processing engine

[0071] Figure 4 is a block diagram of a graphics processing engine 410 of a graphics processor according to some embodiments. In one embodiment, the graphics processing engine (GPE) 410 is Figure 3 A version of the GPE 310 is shown in FIG. Figure 4 Those elements having the same reference numerals (or names) as elements of any other figure herein may operate or function in any manner similar to that described elsewhere herein, but are not limited thereto. Figure 3410. The 3D pipeline 312 and the media pipeline 316 of the GPE 410. The media pipeline 316 is optional in some embodiments of the GPE 410 and may not be explicitly included in the GPE 410. For example and in at least one embodiment, separate media and / or image processors are coupled to the GPE 410.

[0072] In some embodiments, GPE 410 is coupled to or includes command stream converter 403, which provides command stream to 3D pipeline 312 and / or media pipeline 316. In some embodiments, command stream converter 403 is coupled to memory, which can be system memory, or one or more of internal cache memory and shared cache memory. In some embodiments, command stream converter 403 receives commands from memory and sends these commands to 3D pipeline 312 and / or media pipeline 316. These commands are instructions taken from the ring buffer for storing 3D pipeline 312 and media pipeline 316. In one embodiment, the ring buffer can additionally include a batch command buffer storing multiple batches of multiple commands. Commands for 3D pipeline 312 can also include references to data stored in memory, such as but not limited to vertex data and geometric data for 3D pipeline 312 and / or image data and memory objects for media pipeline 316. The 3D pipeline 312 and the media pipeline 316 process commands and data by executing operations via logic within the corresponding pipelines or dispatching one or more execution threads to the graphics core array 414. In one embodiment, the graphics core array 414 includes one or more graphics core blocks (e.g., (multiple) graphics cores 415A, (multiple) graphics cores 415B), each block including one or more graphics cores. Each graphics core includes a set of graphics execution resources, which include: general execution logic and graphics-specific execution logic for performing graphics operations and computing operations; and fixed-function texture processing logic and / or machine learning and artificial intelligence acceleration logic.

[0073] In various embodiments, the 3D pipeline 312 includes fixed function logic and programmable logic for processing one or more shader programs, such as vertex shaders, geometry shaders, pixel shaders, fragment shaders, compute shaders, or other shader programs, by processing instructions and dispatching execution threads to the graphics core array 414. The graphics core array 414 provides a unified execution resource block for use in processing these shader programs. The multi-purpose execution logic (e.g., execution unit) within the (multiple) graphics cores 415A-414B of the graphics core array 414 includes support for various 3D API shader languages ​​and can execute multiple simultaneous execution threads associated with multiple shaders.

[0074] In some embodiments, graphics core array 414 also includes execution logic for performing media functions such as video and / or image processing. In one embodiment, the execution unit additionally includes general purpose logic that can be programmed to perform parallel general purpose computing operations in addition to graphics processing operations. The general purpose logic can be connected to Figure 1 (multiple) processor cores 107 or Figure 2 The general logic within the cores 202A-202N in the process performs processing operations in parallel or in combination.

[0075] Output data generated by threads executing on graphics core array 414 may output data to memory in unified return buffer (URB) 418. URB 418 may store data for multiple threads. In some embodiments, URB 418 may be used to send data between different threads executing on graphics core array 414. In some embodiments, URB 418 may additionally be used for synchronization between threads on the graphics core array and fixed function logic within shared function logic 420.

[0076] In some embodiments, graphics core array 414 is scalable such that the array includes a variable number of graphics cores that each have a variable number of execution units based on the target power and performance level of GPE 410. In one embodiment, execution resources are dynamically scalable such that the execution resources can be enabled or disabled as needed.

[0077] The graphics core array 414 is coupled to shared function logic 420, which includes a plurality of resources shared between the graphics cores in the graphics core array. The shared functions within the shared function logic 420 are hardware logic units that provide specialized, complementary functions to the graphics core array 414. In various embodiments, the shared function logic 420 includes, but is not limited to, sampler 421, math 422, and inter-thread communication (ITC) 423 logic. In addition, some embodiments implement one or more caches 425 within the shared function logic 420.

[0078] Shared functionality is implemented in situations where the demand for a given specialized functionality is insufficient to be included in the graphics core array 414. Instead, a single instantiation of the specialized functionality is implemented as an independent entity in the shared functionality logic 420 and shared between execution resources within the graphics core array 414. The precise set of functionality shared between the graphics core array 414 and included within the graphics core array 414 varies across embodiments. In some embodiments, specific shared functionality within the shared functionality logic 420 that is widely used by the graphics core array 414 may be included within the shared functionality logic 416 within the graphics core array 414. In various embodiments, the shared functionality logic 416 within the graphics core array 414 may include some or all of the logic within the shared functionality logic 420. In one embodiment, all logic elements within the shared functionality logic 420 may be repeated within the shared functionality logic 416 of the graphics core array 414. In one embodiment, the shared functionality logic 420 is excluded in favor of the shared functionality logic 416 within the graphics core array 414.

[0079] Figure 5 is a block diagram of the hardware logic of graphics processor core 500 according to some embodiments described herein. Figure 5 Elements having the same reference numerals (or names) as elements of any other figure herein may operate or function in any manner similar to that described elsewhere herein, but are not limited thereto. In some embodiments, the illustrated graphics processor core 500 is included in Figure 4 4. A graphics processor core 500, sometimes referred to as a core slice, may be one or more graphics cores within a modular graphics processor. Graphics processor core 500 is an exemplary one graphics core slice, and a graphics processor as described herein may include multiple graphics core slices based on target power and performance envelopes. Each graphics core 500 may include a fixed function block 530 coupled to multiple sub-cores 501A-501F (also referred to as sub-slices), which include modular general purpose and fixed function logic blocks.

[0080] In some embodiments, fixed function block 530 includes a geometry / fixed function pipeline 536, which may be shared by all sub-cores in graphics core 500, for example, in low-performance and / or low-power graphics processor implementations. In various embodiments, geometry / fixed function pipeline 536 includes a 3D fixed function pipeline (e.g., Figure 3 and Figure 4 3D pipeline 312 in ), video front end unit, thread generator and thread dispatcher, and unified return buffer manager, which manages such as Figure 4 A unified return buffer such as unified return buffer 418 of .

[0081] In one embodiment, fixed function block 530 also includes graphics SoC interface 537, graphics microcontroller 538, and media pipeline 539. Graphics SoC interface 537 provides an interface between graphics core 500 and other processor cores within the system on chip integrated circuit. Graphics microcontroller 538 is a programmable subprocessor that can be configured to manage various functions of graphics processor 500, including thread dispatching, scheduling, and pre-emption. Media pipeline 539 (e.g., Figure 3 and Figure 4 The media pipeline 316 of the embodiment includes logic for facilitating decoding, encoding, pre-processing and / or post-processing of multimedia data including image data and video data. The media pipeline 539 implements media operations via requests to computational or sampling logic within the sub-cores 501-501F.

[0082] In one embodiment, the SoC interface 537 enables the graphics core 500 to communicate with a general application processor core (e.g., a CPU) and / or other components within the SoC, including memory hierarchy elements such as shared last-level cache memory, system RAM, and / or embedded on-chip or on-package DRAM. The SoC interface 537 may also enable communication with fixed-function devices within the SoC, such as a camera imaging pipeline, and enable the use and / or implementation of global memory atomics that may be shared between the graphics core 500 and the CPU within the SoC. The SoC interface 537 may also implement power management control for the graphics core 500 and implement an interface between the clock domain of the graphics core 500 and other clock domains within the SoC. In one embodiment, the SoC interface 537 implements the reception of a command buffer from a command stream converter and a global thread dispatcher configured to provide commands and instructions to each of one or more graphics cores within a graphics processor. These commands and instructions may be dispatched to the media pipeline 539 when media operations are to be performed, or to the geometry and fixed function pipelines (e.g., geometry and fixed function pipeline 536, geometry and fixed function pipeline 514) when graphics processing operations are to be performed.

[0083] The graphics microcontroller 538 may be configured to perform various scheduling and management tasks for the graphics core 500. In one embodiment, the graphics microcontroller 538 may perform graphics and / or computational workload scheduling for each graphics parallel engine within the execution unit (EU) arrays 502A-502F, 504A-504F within the sub-cores 501A-501F. In this scheduling model, host software executing on the CPU core of the SoC including the graphics core 500 may submit a workload via one of a plurality of graphics processor doorbells, which invokes a scheduling operation to the appropriate graphics engine. The scheduling operation includes determining which workload to run next, submitting the workload to the command stream converter, preempting an existing workload running on the engine, monitoring the progress of the workload, and notifying the host software when the workload is completed. In one embodiment, graphics microcontroller 538 may also facilitate a low power or idle state for graphics core 500, thereby providing graphics core 500 with the ability to save and restore registers within graphics core 500 independently of transitioning across a low power state by an operating system and / or graphics driver software on the system.

[0084] The graphics core 500 may have more or fewer than the illustrated sub-cores 501A-501F, up to N modular sub-cores. For each set of N sub-cores, the graphics core 500 may also include shared function logic 510, shared and / or cache memory 512, geometry / fixed function pipelines 514, and additional fixed function logic 516 for accelerating various graphics and compute processing operations. The shared function logic 510 may include the following: Figure 4 4, and / or associated logic units (e.g., sampler logic, math logic, and / or inter-thread communication logic) associated with the shared function logic 420 of the graphics core 500, which can be shared by each of the N sub-cores within the graphics core 500. The shared and / or cache memory 512 can be a last-level cache for the set of N sub-cores 501A-501F within the graphics core 500, and can also serve as a shared memory accessible by multiple sub-cores. The geometry / fixed function pipeline 514 can be included in the fixed function block 530 instead of the geometry / fixed function pipeline 536, and can include the same or similar logic units.

[0085] In one embodiment, the graphics core 500 includes additional fixed function logic 516, which may include various fixed function acceleration logic for use by the graphics core 500. In one embodiment, the additional fixed function logic 516 includes an additional geometry pipeline for use in position-only shading. In position-only shading, there are two geometry pipelines: a full geometry pipeline within the geometry / fixed function pipelines 516, 536; and a picking pipeline, which is an additional geometry pipeline that may be included in the additional fixed function logic 516. In one embodiment, the picking pipeline is a streamlined version of the full geometry pipeline. The full pipeline and the picking pipeline can execute different instances of the same application, each with a separate context. Position-only shading can hide long picking runs for discarded triangles, allowing shading to be completed earlier in some instances. For example and in one embodiment, the picking pipeline logic within the attachment fixed function logic 516 can execute the position shader in parallel with the main application and generally generate key results faster than the full pipeline because the full pipeline only fetches and shades the position attributes of the vertices without performing rasterization and rendering of pixels to the frame buffer. The picking pipeline can use the generated key results to calculate visibility information for all triangles, regardless of whether those triangles are picked. The full pipeline (which may be referred to as a replay pipeline in this example) can consume visibility information in order to skip picked triangles so as to only shade visible triangles that are ultimately passed to the rasterization stage.

[0086] In one embodiment, the additional fixed-function logic 516 may also include machine learning acceleration logic, such as fixed-function matrix multiplication logic, for implementations including optimizations for machine learning training or inference.

[0087] Each graphics sub-core 501A-501F includes a set of execution resources that can be used to perform graphics operations, media operations, and computing operations in response to requests from a graphics pipeline, a media pipeline, or a shader program. The graphics sub-core 501A-501F includes: multiple EU arrays 502A-502F, 504A-504F; thread dispatch and inter-thread communication (TD / IC) logic 503A-503F; 3D (e.g., texture) samplers 505A-505F; media samplers 506A-506F; shader processors 507A-507F; and shared local memory (SLM) 508A-508F. The EU arrays 502A-502F, 504A-504F each include a plurality of execution units, which are general purpose graphics processing units capable of performing floating point logic operations and integer / fixed point logic operations to serve graphics operations, media operations, or computing operations (including graphics programs, media programs, or computing shader programs). The TD / IC logic 503A-503F performs local thread dispatch and thread control operations for the execution units within the sub-core, and facilitates communication between threads executed on the execution units of the sub-core. The 3D samplers 505A-505F can read textures or other 3D graphics related data into memory. The 3D samplers can read texture data in different ways based on the configured sample state and the texture format associated with a given texture. The media samplers 506A-506F can perform similar read operations based on the type and format associated with the media data. In one embodiment, each graphics sub-core 501A-501F can alternatively include a unified 3D and media sampler. Threads executing on execution units within each of the sub-cores 501A-501F may utilize shared local memory 508A-508F within each sub-core to enable threads executing within a thread group to execute using a common pool of on-chip memory.

[0088] Execution Unit

[0089] Figure 6A-6B Thread execution logic 600 is illustrated including an array of processing elements employed in a graphics processor core according to embodiments described herein. Figure 6A-6B Elements having the same reference number (or name) as elements of any other figure herein may operate or function in any manner similar to that described elsewhere herein, but are not limited thereto. Fig. 6A FIG. 6 shows an overview of thread execution logic 600, which may include a processor illustrated as having Figure 5 A variation of the hardware logic of each sub-core 501A-501F. Figure 6B The diagram illustrates exemplary internal details of an execution unit.

[0090] like Fig. 6A As shown in FIG. 6 , in some embodiments, thread execution logic 600 includes a shader processor 602, a thread dispatcher 604, an instruction cache 606, a scalable execution unit array including a plurality of execution units 608A-608N, a sampler 610, a data cache 612, and a data port 614. In one embodiment, the scalable execution unit array can be dynamically scaled by enabling or disabling one or more execution units (e.g., any one of execution units 608A, 608B, 608C, 608D to 608N-1 and 608N) based on the computational requirements of the workload. In one embodiment, the included components are interconnected via an interconnect structure that links to each of these components. In some embodiments, thread execution logic 600 includes one or more connections to a memory (such as a system memory or a cache memory) through an instruction cache 606, a data port 614, a sampler 610, and one or more of the execution unit arrays 608A-608N. In some embodiments, each execution unit (e.g., 608A) is an independently programmable general-purpose computing unit capable of executing multiple simultaneous hardware threads while processing multiple data elements for each thread in parallel. In various embodiments, the array of execution units 608A-608N is scalable to include any number of separate execution units.

[0091] In some embodiments, execution units 608A-608N are mainly used to execute shader programs. Shader processor 602 can process various shader programs and dispatch execution threads associated with shader programs via thread dispatcher 604. In one embodiment, thread dispatcher includes logic for arbitrating thread initiation requests from graphics and media pipelines and instantiating requested threads on one or more execution units in execution units 608A-608N. For example, a geometry pipeline can dispatch vertices, tessellation or geometry shaders to thread execution logic for processing. In some embodiments, thread dispatcher 604 can also process runtime thread generation requests from executing shader programs.

[0092] In some embodiments, execution units 608A-608N support an instruction set including native support for many standard 3D graphics shader instructions, so that shader programs from graphics libraries (e.g., Direct 3D and OpenGL) can be executed with minimal conversion. These execution units support vertex and geometry processing (e.g., vertex programs, geometry programs, vertex shaders), pixel processing (e.g., pixel shaders, fragment shaders), and general processing (e.g., compute and media shaders). Each execution unit in execution units 608A-608N is capable of performing multiple-issue single instruction multiple data (SIMD), and multi-threaded operations can achieve an efficient execution environment in the face of high latency memory access. Each hardware thread within each execution unit has a dedicated high-bandwidth register file and associated independent thread state. For pipelines capable of integer, single-precision floating-point operations and double-precision floating-point operations, SIMD branch functions, logical operations, transcendental operations, and other miscellaneous operations, execution is multiple-issued per clock. While waiting for data from memory or one of the shared functions, dependency logic within execution units 608A-608N puts the waiting thread to sleep until the requested data has been returned. While the waiting thread is sleeping, hardware resources can be dedicated to processing other threads. For example, during a delay associated with a vertex shader operation, an execution unit can perform operations for a pixel shader, a fragment shader, or another type of shader program including a different vertex shader.

[0093] Each of execution units 608A-608N operates on an array of data elements. The number of data elements is the "execution size," or number of lanes of an instruction. An execution lane is a logical unit that performs data element access, masking, and flow control within an instruction. The number of lanes may be independent of the number of physical arithmetic logic units (ALUs) or floating point units (FPUs) for a particular graphics processor. In some embodiments, execution units 608A-608N support integer and floating point data types.

[0094] The execution unit instruction set includes SIMD instructions. Various data elements can be stored in registers as compact data types, and the execution unit will process various elements based on the data size of the element. For example, when a 256-bit wide vector is operated, the 256-bit vector is stored in a register, and the execution unit operates on the vector as four separate 64-bit compact data elements (data elements of quad-word (QW) size), eight separate 32-bit compact data elements (data elements of double-word (DW) size), sixteen separate 16-bit compact data elements (data elements of word (W) size), or thirty-two separate 8-bit data elements (data elements of byte (B) size). However, different vector widths and register sizes are possible.

[0095] In one embodiment, one or more execution units may be combined into fused execution units 609A-609N, which have thread control logic (607A-607N) common to the fused EUs. Multiple EUs may be fused into EU groups. Each EU in the fused EU group may be configured to execute a separate SIMD hardware thread. The number of EUs in the fused EU group may vary according to the embodiment. In addition, different SIMD widths may be executed per EU, including but not limited to SIMD8, SIMD16, and SIMD32. Each fused graphics execution unit 609A-609N includes at least two execution units. For example, the fused execution unit 609A includes a first EU 608A, a second EU 608B, and a thread control logic 607A common to the first EU 608A and the second EU 608B. The thread control logic 607A controls the threads executed on the fused graphics execution unit 609A, thereby allowing each EU in the fused execution unit 609A-609N to execute using a common instruction pointer register.

[0096] One or more internal instruction caches (e.g., 606) are included in the thread execution logic 600 to cache thread instructions for the execution unit. In some embodiments, one or more data caches (e.g., 612) are included to cache thread data during thread execution. In some embodiments, a sampler 610 is included to provide texture sampling for 3D operations and media sampling for media operations. In some embodiments, the sampler 610 includes specialized texture or media sampling functions to process texture or media data during the sampling process before providing the sampled data to the execution unit.

[0097] During execution, the graphics and media pipeline sends thread initiation requests to the thread execution logic 600 via the thread generation and dispatch logic. Once a set of geometric objects has been processed and rasterized into pixel data, the pixel processor logic (e.g., pixel shader logic, fragment shader logic, etc.) within the shader processor 602 is called to further calculate output information and cause the results to be written to the output surface (e.g., color buffer, depth buffer, stencil buffer, etc.). In some embodiments, the pixel shader or fragment shader calculates the value of each vertex attribute, which is interpolated across the rasterized object. In some embodiments, the pixel processor logic within the shader processor 602 then executes the pixel or fragment shader program supplied by the application programming interface (API). To execute the shader program, the shader processor 602 dispatches the thread to the execution unit (e.g., 608A) via the thread dispatcher 604. In some embodiments, the shader processor 602 uses the texture sampling logic in the sampler 610 to access the texture data in the texture map stored in the memory. Arithmetic operations on texture data and input geometry data compute pixel color data for each geometry fragment, or discard one or more pixels without further processing.

[0098] In some embodiments, data port 614 provides a memory access mechanism for thread execution logic 600 to output processed data to memory for further processing in the graphics processor output pipeline. In some embodiments, data port 614 includes or is coupled to one or more cache memories (e.g., data cache 612) to cache data accessed via the data port.

[0099] like Figure 6B As illustrated in FIG. 6 , the graphics execution unit 608 may include an instruction fetch unit 637, a general register file array (GRF) 624, an architectural register file array (ARF) 626, a thread arbiter 622, an issue unit 630, a branch unit 642, a set of SIMD floating point units (FPUs) 634, and a set of dedicated integer SIMD ALUs 635 in one embodiment. The GRF 624 and ARF 626 include a set of general register files and architectural register files associated with each simultaneous hardware thread that may be active in the graphics execution unit 608. In one embodiment, a per-thread architectural state is maintained in the ARF 626, while data used during thread execution is stored in the GRF 624. The execution state of each thread, including an instruction pointer for each thread, may be maintained in thread-specific registers in the ARF 626.

[0100] In one embodiment, graphics execution unit 608 has an architecture that is a combination of simultaneous multithreading (SMT) and fine-grained interleaved multithreading (IMT). The architecture has a modular configuration that can be fine-tuned at design time based on the target number of simultaneous threads and the target number of registers per execution unit, in which execution unit resources are divided across logic for executing multiple simultaneous threads.

[0101] In one embodiment, the graphics execution unit 608 can issue multiple instructions together, and these instructions can be different instructions respectively. The thread arbiter 622 of the graphics execution unit thread 608 can dispatch the instruction to one of the following items for execution: the send unit 630, the branch unit 632 or (multiple) SIMD FPU 634. Each execution thread can access 128 general registers in the GRF624, wherein each register can store 32 bytes accessible as a SIMD 8 element vector with 32-bit data elements. In one embodiment, each execution unit thread accesses 4 kilobytes in the GRF 624, but the embodiment is not limited thereto, and more or less register resources can be provided in other embodiments. In one embodiment, up to seven threads can be executed simultaneously, but the number of threads per execution unit can also vary according to the embodiment. In an embodiment where seven threads can access 4 kilobytes, the GRF 624 can store a total of 28 kilobytes. The flexible addressing mode can allow multiple registers to be addressed together, thereby efficiently establishing wider registers or representing strided rectangular block data structures.

[0102] In one embodiment, memory operations, sampler operations, and other longer latency system communications are dispatched via "send" instructions executed by message passing send unit 630. In one embodiment, branch instructions are dispatched to a dedicated branch unit 632 to facilitate SIMD divergence and eventual convergence.

[0103] In one embodiment, the graphics execution unit 608 includes one or more SIMD floating point units (FPUs) 634 for performing floating point operations. In one embodiment, the FPUs 634 also support integer calculations. In one embodiment, the FPUs 634 can SIMD perform up to a number M of 32-bit floating point (or integer) operations, or SIMD perform up to 2M 16-bit integer or 16-bit floating point operations. In one embodiment, at least one of the FPUs provides extended math capabilities that support high throughput transcendental math functions and double precision 64-bit floating points. In some embodiments, a set of 8-bit integer SIMD ALUs 635 also represents and can also be specifically optimized to perform operations associated with machine learning calculations.

[0104] In one embodiment, an array of multiple instances of graphics execution unit 608 may be instantiated when graphics sub-cores are grouped (e.g., sub-slices). For scalability, product architectures may choose the exact number of execution units per sub-core grouping. In one embodiment, execution unit 608 may execute instructions across multiple execution lanes. In further embodiments, each thread executed on graphics execution unit 608 is executed on a different lane.

[0105] Figure 7 700 is a block diagram illustrating a graphics processor instruction format 700 according to some embodiments. In one or more embodiments, a graphics processor execution unit supports an instruction set having instructions in a variety of formats. Solid line blocks illustrate components that are generally included in execution unit instructions, while dashed lines include components that are optional or included only in a subset of instructions. In some embodiments, the instruction format 700 described and illustrated are macroinstructions because they are instructions supplied to the execution unit, as opposed to micro-operations generated from instruction decoding that occurs once the instruction is processed.

[0106] In some embodiments, the graphics processor execution unit natively supports instructions using the 128-bit instruction format 710. The 64-bit compact instruction format 730 can be used for some instructions based on the selected instruction, multiple instruction options, and the number of operands. The native 128-bit instruction format 710 provides access to all instruction options, while some options and operations are limited in the 64-bit format 730. The native instructions available in the 64-bit format 730 vary according to the embodiment. In some embodiments, the instructions are partially compressed using a set of index values ​​in the index field 713. The execution unit hardware references a set of compression tables based on the index values ​​and uses the compression table output to reconstruct the native instructions using the 128-bit instruction format 710.

[0107] For each format, the instruction opcode 712 defines the operation to be performed by the execution unit. The execution unit executes each instruction in parallel across multiple data elements of each operand. For example, in response to an add instruction, the execution unit performs a synchronous add operation across each color channel of an identified texture element or picture element. By default, the execution unit executes each instruction across all data channels of the operand. In some embodiments, the instruction control field 714 implements control of certain execution options, such as channel selection (e.g., assertion) and data channel sorting (e.g., mixing). For instructions using the 128-bit instruction format 710, the execution size field 716 limits the number of data channels that will be executed in parallel. In some embodiments, the execution size field 716 is not available for the 64-bit compact instruction format 730.

[0108] Some execution unit instructions have up to three operands, including two source operands src0 720, src1 722, and a destination 718. In some embodiments, the execution unit supports dual destination instructions, where one of the destinations is implicit. Data manipulation instructions may have a third source operand (e.g., SRC2 724), where the instruction opcode 712 determines the number of source operands. The last source operand of an instruction may be an immediate (e.g., hard-coded) value passed with the instruction.

[0109] In some embodiments, the 128-bit instruction format 710 includes an access / address mode field 726 that specifies, for example, whether direct register addressing mode or indirect register addressing mode is used. When direct register addressing mode is used, the register address of one or more operands is provided directly by bits in the instruction.

[0110] In some embodiments, the 128-bit instruction format 710 includes an access / address mode field 726 that specifies an address mode and / or access mode for the instruction. In one embodiment, the access mode is used to define the data access alignment for the instruction. Some embodiments support access modes including a 16-byte aligned access mode and a 1-byte aligned access mode, wherein the byte alignment of the access mode determines the access alignment of the instruction operand. For example, when in a first mode, the instruction may use byte-aligned addressing for source operands and destination operands, and when in a second mode, the instruction may use 16-byte aligned addressing for all source operands and destination operands.

[0111] In one embodiment, the address mode portion of the access / address mode field 726 determines whether the instruction uses direct addressing or indirect addressing. When direct register addressing mode is used, the bits in the instruction directly provide the register address of one or more operands. When indirect register addressing mode is used, the register address of one or more operands can be calculated based on the address register value and the address immediate field in the instruction.

[0112] In some embodiments, instructions are grouped based on the bit field of the opcode 712 to simplify the opcode decoding 740. For 8-bit opcodes, bits 4, 5, and 6 allow the execution unit to determine the type of the opcode. The exact opcode grouping shown is only an example. In some embodiments, the move and logic opcode group 742 includes data movement and logic instructions (e.g., move (mov), compare (cmp)). In some embodiments, the move and logic group 742 shares five most significant bits (MSBs), wherein the move (mov) instruction adopts the form of 0000xxxxb, and the logic instruction adopts the form of 0001xxxxb. The flow control instruction group 744 (e.g., call (call), jump (jmp)) includes instructions using the 0010xxxxb form (e.g., 0x20). The miscellaneous instruction group 746 includes a mixture of instructions, including synchronization instructions (e.g., wait (wait), send (send)) using the 0011xxxxb form (e.g., 0x30). The parallel math instruction group 748 includes component-wise arithmetic instructions (e.g., add, mul) in the form of 0100xxxxb (e.g., 0x40). The parallel math group 748 performs arithmetic operations in parallel across data lanes. The vector math group 750 includes arithmetic instructions (e.g., dp4) in the form of 0101xxxxb (e.g., 0x50). The vector math group performs arithmetic operations on vector operands, such as dot product calculations.

[0113] Graphics Pipeline

[0114] Figure 8 is a block diagram of another embodiment of a graphics processor 800 . Figure 8 Elements having the same reference number (or name) as elements in any other figure herein may operate or function in any manner similar to that described elsewhere herein, but are not limited thereto.

[0115] In some embodiments, the graphics processor 800 includes a geometry pipeline 820, a media pipeline 830, a display engine 840, a thread execution logic 850, and a rendering output pipeline 870. In some embodiments, the graphics processor 800 is a graphics processor within a multi-core processing system that includes one or more general-purpose processing cores. The graphics processor is controlled by register writes to one or more control registers (not shown) or via commands issued to the graphics processor 800 through the ring interconnect 802. In some embodiments, the ring interconnect 802 couples the graphics processor 800 to other processing components, such as other graphics processors or general-purpose processors. Commands from the ring interconnect 802 are interpreted by a command stream converter 803, which supplies instructions to individual components of the geometry pipeline 820 or the media pipeline 830.

[0116] In some embodiments, command stream converter 803 directs the operation of vertex fetcher 805, which reads vertex data from memory and executes vertex processing commands provided by command stream converter 803. In some embodiments, vertex fetcher 805 provides vertex data to vertex shader 807, which performs coordinate space transformation and lighting operations on each vertex. In some embodiments, vertex fetcher 805 and vertex shader 807 execute vertex processing instructions by dispatching execution threads to execution units 852A-852B via thread dispatcher 831.

[0117] In some embodiments, execution units 852A-852B are vector processor arrays with instruction sets for performing graphics and media operations. In some embodiments, execution units 852A-852B have an additional L1 cache 851 dedicated to each array or shared between multiple arrays. The cache can be configured as a data cache, an instruction cache, or a single cache that is partitioned to contain data and instructions in different partitions.

[0118] In some embodiments, the geometry pipeline 820 includes a tessellation component for performing hardware-accelerated tessellation of 3D objects. In some embodiments, a programmable hull shader 811 configures the tessellation operation. A programmable domain shader 817 provides back-end evaluation of the tessellation output. The tessellation 813 operates under the direction of the hull shader 811 and contains dedicated logic for generating a collection of detailed geometric objects based on a coarse geometric model provided as input to the geometry pipeline 820. In some embodiments, if tessellation is not used, the tessellation components (e.g., hull shader 811, tessellation 813, domain shader 817) can be bypassed.

[0119] In some embodiments, the complete geometric object may be processed by the geometry shader 819 via one or more threads dispatched to the execution units 852A-852B, or may proceed directly to the clipper 829. In some embodiments, the geometry shader operates on the entire geometric object, rather than on vertices or vertex patches as in previous stages of the graphics pipeline. If tessellation is disabled, the geometry shader 819 receives input from the vertex shader 807. In some embodiments, the geometry shader 819 is programmable by a geometry shader program to perform geometry tessellation when the tessellation unit is disabled.

[0120] Before rasterization, the clipper 829 processes the vertex data. The clipper 829 can be a fixed function clipper or a programmable clipper with clipping and geometry shader functions. In some embodiments, the rasterizer and depth test component 873 in the render output pipeline 870 dispatches a pixel shader to convert the geometric objects into a per-pixel representation. In some embodiments, the pixel shader logic is included in the thread execution logic 850. In some embodiments, the application can bypass the rasterizer and depth test component 873 and access the unrasterized vertex data via the outflow unit 823.

[0121] The graphics processor 800 has an interconnect bus, interconnect structure or some other interconnect mechanism that allows data and messages to be passed between the main components of the processor. In some embodiments, the execution units 852A-852B and associated logic units (e.g., L1 cache 851, sampler 854, texture cache 858, etc.) are interconnected via data ports 856 to perform memory accesses and communicate with the rendering output pipeline components of the processor. In some embodiments, the sampler 854, caches 851, 858, and execution units 852A-852B each have a separate memory access path. In one embodiment, the texture cache 858 can also be configured as a sampler cache.

[0122] In some embodiments, the rendering output pipeline 870 includes a rasterizer and depth test component 873, which converts vertex-based objects into associated pixel-based representations. In some embodiments, the rasterizer logic includes a windower / masker unit for performing fixed-function triangle and line rasterization. An associated rendering cache 878 and depth cache 879 are also available in some embodiments. The pixel operation component 877 performs pixel-based operations on data, but in some instances, pixel operations associated with 2D operations (e.g., using mixed bit block image transfer) are performed by the 2D engine 841, or replaced by the display controller 843 using overlapping display planes at display time. In some embodiments, a shared L3 cache 875 can be used for all graphics components, allowing data to be shared without using main system memory.

[0123] In some embodiments, the graphics processor media pipeline 830 includes a media engine 837 and a video front end 834. In some embodiments, the video front end 834 receives pipeline commands from the command stream converter 803. In some embodiments, the media pipeline 830 includes a separate command stream converter. In some embodiments, the video front end 834 processes the media commands before sending the commands to the media engine 837. In some embodiments, the media engine 837 includes a thread generation function for generating threads for dispatching to the thread execution logic 850 via the thread dispatcher 831.

[0124] In some embodiments, the graphics processor 800 includes a display engine 840. In some embodiments, the display engine 840 is external to the processor 800 and is coupled to the graphics processor via the ring interconnect 802, or some other interconnect bus or structure. In some embodiments, the display engine 840 includes a 2D engine 841 and a display controller 843. In some embodiments, the display engine 840 includes dedicated logic that can operate independently of the 3D pipeline. In some embodiments, the display controller 843 is coupled to a display device (not shown), which can be a system-integrated display device (such as in a laptop computer) or an external display device attached via a display device connector.

[0125] In some embodiments, geometry pipeline 820 and media pipeline 830 can be configured to perform operations based on multiple graphics and media programming interfaces, and are not dedicated to any application programming interface (API). In some embodiments, the driver software of the graphics processor converts API calls dedicated to specific graphics or media libraries into commands that can be processed by the graphics processor. In some embodiments, support is provided for open graphics library (OpenGL), open computing language (OpenCL) and / or Vulkan graphics and computing API (all from Khronos Group). In some embodiments, support can also be provided for the Direct3D library from Microsoft. In some embodiments, the combination of these libraries can be supported. Support can also be provided for the open source computer vision library (OpenCV). If a mapping from the pipeline of future API to the pipeline of graphics processor can be made, the future API with compatible 3D pipelines will also be supported.

[0126] Graphics Pipeline Programming

[0127] Fig. 9A is a block diagram illustrating a graphics processor command format 900 according to some embodiments. Fig. 9B is a block diagram illustrating a graphics processor command sequence 910 according to an embodiment. Fig. 9AThe solid-line boxes in the figure show components that are generally included in the graphics commands, while the dashed lines include components that are optional or included only in a subset of the graphics commands. Fig. 9A The exemplary graphics processor command format 900 includes a data field for identifying the client 902, a command operation code (opcode) 904, and data for the command 906. Some commands also include a sub-opcode 905 and a command size 908.

[0128] In some embodiments, client 902 specifies the client unit of the graphics device that processes the command data. In some embodiments, the graphics processor command parser checks the client field of each command to adjust the further processing of the command and routes the command data to the appropriate client unit. In some embodiments, the graphics processor client unit includes a memory interface unit, a rendering unit, a 2D unit, a 3D unit, and a media unit. Each client unit has a corresponding processing pipeline that processes the command. Once the command is received by the client unit, the client unit reads the opcode 904 and the sub-opcode 905 (if present) to determine the operation to be performed. The client unit uses the information in the data field 906 to execute the command. For some commands, an explicit command size 908 is expected to specify the size of the command. In some embodiments, the command parser automatically determines the size of at least some of the commands in the command based on the command opcode. In some embodiments, the command is aligned via a multiple of a double word.

[0129] Fig. 9B 910. In some embodiments, software or firmware of a data processing system featuring an embodiment of a graphics processor uses the illustrated version of the command sequence to initiate, execute, and terminate a set of graphics operations. The sample command sequence is shown and described for exemplary purposes only, as the embodiments are not limited to these particular commands, nor to this command sequence. Moreover, the commands may be issued as a batch of commands in a command sequence so that the graphics processor will process the command sequence in an at least partially simultaneous manner.

[0130] In some embodiments, the graphics processor command sequence 910 may begin with: a pipeline dump clear command 912 to cause any active graphics pipeline to complete the currently pending commands of that pipeline. In some embodiments, the 3D pipeline 922 and the media pipeline 924 do not operate simultaneously. The pipeline dump clear is performed to cause the active graphics pipeline to complete any pending commands. In response to the pipeline dump clear, the command parser for the graphics processor will suspend command processing until the active drawing engine completes the pending operation and the associated read cache is invalidated. Optionally, any data marked as 'dirty' in the rendering cache can be dumped to memory. In some embodiments, the pipeline dump clear command 912 can be used for pipeline synchronization or before placing the graphics processor in a low power state.

[0131] In some embodiments, when a command sequence requires the graphics processor to explicitly switch between pipelines, pipeline select command 913 is used. In some embodiments, pipeline select command 913 is only required once in the execution context before issuing pipeline commands, unless the context is to issue commands for both pipelines. In some embodiments, pipeline flush command 912 is required just before a pipeline switch via pipeline select command 913.

[0132] In some embodiments, pipeline control commands 914 configure the graphics pipeline for operation and are used to program the 3D pipeline 922 and the media pipeline 924. In some embodiments, pipeline control commands 914 configure the pipeline state for the active pipeline. In one embodiment, pipeline control commands 914 are used for pipeline synchronization and for clearing data from one or more cache memories within the active pipeline before processing a batch of commands.

[0133] In some embodiments, the return buffer state command 916 is used to configure a set of return buffers for the corresponding pipeline to write data. Some pipeline operations require the allocation, selection, or configuration of one or more return buffers, and during processing, these operations write intermediate data to the one or more return buffers. In some embodiments, the graphics processor also uses one or more return buffers to store output data and perform cross-thread communication. In some embodiments, the return buffer state 916 includes the size and number of return buffers selected for the set of pipeline operations.

[0134] The remaining commands in the command sequence differ based on the active pipeline for operation. Based on pipeline decision 920 , the command sequence is tailored for either 3D pipeline 922 starting with 3D pipeline state 930 , or media pipeline 924 starting with media pipeline state 940 .

[0135] The commands for configuring the 3D pipeline state 930 include 3D state setting commands for vertex buffer state, vertex element state, constant color state, depth buffer state, and other state variables to be configured before processing 3D primitive commands. The values ​​of these commands are determined at least in part based on the specific 3D API in use. In some embodiments, the 3D pipeline state 930 commands can also selectively disable or bypass certain pipeline elements if they will not be used.

[0136] In some embodiments, 3D primitive 932 commands are used to submit 3D primitives to be processed by the 3D pipeline. The commands and associated parameters passed to the graphics processor via the 3D primitive 932 commands will be forwarded to the vertex acquisition function in the graphics pipeline. The vertex acquisition function uses the 3D primitive 932 command data to generate a vertex data structure. The vertex data structure is stored in one or more return buffers. In some embodiments, 3D primitive 932 commands are used to perform vertex operations on the 3D primitives via vertex shaders. In order to process vertex shaders, the 3D pipeline 922 dispatches the shader execution thread to the graphics processor execution unit.

[0137] In some embodiments, the 3D pipeline 922 is triggered via an execution 934 command or event. In some embodiments, a register write triggers the command execution. In some embodiments, the execution is triggered via a 'go' or 'kick' command in a command sequence. In one embodiment, the command execution is triggered using a pipeline synchronization command to flush the command sequence through the graphics pipeline. The 3D pipeline will perform geometry processing for the 3D primitives. Once the operation is completed, the resulting geometric object is rasterized and the pixel engine shades the resulting pixels. For those operations, additional commands for controlling pixel shading and pixel backend operations may also be included.

[0138] In some embodiments, when performing media operations, the graphics processor command sequence 910 follows the media pipeline 924 path. Generally, the specific purpose and manner of programming for the media pipeline 924 depends on the media or computing operation to be performed. During media decoding, specific media decoding operations can be offloaded to the media pipeline. In some embodiments, the media pipeline can also be bypassed, and the resources provided by one or more general processing cores can be used to perform media decoding in whole or in part. In one embodiment, the media pipeline also includes elements for general purpose graphics processor unit (GPGPU) operations, wherein the graphics processor is used to perform SIMD vector operations using a compute shader program that is not explicitly related to rendering graphics primitives.

[0139] In some embodiments, the media pipeline 924 is configured in a similar manner to the 3D pipeline 922. A set of commands for configuring the media pipeline state 940 is dispatched or placed into the command queue, before the media object commands 942. In some embodiments, the commands for the media pipeline state 940 include data for configuring the media pipeline elements that will be used to process the media objects. This includes data for configuring the video decoding and video encoding logic within the media pipeline, such as encoding or decoding formats. In some embodiments, the commands for the media pipeline state 940 also support the use of one or more pointers for "indirect" state elements that contain batches of state settings.

[0140] In some embodiments, media object command 942 supplies a pointer to a media object for processing by the media pipeline. The media object includes a memory buffer that contains video data to be processed. In some embodiments, all media pipeline states must be valid before issuing media object command 942. Once the pipeline state is configured and media object command 942 is queued, media pipeline 924 is triggered via execution 944 commands or equivalent execution events (e.g., register writes). The output from media pipeline 924 can then be post-processed by operations provided by 3D pipeline 922 or media pipeline 924. In some embodiments, GPGPU operations are configured and executed in a manner similar to media operations.

[0141] Graphics software architecture

[0142] Fig.10 An exemplary graphics software architecture for data processing system 1000 is shown in accordance with some embodiments. In some embodiments, the software architecture includes 3D graphics application 1010, operating system 1020, and at least one processor 1030. In some embodiments, processor 1030 includes graphics processor 1032 and one or more general purpose processor cores 1034. Graphics application 1010 and operating system 1020 are each executed in system memory 1050 of the data processing system.

[0143] In some embodiments, the 3D graphics application 1010 includes one or more shader programs that include shader instructions 1012. The shader language instructions may be in a high-level shader language, such as High Level Shader Language (HLSL) or OpenGL Shader Language (GLSL). The application also includes executable instructions 1014 in a machine language suitable for execution by the general purpose processor core 1034. The application also includes graphics objects 1016 defined by vertex data.

[0144] In some embodiments, operating system 1020 is from Microsoft Corporation Operating system 1020 can support graphics API 1022, such as Direct3D API, OpenGL API or Vulkan API. When Direct3D API is in use, operating system 1020 uses front-end shader compiler 1024 to compile any shader instruction 1012 using HLSL into a lower-level shader language. Compilation can be just-in-time (JIT) compilation, or application can perform shader precompilation. In some embodiments, during compilation of 3D graphics application 1010, high-level shaders are compiled into low-level shaders. In some embodiments, shader instructions 1012 are provided in an intermediate form (such as a version of the standard portable intermediate representation (SPIR) used by Vulkan API).

[0145] In some embodiments, the user mode graphics driver 1026 includes a backend shader compiler 1027 for converting shader instructions 1012 into hardware-specific representations. When the OpenGL API is in use, the shader instructions 1012 in the GLSL high-level language are passed to the user mode graphics driver 1026 for compilation. In some embodiments, the user mode graphics driver 1026 uses the operating system kernel mode function 1028 to communicate with the kernel mode graphics driver 1029. In some embodiments, the kernel mode graphics driver 1029 communicates with the graphics processor 1032 to dispatch commands and instructions.

[0146] IP Core Implementation

[0147] One or more aspects of at least one embodiment may be implemented by representative code stored on a machine-readable medium that represents and / or defines logic within an integrated circuit (such as a processor). For example, a machine-readable medium may include instructions representing the various logics within a processor. When read by a machine, the instructions may cause the machine to manufacture logic for performing the techniques described herein. Such representations (referred to as "IP cores") are reusable units of logic for an integrated circuit that may be stored on a tangible machine-readable medium as a hardware model that describes the structure of the integrated circuit. The hardware model may be supplied to various consumers or manufacturing facilities that load the hardware model on a manufacturing machine that manufactures the integrated circuit. The integrated circuit may be manufactured so that the circuit performs the operations described in association with any of the embodiments described herein.

[0148] Fig.11A 1 is a block diagram illustrating an IP core development system 1100 that can be used to manufacture an integrated circuit to perform operations according to an embodiment. The IP core development system 1100 can be used to generate a modular, reusable design that can be incorporated into a larger design or used to build an entire integrated circuit (e.g., a SOC integrated circuit). The design facility 1130 can use a high-level programming language (e.g., C / C++) to generate a software simulation 1110 for the IP core design. The software simulation 1110 can be used to design, test and verify the behavior of the IP core using a simulation model 1112. The simulation model 1112 can include functional, behavioral and / or timing simulations. Then a register transfer level (RTL) design 1115 can be created or synthesized from the simulation model 1112. The RTL design 1115 is an abstraction of the behavior of an integrated circuit (including associated logic executed using the modeled digital signals) that models the flow of digital signals between hardware registers. In addition to the RTL design 1115, a lower-level design at the logic level or transistor level can also be created, designed or synthesized. Thus, the specific details of the initial design and simulation can be different.

[0149] The RTL design 1115 or an equivalent solution may be further synthesized by the design facility into a hardware model 1120, which may be in a hardware description language (HDL) or some other representation of physical design data. The HDL may be further simulated or tested to verify the IP core design. Non-volatile memory 1140 (e.g., a hard disk, flash memory, or any non-volatile storage medium) may be used to store the IP core design for delivery to a third-party manufacturing facility 1165. Alternatively, the IP core design may be transmitted (e.g., via the Internet) via a wired connection 1150 or a wireless connection 1160. The manufacturing facility 1165 may then manufacture an integrated circuit based at least in part on the IP core design. The manufactured integrated circuit may be configured to perform operations according to at least one embodiment described herein.

[0150] Fig. 11BThe figure shows a cross-sectional side view of an integrated circuit package assembly 1170 according to some embodiments described herein. The integrated circuit package assembly 1170 illustrates an implementation of one or more processors or accelerator devices as described herein. The package assembly 1170 includes a plurality of hardware logic units 1172, 1174 connected to a substrate 1180. The logic 1172, 1174 may be implemented at least in part in configurable logic or fixed-function logic hardware, and may include one or more portions of any of (multiple) processor cores, (multiple) graphics processors, or other accelerator devices described herein. Each logic unit 1172, 1174 may be implemented within a semiconductor die and coupled to the substrate 1180 via an interconnect structure 1173. The interconnect structure 1173 may be configured to route electrical signals between the logic 1172, 1174 and the substrate 1180, and may include interconnects such as, but not limited to, bumps or pillars. In some embodiments, the interconnect structure 1173 may be configured to route electrical signals, such as, for example, input / output (I / O) signals and / or power or ground signals associated with the operation of the logic 1172, 1174. In some embodiments, the substrate 1180 is an epoxy-based laminate substrate. In other embodiments, the package assembly 1170 may include other suitable types of substrates. The package assembly 1170 may be connected to other electrical devices via the package interconnect 1183. The package interconnect 1183 may be coupled to the surface of the substrate 1180 so as to route electrical signals to other electrical devices, such as a motherboard, other chipsets, or multi-chip modules.

[0151] In some embodiments, the logic units 1172, 1174 are electrically coupled to a bridge 1182 configured to route electrical signals between the logic units 1172, 1174. The bridge 1182 may be a dense interconnect structure that provides routing for electrical signals. The bridge 1182 may include a bridge substrate composed of glass or a suitable semiconductor material. Circuit features may be formed on the bridge substrate to provide chip-to-chip connections between the logic units 1172, 1174.

[0152] Although two logic units 1172, 1174 and bridge 1182 are illustrated, the embodiments described herein may include more or fewer logic units on one or more dies. One or more dies may be connected by zero or more bridges, as bridge 1182 may be excluded when logic is included on a single die. Alternatively, multiple dies or logic units may be connected by one or more bridges. In addition, multiple logic units, dies, and bridges may be connected together in other possible configurations including three-dimensional configurations.

[0153] Exemplary System-on-Chip Integrated Circuit

[0154] Figure 12-14B The figure shows an exemplary integrated circuit and associated graphics processor that can be manufactured using one or more IP cores according to various embodiments described herein. In addition to what is shown, other logic and circuits may also be included, including additional graphics processors / cores, peripheral interface controllers, or general-purpose processor cores.

[0155] Fig.12 1 is a block diagram illustrating an exemplary system-on-chip integrated circuit 1200 that can be manufactured using one or more IP cores according to an embodiment. The exemplary integrated circuit 1200 includes one or more application processors 1205 (e.g., CPUs), at least one graphics processor 1210, and may additionally include an image processor 1215 and / or a video processor 1220, any of which may be modular IP cores from the same or multiple different design facilities. The integrated circuit 1200 includes peripheral or bus logic that includes a USB controller 1225, a UART controller 1230, a SPI / SDIO controller 1235, and an I2S / I2C controller 1240. In addition, the integrated circuit may include a display device 1245 that is coupled to one or more of a high-definition multimedia interface (HDMI) controller 1250 and a mobile industry processor interface (MIPI) display interface 1255. Storage may be provided by a flash memory subsystem 1260 (including flash memory and a flash memory controller). A memory interface may be provided via a memory controller 1265 to access SDRAM or SRAM memory devices. Additionally, some integrated circuits additionally include an embedded security engine 1270 .

[0156] Figure 13A-13B is a block diagram illustrating an exemplary graphics processor for use within a SoC according to embodiments described herein. Fig.13A An exemplary graphics processor 1310 of a system on chip integrated circuit that may be fabricated using one or more IP cores according to an embodiment is shown. Fig. 13B An additional exemplary graphics processor 1340 of a system on a chip integrated circuit that may be fabricated using one or more IP cores in accordance with an embodiment is shown. Fig.13A Graphics processor 1310 is an example of a low-power graphics processor core. Fig. 13B The graphics processor 1340 of FIG. 1 is an example of a higher performance graphics processor core. Each of the graphics processors 1310 and 1340 may be Fig.12 A variant of graphics processor 1210.

[0157] like Fig.13AAs shown in , the graphics processor 1310 includes a vertex processor 1305 and one or more fragment processors 1315A-1315N (e.g., 1315A, 1315B, 1315C, 1315D to 1315N-1 and 1315N). The graphics processor 1310 can execute different shader programs via separate logic so that the vertex processor 1305 is optimized to perform operations for the vertex shader program, while one or more fragment processors 1315A-1315N perform fragment (e.g., pixel) shading operations for fragment or pixel shader programs. The vertex processor 1305 performs the vertex processing stage of the 3D graphics pipeline and generates primitives and vertex data. (Multiple) fragment processors 1315A-1315N use the primitives and vertex data generated by the vertex processor 1305 to generate a frame buffer displayed on a display device. In one embodiment, the fragment processor(s) 1315A-1315N are optimized to execute fragment shader programs as provided for use in the OpenGL API, the fragment processors 1315A-1315N may be used to perform similar operations as pixel shader programs as provided for use in the Direct 3D API.

[0158] The graphics processor 1310 additionally includes one or more memory management units (MMUs) 1320A-1320B, cache(s) 1325A-1325B, and circuit interconnect(s) 1330A-1330B. The one or more MMUs 1320A-1320B provide virtual to physical address mappings for the graphics processor 1310, including virtual to physical address mappings for the vertex processor 1305 and / or the fragment processor(s) 1315A-1315N, which may reference vertex or image / texture data stored in memory in addition to the vertex or image / texture data stored in the one or more caches 1325A-1325B. In one embodiment, the one or more MMUs 1320A-1320B may be synchronized with other MMUs within the system so that each processor 1205-1220 may participate in a shared or unified virtual memory system, including with other MMUs. Fig.12 One or more MMUs associated with one or more application processors 1205, image processor 1215, and / or video processor 1220. According to an embodiment, one or more circuit interconnects 1330A-1330B enable graphics processor 1310 to interface with other IP cores within the SoC via an internal bus of the SoC or via a direct connection.

[0159] like Fig. 13B As shown in FIG. 1 , the graphics processor 1340 includes Fig.13AOne or more MMUs 1320A-1320B, caches 1325A-1325B, and circuit interconnects 1330A-1330B of a graphics processor 1310. Graphics processor 1340 includes one or more shader cores 1355A-1355N, which one or more shader cores 1355A-1355N (e.g., 1355A, 1355B, 1355C, 1355D, 1355E, 1355F to 1355N-1 and 1355N) provide a unified shader core architecture in which a single core or a single type of core can execute all types of programmable shader code, including shader program code for implementing vertex shaders, fragment shaders, and / or compute shaders. The exact number of shader cores present may vary between embodiments and implementations. In addition, the graphics processor 1340 includes an inter-core task manager 1345 and a tiling unit 1358, wherein the inter-core task manager 1345 acts as a thread dispatcher to dispatch execution threads to one or more shader cores 1355A-1355N, and the tiling unit 1358 is used to accelerate tiling operations for tile-based rendering, in which rendering operations on a scene are subdivided in image space, such as to exploit local spatial consistency within a scene, or to optimize the use of internal caches.

[0160] Figure 14A-14B Additional exemplary graphics processor logic is illustrated according to embodiments described herein. Fig.14A The figure shows a graphics core 1400, which may include Fig.12 The graphics processor 1210 may be Fig. 13B Unified shader cores 1355A-1355N in. Fig. 14B A highly parallel general purpose graphics processing unit 1430 suitable for deployment on a multi-chip module is shown.

[0161] like Fig.14AAs shown in , graphics core 1400 includes a shared instruction cache 1402, texture units 1418, and cache memory / shared memory 1420 that are common to execution resources within graphics core 1400. Graphics core 1400 may include multiple slices 1401A-1401N or partitions for each core, and the graphics processor may include multiple instances of graphics core 1400. Slices 1401A-1401N may include support logic including local instruction caches 1404A-1404N, thread schedulers 1406A-1406N, thread dispatchers 1408A-1408N, and a set of registers 1410A. To perform logical operations, slices 1401A-1401N may include a set of additional function units (AFU 1412A-1412N), floating point units (FPU 1414A-1414N), integer arithmetic logic units (ALU 1416A-1416N), addressing calculation units (ACU 1413A-1413N), double precision floating point units (DPFPU 1415A-1415N), and matrix processing units (MPU 1417A-1417N).

[0162] Some of these computing units operate with specific precision. For example, FPU 1414A-1414N can perform single-precision (32-bit) and half-precision (16-bit) floating-point operations, while DPFPU 1415A-1415N performs double-precision (64-bit) floating-point operations. ALU 1416A-1416N can perform variable-precision integer operations with 8-bit precision, 16-bit precision, and 32-bit precision, and can be configured for mixed-precision operations. MPU 1417A 1417N can also be configured for mixed-precision matrix operations, including half-precision floating-point operations and 8-bit integer operations. MPU 1417A-1417N can perform a variety of matrix operations to accelerate machine learning application frameworks, including implementation of support for accelerated general matrix-matrix multiplication (GEMM). AFU 1412A-1412N can perform additional logical operations that are not supported by floating-point units or integer units, including trigonometric operations (e.g., sine, cosine, etc.).

[0163] like Fig. 14BAs shown in , a general purpose processing unit (GPGPU) 1430 can be configured to enable highly parallel computing operations to be performed by a graphics processing unit array. In addition, GPGPU 1430 can be directly linked to other instances of GPGPU to create a multi-GPU cluster, thereby improving the training speed of deep neural networks in particular. GPGPU 1430 includes a host interface 1432 for implementing a connection with a host processor. In one embodiment, host interface 1432 is a PCI express interface. However, the host interface can also be a supplier-specific communication interface or communication structure. GPGPU 1430 receives commands from the host processor and uses a global scheduler 1434 to distribute the execution threads associated with those commands to a group of computing clusters 1436A-1436H. Computing clusters 1436A-1436H share cache memory 1438. Cache memory 1438 can act as a higher-level cache of cache memory within computing clusters 1436A-1436H.

[0164] GPGPU 1430 includes memory 1434A-1434B coupled to compute clusters 1436A-1436H via a set of memory controllers 1442A-1442B. In various embodiments, memory 1434A-1434B may include various types of memory devices, including dynamic random access memory (DRAM) or graphics random access memory, such as synchronous graphics random access memory (SGRAM), including graphics double data rate (GDDR) memory.

[0165] In one embodiment, computing clusters 1436A-1436H each include a set of graphics cores (e.g. Fig.14A The graphics cores 1400 may include multiple types of integer logic units and floating point logic units that may perform computational operations within a range of precision, including those suitable for machine learning computations. For example and in one embodiment, at least a subset of the floating point units in each of the computing clusters 1436A-1436H may be configured to perform 16-bit or 32-bit floating point operations, while a different subset of the floating point units may be configured to perform 64-bit floating point operations.

[0166] Multiple instances of GPGPU 1430 can be configured to operate as a computing cluster. The communication mechanism used by the computing cluster for synchronization and data exchange varies across embodiments. In one embodiment, multiple instances of GPGPU 1430 communicate through a host interface 1432. In one embodiment, GPGPU 1430 includes an I / O hub 1439, which couples GPGPU 1430 with a GPU link 1440 that implements direct connection to other instances of GPGPU. In one embodiment, GPU link 1440 is coupled to a dedicated GPU to GPU bridge that implements communication and synchronization between multiple instances of GPGPU 1430. In one embodiment, GPU link 1440 is coupled to a high-speed interconnect to transmit and receive data to other GPGPUs or parallel processors. In one embodiment, multiple instances of GPGPU 1430 are located in separate data processing systems and communicate via a network device that can be accessed via host interface 1432. In one embodiment, GPU link 1440 may be configured to enable connection to a host processor in addition to or as an alternative to host interface 1432 .

[0167] Although the illustrated configuration of GPGPU 1430 can be configured to train a neural network, one embodiment provides an alternative configuration of GPGPU 1430 that can be configured to be deployed within a high-performance or low-power inference platform. In the inference configuration, GPGPU 1430 includes fewer of the compute clusters 1436A-1436H that are associated with the training configuration. In addition, the memory technology associated with memory 1434A-1434B may be different between the inference configuration and the training configuration, wherein a higher bandwidth memory technology is dedicated to the training configuration. In one embodiment, the inference configuration of GPGPU 1430 may support inference specific instructions. For example, the inference configuration may provide support for one or more 8-bit integer dot product instructions that are typically used during inference operations of a deployed neural network.

[0168] Immersive Video Overview

[0169] Fig.15AVarious forms of immersive video are shown. Immersive video can be presented in a variety of forms, depending on the degrees of freedom available to the viewer. Degrees of freedom refer to the number of different directions an object can move in three-dimensional (3D) space. Immersive video can be viewed via a head-mounted display that includes tracking of position and orientation. Example forms of immersive video include 3DoF 1502, 3DoF+1504, and full 6DoF 1506. In addition to immersive video in full 6DoF 1506, 6DoF immersive video also includes omnidirectional 6DoF 1507 and windowed 6DoF 1508.

[0170] For video in 3DoF 1502 (e.g., 360-degree video), the viewer can change orientation (e.g., yaw, pitch, roll) but not position. For video in 3DoF+ 1504, the viewer can change orientation and make minor changes to position changes. For video in 6DoF 1506, the viewer can change orientation and change position. More limited forms of 6DoF video are also available. Video in omnidirectional 6DoF 1507 enables the viewer to take multiple steps in a virtual scene. Video in windowed 6DoF 1508 allows the viewer to change orientation and position, but the viewer is constrained to a limited viewing area. Increasing the available degrees of freedom in immersive video typically includes increasing the amount and complexity of data involved in video generation, encoding, decoding, and playback.

[0171] Fig. 15B Image projection and texture planes for immersive video are shown. A 3D view 1510 of video content can be generated using data from multiple cameras. Multiple projection planes 1512 can be used to generate geometric data for the video content. Multiple texture planes 1514 can be derived for the projection planes 1512 used to generate the geometric data. The texture planes 1514 can be applied to a 3D model that is pre-generated or generated based on a point cloud derived from the video data. The multiple projection planes 1512 can be used to generate multiple two-dimensional (2D) projections, each 2D projection being associated with a projection plane.

[0172] Fig.16A client-server system is presented by which immersive video content can be generated and encoded by a server 1620 infrastructure for transmission to one or more client 1630 devices. The client 1630 devices can then decompress and render the immersive video content. In one embodiment, one or more server 1620 devices may include input from one or more optical cameras 1601 with depth sensors 1602. Parallel computing 1604 resources can decompose video and depth data into point clouds 1605 and / or texture triangles 1606. Data for generating texture triangles 1606 can also be provided by a pre-generated 3D model 1603 of the scene. Point clouds 1605 and / or texture triangles 1606 can be compressed for transmission to one or more client devices that can render the content locally. In one embodiment, various compression units 1607, 1608 can compress the generated content using various compression algorithms for transmission from the server 1620 to one or more client 1630 devices via a delivery medium. The decompression units 1609, 1610 on the client 1630 device can decompress and decode the incoming bitstream into video / texture and geometry data. For example, the decompression unit 1609 can decode the compressed point cloud data and provide the decompressed point cloud data to the viewpoint interpolation unit 1611. The interpolated viewpoint data can be used to generate bitmap data 1613. The decompressed point cloud data can be provided to the geometry reconstruction unit 1612 to reconstruct the geometry data of the scene. The reconstructed geometry data can be textured by the decoded texture data (texture triangles 1614) to generate a 3D rendering 1616 for viewing by the client 1630.

[0173] FIG. 17A to FIG. 17B Systems 1700, 1710 for encoding and decoding 3DoF+ content are shown. System 1700 may be implemented by, for example, Fig.16 The system 1710 may be implemented by hardware and software of the server 1620 infrastructure in FIG. Fig.16 The hardware and software of the client 1630 are implemented.

[0174] like Fig.17AAs shown, the system 1700 can be used to encode the video data 1702 of the base view 1701 and the video data 1705A to 1705C of the additional view 1704. Multiple cameras can provide input data including video data and depth data, wherein each frame of the video data can be converted into a texture. A set of reprojection 1706 and occlusion detection 1707 units can operate on the received video data and output the processed data to the patch forming 1708 unit. The patch formed by the patch forming 1708 unit can be provided to the patch packing 1709 unit. The video data 1702 of the base view 1701 can be encoded, for example, via a high-efficiency video coding (HEVC) encoder 1703A. A variant of the HEVC encoder 1703A can also be used to encode the patch video data output from the patch packing 1709 unit. Metadata for reconstructing the video from the encoded patch can be encoded by a metadata encoding 1703B unit. Multiple encoded videos and metadata streams can then be transmitted to a client device for viewing.

[0175] like Fig. 17B As shown, multiple video data streams can be received, decoded and reconstructed into immersive video by system 1710. The multiple video streams include a stream of base video and a stream of packaged data containing additional views. Encoded metadata is also received. In one embodiment, the multiple video streams can be decoded via the HEVC 1713A decoder. The metadata can be decoded via the metadata 1713B decoder. The decoded metadata is then used to unpack the decoded additional views via the patch unpacking 1719 logic. The decoded texture and depth data (video 0 1712, videos 1 to 3 1714A to 1715C) of the base view 1701 and the additional view 1704 are received by the client (e.g., Fig.16 The decoded videos 1712, 1715A to 1715C may be provided as texture and depth data to an intermediate view renderer 1714, which may be used to render an intermediate view of a head mounted display 1711. The head mounted display position information 1716 may be provided as feedback to the intermediate view renderer 1714, which may render an updated view of the displayed viewport presented via the head mounted display 1711.

[0176] FIG. 18A to FIG. 18B A system for encoding and decoding 6DoF texture geometry data is presented. Fig.18A A 6DoF texture geometry encoding system 1800 is shown. Fig.18BA 6DoF texture geometry decoding system 1820 is shown. 6DoF texture geometry encoding and decoding can be used to enable 6DoF immersive video in which video data is applied as a texture to a variation of geometry data, allowing new intermediate views to be rendered based on the position and orientation of the head mounted display. Data recorded by multiple video cameras can be combined with 3D models (specifically, of static objects).

[0177] like Fig.18A As shown, a 6DoF texture geometry coding system 1800 may receive video data 1802 of a base view and video data 1805A to 1805C of additional views. The video data 1802, 1805A-1805C include texture and depth data that may be processed by a reprojection and occlusion detection unit 1806. The output from the reprojection and occlusion detection unit 1806 may be provided to a patch decomposition unit 1807 and a geometry image generator 1808. The output from the patch decomposition unit 1807 is provided to a patch packing unit 1809 and an auxiliary patch information compressor 1813. The auxiliary patch information (patch information) provides information of patches used to reconstruct video texture and depth data. The patch packing unit 1809 outputs the packed patch data to the geometry image generator 1808, the texture image generator 1810, the attribute image generator 1811, and the occupancy map compressor 1812.

[0178] The geometry image generator 1808, the texture image generator 1810, and the attribute image generator 1811 output data to the video compressor 1814. The geometry image generator 1808 may receive input from the reprojection and occlusion detection unit 1806, the patch decomposition unit 1807, and the patch packing unit 1809, and generate geometry image data. The texture image generator 1810 may receive packed patch data from the patch packing unit 1809, and receive video texture and depth data from the reprojection and occlusion detection unit 1806. The attribute image generator 1811 generates an attribute image based on the video texture and depth data received from the reprojection and occlusion detection unit 1806 and the packed patch data received from the patch packing unit 1809.

[0179] The occupancy map may be generated by an occupancy map compressor 1812 based on the packed patch data output from the patch packing unit 1809. The auxiliary patch information may be generated by an auxiliary patch information compressor 1813. The compressed occupancy map and auxiliary patch information data may be multiplexed by a multiplexer 1815 with the compressed and / or encoded video data output from the video compressor 1814 into a decompressed bitstream 1816. The compressed video data output from the video compressor 1814 includes compressed geometry image data, texture image data, and attribute image data. The compressed bitstream 1816 may be stored or provided to a client device for decompression and viewing.

[0180] like Fig.18B As shown, the 6DoF texture geometry decoding system 1820 can be used to decode the texture using Fig.18A The 6DoF content generated by the encoding system 1800 is decoded. The compressed bitstream 1816 is received by the demultiplexer 1835 and demultiplexed into multiple video decoded streams, occupancy maps, and auxiliary patch information. The multiple video streams are decoded / decompressed by the video decoders 1834A to 1834B. The occupancy map data is decoded / decompressed by the occupancy map decoder 1832. The decoded video data and occupancy map data are output by the video decoders 1834A to 1834B and the occupancy map decoder 1832 to the unpacking unit 1829. The unpacking unit performs the unpacking of the decoded video data and occupancy map data by the video decoders 1834A to 1834B and the occupancy map decoder 1832. Fig.18A The video patch data packed by the patch packing unit 1809 is unpacked. The auxiliary patch information from the auxiliary patch information decoder 1833 is provided to the occlusion filling unit 1826, which can be used to fill the patch according to the occluded portion of the object that may be missing from the specific view of the video data. The respective video streams 1822, 1825A to 1825C with texture and depth data are output from the occlusion filling unit 1826 and provided to the intermediate view renderer 1823, which can render a view for display on the head mounted display 1824 based on the position and orientation information provided by the head mounted display 1824.

[0181] FIG. 19A to FIG. 19B A system for encoding and decoding 6DoF point cloud data is demonstrated. Fig.19A The 6DoF point cloud encoding system 1900 is demonstrated. Fig.19B A 6DoF point cloud decoding system 1920 is shown. A 6DoF video can be represented using a point cloud, where for a point cloud video sequence, there are new point cloud frames at regular time intervals (e.g., 60 Hz). Each point in a point cloud data frame is represented by six parameters: (X, Y, Z) geometric position and (R, G, B or Y, U, V) texture data. Fig.19AIn the encoding system 1900 of FIG. 1 , the point cloud frame is projected onto a plurality of two-dimensional (2D) planes, each 2D plane corresponding to a certain projection angle. The projection planes may be similar to Fig. 15B Projection plane 1512. In some embodiments, six projection angles are used in the PCC standard test model, where the projection angles correspond to angles pointing to the centers of the six faces of the rectangular solid that defines the object represented by the point cloud data. Although six projection angles are described, other numbers of angles may be used in different embodiments.

[0182] A texture and depth 2D image patch representation is formed at each projection angle. A 2D patch image representation of a projection angle can be created by projecting only those points with the closest normal to the projection angle. In other words, a 2D patch image representation is adopted for the point that maximizes the dot product of the point normal and the plane normal. Texture patches from separate projections are combined into a single texture image, which is called a geometric image. Metadata used to represent patches and how they are packed into frames is described in the occupancy map and auxiliary patch information. The occupancy map metadata includes an indication of which image sample positions are empty (e.g., do not contain corresponding point cloud information). The auxiliary patch information indicates the projection plane to which the patch belongs and can be used to determine the projection plane associated with a given sample position. Texture images and depth images are encoded using a 2D conventional video encoder such as a high-efficiency video coding (HEVC) encoder. Metadata can be compressed separately using metadata encoding logic. In the test model decoder, the texture image and depth image are decoded using an HEVC video decoder. Point clouds are reconstructed using decoded texture and depth images as well as occupancy maps and auxiliary patch information metadata.

[0183] like Fig.19A As shown, the input frame of point cloud data can be decomposed into patch data. Fig.18A The point cloud data and the decomposed patch data are encoded in a similar manner to the video texture and depth data in the image. The input data including the point cloud frame 1906 can be provided to the patch decomposition unit 1907. The input point cloud data and its decomposed patches can be used by the packing unit 1909, the geometry image generator 1908, the texture image generator 1910, the attribute image generator 1911, the occupancy map compressor 1912 and the auxiliary patch information compressor 1913 using the same method as the one used by the point cloud frame 1906. Fig.18AThe video compressor 1914 may encode and / or compress the geometry image, texture image, and attribute image data. The compressed and / or encoded video data from the video compressor 1914 may be multiplexed by a multiplexer 1915 with the occupancy map and auxiliary patch information data into a compressed bitstream 1916, which may be stored or transmitted for display.

[0184] Depend on Fig.19A The compressed bit stream output by system 1900 can be obtained by Fig.19B The point cloud decoding system 1920 shown in FIG. Fig.19B As shown, the compressed bitstream 1916 can be demultiplexed into multiple encoded / compressed video streams, occupancy map data, and auxiliary patch information. The video streams can be decoded / decompressed by a multi-stream video decoder 1934, which can output texture and geometry data. The occupancy map and auxiliary patch information can be decompressed / decoded by an occupancy map decoder 1932 and an auxiliary patch information decoder 1933.

[0185] Geometry reconstruction, smoothing, and texture reconstruction can then be performed to reconstruct the Fig.19A The point cloud data of the 6DoF point cloud encoding system 1900 of the multi-stream video decoder 1934 can be reconstructed by a geometry reconstruction unit 1936 based on the geometry data decoded from the video stream of the multi-stream video decoder 1934, and the output of the occupancy map decoder 1932 and the auxiliary patch information decoder 1933. The reconstructed geometry data can be smoothed by a smoothing unit 1937. The smoothed geometry and texture image data decoded from the video stream output by the multi-stream video decoder 1934 is provided to a texture reconstruction unit 1938. The texture reconstruction unit 1938 can output a reconstructed point cloud 1939, which is provided to Fig.19A A variant of the input point cloud frame 1926 of the 6DoF point cloud encoding system 1900.

[0186] Selective patch packaging for immersive video

[0187] In some embodiments, an apparatus, system, or process provides selective patch packing for immersive video. In some embodiments, patches are selected based on which of a plurality of projection directions (the projection directions are content from a given camera or virtual camera) are associated with each of the patches.

[0188] In conventional patch packing for immersive video, all patches are packed and encoded into a coded picture for transmission to the user, who is then required to decode the entire picture containing the patches in order to obtain and blend any required patch or patches with a primary view to render a specific viewport.

[0189] However, when rendering a viewport for viewing based on a particular view position (e.g., HMD (head mounted display) position / orientation), only a subset of projection directions are used to render the viewport, where the projection directions are typically directions close to the viewport position.

[0190] In some embodiments, selective patch packing of patches from various projection directions is applied, enabling the client to decode less patch data when rendering a particular viewport, while still being able to correctly render the required viewport for any view position.

[0191] In some embodiments, an apparatus, system, or process for selective patch filling of immersive video includes one or more of the following:

[0192] (1) Selective patch packing provides for generating a plurality of coded pictures, wherein each coded picture includes selected patches for one or more projection directions, which may include:

[0193] (a) Patches from each projection direction are each packed into a corresponding coded picture, so selective patch padding provides a separate coded picture for each projection direction.

[0194] (b) Projection direction sets (sets / groups) are established, and a separate coded picture is generated for packing patches associated with each set of projection directions.

[0195] (2) Selective patch packing uses tiles to contain selected patches for one or more projection directions, which may include:

[0196] (a) Tiles are applied to selective patch packing for each projection direction, where the entire coded picture contains a tile corresponding to each projection direction and containing the corresponding patch for that projection direction.

[0197] (b) Tiles are applied to selective patch packing for each set of projection directions, wherein the entire coded picture comprises tiles corresponding to each set of projection directions and comprising corresponding patches for said set of projection directions.

[0198] Fig. 20A is a diagram of a processing system for providing selective patch packing according to some embodiments. In some embodiments, the processing system 2000 (eg, Figure 1The processing system 100 shown in FIG. 1 includes one or more processor cores. In some embodiments, the processing system 2000 includes one or more processors 2005 (which may include one or more CPUs (central processing units)), such as Figure 1 The processor 102 shown in FIG. 1 has one or more processor cores, and the processing system further includes one or more GPUs 2010, such as Figure 1 2005, wherein the GPU 2010 may be included in or separate from the one or more processors 2205. However, the embodiment is not limited to this particular processing structure. The one or more processors 2205 may include, for example, Figure 2 The components shown in the processor 200 in FIG. 1 and the one or more GPUs 2010 may include the components shown in FIG. Figure 3 The processing system further includes a memory 2015 for storing data for encoding the immersive video.

[0199] In some embodiments, a processor core (such as a processor core of one or more GPUs 2010) provides patch packing and encoding, as shown in FIG. Fig.17A In some embodiments, patch packing includes selective patch packing based at least in part on the projection direction to generate one or more coded pictures containing selected patches 2025. In some embodiments, selective patch packing may include Fig.24 , Fig.26 , Fig.27A or Fig.27B The packing shown in the encoded picture provided in any one of .

[0200] Fig. 20B 2050 (e.g., Figure 1 The processing system 100 shown in FIG. 1 includes one or more processor cores. In some embodiments, the client system 2050 includes one or more processors 2055 (which may include one or more CPUs (central processing units)), such as Figure 1 The processor 102 shown in FIG. 1 has one or more processor cores, and the client system further includes one or more GPUs 2060, such as Figure 12060 may be included in or separate from the one or more processors 2055. However, the embodiment is not limited to this particular processing structure. The one or more processors 2055 may include, for example, Figure 2 The components shown in the processor 200 in FIG. 1 and the one or more GPUs 2060 may include the components shown in FIG. Figure 3 The client system 2050 further includes a memory 2065 for storing data for providing immersive video, such as data for a head mounted display (HMD).

[0201] In some embodiments, the client system 2050 receives one or more encoded pictures with selected patches for certain projection directions 2070, such as Fig. 20A 2025. A processor core (such as a processor core of one or more GPUs 2060) provides limited decoding and patch unpacking 2075 for one or more encoded pictures having selected patches, as shown in FIG. Fig.17A HEVC decoding 1713A and metadata decoding 1713B and patch unpacking 1719 in . In some embodiments, decoding and patch unpacking include decoding and unpacking limited to patches associated with a particular viewport to be rendered by the client system, as illustrated by the patches required for the viewport 2080 provided to the viewport renderer 2085, so as to minimize the amount of decoding required to provide immersive video. In some embodiments, limited decoding and unpacking of patches 2075 may include decoding and unpacking of patches as illustrated in . Fig.24 , Fig.26 , Fig.27A or Fig.27B The decoding and unpacking required in the coded pictures shown in any one of .

[0202] Fig.21 is an illustration of an image of an immersive video with selective patch packing according to some embodiments. In image 2100, there may be multiple foreground elements, such as multiple players in a football game or other sporting event. The football game may be recorded by any number of cameras, where each camera may define a particular view. In a specific example, there may be 36 images per frame, where each frame has a resolution of 5K x 3K (5120 x 2880).

[0203] Generates patches based on objects visible from a specific camera view. Fig. 22 is an exemplary set of images (such as including Fig.21. The number of patches depends on the camera view and the nature of the image, but can include very large amounts of high-definition data packed into the encoded picture. Images called atlases are copied from multiple images at the same frame time but from different positions / viewpoints. In conventional systems, the client is required to decode all patches in the atlas, thus incurring huge processing costs. In some embodiments, the system provides selective patch packing based on projection direction to reduce the amount of decoding required by the client to generate a particular viewpoint. In some embodiments, only areas occluded from other projection directions are included in the patch for a particular camera projection direction. In some embodiments, selective patch packing can include Fig.24 , Fig.26 , Fig.27A or Fig.27B The packing shown in the encoded picture provided in any one of .

[0204] Fig.23 is an illustration of an exemplary viewpoint group of a system including selective patch packing according to some embodiments. Although embodiments of the system may include any number of cameras, Fig.23 An example is shown that includes 24 cameras (cameras C01 to C24) establishing 24 different viewpoints. Actual systems may include a greater number of cameras, and these cameras may be distributed in any arrangement depending on the needs of the particular subject of the immersive video.

[0205] In some embodiments, selective patch packing of immersive video is performed based on various viewpoints in the system, where patches associated with each viewpoint are selected. Fig.24 is an illustration of selective patch packing for providing coded pictures for each viewpoint according to some embodiments. In some embodiments, the system selects patches according to each available viewpoint, and packs patches associated with each viewpoint and encodes these patches into separate coded pictures.

[0206] In the specific example where there are n viewpoints, such as Fig.23 As shown in Figure 1, the system generates n coded pictures for n=24 viewpoints. Fig.24 As shown, the result of the selective patch packing is n coded pictures (and therefore n atlases of the image) such as patch-1 1410, patch-2, and continuing to patch-n. In some embodiments, the client can decode one or more of the multiple coded pictures 1410 to 2430 as needed when generating immersive video.

[0207] In some embodiments, selective patch packing for immersive video is performed based on various viewpoints in the system, wherein patches associated with each of multiple groups of viewpoints are selected. FIG. 25A to FIG. 25DMultiple sets of viewpoints for selective patch packing according to some embodiments are shown. FIG. 25A to FIG. 25D Demonstrated based on Fig.23 The viewpoints shown in establish specific examples of certain groups of viewpoints.

[0208] In this example, Fig.25A A first set of viewpoints of cameras C01 to C06 is provided; Fig.25B A second set of viewpoints of cameras C07 to C12 is provided; Fig.25C A third set of viewpoints of cameras C13 to C18 is provided; and Fig.25D A fourth set of viewpoints is provided for cameras C19 to C24. Although in this example for ease of illustration the cameras are evenly spaced and there are 6 viewpoints in each of the 4 sets of viewpoints, embodiments are not limited to any particular number or location of viewpoints and may include sets of viewpoints of varying sizes and locations.

[0209] Fig.26 is an illustration of selective patch packing for providing coded pictures for each set of views according to some embodiments. In some embodiments, the system selects patches according to each set of views, and packs and encodes the patches associated with each set of views into separate coded pictures (atlases representing subsets of images).

[0210] Fig.26 A plurality of coded pictures are shown, where, in one possible example, each coded picture has FIG. 25A to FIG. 25D The patches of one of the multiple sets of viewpoints shown in FIG. Fig.26 As shown in , the first coded picture (coded picture-1 2610) includes Fig.25A The selected patches of the first set of viewpoints shown in ; the second coded picture (coded picture-2 2620) includes Fig.25B The selected patches of the second set of viewpoints shown in FIG. 3; the third coded picture (coded picture-32630) comprises Fig.25C ; and a fourth coded picture (coded picture-4 2640) comprising Fig.25D The selected patches for the fourth set of viewpoints shown in .

[0211] In some embodiments, selective patch packing for immersive video is performed based on various viewpoints in the system, where a patch associated with each viewpoint is selected for a tile in an encoded picture. Fig.27AIt is an illustration of selectively packing patches into tiles according to projection direction according to some embodiments. In some embodiments, tiles are applied to selective patch packing for each projection direction. For example, the HEVC (High Efficiency Video Coding) motion constrained tile set (MCTS) feature can be applied so that the selected patch is included in a specific motion constrained tile. In this way, the decoder can decode the tiles individually without decoding the entire encoded picture. For example, patches from each projection direction can each be packed and encoded into its own tile, where the entire encoded picture contains tiles corresponding to each projection direction containing the associated patch.

[0212] exist Fig.27A In the particular example shown, an apparatus, system, or process performs selective patch packing based on projection direction (e.g., from Fig. 22 ) to encode a tiled coded picture 2700 including a tile for each projection direction. In some embodiments, each tile of the coded picture 2700 includes a patch associated with a projection direction. For example, tile-1 includes a patch for a first projection direction, in one example, Fig.23 The projection direction of camera C01 shown in .

[0213] In some embodiments, selective patch packing for immersive video is performed based on various viewpoints in the system, wherein patches associated with each of a plurality of viewpoints are selected for a tile in an encoded picture. Fig.27B is an illustration of selectively packing patches into tiles according to multiple groups of projection directions according to some embodiments. In some embodiments, tiles are applied for selective patch packing for each of the multiple groups of projection directions. For example, HEVCMCTS features can be applied so that selected patches for a set of projection directions are included in a specific motion constrained tile. In this way, the decoder can decode tiles individually without decoding the entire encoded picture. For example, patches from each group of projection directions can each be packed and encoded into its own tile, where the entire encoded picture contains tiles corresponding to each group of projection directions that contain associated patches. Multiple groups of projection directions can be, for example, FIG. 25A to FIG. 25D What is shown.

[0214] exist Fig.27B In the particular example shown, an apparatus, system, or process performs selective patch packing based on multiple sets of projection directions (e.g., from Fig. 22 ) to encode a tiled coded picture 2750 including tiles for each set of projection directions. In some embodiments, each tile of the coded picture 2750 includes a patch associated with a particular set of projection directions. For example, tile-1 includes a patch for the first set of projection directions, such as Fig.25ASimilarly, tile-2 may include a set of projection directions for Fig.25B The set of projection direction patches shown in FIG. 3 may include patches for Fig.25C The set of projection direction patches shown in FIG. 4 may include patches for Fig.25D However, embodiments are not limited to this particular arrangement, and may include different numbers and sizes of tiles representing different sets of projection directions.

[0215] Fig.28A 28 is a flowchart illustrating a process for selective patch packing according to some embodiments. In some embodiments, a process for immersive video includes generating patches for multiple viewpoints 2810. The process may include detecting occlusions and generating patches for the detected occlusions, as shown as Fig.17A In some embodiments, the process includes selecting patches based on projection directions 2815. In some embodiments, the process may include selecting patches based on each projection direction or multiple groups of projection directions (e.g., FIG. 25A to FIG. 25D In some embodiments, only areas occluded from other projection directions are included in the patch for a particular camera projection direction.

[0216] In some embodiments, the selected patches are packed and encoded into one or more coded pictures 2820, wherein the selected patches are packed in a manner that enables the client to decode relevant portions of the patches for the viewpoint rather than being required to decode all patches. In some embodiments, packing the selected patches may include:

[0217] 2822: Pack the patches for each projection direction into separate coded pictures;

[0218] 2824: Packing patches for each set of projection directions into separate coded pictures;

[0219] 2826: Packing patches into a coded picture comprising a plurality of tiles, wherein each tile comprises patches for one of the projection directions; or:

[0220] 2828: Packing patches into a coded picture comprising a plurality of tiles, wherein each tile comprises patches for a set of projection directions.

[0221] In some embodiments, one or more encoded pictures based on the selected patch are provided to the client for decoding and unpacking 2830 .

[0222] Fig.28B28 is a flowchart illustrating a process for decoding and unpacking selectively packed patches according to some embodiments. The process of presenting immersive video includes receiving one or more encoded pictures including selectively packed patches 2850. In some embodiments, the selectively packed patches 2850 may be decoded based on each projection direction or multiple groups of projection directions (e.g., FIG. 25A to FIG. 25D In some embodiments, packing the selected patches may include:

[0223] 2852: Pack patches for each projection direction into separate coded pictures;

[0224] 2854: Pack patches for each set of projection directions into separate coded pictures;

[0225] 2856: A patch in a coded picture comprises a plurality of tiles, wherein each tile comprises a patch for one of a plurality of projection directions; or:

[0226] 2858: A patch in a coded picture comprises a plurality of tiles, wherein each tile comprises patches for a set of projection directions.

[0227] In some embodiments, one or more coded pictures or tiles of coded pictures are identified based on a projection direction associated with the coded pictures or tiles and a current or anticipated viewpoint 2860. In some embodiments, the identified one or more coded pictures or tiles of coded pictures are decoded and unpacked 2865. In this way, the amount of decoding required by the client system is reduced compared to conventional systems by limiting decoding to patches associated with directions close to the viewpoint.

[0228] In some embodiments, a processing system includes: one or more processor cores; and a memory for storing data of an immersive video, the data including a plurality of patches for a plurality of projection directions. The system is configured to select patches for packing, the selection of patches being based at least in part on which projection direction of the plurality of projection directions is associated with each of the patches; and the system is configured to encode the patches into one or more encoded pictures according to the selection of the patches.

[0229] In some embodiments, the one or more encoded pictures comprises a plurality of encoded pictures, wherein each of the plurality of encoded pictures comprises a patch for one or more projection directions.

[0230] In some embodiments, each encoded picture of the plurality of encoded pictures comprises a patch for one of the plurality of projection directions.

[0231] In some embodiments, the multiple projection directions are divided into multiple groups of projection directions, each group of projection directions includes one or more projection directions, and wherein each of the multiple encoded pictures includes patches for a group of projection directions in the multiple groups of projection directions.

[0232] In some embodiments, the one or more encoded pictures comprise an encoded picture comprising a plurality of tiles, wherein each tile of the plurality of tiles comprises a patch for one or more projection directions.

[0233] In some embodiments, each tile of the plurality of tiles comprises a patch for one of the plurality of projection directions.

[0234] In some embodiments, the multiple projection directions are divided into multiple groups of projection directions, each group of projection directions includes one or more projection directions, and wherein each of the multiple tiles includes a patch for a group of projection directions in the multiple groups of projection directions.

[0235] In some embodiments, each tile of the plurality of tiles is a motion constrained tile in a HEVC (High Efficiency Video Coding) motion constrained tile set (MCTS).

[0236] In some embodiments, only areas that are occluded from other projection directions of the plurality of projection directions are included in the patch for a particular projection direction.

[0237] In some embodiments, the system is used to provide the one or more encoded pictures to a client system for decoding.

[0238] In some embodiments, the immersive video is a 6DoF (6 degrees of freedom) video.

[0239] In some embodiments, the one or more processor cores include one or more graphics processor cores of a graphics processing unit (GPU).

[0240] In some embodiments, a client system includes: one or more processor cores; and a memory for storing data of an immersive video. The client system is configured to receive one or more encoded pictures, the one or more encoded pictures including a plurality of patches for a plurality of projection directions, encode the one or more encoded pictures according to a selection of patches based at least in part on which projection direction of the plurality of projection directions is associated with each of the patches, and the client system is configured to decode a subset of the plurality of patches to render a viewport, the client system being configured to select the subset of patches from the one or more encoded pictures based at least in part on a viewport to be rendered by the client system.

[0241] In some embodiments, the one or more encoded pictures comprises a plurality of encoded pictures, wherein each encoded picture of the plurality of encoded pictures comprises a patch for one or more projection directions.

[0242] In some embodiments, each encoded picture of the plurality of encoded pictures comprises a patch for one of the plurality of projection directions.

[0243] In some embodiments, the multiple projection directions are divided into multiple groups of projection directions, each group of projection directions includes one or more projection directions, and wherein each of the multiple encoded pictures includes patches for a group of projection directions in the multiple groups of projection directions.

[0244] In some embodiments, the one or more encoded pictures comprise an encoded picture comprising a plurality of tiles, wherein each tile of the plurality of tiles comprises a patch for one or more projection directions.

[0245] In some embodiments, each tile of the plurality of tiles comprises a patch for one of the plurality of projection directions.

[0246] In some embodiments, the multiple projection directions are divided into multiple groups of projection directions, each group of projection directions includes one or more projection directions, and wherein each of the multiple tiles includes a patch for a group of projection directions in the multiple groups of projection directions.

[0247] In some embodiments, each tile of the plurality of tiles is a motion constrained tile in a HEVC (High Efficiency Video Coding) motion constrained tile set (MCTS).

[0248] In some embodiments, a non-transitory computer-readable storage medium has stored thereon data representing a sequence of instructions that, when executed by one or more processors, causes the one or more processors to perform operations including: detecting one or more objects from multiple projection directions of an immersive video; generating multiple patches for the detected objects; selecting the multiple patches for packing based at least in part on which of the multiple projection directions is associated with each of the patches; and encoding the patches into one or more encoded pictures based on the selection of the patches.

[0249] In some embodiments, the one or more encoded pictures comprises a plurality of encoded pictures, wherein each encoded picture of the plurality of encoded pictures comprises a patch for one or more projection directions.

[0250] In some embodiments, the one or more encoded pictures comprise an encoded picture comprising a plurality of tiles, wherein each tile of the plurality of tiles comprises a patch for one or more projection directions.

[0251] In some embodiments, only areas that are occluded from other projection directions of the plurality of projection directions are included in the patch for a particular projection direction.

[0252] In some embodiments, a device includes a device for detecting one or more objects from multiple projection directions of an immersive video; a device for generating multiple patches for the detected objects; selecting the multiple patches for packing based at least in part on which of the multiple projection directions is associated with each of the patches; and a device for encoding the patches into one or more encoded pictures based on the selection of the patches.

[0253] In some embodiments, the one or more encoded pictures comprises a plurality of encoded pictures, wherein each encoded picture of the plurality of encoded pictures comprises a patch for one or more projection directions.

[0254] In some embodiments, the one or more encoded pictures comprise an encoded picture comprising a plurality of tiles, wherein each tile of the plurality of tiles comprises a patch for one or more projection directions.

[0255] In some embodiments, only areas that are occluded from other projection directions of the plurality of projection directions are included in the patch for a particular projection direction.

[0256] In some embodiments, a method includes: detecting one or more objects from multiple projection directions of an immersive video; generating multiple patches for the detected objects; selecting the multiple patches for packing based at least in part on which of the multiple projection directions is associated with each of the patches; and encoding the patches into one or more encoded pictures based on the selection of the patches.

[0257] In some embodiments, the one or more encoded pictures comprises a plurality of encoded pictures, wherein each encoded picture of the plurality of encoded pictures comprises a patch for one or more projection directions.

[0258] In some embodiments, the one or more encoded pictures comprise an encoded picture comprising a plurality of tiles, wherein each tile of the plurality of tiles comprises a patch for one or more projection directions.

[0259] In some embodiments, only areas that are occluded from other projection directions of the plurality of projection directions are included in the patch for a particular projection direction.

[0260] In some embodiments, a non-transitory computer-readable storage medium having stored thereon data representing a sequence of instructions that, when executed by one or more processors, causes the one or more processors to perform operations comprising: receiving one or more encoded pictures, the one or more encoded pictures comprising a plurality of patches for a plurality of projection directions, encoding the one or more encoded pictures based on a selection of patches based at least in part on which of the plurality of projection directions is associated with each of the patches; and selecting a subset of the plurality of patches from the one or more encoded pictures based at least in part on a viewport to be rendered for immersive video; and decoding the subset of patches for rendering the viewport.

[0261] In some embodiments, the one or more encoded pictures comprises a plurality of encoded pictures, wherein each encoded picture of the plurality of encoded pictures comprises a patch for one or more projection directions.

[0262] In some embodiments, the one or more encoded pictures comprise an encoded picture comprising a plurality of tiles, wherein each tile of the plurality of tiles comprises a patch for one or more projection directions.

[0263] In some embodiments, a device includes: a device for receiving one or more encoded pictures, the one or more encoded pictures including multiple patches for multiple projection directions, encoding the one or more encoded pictures based on a selection of patches based at least in part on which of the multiple projection directions is associated with each of the patches; and a device for selecting a subset of the multiple patches from the one or more encoded pictures based at least in part on a viewport to be rendered for immersive video; and decoding the subset of patches for rendering the viewport.

[0264] In some embodiments, the one or more encoded pictures comprises a plurality of encoded pictures, wherein each encoded picture of the plurality of encoded pictures comprises a patch for one or more projection directions.

[0265] In some embodiments, the one or more encoded pictures comprise an encoded picture comprising a plurality of tiles, wherein each tile of the plurality of tiles comprises a patch for one or more projection directions.

[0266] In some embodiments, a method includes: receiving one or more encoded pictures, the one or more encoded pictures including multiple patches for multiple projection directions, encoding the one or more encoded pictures based on a selection of patches based at least in part on which of the multiple projection directions is associated with each of the patches; and selecting a subset of the multiple patches from the one or more encoded pictures based at least in part on a viewport to be rendered for immersive video; and decoding the subset of patches for rendering the viewport. In some embodiments, the one or more encoded pictures comprises a plurality of encoded pictures, wherein each encoded picture of the plurality of encoded pictures comprises a patch for one or more projection directions.

[0267] In some embodiments, the one or more encoded pictures comprise an encoded picture comprising a plurality of tiles, wherein each tile of the plurality of tiles comprises a patch for one or more projection directions.

[0268] In the above description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of the described embodiments. However, it will be apparent to those skilled in the art that embodiments may be practiced without some of these specific details. In other examples, well-known structures and devices are shown in block diagram form. There may be intermediate structures between the displayed components. The components described or displayed herein may have additional inputs or outputs that are not displayed or described.

[0269] Various embodiments may include various processes. These processes may be performed by hardware components or may be embodied in computer programs or machine executable instructions, which may be used to cause a general purpose processor or a special purpose processor or a logic circuit programmed with the instructions to perform the process. Alternatively, the process may be performed by a combination of hardware and software.

[0270] Parts of the various embodiments may be provided as a computer program product, which may include a computer-readable medium having computer program instructions stored thereon, which may be used to program a computer (or other electronic device) for execution by one or more processors to perform processes according to certain embodiments. Computer-readable media may include, but are not limited to, disks, optical disks, read-only memories (ROMs), random access memories (RAMs), erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, flash memory, or other types of computer-readable media suitable for storing electronic instructions. In addition, the embodiments may also be downloaded as a computer program product, wherein the program may be transferred from a remote computer to a requesting computer. In some embodiments, data representing a sequence of instructions is stored on a non-transitory computer-readable storage medium, which causes the processor to perform certain operations when executed by the processor.

[0271] Many of the methods are described in their most basic form, but processes may be added to or deleted from any method, and information may be added to or reduced from any of the described messages without departing from the basic scope of the embodiments of the invention. It will be apparent to those skilled in the art that many additional modifications and adaptations may be made. The specific embodiments are not provided to limit the concepts, but to illustrate the concepts. The scope of the embodiments is not to be determined by the specific embodiments provided above, but only by the following claims.

[0272] If element "A" is said to be coupled to element "B" or coupled with element "B", element A can be directly coupled to element B or indirectly coupled through, for example, element C. When the specification or claims state that part, feature, structure, process or characteristic A "causes" part, feature, structure, process or characteristic B, it means that "A" is at least partially the cause of "B", but there may also be at least one other part, feature, structure, process or characteristic that contributes to causing "B". If the specification indicates that a part, feature, structure, process or characteristic "may", "might", or "could" be included, that particular part, feature, structure, process or characteristic is not required to be included. If the specification or claims mention "a" or "an" element, this does not mean that there is only one of the described elements.

[0273] An embodiment is an implementation or example. References in the specification to "an embodiment", "one embodiment", "some embodiments", or "other embodiments" mean that the specific features, structures, or characteristics described in conjunction with the embodiment are included in at least some embodiments, but not necessarily all embodiments. Multiple occurrences of "an embodiment", "one embodiment", or "some embodiments" do not necessarily all refer to the same embodiment. It should be recognized that in the foregoing description of exemplary embodiments, different features are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and helping to understand one or more different novel aspects. However, the method of the present disclosure is not to be interpreted as reflecting the following intention: the claimed embodiment requires more features than those explicitly stated in each claim. On the contrary, as reflected in the following claims, the novel aspect lies in less than all the features of a single aforementioned disclosed embodiment. Therefore, the claims are hereby expressly incorporated into this specification, wherein each claim independently represents a separate embodiment.

Claims

1. At least one memory including instructions for causing at least one processor circuit to perform at least the following operations: Packing a first patch of data into a first atlas associated with a first camera; packing a second patch of data into a second atlas associated with a second camera; and The information of the first atlas and the information of the second atlas are encoded into a video bitstream.

2. At least one memory as claimed in claim 1, wherein: The video bitstream is an immersive video bitstream.

3. At least one memory as claimed in claim 1, wherein: The first camera is associated with a first projection viewpoint and the second camera is associated with a second projection viewpoint.

4. At least one memory as claimed in claim 1, wherein: The first atlas is a first image and the second atlas is a second image.

5. At least one memory as claimed in claim 1, wherein: The first patch includes corresponding two-dimensional image data.

6. At least one memory as claimed in claim 1, wherein: The information of the first atlas includes metadata for reconstructing a video from a first patch of the data.

7. At least one memory as claimed in claim 6, wherein: The instructions are used to cause one or more processor circuits of the at least one processor circuit to perform the following operations: encoding the first atlas in a first stream of the video bitstream; as well as The metadata is encoded in a second stream of the video bitstream.

8. An apparatus comprising: Interface circuit; instruction; as well as At least one processor circuit is programmed based on the instructions to perform the following operations: Packing a first patch of data into a first atlas associated with a first camera; packing a second patch of data into a second atlas associated with a second camera; as well as The information of the first atlas and the information of the second atlas are encoded into a video bitstream.

9. The device of claim 8, wherein: The video bitstream is an immersive video bitstream.

10. The device of claim 8, wherein: The first camera is associated with a first projection viewpoint and the second camera is associated with a second projection viewpoint.

11. The device of claim 8, wherein: The first atlas is a first image and the second atlas is a second image.

12. The device of claim 8, wherein: The first patch includes corresponding two-dimensional image data.

13. The device of claim 8, wherein: The information of the first atlas includes metadata for reconstructing a video from a first patch of the data.

14. The device of claim 13, wherein: One or more processor circuits of the at least one processor circuit are configured to perform the following operations: encoding the first set of atlases in a first stream of the video bitstream; and The metadata is encoded in a second stream of the video bitstream.

15. A system comprising: means for packing, the means for packing being configured to pack a first patch of data into a first atlas associated with a first camera and to pack a second patch of data into a second atlas associated with a second camera; as well as The device for encoding is configured to encode the information of the first set of images and the information of the second set of images into a video bitstream.

16. The system of claim 15, wherein: The video bitstream is an immersive video bitstream.

17. The system of claim 15, wherein: The first camera is associated with a first projection viewpoint and the second camera is associated with a second projection viewpoint.

18. The system of claim 15, wherein: The first atlas is a first image and the second atlas is a second image.

19. The system of claim 15, wherein: The first patch includes corresponding two-dimensional image data.

20. The system of claim 15, wherein: The information of the first atlas includes metadata for reconstructing a video from a first patch of the data, and the means for encoding is configured to: encoding the first atlas in a first stream of the video bitstream; as well as The metadata is encoded in a second stream of the video bitstream.

Images