Variable Rasterization Rate
By dynamically adjusting the rasterization rate of the tile, the rasterization rate management problem in high-resolution and high-frame rate graphic display is solved, and the graphics rendering efficiency and image quality are improved. It is suitable for three-dimensional and virtual reality applications.
Patent Information
- Application Number
- CN202010451724.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-05-31
- Filing Date
- 2020-05-25
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2040-05-25
AI Technical Summary
In high resolution and high frame rate graphic display, the prior art is difficult to effectively manage the rasterization rate, resulting in a decline in image quality, especially in three-dimensional graphics and virtual reality applications, and it is necessary to improve the graphics rendering efficiency to avoid skipping frames. Mismatched images and lost 3D effects.
By receiving a function indicating the desired sampling rate of the image content, the image is divided into multiple tiles according to different axial positions, and its rasterization rate is determined based on the expected sampling rate of each tiles, and the rasterization rate is dynamically adjusted to match lens distortion. Effects and other resolution reduction scenarios to achieve variable rasterization rate.
Improves graphics rendering efficiency, reduces storage and bandwidth requirements, and maintains image quality, especially in virtual reality and 3D graphics applications, providing a higher immersive experience.
Smart Images

Figure CN112017101B_ABST
Abstract
Description
Background Art
[0001] The subject matter disclosed herein relates to the field of graphics processing. More specifically, but not by way of limitation, the subject matter disclosed herein relates to using variable rasterization rates when displaying computer graphics.
[0002] Computers and other computing devices typically have at least one programmable processing element, commonly referred to as a central processing unit (CPU). They also typically have other programmable processors for various types of specialized processing, such as graphics processing operations, which may be performed by a graphics processing unit (GPU). A GPU typically includes multiple cores or processing elements designed to execute the same instructions on parallel data streams, making the GPU more efficient than a general-purpose CPU for algorithms in which large blocks of data are processed in parallel. Generally speaking, the CPU acts as a host and hands specialized parallel tasks to the GPU.
[0003] To render frames on a display, the GPU and CPU typically work together. The number of frames displayed per second (FPS) is called the frame rate. At lower frame rates, the human eye can discern still frames displayed in rapid succession. However, at higher frame rates, the individual frames are imperceptible to humans and appear as seamless motion. Therefore, with everything else being the same (such as resolution), a display with a higher frame rate provides higher graphics quality.
[0004] Generally speaking, as computer displays improve, the resolution of these computer displays also increases. The resolution of a display refers to the number of pixels contained in the display on the vertical and horizontal axes. In order to utilize a higher resolution display, the GPU and CPU must provide frames at a higher resolution. In addition, in order to display three-dimensional (3-D) graphics or virtual reality (VR) graphics, two separate frames (e.g., one for the right eye and one for the left eye) can be rendered for display, rather than a single image. This higher resolution and number of frames generally require improving the efficiency of the CPU and GPU for rendering frames. Otherwise, due to, for example, skipped frames, mismatched right / left images, lower spatial resolution, loss of 3-D effects, etc., the image may be perceived by a human user as an image of lower quality. Summary of the invention
[0005] A disclosed embodiment includes a method for graphics processing. The method includes receiving a first function, wherein the first function indicates an expected sampling rate of image content, wherein the expected sampling rate is different in a first position along a first axial direction and a second position along the first axial direction, and wherein the image content is divided into a plurality of tiles. The method also includes determining a first rasterization rate for each of the plurality of tiles based on the expected sampling rate indicated by the first function corresponding to each respective tile. As discussed in detail below, the rasterization rate is the rate at which objects in object space are sampled for projection to a viewpoint. The method also includes receiving one or more primitives associated with content for display. The method also includes rasterizing at least a portion of the primitive associated with the respective tile based on the determined first rasterization rate for the respective tile. The method also includes displaying an image based on the rasterized portion of the primitive.
[0006] Another aspect of the present disclosure relates to a non-volatile program storage device including instructions stored thereon for causing one or more graphics processors to perform the following operations: receiving a first function, wherein the first function indicates an expected sampling rate for image content, wherein the expected sampling rate is different in a first position along a first axial direction and a second position along the first axial direction, and wherein the image content is divided into a plurality of tiles; determining a first rasterization rate for each of the plurality of tiles based on the expected sampling rate indicated by the first function corresponding to each respective tile; receiving one or more primitives associated with content for display; rasterizing at least a portion of the primitive associated with the respective tile based on the determined first rasterization rate for the respective tile; and displaying an image based on the rasterized portion of the primitive.
[0007] Another aspect of the present disclosure relates to an electronic device, which includes a memory, a display, a user interface, and one or more graphics processors operatively coupled to the memory, wherein the one or more graphics processors are configured to execute instructions, the instructions causing the one or more graphics processors to: receive a first function, wherein the first function indicates an expected sampling rate for image content, wherein the expected sampling rate is different in a first position along a first axial direction and a second position along the first axial direction, and wherein the image content is divided into a plurality of tiles; determine a first rasterization rate for each of the plurality of tiles based on the expected sampling rate indicated by the first function corresponding to each respective tile; receive one or more primitives associated with content for display; rasterize at least a portion of the primitive associated with the respective tile based on the determined first rasterization rate for the respective tile; and display an image based on the rasterized portion of the primitive.
[0008] In other embodiments, each of the above methods and their variants can be implemented as a series of computer executable instructions. Such instructions can use any one or more convenient programming languages. Such instructions can be collected into an engine and / or program and can be stored in any medium that is readable and executable by a computer system or other programmable control device. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] Figure 1 is a block diagram illustrating a computer system that may be used, for example, as an end-user or developer computer system according to one embodiment of the present disclosure.
[0010] Figure 2 is a block diagram illustrating a network environment that can be associated with one or more embodiments of the disclosed subject matter, according to one embodiment of the present disclosure.
[0011] Figure 3 is a block diagram showing an illustrative software architecture diagram according to one embodiment of the present disclosure, according to one or more embodiments of the disclosed subject matter.
[0012] Figure 4 is a block diagram illustrating a computer system for implementing one or more aspects of the disclosed subject matter, according to one embodiment of the present disclosure.
[0013] Figure 5A and Figure 5B A conceptual example of rasterization according to aspects of the present disclosure is shown.
[0014] Fig. 6A and Figure 6B The effect of a lens according to aspects of the present disclosure is shown.
[0015] Figure 7 is a graph illustrating an exemplary sampling rate function according to aspects of the present disclosure.
[0016] Fig. 8A and Figure 8B is a graph showing a function describing a desired sampling rate for a display according to aspects of the present disclosure.
[0017] Fig. 9 An exemplary mapping between coordinates of a view in object space and screen space is shown according to aspects of the present disclosure.
[0018] Fig.10 An exemplary mapping between coordinates of a view in object space and screen space is shown according to aspects of the present disclosure.
[0019] Fig.11 is a flow chart illustrating a technique for graphics processing according to aspects of the present disclosure. DETAILED DESCRIPTION
[0020] In the following description, for the purpose of explanation, a lot of specific details are set forth in order to provide a thorough understanding of the present invention. However, it is obvious to those skilled in the art that the disclosed examples can be practiced in the absence of these specific details. In other examples, the structures and devices are shown in the form of block diagrams to avoid blurring the present invention. References to numbers without subscripts or suffixes should be understood as references to all subscripts and suffixes corresponding to the figure marks. In addition, the language used in this disclosure has been selected primarily for readability and instructive purposes, and may not be selected to delimit or limit the subject matter of the present invention, thereby resorting to the necessary claims to determine such subject matter of the invention. Mentioning "one embodiment" or "one embodiment" in the specification means that in at least one embodiment, a specific feature, structure or characteristic described in conjunction with the embodiment is included, and multiple references to "one embodiment" or "one embodiment" should not be understood as necessarily all referring to the same embodiment.
[0021] As used herein, the term "computer system" refers to a single computer system or multiple computer systems working together to perform the functions described as being performed on or by the computer system. Similarly, a machine-readable medium may refer to a single physical medium or multiple media that may together contain the specified information stored thereon. Reference to a processor may refer to a single processing element or multiple processing elements implemented on a single chip or on multiple processing chips.
[0022] It should be understood that in the development of any actual implementation (as in any development project), many decisions must be made to achieve the developer's specific goals (e.g., to meet system and business-related constraints), and that these goals may vary between different implementations. It should also be understood that such a development effort might be complex and time consuming, but would nevertheless be a routine undertaking for a person of ordinary skill in system design and implementation who has the benefit of this disclosure and has ordinary skills in designing and implementing computing systems and / or graphics systems.
[0023] See also Figure 1 , the disclosed embodiments may be performed by a representative computer system 100. For example, the representative computer system 100 may serve as a software development platform or an end-user device. Figure 1 Various components of the computer system are shown, but they are not intended to represent any particular architecture or manner of interconnecting these components, as such details are not germane to the present disclosure. Network computers and other data processing systems (e.g., handheld computers, personal digital assistants (PDAs), cellular phones, entertainment systems, and other consumer electronic devices, etc.) with fewer or perhaps more components may also be used to implement one or more embodiments.
[0024] like Figure 1 As shown, the computer system 100 includes a bus 122 that is coupled to one or more processors 116 (which may be a CPU and / or a digital signal processor (DSP)), a memory 112 (which may include one or both of a volatile read / write random access memory (RAM) and a read-only memory (ROM)), and a non-volatile storage device 114 (e.g., a solid-state storage drive). The processor 116 can retrieve instructions from the memory 112 and / or the storage device 114 and execute the instructions to perform the operations described herein. The bus 122 interconnects these various components together and also interconnects the processor 116, the memory 112, and the storage device 114 to the display 120, the I / O port 102, and peripheral devices such as input / output (I / O) devices 104. The I / O device 104 may be a pointing device such as a mouse or stylus, a keyboard, a touch screen (e.g., the display 120 and the I / O device 104 may be implemented as a single touch-sensitive display), a modem, a network interface, a printer, and other devices. Typically, the input / output device 104 is coupled to the system through an input / output controller.
[0025] The computer system 100 may also include or be coupled to device sensors 124. The device sensors 124 may include one or more of the following: a depth sensor (such as a depth camera), a three-dimensional (3D) depth sensor, an imaging device (such as a fixed and / or video-enabled image capture unit), a red, green, and blue (RGB) sensor, a proximity sensor, an ambient light sensor, an accelerometer, a gyroscope, any type of still or video camera, a light detection and ranging (LIDAR) device, a global positioning system (GPS), a microphone, a charge coupled device (CCD) (or other image sensor), an infrared sensor, a thermometer, etc. These and other sensors may work in conjunction with one or more GPUs, digital signal processors (DSPs), or conventional microprocessors with appropriate programming so that the sensor outputs may be properly interpreted and / or combined and interpreted.
[0026] If volatile RAM is included in memory 112, the RAM may be implemented as dynamic RAM (DRAM) that requires continuous power to refresh or retain data in the memory. Graphics hardware 106 may be specialized computing hardware for processing graphics and / or assisting processor 116 in performing computing tasks. In some embodiments, graphics hardware 106 may include CPU-integrated graphics and / or one or more programmable GPUs.
[0027] Storage device 114 may be a magnetic hard drive, an optical drive, a non-volatile solid-state storage drive, or other type of storage system that retains data (e.g., large amounts of data) even after the system is powered off (i.e., is non-volatile). Figure 1 Storage device 114 is shown as a local device directly coupled to the rest of the components in computer system 100, but embodiments may utilize non-volatile storage devices remote from computer system 100, such as network storage devices coupled to computer system 100 via network interface 110. The network interface may be a wired or wireless network interface. Bus 122 may include one or more links interconnected by various bridges, controllers, and / or adapters as are well known in the art. Although not shown for clarity, Figure 1 Only a single element of each type is shown, but multiple elements of any or all of the various element types may be used as desired.
[0028] Now go to Figure 2 , the block diagram shows a network 200 of interconnected programmable devices, including a server 230 and an associated data repository 240, as well as a desktop computer system 210, a notebook computer system 212, a tablet computer system 214, and a mobile phone 216. Other types of computer devices may also be included. Any of these programmable devices may be illustrated as Figure 1 The network 220 that interconnects the programmable devices can be any type of network, wired or wireless, local or wide area, public or private, for transmitting data from one system to another using any desired network communication protocol. Although illustrated as a single network 220, any number of interconnected networks can be used to connect the various programmable devices, and each can employ different network technologies.
[0029] In one example, desktop system 210 may be a developer system that distributes a graphics application to server 230, which in turn may distribute the graphics application to multiple devices, such as systems 212, 214, and 216, each of which may employ a separate GPU. When a graphics application is launched, one action performed by the application may be to create a collection of pipeline objects that may include state information, fragment shaders, and vertex shaders.
[0030] As noted above, embodiments of the subject matter disclosed herein include the use and execution of software. Figure 3 An illustrative description of the computing software architecture is provided in the layer diagram in . Similar to the hardware example, the Figure 3The software architecture in the example is not intended to be exclusive in any way, but rather illustrative. This is particularly true for layer type diagrams, which software developers tend to express in slightly different ways. In this case, the description begins with the layer starting from the base hardware layer 395 that illustrates the hardware 340, which may include the CPU and GPU or other processing and / or computer hardware described above. Above the hardware layer is the operating system (O / S) kernel layer 390, which shows an example as the O / S kernel 345, which is the kernel software that can perform memory management, device management, and system calls (often permissions for hardware drivers). The markings used here are generally intended to illustrate that the software elements shown in the layer use resources from the layers below and provide services to the layers above. However, all components of a particular software element may not function exactly in this manner.
[0031] return Figure 3 , layer 385 is an O / S service layer illustrated by O / S service 350. O / S service 350 can provide core O / S functions in a protected environment. In addition, O / S service 350 shown in layer 385 may include 351, 352, a software ray tracer 353, and a pure software rasterizer 354 (OpenGL is a registered trademark of Silicon Graphics, Inc., and Metal is a registered trademark of Apple, Inc.). These specific examples are all related to graphics and / or graphics libraries, which are all related to graphics processing. These specific examples also represent graphics frameworks / libraries that can operate in the lower levels of the framework, allowing developers to use shading and graphics primitives and / or obtain fairly tightly coupled control of the graphics hardware. In addition, the specific examples described in layer 385 can pass their work products directly to the hardware or a hardware driver, which can be software tightly coupled to the hardware.
[0032] Reference again Figure 3 , 351 represents an example of a well-known library and application programming interface (API) for graphics rendering, including two-dimensional (2D) and 3D graphics. 352 also represents the graphics libraries and frameworks released, but it is more 351 is a low level layer that supports fine-grained, low-level control of organizing, processing, and submitting graphics and compute commands, and managing the associated data and resources of those commands. A software ray tracer 353 is software for generating image information based on a process of tracing the path of light through pixels in an image plane. A pure software rasterizer 354 generally refers to software for generating graphics information such as pixels without specialized graphics hardware (e.g., using only the CPU). The libraries or frameworks shown within the O / S services layer 385 are merely illustrative and are intended to illustrate the general hierarchy of this layer and how it relates to other software (e.g., lower-level kernel operations and higher-level application services 360) in a sample arrangement. In addition, 352 indicates a published framework / library of Apple Inc. that can be used by developers of graphics applications.
[0033] Above the O / S service layer 385 is the application service layer 380, which includes the game engine 361, the 3D rendering engine 362, the animation engine 363, and the rendering engine 364. The O / S service layer 385 represents a higher-level framework that is directly accessed by the application. In some embodiments, the O / S service layer 385 includes graphics-related frameworks that are high-level because they are agnostic to the underlying graphics libraries (such as those discussed with reference to layer 385). In such embodiments, these higher-level graphics frameworks are intended to provide developers with access to graphics functionality in a more user- and developer-friendly manner and allow developers to avoid working with shading and graphics primitives. By way of example, the game engine 361 can be a graphics rendering and animation infrastructure and can be used to display two-dimensional (2D) texture images in an animated manner. The 3D rendering engine 362 can be a framework that helps to display graphics in a more user- and developer-friendly manner than frameworks such as 3D rendering engine 362 with similar capabilities. The 3D rendering framework for high-level import, manipulation and rendering of 3D assets. The animation engine 363 can be a graphics rendering and animation infrastructure and can be used to display the views and other visual elements of the application in an animated manner. The rendering engine 364 can be a two-dimensional drawing engine for providing 2D rendering for the application.
[0034] The application layer 375 is located above the application service layer 380. The application layer 375 includes any number and type of application programs. By way of example, Figure 3 Three specific applications are shown: Photos 371 (a photo management, editing, and sharing program), Financial Management Application 372, and Movie Application 373 (a movie making and sharing program). Application layer 375 also shows two general applications A 370 and B 374, which represent any other applications that may interact with or be a part of the disclosed embodiments. In general, embodiments of the disclosed subject matter employ and / or interact with applications that generate displayable / viewable content.
[0035] In evaluating the O / S service layer 385 and the application service layer 380, it may be useful to recognize that different frameworks have higher or lower level application programming interfaces, even if the framework is represented in Figure 3 The same is true in the same layer of the schematic. Figure 3 The examples are used to provide general guidance and to present illustrative frameworks that may be discussed herein. In addition, in some examples, the frameworks in layer 380 utilize the libraries represented in layer 385. Thus, Figure 3 Provides knowledge reinforcement for these examples. Figure 3 It is not intended to limit the types of frameworks or libraries that may be used in any particular manner or in any particular implementation.In general, various embodiments of the present disclosure present software activities and architectures in the layers between the hardware 340 and application 375 layers.
[0036] Reference again Figure 3 , some embodiments include the use of higher-level frameworks, such as those shown in application service layer 380. The high-level framework can perform intelligent analysis on a specific graphics request from an application. The high-level framework can then select specific hardware and / or specific libraries or low-level frameworks to help process the request. In these embodiments, the intelligent analysis can provide runtime decisions about the best path for a graphics request to follow down to the hardware.
[0037] See now Figure 4 , a block diagram of a computing system 400 further illustrates a computer system according to one embodiment in detail. The computing system 400 includes a CPU 401, a graphics processing system 403, a display 402, a power management unit (PMU) 404, and a system memory 430. In one embodiment, the CPU 401 and the graphics processing system 403 are included on separate integrated circuits (ICs) or IC packages. However, in other embodiments, the CPU 401 and the graphics processing system 403 or their collective functions can be included in a single IC or package.
[0038] A representative graphics processing system 403 may be used to process application data and render a graphical representation of a virtual object to a display 402. For example, the CPU 401 may receive a request from application code (not shown) to render a graphic. The request may be via an internal or third-party graphics library and framework. The graphic may be part of a model of a virtual object including one or more polygons, such as triangles. The request may reference data stored in, for example, system memory 430 or video memory 425.
[0039] Data bus 405 connects various elements of computing system 400, including CPU 401, system memory 430, and graphics processing system 403. In one embodiment, system memory 430 includes instructions for CPU 401 and / or graphics processing system 403 to perform the functions assigned to them in the present disclosure. More specifically, graphics processing system 403 can receive instructions transmitted by CPU 401 and process these instructions to render and display graphic images on display 402.
[0040] System memory 430 may include application programs 431 and GPU drivers 432. In this example, graphics processing system 403 includes frame buffer 424, GPU 420, and video memory 425. GPU 420 may include a graphics pipeline including one or more vertex shaders 421, one or more rasterizers 422, one or more fragment shaders 423, and one or more geometry shaders 426. In some embodiments, a unified memory model may be supported, where system memory 430 and video memory 425 comprise a single memory utilized by both GPU 420 and CPU 401, rather than separate memory systems. As used herein, application code may refer to code executed on CPU 401 during application runtime, separate from graphics functions that may be executed on GPU 420. Graphics functions may be executed on the GPU, for example, because hardware components of GPU 420 (such as shaders) may be programmable, thereby allowing graphics functions to be executed on GPU 420. Application programming interface (API) and driver software executing on CPU 401 may facilitate interaction between application code and graphics functions, such as by providing an interface between application code and GPU 420 and allowing application code to set up and execute graphics functions on GPU 420 .
[0041] In some cases, the frame buffer 424 may be located in the system memory 430. In some embodiments, the frame buffer 424 may be located in the video memory 425 or as a dedicated memory. In one embodiment, the application 431 includes code written using an API. The API includes a predetermined, standardized set of commands executed by associated hardware. The application 431 generates API commands to render images for display by one or more shading engines and / or rasterizers of the GPU 420. The GPU driver 432 converts high-level API commands into machine code programs that can be executed by the GPU 420.
[0042] In one embodiment, the CPU 401 transmits API commands to the GPU 420 to render graphics data and store the rendered image in the frame buffer 424 for display on the display 402. The image can be rendered by dividing the image into multiple segments of a grid, each of which is called a tile. Each tile can be rendered separately to the video memory 425 by the GPU 420. Rendering a single tile instead of rendering the entire frame at once helps to reduce the amount of memory and bandwidth required for rendering. In some cases, multiple tiles can be rendered independently, such as in a parallel graphics pipeline. When all tiles of the frame are completed, the frame buffer 424 can output the image to the display 402. Common tile sizes include 16×16 pixels and 32×32 pixels, but tiles of any size can also be used.
[0043] GPU 420 may include multiple multiprocessors configured to execute multiple threads in parallel. In some cases, the multiprocessor may be configured as a shader and a rasterizer. Generally speaking, GPU 420 may render a view of a virtual object using a model coordinate system of the virtual object. The virtual object may be presented from the perspective of a camera at a specified position. Vertex shader 421 performs matrix operations on the coordinates of a specific polygon to determine the coordinates of the polygon rendered from the perspective of the camera based on the model coordinates. Unlike vertex shader 421 operating on a single vertex, the input received by geometry shader 426 is the vertex of a complete primitive, such as two vertices of a line, three vertices of a triangle, or a single vertex of a point. Rasterizer 422 then determines which pixels of the display intersect with the polygon. Fragment shader 423 then assigns a color value to each pixel intersecting with the polygon. The color value may be based on the content of a specific texture read from a memory, for example. The texture may be stored in memory 430 or video memory 425. Shaders may be programmed as part of a programmable GPU pipeline using shader functions to allow increased flexibility and functionality of shaders. This programmability also allows the GPU to perform non-graphical data parallel tasks. In some embodiments, the rasterizer 422 can be a fixed function of the GPU pipeline to allow improved performance. The function of the rasterizer 422 can be adjusted by parameters or commands passed to the rasterizer 422 by, for example, an API or a GPU driver 432. After the polygon is shaded, the polygon can be written to a frame buffer in the video memory 424 for use by the display 402. As will be described in further detail below, by intelligently changing the rasterization rate, the graphics rendering efficiency can be improved while still implementing a minimum quality standard.
[0044] PMU 404 is responsible for allocating power between different components of computing system 400. Powering on GPU 420 is part of the initialization operation that prepares GPU 420 for executing graphics commands. In one embodiment, PMU 404 can access power management policies regarding the power consumption of CPU 401 and GPU 420. For example, workloads can be allocated to CPU 401, GPU 420, or a combination of the two. Then, considering the workload required for each component, PMU 404 can optimize power allocation to save the most energy. In one example, when no workload is allocated to GPU 420 for execution or when GPU 420 is waiting for the next workload to be idle, PMU 404 can put GPU 420 into sleep mode and can minimize the power consumed by GPU 420 (if any).
[0045] Figure 5A and Figure 5B A conceptual example of rasterization according to aspects of the present disclosure is shown. Generally speaking, a three-dimensional (3D) virtual object is constructed using a set of polygons composed of primitives such as lines, points, triangles, quadrilaterals, etc. 3D virtual objects occupy an object space including a virtual space in a memory, which is defined by how the 3D virtual objects are related to each other and their virtual environment. Rasterization converts primitives such as lines and triangles into a two-dimensional image that can be displayed, for example, on a display screen 502. The display screen 502 defines a screen space. Rasterization projects a view 520 of an object space 522, which may include a three-dimensional shape, such as a shape 504 for a viewpoint 506 on the display screen 502. The viewpoint represents an imaginary point where an eye can be located. The display screen 502 can be considered as a window into the object space. The size and shape of the display screen 502 help define the view into the object space. The view is defined by a frustum, which is an imaginary pyramid with sides 508 extending from the viewpoint 506 to the corners of the display screen 502 and into the object space. Points along the boundary of the shape 504 can be sampled at the rasterization rate and projected from the object space to the display screen 502 via an imaginary line 510 extending from the point in the object space to the viewpoint 506. The places where the corresponding imaginary lines 510 pass through the display screen 502 define projected points 512 of the shape 504 on the display screen 502. The display grid 514 is a representation of the display screen 502 including a grid of pixels. The projected points 512 define fragments 516, and this fragment 516 can be shaded by a fragment shader.
[0046] Generally speaking, in order to provide a more immersive experience for the user, a display with a wider field of view can be used. In some cases, a large display can be used to provide a wide field of view. One disadvantage of a large display is that a large display is generally not easy to move, and may not be suitable, for example, in a virtual reality (VR) and / or augmented reality (AR) head-mounted display (HMD). The HMD is usually worn on the user's head and includes a display worn near the user's eyes. However, the human eye has a wide field of view, and a display large enough to fill the field of view may be too large to be easily worn. In addition, such a large display may be very close to the user, thereby potentially reducing the immersion of the HMD. In some cases, a lens can be used to allow the use of a smaller display and make the HMD display feel farther away from the user. However, placing a lens between the user and the display that can fill the field of view can introduce a certain amount of distortion in the view of the display.
[0047] like Fig. 6A As shown, generally speaking, placing lens 602 between a user and display 604 produces a condensed image 606 (e.g., an image exhibiting pincushion distortion) in the display seen by the user. To correct for this effect, the HMD often displays a barrel distorted image 608 of image 610, such as Figure 6B The barrel distorted image 608 helps correct the pincushion distortion introduced by the lens 612, thereby helping the user perceive the undistorted image 614. The barrel distortion is perceived as expanding the size of the central portion of the image 610 while reducing the size of the peripheral portions of the image 610. Reducing the size of the peripheral portions reduces the effective resolution of the peripheral portions compared to the central portion in the barrel distorted image 608.
[0048] That is, the entire display 604 has a certain fixed resolution (e.g., native or set resolution). Generally speaking, the central portion of the distorted image 608 is rendered at a fixed resolution to allow the highest quality image to be displayed. Each pixel in the central portion of the display 604 represents a specific portion of the view entering the object space, and the size of each pixel of the display 604 defines the amount of the view entering the object space represented by the pixel. In some cases, the fixed or set resolution of the display 604 may be high enough so that increasing the resolution of the display will be imperceptible to most users. In the barrel-distorted image 608, the size of the peripheral portion of the distorted image 608 is reduced compared to the central portion. This distortion effectively squeezes the peripheral portion of the image into a smaller space. Conceptually, if the same level of detail as the central portion is to be maintained in these peripheral portions (e.g., obtained by sampling from the object space at the same rate), the pixels of the display 604 will also have to be squeezed into the smaller space. That is, the resolution in the peripheral portion will have to be increased to maintain the same level of detail. However, since the display 604 has a fixed resolution, it is not possible to perform an increase in the resolution in the peripheral portion. In contrast, each pixel in the peripheral portion of the distorted image 608 represents a larger portion of the view into object space than a pixel in the central portion of the distorted image 608, thereby reducing the effective resolution in the peripheral portion compared to the central portion.
[0049] Reducing the effective resolution in the peripheral portion can be performed in any desired manner. For example, in some embodiments, the reduced effective resolution in the peripheral portion can be achieved by sampling from the object space at a constant rate over the central portion of the display and then substantially discarding some samples in the peripheral portion, such as by averaging or otherwise combining the values of multiple samples together. According to certain aspects of the present disclosure, rather than sampling and discarding some samples, the sampling rate can be dynamically adjusted in portions of the distorted image 608.
[0050] Figure 7 An exemplary sampling rate function 700 according to aspects of the present disclosure is shown. Figure 7 In FIG. 1 , the X-axis represents the pixel coordinates of the screen across the original image in a single dimension. Although shown as a one-dimensional diagram, such as a horizontal set of pixel coordinates, it should be understood that a similar representation in another dimension, such as a vertical set of pixel coordinates, can be combined with Figure 7exists. The Y-axis represents the effective sampling rate as a multiplier of the full resolution of the displayed image once barrel distortion is applied. As shown, the maximum value on the Y-axis is 1, which indicates sampling at the full or highest rate required by the resolution of the display or image to be displayed. In this example, after barrel distortion is applied, the central portion of the image is sampled and rendered at full resolution, while the peripheral portions are sampled and rendered at a lower effective resolution. The sampling rate 702 includes the central portion displayed at full resolution (x1.0), while the effective resolution drops in the displayed peripheral portions (i.e., x0.75, x0.5, etc.).
[0051] It will be appreciated that in this case, the sampling rate 702 is based on the pincushion distortion caused by the lens and indicates the amount of barrel distortion that can be applied to correct the pincushion distortion. The sampling rate should be matched to the lens parameters to correspond to and correct the way the lens distorts the displayed image. Since the sampling rate 702 is based on the distortion of the lens, the specific curve of the sampling rate 702 may be different between different lenses or lens groups. As the effective resolution decreases, it is not necessary to render and apply shading to objects in the peripheral portion at the same effective resolution as applied to the full resolution area, because in the case of distortion, the effective resolution is reduced due to the distortion effect. It should be noted that although described based on the pincushion distortion caused by the lens, the sampling rate 702 can also be used to describe other resolution reduction scenarios, such as those related to other types of distortion, shadow mapping, extended drawing distance, and foveal imaging. For foveal imaging, the sensitivity difference between the area viewed by the fovea of the human eye and the area viewed by the fovea periphery and the peripheral area of the human eye is reduced. As the eye sensitivity decreases, a lower resolution image can be displayed without a perceived reduction in resolution or immersion. In other use cases, such as foveal imaging, areas outside the foveal view provide less visual acuity than the fovea, so the amount of graphical detail in those areas can be reduced without affecting the image quality perceived by the user.
[0052] In general, as the effective resolution decreases, the rasterization rate can be reduced. Reducing the rasterization rate reduces the number of points of a 3D object that are projected into a given area (such as a tile) in display space. This reduced number of points reduces the number of fragments that need to be shaded by the fragment shader. This reduced number of fragments and resolution also helps reduce the memory footprint required to store shaded fragments and textures. For example, in areas where the number of fragments is reduced, lower quality textures can be used. These lower quality textures are typically smaller than higher quality textures and have a smaller memory footprint.
[0053] According to aspects of the present disclosure, a variable rasterization rate (VRR) may be used to vary the rasterization rate of primitives (and therefore also fragments and textures of those fragments) based on where the primitives are located. Fig. 8A and Figure 8B 800 and 850 are diagrams illustrating a function (such as a piecewise defined linear function) describing a desired sampling rate to use across the axis of a display. According to aspects of the present disclosure, the function can be used to address a given distortion effect, such as barrel distortion. Figure 7 similar, Fig. 8A and Figure 8B The X-axis in represents the pixel coordinates of the screen across the initial image in a single dimension, and the Y-axis represents the effective sampling rate, which can be expressed as a multiplier of the full resolution sampling rate of the displayed image. The desired sampling rate 802 can be characterized by one or more functions 804A-804C (collectively referred to as 804) in one dimension. For example, function 804 can be a single function or multiple functions for approximating the desired sampling rate 802 in the X-axis, and another one or more functions (not shown) can be used to approximate the effective sampling rate of the Y-axis. In some cases, function 804 may be a piecewise linear function defining multiple line segments. For example, function 804 can be expressed as a single piecewise linear function with three segments (a first segment 804A with a positive slope, a second segment 804B with a zero slope, and a third segment 804C with a negative slope). Since the distortion effect may not be symmetrical along the axis, the slope of the first segment 804A may not be the inverse of the slope of the third segment 804C. Linear function 804 can be used to map between a coordinate system of screen space to which barrel distortion is applied and a coordinate system of object space, and vice versa. Although linear function 804 is shown as helpful for modeling resolution reduction experienced with barrel distortion, linear function 804 can also be used to model other situations where resolution changes can be used, such as for foveal imaging or other distortion effects. Linear function 804 can be user defined, such as defined by a programmer via an application programming interface (API) to be similar to a desired sampling rate. In some cases, linear function 804 can be defined on a per-image or per-layer basis. For example, when performing stereo rendering, where an image is rendered for the right eye and another image is rendered for the left eye, a separate set of linear functions can be specified for the image rendered in the right eye and the image rendered in the left eye. In the case where images are rendered as layers, separate linear functions can be specified for the layers.
[0054] Some graphics processing systems can generate images for display by effectively dividing the screen space into a grid of tiles and rendering each tile separately. Generally speaking, the size of the tiles is set to be significantly smaller than the size of the screen in order to reduce the memory and bandwidth requirements for processing each tile. Generally speaking, these tile sizes can be set before the image is displayed and are fixed on the image. Example tile sizes include 16×16 pixel and 32×32 pixel tiles, but tiles of any size can also be used. Generally speaking, when rendering a tile, the entire tile is rendered in one pass, and multiple tiles can be rendered in parallel. After rendering, the tiles can then be combined to form the final image for display.
[0055] In some cases, the rasterization rate 806 can be adjusted at the tile level so that the rasterization rate within the tile is constant, but can be adjusted across multiple tiles. For example, the rasterization rate can be set on a per-tile basis so that all pixels in a given tile have the same rasterization rate. Setting a single rasterization rate across tiles helps allow tiles to be efficiently processed by the graphics pipeline while still approximating the linear function 804. As shown, the rasterization rate 806 can be determined based on the linear function 804 so that the rasterization rate 806 approximates the linear function. The rasterization rate 806 can be adjusted in steps 808, where the rasterization rate 806 is changed for each step 808. Each step 808 can represent one or more tiles on a specific axis (here, the x-axis). In some cases, the highest rasterization rate (i.e., the highest sampling quality) of the tile corresponding to the linear function 804 can be used. For example, the highest rasterization rate of any point in the tile defined by the linear function 804 can be used as the rasterization rate of the entire tile. This ensures that the minimum quality of the tile at least matches the quality specified by the linear function 804 .
[0056] Fig. 9 An exemplary mapping 900 between view coordinates in object space 902 and screen space 904 in which a uniform rasterization rate is applied is shown in accordance with aspects of the present disclosure. The numbers along the x-axis and y-axis (all ones ("1") in this example) represent the sampling rate of the tile for the corresponding row or column. In this example, there is a one-to-one mapping on both the x-axis and the y-axis from each pixel in the tile in model space 902 to each pixel in the corresponding tile in screen space 904. For example, for tile 906, given a tile size of 32×32 pixels in model space 902, rasterization would be performed at 32 points of the model in model space 902 along each axis to draw 32 pixels of tile 907 in screen space 904 along each axis.
[0057] Fig.101000 between view coordinates in object space 1002 and screen space 1004 to which a variable rasterization rate has been applied according to aspects of the present disclosure. It should be noted that for clarity, the exemplary mapping 1000 is illustrated as being based on a linear function (as described above with reference to Figure 8B The invention relates to a method for varying the rasterization rate for only a small subset of tiles (e.g., two tiles in the x-axis direction and one tile in the y-axis direction) rather than varying the rasterization rate over the entire set of tiles in a given axial direction. Since objects in object space can be rasterized into screen space at different rates on a per-tile basis, a mapping between object space and screen space can be provided to a user, for example, via an API. The numbers 1006 along the Y-axis and 1008 along the X-axis of object space 1002 represent the multiplicative inverse of the rasterization rate applied to the tile. For example, tile 1010 of model space 1002 has a column value of '5' along the X-axis and a row value of '4' along the Y-axis. This indicates that each point in model space in tile 1010 will be sampled along the X-axis at 1 / 5 of the rate of tile 1011 (i.e., an exemplary tile with a "normal" or "1" rasterization rate in the x-axis direction and the y-axis direction). Similarly, each point in model space in tile 1010 will be sampled along the Y axis at 1 / 4 the rate of tile 1011. That is, if tile 1011 is sampled at 32 points on the X and Y axes, tile 1010 will be sampled at 6 points on the X axis (e.g., every 6th point is sampled compared to the points in tile 1011) and at 8 points on the Y axis (e.g., every 8th point is sampled compared to the points in tile 1011). Each point is then rasterized into screen space and expanded into a number of pixels associated with number 1006 along the Y axis and number 1008 along the X axis of object space 1002. For example, 6 sampled points on the X axis from tile 1010 can be expanded to 32 pixels on the X axis of tile 1012 in screen space. Similarly, the 8 sample points on the Y axis from tile 1010 may be extended to 32 pixels on the Y axis of tile 1012 in screen space. In other words, during rasterization, only every fifth pixel of tile 1012 in screen space 1004 is effectively sampled on the X axis from tile 1010 in object space 1002, and only every fourth pixel of tile 1012 in screen space 1004 is sampled on the Y axis from tile 1010 in object space 1002. In some embodiments, pixels in screen space 1004 may simply be copied from the sampled pixels to fill tile 1012 in screen space 1004. The copied pixels may be imperceptible because, for example, barrel distortion reduces the effective resolution of tile 1012. Similarly, for other implementations, such as those with foveated imaging, the tile including the copied pixels should be located in an area where the eye's sensitivity is reduced and the copied pixels may be imperceptible.
[0058] Fig.11 1 is a flow chart illustrating a technique for graphics processing according to aspects of the present disclosure. At step 1102, the technique is performed by receiving a first function. The first function indicates a desired sampling rate of image content, and the desired sampling rate is different for positions along an axis. For example, a user (such as a programmer via an API) may provide one or more piecewise linear functions based on lens parameters that reflect the distortion effects of a lens in one or more axial directions. At step 1104, a first rasterization rate for each of the plurality of tiles is determined by sampling corresponding portions of the piecewise linear functions. For example, the piecewise linear functions can be sampled at a desired degree of granularity to obtain a set of numbers indicating corresponding rasterization rates for various portions (e.g., tiles or sets of tiles) of an image across a given axis. Thus, the set of numbers may represent rasterization rates for various segments of the piecewise linear function for corresponding portions of the image. In some cases, the piecewise linear functions may be sampled so that the selected rasterization rate assigned to any given tile is at least as high as the highest value of the piecewise linear function corresponding to the given tile. At block 1106, one or more primitives associated with content for display are received. For example, the content may include one or more virtual objects having one or more primitives. Portions of these one or more primitives are located within a tile. At block 1108, at least a portion of the primitives associated with the corresponding tile are rasterized based on the determined rasterization rate of the corresponding tile. For example, the rasterizer may determine which pixels of the display intersect the primitives at the determined rasterization rate. As a more specific example, (e.g., as described above in conjunction with Fig.10 A tile rasterized at a rate of '5' (defined in the example discussed) may map every five pixels of the display to a single point in the tile. The pixels of the display may then be replicated based on the mapped pixels. At block 1110, an image based on the rasterized portion of the primitive is displayed. For example, a fragment shader may assign color values to the rasterized pixels and / or a texture may be mapped to the shaded fragment. The mapped texture may be at a reduced resolution compared to the full resolution texture used in other portions of the image. The resulting pixels may then be output for display.
[0059] It should be understood that the above description is intended to be exemplary and not restrictive. For example, the above embodiments may be used in combination with each other. Upon reviewing the above description, many other embodiments will be apparent to those skilled in the art. Therefore, the scope of the present invention should be determined with reference to the attached claims and the full scope of equivalents to such claims.
Claims
1. A method for graphics processing, include: receiving a first function, wherein the first function indicates a desired sampling rate of image content, wherein the first function is a piecewise linear function defining a plurality of line segments, wherein at least one line segment of the plurality of line segments is defined differently compared to another line segment of the plurality of line segments, wherein the desired sampling rate differs between a first position along a first axial direction and a second position along the first axial direction, and wherein the image content is divided into a plurality of tiles; determining a first rasterization rate for each tile of the plurality of tiles based on the desired sampling rate indicated by the first function corresponding to each respective tile, wherein a rasterization rate for a tile having multiple values of the first function across the respective tiles is determined based on a highest value of the first function corresponding to the respective tile; receiving one or more primitives associated with content for display; rasterizing at least a first portion of a primitive of the one or more primitives associated with the corresponding tile based on the determined first rasterization rate for the corresponding tile; as well as An image based on the rasterized first portion of the primitive is displayed. The method according to claim 1 , wherein the first function corresponds to a lens parameter of a lens.
3. The method according to claim 1, further comprising: include: receiving a second function indicating a desired sampling rate in a second axial direction, the second axial direction being orthogonal to the first axial direction; determining a second rasterization rate for each tile of the plurality of tiles based on the desired sampling rate for the second axial direction indicated by the second function corresponding to each respective tile; and At least a second portion of the primitive associated with the corresponding tile is rasterized based on the determined second rasterization rate for the corresponding tile.
4. The method of claim 3, wherein the second function comprises another piecewise linear function defining a plurality of line segments, wherein at least one line segment of the plurality of line segments of the another piecewise linear function is defined differently than another line segment of the another piecewise linear function.
5. The method according to claim 4, further comprising: include: providing a mapping between coordinates in a model space associated with the image content and coordinates in a screen space associated with the display; Receives a set of coordinates; as well as The set of coordinates is mapped between the coordinates in the model space and the coordinates in the screen space. 6 . The method of claim 1 , further comprising sampling the first portion of the primitive based on the determined rasterization rate.
7. A non-transitory program storage device comprising instructions stored thereon, the instructions causing one or more graphics processors to: receiving a first function, wherein the first function indicates a desired sampling rate of image content, wherein the first function is a piecewise linear function defining a plurality of line segments, wherein at least one line segment of the plurality of line segments is defined differently compared to another line segment, wherein the desired sampling rate differs between a first position along a first axial direction and a second position along the first axial direction, and wherein the image content is divided into a plurality of tiles; determining a first rasterization rate for each tile of the plurality of tiles based on the desired sampling rate indicated by the first function corresponding to each respective tile, wherein a rasterization rate for a tile having multiple values of the first function across the respective tiles is determined based on a highest value of the first function corresponding to the respective tile; receiving one or more primitives associated with content for display; rasterizing at least a first portion of a primitive of the one or more primitives associated with the corresponding tile based on the determined first rasterization rate for the corresponding tile; as well as An image based on the rasterized first portion of the primitive is displayed.
8. The non-transitory program storage device of claim 7, wherein the first function corresponds to a lens parameter of a lens.
9. The non-transitory program storage device of claim 7, wherein the instructions stored thereon further cause one or more graphics processors to: receiving a second function indicating a desired sampling rate in a second axial direction, the second axial direction being orthogonal to the first axial direction; determining a second rasterization rate for each tile of the plurality of tiles based on the desired sampling rate for the second axial direction indicated by the second function corresponding to each respective tile; and At least a second portion of the primitive associated with the corresponding tile is rasterized based on the determined second rasterization rate for the corresponding tile.
10. The non-transitory program storage device of claim 9, wherein the second function comprises another piecewise linear function defining a plurality of line segments, wherein at least one line segment of the plurality of line segments of the another piecewise linear function is defined differently than another line segment of the another piecewise linear function.
11. The non-transitory program storage device of claim 10, wherein the program storage device further comprises instructions for causing the one or more graphics processors to: providing a mapping between coordinates in a model space associated with the image content and coordinates in a screen space associated with the display; receiving a set of coordinates; and The set of coordinates is mapped between the coordinates in the model space and the coordinates in the screen space.
12. The non-transitory program storage device of claim 7, wherein the program storage device further comprises instructions for causing the one or more graphics processors to sample the first portion of the primitive based on the determined rasterization rate.
13. An electronic device, include: Memory; monitor; user interface; and one or more graphics processors operatively coupled to the memory, wherein the one or more graphics processors are configured to execute instructions that cause the one or more graphics processors to: receiving a first function, wherein the first function indicates a desired sampling rate of image content, wherein the first function is a piecewise linear function defining a plurality of line segments, wherein at least one line segment of the plurality of line segments is defined differently compared to another line segment of the plurality of line segments, wherein the desired sampling rate differs between a first position along a first axial direction and a second position along the first axial direction, and wherein the image content is divided into a plurality of tiles; determining a first rasterization rate for each tile of the plurality of tiles based on the desired sampling rate indicated by the first function corresponding to each respective tile, wherein a rasterization rate for a tile having multiple values of the first function across the respective tiles is determined based on a highest value of the first function corresponding to the tile; receiving one or more primitives associated with content for display; rasterizing at least a first portion of a primitive of the one or more primitives associated with the corresponding tile based on the determined first rasterization rate for the corresponding tile; as well as An image based on the rasterized first portion of the primitive is displayed. The electronic device of claim 13 , wherein the first function corresponds to a lens parameter of a lens.
15. The electronic device of claim 13, wherein the instructions stored thereon further cause the one or more graphics processors to: receiving a second function indicating a desired sampling rate in a second axial direction, the second axial direction being orthogonal to the first axial direction; determining a second rasterization rate for each tile of the plurality of tiles based on the desired sampling rate for the second axial direction indicated by the second function corresponding to each respective tile; and At least a second portion of the primitive associated with the corresponding tile is rasterized based on the determined second rasterization rate for the corresponding tile.
16. The electronic device of claim 15, wherein the second function comprises another piecewise linear function defining a plurality of line segments, wherein at least one line segment of the another piecewise linear function is defined differently than another line segment of the another piecewise linear function.
17. The electronic device of claim 16, wherein the instructions stored thereon further cause the one or more graphics processors to: providing a mapping between coordinates in a model space associated with the image content and coordinates in a screen space associated with the display; receiving a set of coordinates; and The set of coordinates is mapped between the coordinates in the model space and the coordinates in the screen space.
