Multi-pipeline and jitter rendering method for mobile device

Abstract

Systems and methods provide improved user experience rendering techniques. The disclosed systems and methods may provide a flexible user experience in which a user may indicate snapshots to switch from user experience rendering where the user may think to have a fluent frame rate to a user experience related to higher quality image rendering. Further, the disclosed systems and methods achieve desired visual effects (e.g., anti-aliasing, depth of field) by dithering objects in scene data during image rendering. Object jitter mitigates or avoids undesirable visual effects (e.g., overflow) that may occur when applying other graphical operations to render the user experience.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Application No. 63 / 422,826, filed on November 4, 2022, entitled “INTERACTIVE-RATE PATHTRACING FOR MOBILE,” the entire contents of which are incorporated herein by reference. Technical Field

[0002] The technology of the present disclosure generally relates to methods for rendering digital images on user devices. In particular, the technology of the present disclosure includes methods and systems for applying graphics operations to a virtual environment, wherein these methods and systems implement image snapshot functions, anti-aliasing based on virtual object jitter, and depth of field effects based on virtual object jitter. Background Art

[0003] Image rendering quality, including more realistic simulation of real-world objects, higher resolution, or smoother frame rates, is an ongoing goal for many technologies and applications. This goal applies to all technologies that rely on image rendering to create a user experience, however, users of real-time gaming applications are known to have high expectations for the quality of rendering of their user experience. The trend towards processor-intensive graphics manipulation techniques, such as ray tracing, in real-time gaming applications has put pressure on innovators in the field to develop new approaches that provide the same or similar visual effects with less demand on hardware components. Innovators also seek to implement new features that give users of real-time gaming applications more flexibility to customize their user experience based on their personal preferences.

[0004] Additionally, improving rendering quality is more difficult when creating user experiences on mobile platforms, as mobile platforms often have tighter hardware constraints than comparably priced desktop platforms. Technological improvements that can provide the same or similar rendering quality with lower processor requirements are a critical step in ensuring that mobile devices remain viable in terms of user experience rendering. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] According to one or more different examples, the present disclosure is described in detail with reference to the following drawings. The drawings are for illustration purposes only and only show typical or exemplary examples.

[0006] Figure 1 An example of a user experience rendering system according to an example of the present disclosure is shown.

[0007] Figure 2A-2F Various graphics operations in an example rendering pipeline according to examples of the present disclosure are shown.

[0008] Figure 3An example graphics application generating a real-time user experience according to examples of the present disclosure is shown.

[0009] Figure 4 An example graphics application using an in-game snapshot function according to an example of the present disclosure is shown.

[0010] Figure 5 An example graphics application is shown that generates a real-time user experience and employs an embodiment of a dithered rendering pipeline according to examples of the present disclosure.

[0011] Figure 6A-6B It shows how the dithered rendering pipeline according to an example of the present disclosure can achieve an anti-aliasing effect.

[0012] Figure 7A-7B A comparison is shown between an image rendered using a depth of field effect generated by a dithered rendering pipeline and an image rendered without a depth of field effect according to an example of the present disclosure.

[0013] Figure 8 Additional block diagrams depict example computer systems that may implement various examples described herein according to examples of the present disclosure.

[0014] The drawings are not exhaustive and do not limit the disclosure to the precise forms disclosed. DETAILED DESCRIPTION

[0015] The technology of the present disclosure provides an improved user experience rendering technology. In the present disclosure, "rendering" refers to any process of converting scene data into one or more images. "Scene data" refers to a series of data structures representing a virtual environment. "Virtual environment" refers to one or more virtual objects (e.g., characters, landscape backgrounds, cameras, raster grids, light sources) positioned along a virtual coordinate system space. "User experience" refers to one or more images that can be displayed to a user to simulate a virtual environment. In some examples, the rendered user experience is also interactive. "Interactivity" means that the user can provide inputs that change the virtual environment being simulated and / or how the virtual environment is rendered into one or more images.

[0016] In some examples, the techniques of the present disclosure are organized into a graphics application running on a computer. A "graphics application" herein refers to an instruction set executable by a processor, wherein the instruction set includes instructions that cause the processor to perform graphics operations on scene data. A "graphics operation" herein refers to a set of logical operations that change the data in accordance with changes to the visual representation of the data. In some examples, an additional instruction set may be run within or in parallel with the graphics application, wherein the additional instruction set applies operations to the scene data that reflect user input or other changes to the scene data (e.g., changes to non-visual characteristics of a virtual object, such as a background story of a virtual actor or the simulated quality of a virtual object).

[0017] In one example of a graphics application, a rendering pipeline may be used. A "rendering pipeline" as described herein refers to a series of graphics operations. In some examples, a rendering pipeline may consist of graphics operations that may be conceptualized as placing and positioning virtual objects in a common coordinate system, projecting these objects onto a raster grid, converting virtual optical properties (e.g., color, brightness, transparency) of the projected objects into virtual optical properties stored at fragments within the raster grid, converting the virtual optical properties at each fragment into pixel data written to a frame buffer, and then driving the frame buffer to a display element. A "raster grid" is a virtual two-dimensional grid. A "fragment" is a single unit within a raster grid. A "pixel data" is a data entry that approximates a virtual optical property, wherein the pixel data may be compatibly driven to an associated element of a display device (e.g., a single red / green / blue combination "RGB" light emitting diode (LED) pixel on a computer display screen). A "frame buffer" is a series of locations in a computer-readable medium that corresponds to a display device element. In some examples, a frame buffer is a faster portion of a machine-readable medium (e.g., random-access memory (RAM), processor cache) that can be driven to a display device by a processor faster than a slower portion of the machine-readable medium (e.g., read-only memory (ROM), flash drive). In some examples, when the processor completes execution of a rendering pipeline, the rendering pipeline is said to have been "iterated". Multiple iterations of the rendering pipeline can result in a series of frame buffers being driven to a display device. The rate at which a new frame buffer is driven to a display device, overwriting a previous frame buffer is referred to herein as the "frame rate" of the associated user experience.

[0018] In some examples, graphics applications can provide users with greater flexibility to customize user experience according to personal preferences. This is achieved by using multiple rendering pipelines at the same time. In this context, "simultaneously" means that before executing a certain part of the graphics application, the executable instructions and / or program data associated with the graphics operations in the pipeline can be loaded into the faster part of the memory. The data and / or instructions are transferred to the faster part of the memory so that when switching between multiple rendering pipelines, the relevant data and / or instructions of any rendering pipeline can be passed to the processor faster. Because running the rendering pipeline in this way enables users to switch between multiple rendering pipelines, users gain greater flexibility to customize their user experience. In some examples, this switching is initiated by the user, who indicates that he wants to take a snapshot of the environment. In some examples, the rendering pipeline associated with the snapshot produces higher rendering quality, but the frame rate is lower, effectively enabling the user to flexibly determine when to sacrifice the user experience frame rate in exchange for the user experience rendering quality. In a system using multiple rendering pipelines in the manner disclosed herein, the switching between rendering pipelines can be completed without generating the delay required to read the rendering pipeline related data and / or instructions from the slower part of the memory.

[0019] In addition, examples of rendering pipelines disclosed herein employ improved methods for implementing visual effects in the user experience. In some examples, this is achieved by jittering virtual objects in the virtual environment between iterations of the associated rendering pipeline, thereby achieving effects such as anti-aliasing and depth of field.

[0020] "Aliasing" refers to the phenomenon of jagged edges formed when a virtual object edge passes through a fragment and the fragment is mapped to a set of virtual optical properties. In some embodiments, the fragment is constrained to a set of virtual object properties that can be compatible and ultimately converted into a set of pixel data to drive a single element of a display device.

[0021] "Jittering" refers here to applying offsets to the spatial position of a virtual object (or the points that constitute the vertices of a virtual object), where these offsets change between each iteration of the relevant rendering pipeline. In some examples, the offsets are random, where the virtual object is moved to a random point within a determined volume around its original position. In some examples, the offsets are limited to a determined area on a 2D plane perpendicular to the straight line formed between the virtual camera viewpoint and the point on the virtual object. These offsets may cause the virtual optical properties of a given fragment to change between iterations of the rendering pipeline. This change between iterations causes the pixel data output to the relevant elements of the display device to change in each image in a series of images that make up the user experience. In some embodiments, these changing elements produce a blurred vision in the user experience, thereby smoothing the jagged effect of jagged edges. This effect of reducing jagged artifacts is referred to as "anti-aliasing" here. In some embodiments, when the pixel data of a fragment is written to a frame buffer, the pixel data may be accumulated with the pixel data of the previous rendering pipeline iteration that may already exist in the frame buffer, and the previous pixel data is the result of the previous iteration of the rendering pipeline. This accumulation operation achieves anti-aliasing by performing similar blurring on pixel data in multiple rendering pipeline iterations. This operation of accumulating virtual optical properties or pixel data across multiple rendering pipeline iterations is referred to as "progressive rendering".

[0022] "Depth of field" as used herein refers to the effect of blurring virtual objects that are spatially separated from a focal point. In an example rendering pipeline that implements a depth of field effect, the rendered image encourages a viewer to focus on the focal point because virtual objects that are separated from the focal point are blurred and therefore difficult to visually perceive. In many examples, the blur increases as the distance between the virtual object and the focal point increases. An example of a rendering pipeline implementing a depth of field effect using graphics operations is disclosed herein, wherein the operations jitter the virtual objects based on their distance from the focal point defined in the scene data.

[0023] As an example of the improvement provided by the disclosed example rendering pipeline: in the case where the rendering pipeline uses dithering to achieve the effect in the rendered image, the intended visual effect is more effectively confined to the target virtual object. For example, when applying graphics operations to scene data in a later form of the rendering pipeline (e.g., a raster grid or a frame buffer) to achieve a depth of field effect, because the fragments in the raster grid or the pixel data entries in the frame buffer are not conceptually separated from the virtual objects that may be assigned the depth of field effect, this effect may be applied to multiple objects in the image. Instead, the effect is applied to virtual objects that are not the target. This misuse of the visual effect is referred to herein as "bleed-over". In some examples, further graphics operations are required for these overflow artifacts. By avoiding these further graphics operations, the rendering pipeline can be completed in a shorter time, thereby providing similar depth of field effects at a higher frame rate.

[0024] Figure 1 An example of a user experience rendering system 100 is shown. The user experience rendering system 100 may implement an example rendering pipeline (e.g., a real-time rendering pipeline 131, an offline rendering pipeline 132, an interactive time rendering pipeline 133, a jitter rendering pipeline 134) to render a user experience. The user experience rendering system 100 includes a machine-readable medium 101 (sometimes generally referred to herein as a "memory"), an interconnect system 102, a processor 103, an input interface 104, a display interface 105, and a communication interface 106. In this example, the user experience rendering system 100 is configured to generate one or more images to create a user experience by displaying the one or more images to a user.

[0025] Machine-readable medium 101 may include any form of information storage (RAM, ROM, flash drive, processor cache), covering static and dynamic storage and long-term and short-term storage. Some of the information stored on machine-readable medium 101 can be classified as executable instructions 107 and / or program data 108. Executable instructions 107 may refer to any set of instructions (e.g., compiled program logic, uncompiled program logic, machine code) stored on machine-readable medium 101, which, when executed by processor 103, causes processor 103 to perform the functions described herein. Executable instructions 107 may include operating system 109, application programs 110, and graphics applications 130. Program data 108 may refer to any set of data input to and / or output from processor 103 when processor 103 executes any portion of executable instructions 107. Program data 108 may include operating system data 111, application data 112, graphics application data 135, and rendered image storage 136.

[0026] The machine-readable medium 101 may include a combination of different storage media that are physically and / or logically separated. In addition, the data and / or instructions stored on the machine-readable medium 101 may be stored in portions on multiple storage media. For example, when executing the application 110, the processor 103 may write a portion of the application data 112 from the ROM to the RAM so that the processor 103 can access the portion of the application data 112 more quickly when executing the remaining instructions within the application 110. This operation of writing the application data 112 from the ROM to the RAM does not remove the application data 112 from the machine-readable medium 101, because the machine-readable medium 101 can be collectively referred to as any and all forms of machine-readable media that are accessible to the processor (e.g., RAM, ROM, flash drive, processor cache).

[0027] The interconnection system 102 may refer to one or more communication media that facilitate the interaction between the components of the user experience rendering system 100. Figure 1 In the embodiment, the interconnection system 102 is configured as a bus connected to the machine-readable medium 101, the processor 103, the input interface 104, the display interface 105, and the communication interface 106, but in some examples, one or more of these components can be connected to one or more other components through a dedicated connection. In some examples, one or more of these components can be connected to one or more other components through a network connection.

[0028] Processor 103 may refer to one or more general-purpose processors (e.g., microprocessors) and / or one or more special-purpose processors (e.g., graphics processing units (GPUs), network processors, or application-specific integrated circuits (ASICs)). In addition, in examples where processor 103 represents multiple processors, these processors may work in parallel so that processor 103 can execute multiple instruction sets simultaneously.

[0029] Input device 104 may refer to any device (eg, keyboard, mouse, touch screen) with which a user may interact, where the device converts such interactions into signals that may be interpreted by processor 103 . Figure 1The input device 104 is depicted as a touch screen 113 of a mobile platform, however other examples of the user experience rendering system 100 may be implemented on different platforms that support other types of input devices. For example, when a personal desktop computer is used as a platform for the user experience rendering system 100, the input device 104 may include a keyboard and / or a mouse. In addition, the input device 104 is depicted herein as being associated with only one input device, however the input device 104 may also refer to one or more additional input devices (e.g., side buttons, mobile phone cameras, microphones) that may be operated simultaneously with the mobile platform touch screen.

[0030] Display device 105 may refer to any device that can output a visual experience to a user (e.g., a smartphone screen, a liquid crystal display (LCD), an LED). The rendered image may be output to display device 105 by processor 103, which writes pixel data representing the rendered image to rendered image storage 136, and then display device 105 drives the pixel data stored in rendered image storage 136 to elements of display device 105. In some examples, processor 103 may drive the pixel data stored in rendered image storage 136 to elements of display device 105. Figure 1 Display device 105 is depicted as a mobile platform display screen 114, however other examples of user experience rendering system 100 may be implemented on platforms that support other types of display devices. For example, when a personal desktop computer is used as a platform for user experience rendering system 100, an LCD display may allow for a physically larger and / or higher resolution (more pixels per image) user experience. Additionally, display device 105 is depicted herein as relating to only one display device, however display device 105 may also refer to one or more additional display devices (e.g., LEDs, additional LCD screens) that may operate simultaneously with the mobile platform display screen.

[0031] Communication interface 106 may refer to one or more devices that allow processor 103 to communicate with components not local to processor 103, and / or one or more devices that allow instructions and / or data to be sent from machine-readable medium 101 to a network. Communication interface 106 may include a modem or soft modem, a network interface (e.g., Ethernet, network interface card, WiMedia TM , IEEE TM 802.XX or other interfaces), communication ports (such as Universal Serial Bus (USB) port, infrared (IR) port, recommended standard 232 (RS232) port, The instructions and data transmitted through the communication interface 106 are generally carried on signals, which may be electronic, electromagnetic (including optical), or other signals capable of being exchanged through a given communication interface 106. These signals may be sent from the communication interface 106 via a channel of a wired or wireless communication medium. Some examples of a channel may include a telephone line, a cellular link, a radio frequency (RF) link, an optical link, a network interface, a local area network or a wide area network, and other wired or wireless communication channels. Figure 1 The communication interface is depicted as employing wireless cellular link 115 as a channel, however other examples of user experience rendering system 100 may be implemented on platforms that support other types of communication channels. Figure 1 All components of the user experience rendering system 100 are depicted as being located within the physical housing of a mobile phone and communicating via an interconnect system 102. In some examples, one or more of the disclosed components may be located remotely (outside of any associated physical housing surrounding the one or more components), and the remaining local components may communicate with the remote components via a communication interface 106.

[0032] Referring now to the executable instructions 107 and program data 108 located on the machine-readable medium 101, the operating system 109 manages the various hardware and software components of the system and provides common interface services. The operating system 109 may include any known operating system (e.g., Windows TM 、MacOS TM 、Linux TM ), may be custom written for the system, or may not include an operating system at all, with hardware and software components providing their own interface services. When executing instructions associated with the operating system 109, the processor 103 may read input from and / or output to the operating system data 111. The operating system data 111 may refer to data related to the interface of the components of the computer system (e.g., executable instruction locations, program data locations, interface preferences, device driver locations).

[0033] Applications 110 may include one or more software programs designed to perform functions other than rendering a user experience (e.g., email, phone, Internet browser). When executing instructions related to application 110, processor 103 may read input from and / or output to application data 112. Application data 112 may refer to data related to the functions performed by application 110 (e.g., an email address book, a phone number contact list, a list of bookmarked web pages).

[0034] Graphics application 130 may include any software program designed to output one or more rendered images. Examples of graphics applications include interactive video games (e.g., World of Warcraft TM、Final Fantasy XIV TM , Pac-Man TM ) and animation software (e.g., Autodesk TM Maya TM Blender TM , Adobe TM Animate TM When executing instructions associated with graphics application 130, processor 103 may read input from and / or output to graphics application data 135. Graphics application data 135 may refer to data related to visual content displayed to a user (e.g., virtual character settings, login account information, virtual environment objects).

[0035] Rendered image storage 136 is depicted here as being located within graphics application data 135 and is where the processor 103 executes the rendering pipeline to output images. In some examples, rendered image storage 136 is a frame buffer. As described herein, rendered image storage 136 can be composed of a faster portion of the machine-readable medium 101 (e.g., RAM, processor cache). In some examples, such as when an offline rendering pipeline 132 is employed, the rendered image output by the graphics application 130 can be written to a slower portion of the rendered image storage 136 (e.g., ROM) and then later written back to the frame buffer to drive the rendered image on the display device 105 without running a rendering pipeline iteration. Note that rendered image storage 136 can refer to two separate and distinct machine-readable medium devices (e.g., RAM and ROM), both of which have the common feature of being used to store rendered image data for later display on the display device 105.

[0036] The graphics application 130 may instruct the processor 103 to process user input, graphics application data, etc. into changes to the scene data 137 that are associated with the virtual environment, such that the changes to these virtual objects are reflected in one or more images rendered based on the virtual environment. For example, in the context of a video game, the processor 103 may execute the instructions of the graphics application 130 to output changes to the scene data 137 such that the position of a virtual basketball object relative to a virtual basketball court changes as if the virtual basketball is propelled forward and falls under the action of virtual gravity.

[0037] In some examples, not all changes to virtual objects within scene data 137 trigger an iteration of the rendering pipeline and subsequent output to display device 105. In these examples, changes to the virtual basketball data within scene data 137 may occur at a rate that exceeds the rate at which processor 103 can render images and output to display device 105. Alternatively, graphics application 130 instructs processor 103 to run iterations of the rendering pipeline at regular intervals, independent of changes to scene data 137. In these examples, because the number of iterations per second of the rendering pipeline may be less than the number of times processor 103 updates scene data 137, the number of instructions executed by the rendering pipeline is less than the number of times each change to scene data 137 triggers an iteration of the rendering pipeline.

[0038] In some examples, virtual objects within scene data 137 have vertices that define polygons that make up the surface of the object. These objects, vertices, and / or polygons may additionally have associated virtual optical properties (e.g., color, brightness, transparency). In some examples, in order to convert scene data 137 into a visual image displayed on display device 105, the virtual optical properties must be converted into pixel data that can be saved to a frame buffer within rendered image storage 136.

[0039] In some examples, virtual optical properties in scene data 137 are mapped to pixel data entries in a frame buffer by instructing processor 103 to project virtual optical properties of one or more polygons in scene data 137 to locations on a raster grid. Processor 103 then writes the virtual optical properties projected to each fragment of the raster grid to a location in the frame buffer corresponding to an element of display device 105.

[0040] There are many ways to manipulate scene data 137, copies of scene data 137, fragment metadata, and / or frame buffer data to achieve desired effects in a user experience. The order and / or number of these graphics operations and the data to which these graphics operations are applied can be varied to provide a wide range of user experiences with different rendering times and / or different image qualities output by the associated rendering pipeline. Although examples of rendering pipelines are depicted here as a subset of instructions within graphics application 130, in other examples, such rendering pipelines may instead be independent instruction sets that are not part of any particular application, program, or executable instruction set.

[0041] A real-time rendering pipeline 131 may refer to any rendering pipeline that renders an image within a time frame that is intended to provide a smooth visual experience to a user based on scene data 137 that is simultaneously manipulated by user input. Those skilled in the art will appreciate that a real-time rendering pipeline typically needs to complete an iteration within a time frame that allows the entire frame buffer to be updated a minimum number of times per second (e.g., no less than 30 times per second), where the minimum number of times per second is related to a target frame rate. However, a frame rate that is considered sufficiently smooth may vary greatly depending on the graphics application and the target user experience. For example, a competitive first-person shooter video game (e.g., ) may need more than 100 frames per second to be considered smooth enough for competitive gamers, whereas for animation software (e.g. ), where the virtual environment is designed for subsequent refinement and / or viewing, may be sufficiently smooth as long as the animator is still able to position himself in the virtual environment (e.g., less than 10 frames per second may be sufficient).

[0042] The offline rendering pipeline 132 may refer to any rendering pipeline that renders images for any length of time and the scene data 137 is not simultaneously manipulated by user input. The offline rendering pipeline 132 may render one or more images based on the scene data 137 that changes over time, however, these changes are determined before the first image is rendered. In some examples, the purpose of the offline rendering pipeline 132 is to provide a higher quality visual experience for users who are not interacting with the graphics application 130 while rendering. In some examples, the cost of higher quality is often that the rendering process takes longer because the processor 103 may need to complete more steps in each iteration of the offline rendering pipeline 132. In some examples using the offline rendering pipeline 132, one or more rendered images are stored in the rendered image storage 136 and written to the frame buffer after all iterations of the offline rendering pipeline 132 are completed.

[0043] The interactive temporal rendering pipeline 133, similar to the real-time rendering pipeline 131, may refer to any rendering pipeline that renders images within a time frame that is intended to provide a smooth visual experience to the user based on the scene data 137 that is simultaneously manipulated by the user input of the user experiencing the rendering system 100. Unlike the real-time rendering pipeline 131, the interactive temporal rendering pipeline 133 may update the frame buffer at a frame rate that is lower than the real-time rendering pipeline 131 or a certain threshold (e.g., 30 frames per second). In some examples, this threshold is related to a frame rate that is considered to be smooth enough for the target user experience and can vary greatly depending on the target user. For example, a competitive first-person shooter video game (e.g., ) might consider any rendering pipeline with an output frame rate below a threshold of 100 frames per second to be an interactive time rendering pipeline. On the other hand, animation software (e.g. ) may consider the rendering pipeline to be interactive only when the frame rate of the rendering pipeline output is less than 10 frames per second.

[0044] In one example, the interactive time rendering pipeline 133 may be provided together with the real-time rendering pipeline so that the user can selectively switch between the rendering pipelines. This switching allows the user to temporarily sacrifice the number of frames rendered per second (and the smoothness of the user experience) in exchange for an improvement in the image quality (e.g., resolution, virtual lighting fidelity, number of polygons) rendered by the user experience rendering system 100.

[0045] The jittered rendering pipeline 134 may refer to a rendering pipeline that renders images within any time frame, which additionally operates on the spatial positions of virtual objects within the scene data 137 (e.g., the x, y, z coordinates of the center points, vertices, edges, etc. of the virtual objects). In some examples, the processor 103 does not directly operate on the scene data 137 when executing the jittered rendering pipeline 134, but rather operates on a copy of the scene data 137, so that changes in the spatial positions in the copy of the scene data 137 do not affect other instruction sets that may read directly from the scene data 137 (e.g., a physics engine that simulates physics in a virtual environment). This jitters the rendered virtual objects between subsequent iterations of the jittered rendering pipeline 134, so that various visual effects are accumulated in multiple images rendered by the user experience rendering system 100. In one example, the jittered rendering pipeline 134 can be used to implement anti-aliasing in the rendered user experience. In another example, the jittered rendering pipeline 134 can implement a depth of field effect.

[0046] The following Figures 2-7 show different examples of graphics applications 130 and rendering pipelines. Although the processor 103 executes instructions related to the operations disclosed herein, for the sake of language simplicity, these operations may be described as being performed by relevant examples of the graphics application 130, or these operations may not be explicitly described as being performed by any specific executor. In these cases, it should still be understood that the processor 103 executes instructions related to the operation. In some examples, the disclosed steps may be performed in a different order, in parallel on multiple processors, or in addition to other steps related to the graphics operation. Therefore, the term "step" is used for illustrative purposes and should not limit the order of operations discussed herein.

[0047] Figure 2A-2F Various graphics operations within the example rendering pipeline are shown. In some examples, the disclosed rendering pipeline constitutes a complete rendering pipeline. In other examples, the complete rendering pipeline may be composed of removing one or more disclosed graphics operations from the disclosed examples, and / or by adding one or more other graphics operations from the disclosed examples.

[0048] Figure 2A-2CApplying one or more vertex shaders to scene data is shown, wherein the vertex shaders can transform vertices associated with virtual objects in the scene data into points relative to a coordinate system that defines a virtual environment space. Prior to applying the vertex shaders, the scene data can include a list of virtual objects (e.g., characters, landscape backgrounds, cameras, raster grids, light sources) grouped into array data that define characteristics of the virtual objects, including vertices that define boundaries of the virtual objects.

[0049] Figure 2A A virtual cube object 201 is shown, represented as array data containing x, y, z coordinates of each vertex of the virtual cube object 201. Although the virtual cube object 201 is stored as a list of vertices, the virtual cube object 201 may be considered to have additional properties by being included in the associated array data. For example, each vertex may be associated with a set of virtual optical properties (e.g., color, brightness, transparency) and / or a set of virtual physical properties (e.g., mass, heat, velocity, volume).

[0050] Figure 2B 2 shows an example result after applying one or more vertex shaders to the scene data. Based on the spatial coordinates and directions of the virtual camera view defined in the scene data, a virtual space coordinate system 202 can be constructed, wherein the virtual space coordinate system 202 defines the spatial relationships within the virtual environment. Figure 2B-2E is shown in the figure, but it can be understood as being similar to what an observer sees. Figure 2B-2E The virtual environment is populated by the vertices 203 of the virtual cube object 201, as defined by the x, y, z coordinates of each vertex in the scene data. In some examples, objects outside the viewing area defined by the duration, direction, and position relative to the virtual camera's viewing angle can be removed from the scene data.

[0051] Figure 2C 204. An example result after applying a further vertex shader to the scene data is shown, where the further vertex shader includes a graphics operation to connect the vertices of the virtual object into a polygon mesh. As shown, the vertices 203 of the virtual cube object 201 have been connected to form a set of six polygons 204. In some examples, the vertex shader also interpolates virtual optical properties from the vertices of the polygons 204 across the polygons 204. In the disclosed examples, the final result of applying one or more vertex shaders is as follows: Figure 2C As shown, scene data is processed into a plurality of polygons 204 having virtual optical properties, which are located on a virtual space coordinate system 202 .

[0052] Figure 2D-2FThe rasterization process is shown as applied to a plurality of polygons 204 in a virtual environment generated from scene data. Rasterization herein refers to any process that converts a perspective containing a series of virtual objects in a virtual environment into a set of pixel data that can be output to a display device.

[0053] Figure 2D 205 is shown as being applied within the virtual environment and polygon 204 is projected onto raster grid 205. Raster grid may be a two-dimensional grid perpendicular to the viewing direction of the virtual camera. In some examples, raster grid 205 is a virtual object represented in scene data. In some examples, after raster grid is applied to the virtual environment, polygon 204 is flattened in the direction the virtual camera is facing and slides toward the virtual camera viewpoint until flattened polygon 206 intersects with raster grid 205. The final result is shown in FIG. Figure 2D As shown, a flattened polygon 206 can be seen on the raster grid 205. It should be noted that all operations described herein are performed by a processor (e.g., processor 103). Figure 2A-2F These operations are shown in a visual manner in order to facilitate better understanding by the reader, but in a more precise sense, these operations are logical operations performed by the processor on the scene data (or a copy of the scene data) stored in the memory. Those skilled in the art understand the logical operations corresponding to the given graphical operations shown here.

[0054] Figure 2E The application of one or more fragment shaders to the flattened polygons 206 and the raster grid 205 is shown. In this context, the fragment shader can be understood as filling the fragments within the raster grid with virtual optical properties associated with objects in the virtual environment. In the disclosed example, this is achieved by writing the virtual optical properties of the flattened polygons 206 to the fragments 207 that intersect with it. In some examples, each flattened polygon 206 can have optical properties mapped thereon based on the optical properties included in the virtual cube object 201. In examples not shown here, the polygons 206 can receive their optical properties at an earlier stage in the rendering pipeline, for example, when a simulated light from a virtual light source object is introduced when a vertex shader is applied. In some examples, for each fragment 207 within the raster grid 205, one or more points within the fragment 207 are sampled to determine which flattened polygon 206 intersects with the fragment 207. After identifying the intersecting polygons 206, the optical properties of the intersecting polygons 206 can be written to the fragments 207 through one or more logical operations.

[0055] In the disclosed example, each fragment 207 is shaded according to the virtual optical properties of the flattened polygon 206 that intersects the center point of each fragment 207. This can be seen in the jagged edges formed by the entire fragment 207, even if multiple flattened polygons 206 cover a fragment 207 (e.g., fragment 207A), the entire fragment 207 still displays the optical properties of a flattened polygon 206. This jagged edge visual artifact is an example of aliasing, as described above and below with respect to Figure 6A-6B As mentioned above, various anti-aliasing methods can be used to soften such jagged edge visual artifacts.

[0056] Figure 2F The final result of the entire rasterization process and rendering pipeline is shown. In the disclosed example, the fragment 207 is converted into pixel data and written to the frame buffer 208. In some examples, this conversion is accomplished by the processor 103 simply writing the optical properties at the fragment 207 to the corresponding position of the frame buffer 208. The frame buffer 208 is then driven to the display device 105, which in turn presents a series of elements to the user, where the real optical properties and orientations of these elements approximate the virtual optical properties and orientations of the frame buffer 208. Therefore, the real optical properties and orientations of these elements approximate the virtual environment observed from the perspective of the virtual camera.

[0057] Figure 3 An example graphics application 300 that generates a real-time user experience is shown. In the disclosed example, the graphics application 300 employs an embodiment of the real-time rendering pipeline 131.

[0058] At step 301, graphics application 300 is launched. In some examples, this occurs when a user provides input indicating that graphics application 300 should be run. In some examples, executable instructions and data related to graphics operations within real-time rendering pipeline 131 can be loaded into a faster portion (e.g., RAM, cache) of machine-readable medium 101. This avoids delays that may require reading related data and / or instructions from a slower portion (e.g., ROM, flash memory) of machine-readable medium 101 during execution of graphics application 300.

[0059] At step 302, the scene data is updated. In some examples, this occurs when the processor 103 copies the scene data to a faster portion of the machine-readable medium 101 (e.g., processor cache, RAM) in preparation for starting an iteration of the real-time rendering pipeline 131. In some examples, this step is only required when the scene data changes or when the graphics application is first initialized.

[0060] In the disclosed example, the real-time rendering pipeline 131 is a subset of instructions within the graphics application 300. Figure 2A-2EThe disclosed definitions and descriptions describe image operations, but other examples of the real-time rendering pipeline 131 may apply parts of the disclosed graphics operation steps in different ways, and / or may adopt the disclosed graphics operation steps in any order, and / or may adopt one or more disclosed graphics operation steps in parallel on multiple processors, and / or may adopt one or more graphics operation steps multiple times, and / or may adopt the disclosed graphics operation steps in a loop, and / or may adopt graphics operation steps not disclosed herein (e.g., tessellation, depth testing, template testing). In addition, other types of rendering pipelines (e.g., offline rendering pipelines, interactive time rendering pipelines, jitter rendering pipelines) may be used instead of the real-time rendering pipeline 131 to provide different user experiences.

[0061] At step 303, the first step of the real-time rendering pipeline 131 is to apply one or more vertex shaders to the copy of the scene data. In some examples, this occurs when the one or more vertex shaders convert vertices in the copy of the scene data into polygons placed in the virtual environment, where the polygons have associated virtual optical properties. The vertex shader is referred to above in Figure 2A-2C Described in more detail.

[0062] At step 304, the polygons in the scene data are rasterized. In some examples, this occurs when the polygons are flattened and applied to a raster grid. This step in the rasterization process is described above with reference to Figure 2D Described in more detail.

[0063] At step 305, one or more fragment shaders are applied to the flattened polygons and the raster grid. In some examples, this occurs when one or more fragment shaders fill the fragments within the raster grid with the virtual optical properties of the flattened polygons applied to the fragments. The fragment shaders are referred to above in Figure 2E Described in more detail.

[0064] At step 306, the fragment is written to the frame buffer. In some examples, this occurs when the virtual optical properties of the filled fragment are converted into pixel data and then written to the frame buffer. This step in the rasterization process is referred to above in Figure 2F Described in more detail.

[0065] Step 306 completes a subset of the real-time rendering pipeline 131 instructions in the illustrated example of the graphics application 300, however, the graphics application 300 may have only displayed one image to the user so far. In some examples, other graphics operations may be applied directly to the fragment and / or frame buffer before the frame buffer is output to the display device. In some examples, when the pixel data of the fragment is written to the frame buffer, the pixel data may be accumulated with the previous pixel data that may already exist in the frame buffer, and the previous pixel data may be the result of the previous iteration of the rendering pipeline. This accumulation operation blurs the pixel data in multiple rendering pipeline iterations. In some examples, the frame buffer may be output to the display device before an iteration of the real-time rendering pipeline 131 is completed (for example, the processor 103 may output a partially updated frame buffer 208 when a determined maximum amount of time has passed since the frame buffer 208 was previously output to the display device 105, wherein the determined maximum amount of time is related to the determined minimum frame rate). In some examples, the frame buffer may remain as is, waiting for a subsequent iteration of the rendering pipeline, and then output to the display device.

[0066] At step 307, the graphics application 300 checks for input from the user. The input from the user may be in the form of a machine-readable signal generated by a peripheral device (eg, a mouse, a keyboard, a mobile phone touch screen, a mobile phone button).

[0067] If no user input is detected, the graphics application 300 returns to step 302 and begins another iteration of the real-time rendering pipeline 131. In some examples, if part or all of the scene data copy created by step 302 of this iteration is no different from part or all of the scene data copy created by step 302 of the previous iteration (e.g., one or more of the objects, light sources, camera perspectives, etc. in the scene data have not changed), the graphics application 300 can skip the scene data update of step 302 and perform another iteration of the real-time rendering pipeline 131 to render the existing scene data. In some examples, for example, if the real-time rendering pipeline 131 does not employ multi-buffering ("multi-buffering" refers to a configuration in which additional frame buffers store previously completed frame buffer copies) or the real-time rendering pipeline 131 includes functionality to copy the latest frame buffer data to all other frame buffers, the graphics application 300 can completely avoid additional iterations of the real-time rendering pipeline 131 when the scene data has not changed since the previous iteration.

[0068] In some examples, graphics application 300 may concurrently run another application that manipulates scene data and / or another subset of instructions within graphics application 300. Even without user input, these changes to the scene data may be reflected in the copy of the scene data created at step 302. For example, a physics engine may be concurrently run, where the scene data is altered to simulate passive physical forces on virtual objects in a virtual environment (e.g., a virtual cube object moves in the negative y direction to simulate gravity acting on the virtual cube object).

[0069] If user input is detected at step 307 , graphics application 300 proceeds to step 308 .

[0070] At step 308, user input is checked to see if the user wishes to stop the graphics application 300. If the user inputs a command to terminate the execution of the graphics application 300, the graphics application 300 proceeds to step 309. At step 309, the graphics application 300 terminates and the processor 103 stops executing instructions related to the graphics application 300. If the user does not input a command to terminate the execution of the graphics application 300, the graphics application 300 proceeds to step 310.

[0071] At step 310, changes to the scene data are calculated. In some examples, step 310 includes converting user input into physical forces applied in the virtual environment. For example, a user may input that they wish to shoot a virtual bullet at a virtual cube object. In this case, the graphics application 300 can create a moving virtual bullet object in the scene data and then run the scene data through a physics engine that simulates the movement of the virtual bullet object and its collision with the virtual cube object based on the virtual physical properties (e.g., mass, velocity, volume) of the virtual bullet object and the virtual cube object. In the disclosed example, these changes are applied directly to the scene data because the physics engine operates on the scene data, not a copy of the scene data. The above-mentioned real-time rendering pipeline 131 operates on a copy of the scene data to avoid affecting other processes (e.g., a physics engine) that may obtain input from the scene data.

[0072] The graphics application 300 then returns to step 302, where a new copy of the scene data is created in preparation for the next iteration of the real-time rendering pipeline 131. As described above, in some examples, the scene data is copied to a faster portion of the machine-readable medium 101 (e.g., RAM, cache). The graphics application 300 then returns to the real-time rendering pipeline 131 to render the new copy of the scene data into an image that the user can perceive through the display device 105. The disclosed loop from step 302 to step 308, to step 310, and back to step 302, or from step 302 to step 307 and back to step 302, creates an interactive real-time user experience in which the user can perceive the virtual environment to change in response to input provided by the user.

[0073] Figure 4 An example graphics application 400 that employs an in-game snapshot function is shown. In the disclosed example, the graphics application 400 employs embodiments of both the real-time rendering pipeline 131 and the interactive time rendering pipeline 133.

[0074] At step 401, graphics application 400 is launched. In some examples, this occurs when a user provides input indicating that graphics application 400 should be run. In some examples, executable instructions and data related to graphics operations within real-time rendering pipeline 131 and interactive time rendering pipeline 133 can be loaded into a faster portion of machine-readable medium 101 (e.g., RAM, cache). Transferring the underlying instruction set of graphics operations for both pipelines to a faster portion of machine-readable medium 101 allows relevant data and / or instructions for either pipeline to be quickly passed to processor 103 when switching between rendering pipelines in subsequent iterations. This avoids delays that may require reading relevant data and / or instructions from a slower portion of machine-readable medium 101 (e.g., ROM, flash memory) during execution of graphics application 400.

[0075] At step 402, the scene data is updated. In some examples, this occurs when the processor 103 copies the scene data to a faster portion of the machine-readable medium 101 (e.g., processor cache, RAM) in preparation for starting an iteration of the real-time rendering pipeline 131 or the interactive time rendering pipeline 133. In some examples, this step is only required when the scene data changes or when the graphics application is first initialized.

[0076] At step 403, a snapshot flag is checked. In some examples, the snapshot flag is an indication stored in memory that indicates whether a user has entered a command to create an in-game snapshot. An in-game snapshot may refer to a high-quality rendering of a virtual environment, for example, a rendering in which virtual objects are displayed with higher resolution, more realistic lighting, and / or smoother surfaces than a previous rendering.

[0077] In some examples, this high-quality rendering is only possible through more complex graphics operations that require more processing time, causing the associated rendering pipeline iterations to take longer to complete than iterations of the rendering pipeline that output lower-quality renderings. When rendering pipeline iterations take longer to complete, output to the frame buffer may not be sufficient to produce a smooth frame rate. As mentioned above, the industry generally describes 30 frames per second as a smooth frame rate, however, what is considered a smooth frame rate can vary greatly between different industries and target viewing users.

[0078] In some examples, if step 403 is executed for the first time since starting graphics application 400, the snapshot flag is not set. However, subsequent steps of graphics application 400 may allow the user to set the snapshot flag, so that subsequent iterations of step 403 may be executed with the snapshot flag set. If the snapshot flag is not set, the processor continues to execute the example of real-time rendering pipeline 131.

[0079] In the disclosed example, the real-time rendering pipeline 131 is a subset of instructions within the graphics application 400. Graphics operations are referred to above. Figure 2A-2F The disclosed definitions and descriptions are described, however, other examples of the real-time rendering pipeline 131 may apply parts of the disclosed graphics operation steps in different ways, and / or may adopt the disclosed graphics operation steps in any order, and / or may adopt one or more disclosed graphics operation steps in parallel on multiple processors, and / or may adopt one or more graphics operation steps multiple times, and / or may adopt the disclosed graphics operation steps in a loop, and / or may adopt graphics operation steps not disclosed herein (e.g., surface subdivision, depth test, template test). In addition, other types of rendering pipelines (e.g., offline rendering pipelines, interactive time rendering pipelines, jitter rendering pipelines) may be used instead of the real-time rendering pipeline 131 to provide different user experiences.

[0080] At step 404, the first step of the real-time rendering pipeline 131 is to apply one or more vertex shaders to the copy of the scene data. In some examples, this occurs when the one or more vertex shaders convert vertices in the copy of the scene data into polygons placed in the virtual environment, where the polygons have associated virtual optical properties. The vertex shader is referred to above in Figure 2A-2C Described in more detail.

[0081] At step 405, the polygons in the scene data are rasterized. In some examples, this occurs when the polygons are flattened and applied to a raster grid. This step in the rasterization process is described above with reference to Figure 2D Described in more detail.

[0082] At step 406, one or more fragment shaders are applied to the flattened polygons and the raster grid. In some examples, this occurs when the one or more fragment shaders fill the fragments within the raster grid with the virtual optical properties of the flattened polygons applied to the fragments. The fragment shaders are referred to above in Figure 2E Described in more detail.

[0083] At step 407, the fragment is written to the frame buffer. In some examples, this occurs when the virtual optical properties of the filled fragment are converted into pixel data and then written to the frame buffer. This step in the rasterization process is referred to above. Figure 2F Described in more detail.

[0084] Step 407 completes a subset of the real-time rendering pipeline 131 instructions in the illustrated example of the graphics application 400, however, the graphics application 400 may have only displayed one image to the user so far. In some examples, other graphics operations may be applied directly to the fragment and / or frame buffer before the frame buffer is output to the display device. In some examples, when the pixel data of the fragment is written to the frame buffer, the pixel data may be accumulated with the previous pixel data that may already exist in the frame buffer, and the previous pixel data may be the result of the previous iteration of the rendering pipeline. This accumulation operation blurs the pixel data in multiple rendering pipeline iterations. In some examples, the frame buffer may be output to the display device before an iteration of the real-time rendering pipeline 131 is completed (for example, the processor 103 may output a partially updated frame buffer 208 when a determined maximum amount of time has passed since the frame buffer 208 was previously output to the display device 105, wherein the determined maximum amount of time is related to the determined minimum frame rate). In some examples, the frame buffer may remain as is, waiting for a subsequent iteration of the rendering pipeline, and then output to the display device.

[0085] At step 408, the graphics application 400 checks for user input. The user input may be in the form of machine-readable signals generated by a peripheral device (eg, a mouse, a keyboard, a cell phone touch screen, a cell phone button).

[0086] If no user input is detected, the graphics application 400 returns to step 402 and starts another iteration of the real-time rendering pipeline 131 or the interactive time rendering pipeline 133. In some examples, if part or all of the copy of the scene data created by step 402 of this iteration is no different from part or all of the copy of the scene data created by step 402 of the previous iteration (for example, one or more of the objects, light sources, camera perspectives, etc. in the scene data have not changed), the graphics application 400 can skip the scene data update of step 402 and perform another iteration of the real-time rendering pipeline 131 or the interactive time rendering pipeline 133 to render the existing scene data. In some examples, for example, where neither the real-time rendering pipeline 131 nor the interactive time rendering pipeline 133 employs multi-buffering ("multi-buffering" refers to a configuration in which additional frame buffers store previously completed copies of frame buffers), or where both the real-time rendering pipeline 131 and the interactive time rendering pipeline 133 include functionality to copy the latest frame buffer data to all other frame buffers, the graphics application 400 can completely avoid additional iterations of the real-time rendering pipeline 131 or the interactive time rendering pipeline 133 when the scene data has not changed since the previous iteration.

[0087] In some examples, graphics application 400 may concurrently run another application and / or a subset of instructions within graphics application 400 that causes changes to the scene data. Even without user input, these changes to the scene data may be reflected in the copy of the scene data created at step 402. For example, a physics engine may be concurrently run, where the scene data is altered to simulate passive physical forces on virtual objects in a virtual environment (e.g., a virtual cube object moves in the negative y direction to simulate gravity acting on the virtual cube object).

[0088] If user input is detected at step 408 , graphics application 400 continues to step 409 .

[0089] At step 409, user input is checked to see if the user wishes to stop the graphics application 400. If the user inputs a command to terminate the operation of the graphics application 400, the graphics application 400 proceeds to step 410. At step 410, the graphics application 400 is terminated, and the processor 103 stops executing instructions related to the graphics application 400. If the user does not input a command to terminate the operation of the graphics application 400, the graphics application 400 proceeds to step 411.

[0090] In step 411, it is checked whether the user input includes a command to set a snapshot flag. If the user input includes a command to set a snapshot flag, the graphics application 400 proceeds to step 412. If the user input does not include a command to set a snapshot flag, the graphics application 400 directly proceeds to step 413.

[0091] At step 412, a snapshot flag is set to indicate that the user has commanded a snapshot to be taken. In some examples, if the snapshot flag was not set at step 403 of the previous iteration, then during step 412, the frame buffer is replaced with uniform pixel data (e.g., all fragments in the frame buffer indicate the same color for output to the display device). This flashing of the frame buffer may produce a "camera shutter" effect, as the user experiences a brief flash of one color while the displayed pixels are repopulated with virtual optical properties output by a subsequent iteration of the interactive temporal rendering pipeline (described in further detail below in steps 414-418). The graphics application 400 then proceeds to step 413.

[0092] In step 413, changes to the scene data are calculated. In some examples, step 413 includes converting user input into physical forces applied in the virtual environment. For example, the user may input that they wish to shoot a virtual bullet at a virtual cube object. In this case, the graphics application 400 can create a moving virtual bullet object in the scene data and then run the scene data through a physics engine that simulates the movement of the virtual bullet object and its collision with the virtual cube object based on the virtual physical properties (e.g., mass, velocity, volume) of the virtual bullet object and the virtual cube object. In the disclosed example, these changes are applied directly to the scene data because the physics engine operates on the scene data, not a copy of the scene data. The above-mentioned real-time rendering pipeline 131 and interactive time rendering pipeline 133 operate on copies of the scene data to avoid affecting other processes (e.g., physics engines) that may obtain input from the scene data.

[0093] The graphics application 400 then returns to step 402, where a new copy of the scene data is created in preparation for the next iteration of the real-time rendering pipeline 131 or the interactive time rendering pipeline 133. After step 402, the processor 103 again checks the snapshot flag in step 403. In some examples, for example, if the snapshot flag has been set in step 412, the graphics application 400 continues through the instance of the interactive time rendering pipeline 133, rather than the real-time rendering pipeline 131 described above.

[0094] One difference between the real-time rendering pipeline 131 and the interactive time rendering pipeline 133 in the graphics application 400 is that one iteration of the real-time rendering pipeline 131 takes less time to complete than one iteration of the interactive time rendering pipeline 133. As described above, when a snapshot is indicated, the interactive time rendering pipeline 133 is run instead of the real-time rendering pipeline 131, so that the graphics application 400 outputs high-quality images at a lower frame rate until the snapshot flag is unset.

[0095] Those skilled in the art will appreciate that what is considered the real-time rendering pipeline 131 in one example of the graphics application 400 may be considered the interactive time rendering pipeline 133 in another example, and vice versa.

[0096] If the snapshot flag is set, the graphics application 400 proceeds from step 401 to the interactive time rendering pipeline 133. In the disclosed example, the interactive time rendering pipeline 133 is a subset of instructions within the graphics application 400. Graphics operations are referred to above. Figure 2A-2F The disclosed definitions and descriptions are described, however, other examples of the interactive time rendering pipeline 133 may apply parts of the disclosed graphics operation steps in different ways, and / or may adopt the disclosed graphics operation steps in any order, and / or may adopt one or more disclosed graphics operation steps in parallel on multiple processors, and / or may adopt one or more graphics operation steps multiple times, and / or may adopt the disclosed graphics operation steps in a loop, and / or may adopt graphics operation steps not disclosed herein (e.g., surface subdivision, depth test, template test). In addition, other types of rendering pipelines (e.g., offline rendering pipelines, interactive time rendering pipelines, jitter rendering pipelines) may be used instead of the real-time rendering pipeline 131 to provide different user experiences.

[0097] At step 414, the first step of the interactive time rendering pipeline 133 is to apply one or more vertex shaders to the copy of the scene data. In some examples, this occurs when the one or more vertex shaders convert vertices in the copy of the scene data into polygons placed in the virtual environment, where the polygons have associated virtual optical properties. The vertex shader is described above with reference to Figure 2A-2C In the disclosed example of graphics application 400, step 414 is the same as step 404, except that step 414 is performed in interactive time rendering pipeline 133, while step 404 is performed in real-time rendering pipeline 131. In other examples, step 414 and step 404 may be different.

[0098] At step 415, the polygons in the scene data are rasterized. In some examples, this occurs when the polygons are flattened and applied to a raster grid. This step in the rasterization process is described above with reference to Figure 2D In the disclosed example of the graphics application 400, step 415 is the same as step 405, except that step 415 is performed in the interactive time rendering pipeline 133, while step 405 is performed in the real-time rendering pipeline 131. In other examples, step 415 and step 405 may be different.

[0099] At step 416, path tracing graphics operations are employed to simulate lighting on the flattened polygons. Path tracing is described herein (but not illustrated) as an example of a complex graphics operation technique that may take longer for the processor 103 to complete than other graphics operation techniques. Thus, although in the disclosed example of graphics application 400, the real-time rendering pipeline 131 and the interactive temporal rendering pipeline 133 are otherwise identical, the path tracing graphics operations performed at step 416 cause the interactive temporal rendering pipeline 133 to take longer to complete than the real-time rendering pipeline 131.

[0100] In some examples, path tracing includes following one or more lines emanating from a virtual camera perspective through a flattened polygon projected on a raster grid, and then drawing one or more additional lines emanating from a surface of a non-flattened polygon to simulate a light beam. Some of these additional lines intersect with a virtual light source in the virtual environment. If the additional lines intersect with the virtual light source, the processor 103 uses the virtual light source and the virtual optical properties of the non-flattened polygon to calculate new virtual optical properties of the flattened polygon, wherein these new virtual optical properties may better approximate the behavior of real light than the virtual optical properties of the flattened polygon before.

[0101] At step 417, one or more fragment shaders are applied to the flattened polygons and the raster grid. In some examples, this occurs when one or more fragment shaders fill the fragments within the raster grid with the virtual optical properties of the flattened polygons applied to the fragments. The fragment shaders are referred to above in Figure 2E In the disclosed example of the graphics application 400, step 417 is the same as step 406, except that step 417 is performed in the interactive time rendering pipeline 133, while step 406 is performed in the real-time rendering pipeline 131. In other examples, step 417 and step 406 may be different.

[0102] At step 418, the fragment is written to the frame buffer. In some examples, this occurs when the virtual optical properties of the filled fragment are converted into pixel data and then written to the frame buffer. This step in the rasterization process is referred to above in Figure 2F In the disclosed example of the graphics application 400, step 418 is the same as step 407, except that step 418 is performed in the interactive time rendering pipeline 133, while step 407 is performed in the real-time rendering pipeline 131. In other examples, step 418 and step 407 may be different.

[0103] Step 418 completes a subset of instructions of the interactive time rendering pipeline 133 in the illustrated example in the graphics application 400, however, the graphics application 400 may have only displayed one image to the user so far. In some examples, other graphics operations may be applied directly to the fragment and / or frame buffer before the frame buffer is output to the display device. In some examples, when the pixel data of the fragment is written to the frame buffer, the pixel data may be accumulated with previous pixel data that may already exist in the frame buffer, which may be the result of a previous iteration of the rendering pipeline. This accumulation operation blurs the pixel data across multiple rendering pipeline iterations. In some examples, the frame buffer may be output to the display device before an iteration of the interactive rendering pipeline 133 is completed (e.g., the processor 103 may output a partially updated frame buffer 208 when a determined maximum amount of time has passed since the frame buffer was previously output to the display device 105, wherein the determined maximum amount of time is related to the determined minimum frame rate). In some examples, the frame buffer may remain as is, awaiting a subsequent iteration of the rendering pipeline before being output to the display device.

[0104] In some examples, the graphics application 400 then returns to step 402 through steps 408-413 and then performs another iteration of the real-time rendering pipeline 131 or the interactive time rendering pipeline 133, depending on whether the snapshot flag was previously set. In some examples, the snapshot flag may be automatically turned off by a separate set of instructions that turn off the snapshot flag after a predetermined period of time.

[0105] The end result of the graphics application 400 is to create an interactive real-time user experience in which the user can instruct to take a snapshot. This option of taking a snapshot adds flexibility to the user experience because the user can choose when to trade off between frame rate and rendering quality.

[0106] Figure 5 An example graphics application 500 that generates a real-time user experience is shown. In the disclosed example, the graphics application 500 employs an embodiment of the dithered rendering pipeline 134.

[0107] At step 501, graphics application 500 is launched. In some examples, this occurs when a user provides input indicating that graphics application 500 should be run. In some examples, executable instructions and data related to graphics operations within dithered rendering pipeline 134 can be loaded into a faster portion (e.g., RAM, cache) of machine-readable medium 101. This avoids delays that may be required during execution of graphics application 500 to read related data and / or instructions from a slower portion (e.g., ROM, flash memory) of machine-readable medium 101.

[0108] At step 502, scene data is read. In some examples, this occurs when processor 103 copies scene data to a faster portion of machine-readable medium 101 (e.g., processor cache, RAM) in preparation for starting an iteration of the dithered rendering pipeline 134. In some examples, this step is only required when scene data changes or when a graphics application is first initialized.

[0109] In the disclosed example, the dithering rendering pipeline 134 is a subset of instructions within the graphics application 500. Graphics operations are referred to above. Figure 2A-2E The disclosed definitions and descriptions are described, however, other examples of the jitter rendering pipeline 134 may apply parts of the disclosed graphics operation steps in different ways, and / or may adopt the disclosed graphics operation steps in any order, and / or may adopt one or more disclosed graphics operation steps in parallel on multiple processors, and / or may adopt one or more graphics operation steps multiple times, and / or may adopt the disclosed graphics operation steps in a loop, and / or may adopt graphics operation steps not disclosed herein (e.g., surface subdivision, depth test, template test). In addition, other types of rendering pipelines (e.g., offline rendering pipelines, interactive time rendering pipelines, real-time rendering pipelines) can replace the jitter rendering pipeline 134, be used in series or in parallel with the jitter rendering pipeline 134, to provide different user experiences.

[0110] Additionally, the dithered rendering pipeline 134 may be categorized as a real-time rendering pipeline 131, an offline rendering pipeline 132, or an interactive time rendering pipeline 133 based on its frame rate. The dithered rendering pipeline 134 is not distinguished from these other rendering pipelines based on its frame rate, but rather by the graphics operation steps it employs to apply offsets to virtual objects in scene data (described further below).

[0111] At step 503, the first step of the jitter rendering pipeline 134, one or more vertex shaders are applied to the copy of the scene data. In some examples, this occurs when the one or more vertex shaders convert vertices in the copy of the scene data into polygons placed in the virtual environment, where the polygons have associated virtual optical properties. The vertex shader is referred to above in Figure 2A-2C Described in more detail.

[0112] In step 504, the vertices within the scene data converted in step 503 are jittered. In some examples, this occurs when the processor 103 applies an offset to the spatial position of one or more vertices in the copy of the scene data. In some examples, the offset is only applied to vertices within a certain portion of the volume of the virtual environment, which may be within the field of view of the virtual camera perspective. In some examples, the offset is random, wherein the vertex is moved to a random point within a determined volume around its original position. In some examples, the offset is only applied in a direction perpendicular to the direction facing the virtual camera perspective. In some examples, the offset is limited to a determined area on a 2D plane perpendicular to the straight line formed between the virtual camera viewpoint and the vertex. In some examples, the 2D projected position of the vertex is randomly offset by a range of -1 to 1 in the X and Y directions.

[0113] In some examples, the offset is applied to the scene data itself rather than to the copy. In some examples, the scene data is applied to the copy of the scene data created in step 502 before the current iteration of the dithered rendering pipeline 134. In examples where an offset has been applied directly to one or more vertices in the scene data in a previous iteration of the dithered rendering pipeline 134, the processor 103 removes the previous offset from the scene data rather than adding a second offset. This returns the offset vertices to their original spatial positions in the scene data before continuing to run the remaining steps in the current iteration of the dithered rendering pipeline 134.

[0114] In an example where a first offset has been applied directly to one or more vertices within a first copy of the scene data in a previous iteration of the dithered rendering pipeline 134, the processor 103 does not apply a second offset to the same vertex in a second copy of the scene data in a current iteration of the dithered rendering pipeline 134. Not applying the second offset in the second iteration of the dithered rendering pipeline 134 causes the vertex to appear to be returned to a non-offset position specified in the scene data that has been copied but not modified in a previous iteration or the current iteration of the dithered rendering pipeline 134.

[0115] At step 505, the polygons in the scene data are rasterized. In some examples, this occurs when the polygons are flattened and applied to a raster grid. This step in the rasterization process is described above with reference to Figure 2D In some examples of the dithered rendering pipeline 134, a dither offset is instead applied after step 505, offsetting the position of one or more vertices of the flattened polygon, the offset being constrained to within a determined region on the raster grid.

[0116] At step 506, one or more fragment shaders are applied to the flattened polygons and the raster grid. In some examples, this occurs when the one or more fragment shaders fill the fragments within the raster grid with the virtual optical properties of the flattened polygons applied to the fragments. The fragment shaders are referred to above in Figure 2E Described in more detail.

[0117] At step 507, the fragment is written to the frame buffer. In some examples, this occurs when the virtual optical properties of the filled fragment are converted into pixel data and then written to the frame buffer. This step in the rasterization process is referred to above in Figure 2F Described in more detail.

[0118] Step 507 completes the subset of jitter rendering pipeline 134 instructions in the illustrated example of graphics application 500, however, graphics application 500 may have only displayed one image to the user so far. In some examples, other graphics operations may be applied directly to the fragment and / or frame buffer before the frame buffer is output to the display device. In some examples, when the pixel data of the fragment is written to the frame buffer, the pixel data may be accumulated with the previous pixel data that may already exist in the frame buffer, and the previous pixel data may be the result of a previous iteration of the rendering pipeline. This accumulation operation blurs the pixel data in multiple rendering pipeline iterations. In some examples, the frame buffer may be output to the display device before an iteration of the jitter rendering pipeline 134 is completed (for example, the processor 103 may output a partially updated frame buffer 208 when a determined maximum amount of time has passed since the frame buffer 208 was previously output to the display device 105, wherein the determined maximum amount of time is related to the determined minimum frame rate). In some examples, the frame buffer may remain as is, waiting for a subsequent iteration of the rendering pipeline before being output to the display device.

[0119] Those skilled in the art will appreciate that, according to the above example, the rapid succession of iterations of the jitter rendering pipeline 134 can cause one or more vertices that are offset in one or more iterations to appear to jitter between the non-offset position and one or more offset positions of one or more vertices defined in the scene data. This jitter effect, as described below with respect to FIGS. 6-7 , can be used to achieve a visual effect on one or more rendered images.

[0120] At step 508, the graphics application 500 checks for user input. The user input may be in the form of machine-readable signals generated by a peripheral device (eg, a mouse, a keyboard, a cell phone touch screen, a cell phone button).

[0121] If no user input is detected, the graphics application 500 returns to step 502 and begins another iteration of the dithered rendering pipeline 134. In some examples, if part or all of the copy of the scene data created by step 502 of this iteration is no different from part or all of the copy of the scene data created by step 502 of the previous iteration (e.g., one or more of the objects, light sources, camera perspectives, etc. in the scene data have not changed), the graphics application 500 can skip the scene data update of step 502 and perform another iteration of the dithered rendering pipeline 134 to render the existing scene data.

[0122] In some examples, graphics application 500 may concurrently run another application and / or a subset of instructions within graphics application 500 that causes changes to the scene data. Even without user input, these changes to the scene data may be reflected in the copy of the scene data created in step 502. For example, a physics engine may be concurrently run, where the scene data is altered to simulate passive physical forces on virtual objects in a virtual environment (e.g., a virtual cube object moves in the negative y direction to simulate gravity acting on the virtual cube object).

[0123] If user input is detected at step 508 , graphics application 500 continues to step 509 .

[0124] At step 509, user input is checked to see if the user wishes to stop the graphics application 500. If the user inputs a command to terminate the operation of the graphics application 500, the graphics application 500 proceeds to step 510. At step 510, the graphics application 500 terminates and the processor 103 stops executing instructions related to the graphics application 500. If the user does not input a command to terminate the operation of the graphics application 500, the graphics application 500 proceeds to step 511.

[0125] In step 511, changes to the scene data are calculated. In some examples, step 511 includes converting user input into physical forces applied in the virtual environment. For example, the user may input that they wish to shoot a virtual bullet at a virtual cube object. In this case, the graphics application 500 can create a moving virtual bullet object in the scene data, and then run the scene data through a physics engine, which simulates the movement of the virtual bullet object and its collision with the virtual cube object based on the virtual physical properties (e.g., mass, speed, volume) of the virtual bullet object and the virtual cube object. In the disclosed examples, these changes are applied directly to the scene data because the physics engine operates the scene data, rather than a copy of the scene data. In some examples, the above-mentioned jitter rendering pipeline 134 operates a copy of the scene data to avoid affecting other processes (e.g., a physics engine) that may obtain input from the scene data.

[0126] The graphics application 500 then returns to step 502, where a new copy of the scene data is created in preparation for the next iteration of the dithered rendering pipeline 134. As described above, in some examples, the scene data is copied to a faster portion of the machine-readable medium 101 (e.g., RAM, cache). The graphics application 500 then returns to the dithered rendering pipeline 134 to render the new copy of the scene data into an image that the user can perceive through the display device 105. The disclosed loop from step 502 to step 509, to step 511, and back to step 502, or from step 502 to step 508 and back to step 502, dithers vertices in the scene data (or copies of the scene data) between subsequent iterations of the dithered rendering pipeline 134. In some examples, as each iteration of the dithered rendering pipeline 134 is executed, the vertices appear to dither around their designated spatial locations in the virtual environment, as follows Figure 6A-6B As discussed, this causes the edges of a polygon to move within a fragment and change the optical properties of the output to the frame buffer.

[0127] In some examples, the jitter rendering pipeline 134 may apply step 503 before performing a different graphics operation. For example, before performing a path tracing graphics operation (described above in Figure 4 The jittering of step 503 may be applied before step 416 of step 503 (discussed in detail below in FIG. 6-7 ) to achieve aliasing and / or depth of field effects. In such an example, the jittering of the virtual object affects the collision between one or more rays drawn during the path tracing step between iterations of the rendering pipeline. In particular, when using progressive rendering techniques, such jittering may cause the lighting effects simulated by the path tracing graphics operation to become blurred across multiple rendered images that are viewed in succession.

[0128] Figure 6A-6B Shows how the dithered rendering pipeline achieves anti-aliasing. Fig. 6A , the virtual cube object 601 is shown in its idealized form with straight edges. The raster grid 602 is located between the virtual camera perspective (approximately the same perspective as the view of the view) and the cube virtual object 601. The point 603 of a vertex of the virtual cube object 601 and the edge of the virtual cube object 601 are shown as passing through the raster grid 602, but this is only for illustrative purposes - as those skilled in the art will understand, and as described above, each fragment of the raster grid 602 can only have one set of optical properties because the fragment must ultimately map to a single pixel data entry in the frame buffer. The vertex 603 and the edge of the virtual cube object 601 show where the polygons align with each fragment within the raster grid 602. The aliasing effect can be best seen in the figure by contrasting the idealized edge of the virtual cube object 601 with the jagged edge formed by the enforced uniformity of the optical properties on the individual fragments of the raster grid 602.

[0129] Figure 6B 601 in the second iteration of the dithered rendering pipeline. The vertex positions of the cube scene element 601 have shifted from the first position point 603 maintained in the previous iteration of the dithered rendering pipeline. In the current iteration of the dithered rendering pipeline, the vertices of the cube scene element 601 are now located at point 604. As can be seen in Figure 6B As can be seen from the new optical properties shown on the raster grid 602 in FIG. 6 (shown by shading), the dithering of vertices between iterations of the dithered rendering pipeline causes the following Fig. 6A Softening of jagged edges in raster grid 602 is shown. Such softening typically occurs near polygon edges, where a dither offset may be sufficient to change which polygon's optical properties are written to a given fragment of raster grid 602 in a subsequent iteration of the dithered rendering pipeline.

[0130] In some examples, when a second set of virtual optical properties associated with the virtual optical properties of the newly intersected polygon is written to the frame buffer, the second set of virtual optical properties can be accumulated and / or averaged with a previous set of virtual optical properties that already existed in the frame buffer in a previous iteration of the jitter rendering pipeline. This progressive rendering technique allows frame buffer values ​​to be averaged between iterations of the jitter rendering pipeline so that a set of virtual optical properties is output at a frame buffer location associated with a fragment that is sliced ​​by a polygon edge, and the set of virtual optical properties represents the average value of the virtual optical properties on both sides of the polygon edge. If no jitter offset is applied, the optical properties of one or the other polygon will persist in the frame buffer as long as the scene data remains static. From a macro perspective, this creates a smooth visual effect on edges that were previously jagged due to aliasing. In other words, the example of the jitter rendering pipeline 134 can be used to anti-alias the graphics user experience.

[0131] Figure 7A-7B A comparison between an image rendered with a depth effect generated using a jittered rendering pipeline and an image rendered without the depth effect is shown. The depth effect can be implemented in a jittered rendering pipeline where, instead of (or in addition to) vertex jittering to achieve an anti-aliasing effect, an offset is applied to the vertex based on its spatial relationship to a focal point 701. In some examples, the focal point 701 is a virtual object in the scene data. In some examples, as the distance between the vertex and the focal point 701 increases, the magnitude of the applied offset also increases.

[0132] Fig. 7A A rendering produced by a graphics application is shown that does not employ any means to create a depth of field effect. Figure 7B Rendering produced by a graphics application using a dithered rendering pipeline to achieve a depth of field effect is shown. Fig. 7A and Figure 7BIn the figure, the hat-wearing person scene element 702 is close to the focus 701, so the hat-wearing person scene element 702 is in Figure 7A-7B 701 remains substantially unchanged. However, in the rendered output produced by the graphics application employing the depth of field effect based on the dithered rendering pipeline, the bystander scene element 703 and the billboard element 704 become blurry. This is because successive iterations of the dithered rendering pipeline that apply the depth of field effect (as described above) cause the vertices to be jittered according to their distance from the focal point 701, thereby causing the objects associated with the jittered vertices to appear blurry because the objects jump between multiple positions between each iteration of the frame buffer being driven to the display.

[0133] Compared to the hat-wearing person scene element 702, the bystander scene element 703 and the billboard scene element 704 are much farther from the focal point 701. Therefore, in the disclosed example, in each iteration of the dithered rendering pipeline, the vertices of the bystander scene element 703 and the billboard element 704 are offset by a distance greater than the distance that the vertices of the hat-wearing person scene element 702 are offset. In some examples of the dithered rendering pipeline that applies a depth of field effect, scene elements located within a predetermined distance of the focal point 701, such as the hat-wearing person scene element 702, may not be offset at all. In some examples, a progressive rendering technique is used to change the depth of field effect of the dithered rendering pipeline. As shown in the figure, the end result of the depth of field effect is that, in some examples, the viewer's attention is drawn to the unblurred image portion close to the focal point 701.

[0134] Figure 8 A block diagram of an example computer system 800 is shown that can implement various examples described herein. The computer system 800 includes a bus 802 or other communication mechanism for communicating information, and one or more hardware processors 804 are connected to the bus 802 to process information. The hardware processor 804 can be, for example, one or more general-purpose microprocessors.

[0135] The computer system 800 also includes a main memory 806, such as RAM, cache, and other dynamic storage devices, connected to the bus 802 for storing information and instructions for execution by the processor 804. The main memory 806 may also be used to store temporary variables or other intermediate information during the execution of instructions by the processor 804. When stored in a storage medium accessible to the processor 804, the above instructions make the computer system 800 a special-purpose machine customized to perform the operations specified in the instructions.

[0136] The computer system 800 also includes a ROM 808 or other static storage device connected to the bus 802 for storing static information and instructions for execution by the processor 804. A storage device 810, such as a magnetic disk, an optical disk, or a USB flash drive (flash drive), is provided and connected to the bus 802 for storing information and instructions.

[0137] The computer system 800 may be coupled to a display 812, such as an LCD (or touch screen), via bus 802 for displaying information to a computer user. An input device 814, including alphanumeric and other keys, is coupled to bus 802 for communicating information and command selections to processor 804. Another type of user input device is a cursor control 816, such as a mouse, trackball, or cursor direction keys, for communicating directional information and command selections to processor 804 and controlling movement of a cursor on display 812. In some examples, the same directional information and command selections may be achieved by receiving touches on a touch screen without a cursor.

[0138] The computing system 800 may include a user interface module to implement a graphical user interface (GUI), which may be stored in a mass storage device as executable software code that can be executed by a computing device. This module and other modules may include, but are not limited to, components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, programs, subroutines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

[0139] Generally, the terms "component", "engine", "system", "database", "data store", etc. used herein may refer to logic implemented in hardware or firmware, or to a collection of software instructions that may have input and output points, such as in Java. TM , C, or C++ programming languages. Software components can be compiled and linked into executable programs, installed in dynamic link libraries, or written in interpreted programming languages ​​such as BASIC, Perl, TM or Python TM Written. Those skilled in the art will understand that software components can be called by other components, can call themselves, and / or can be called in response to detected events or interrupts. Software components configured to be executed on a computing device may be provided on a computer-readable medium, such as a compact disc, a digital video disc, a flash drive, a disk, or any other tangible medium, or may be provided as a digital download (and may be initially stored in a compressed or installable format that requires installation, decompression, or decryption prior to execution). Such software code may be stored in part or in whole on a storage device of the executing computing device for execution by the computing device. Software instructions may be embedded in firmware, such as an EPROM. Those skilled in the art will also understand that hardware components may consist of connected logic units, such as gates and flip-flops, or may consist of programmable units, such as a programmable gate array or a processor.

[0140] The computer system 800 may implement the techniques described herein using custom hardwired logic, one or more ASICs or field-programmable gate arrays (FPGAs), firmware, and / or program logic that, in conjunction with the computer system, enables the computer system 800 to become or be programmed as a special-purpose machine. According to an example, the techniques described herein are performed by the computer system 800 in response to the processor 804 executing one or more sequences of one or more instructions contained in the main memory 806. These instructions may be read into the main memory 806 from another storage medium such as a storage device 810. Executing the sequence of instructions contained in the main memory 806 causes the processor 804 to perform the processing steps described herein. In alternative examples, hardwired circuits may be used instead of or in combination with software instructions.

[0141] As used herein, "non-transitory media" and similar terms refer to any medium that stores data and / or instructions that cause a machine to operate in a particular manner. Such non-transitory media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage device 810. Volatile media include dynamic memory, such as main memory 806. Common forms of non-transitory media include, for example, floppy disks, removable disks, hard disks, solid-state drives, magnetic tapes, or any other magnetic data storage medium, CD-ROMs, any other optical data storage medium, any physical medium with a pattern of holes, RAM, PROM, EPROM, FLASH-EPROM, NVRAM, any other memory chip or cassette tape, and networked versions thereof.

[0142] Non-transitory media are distinct from, but may be used in conjunction with, transmission media. Transmission media participate in the transmission of information between non-transitory media. For example, transmission media include coaxial cables, copper wire, and optical fiber, including the wires that make up bus 802. Transmission media can also be in the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0143] Computer system 800 also includes a communication interface 818 connected to bus 802. Communication interface 818 provides two-way data communication coupled to one or more network links, which are connected to one or more local area networks. For example, communication interface 818 can be an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem that provides data communication connection with corresponding types of telephone lines. As another example, communication interface 818 can be a local area network (LAN) card, which is used to provide data communication connection with a compatible LAN (or WAN component to communicate with WAN). Wireless links can also be implemented. In any such implementation, communication interface 818 sends and receives electrical signals, electromagnetic signals, or optical signals that carry digital data streams representing various types of information.

[0144] A network link typically provides data communication to other data devices through one or more networks. For example, a network link may be connected to a host computer or data device operated by an Internet Service Provider (ISP) through a local area network. An ISP provides data communication services through a global packet data communication network now commonly referred to as the "Internet." Both local area networks and the Internet use electrical, electromagnetic, or optical signals that carry digital data streams. Signals through various networks and signals that carry digital data in and out of the computer system 800 through network links and communication interface 818 are example forms of transmission media.

[0145] Computer system 800 can send messages and receive data, including program code, through networks, network links and communications interface 818. In the Internet example, a server can transmit the requested code for an application program through the Internet, an ISP, a local area network and communications interface 818.

[0146] The received code may be executed by processor 804 as it is received, and / or stored in storage device 810 or other non-volatile storage for later execution.

[0147] Each of the processes, methods and algorithms described above can be implemented by code components executed by one or more computer systems or computer processors including computer hardware, and can be fully or partially automated by code components. One or more computer systems or computer processors can also support the execution of related operations in a "cloud computing" environment or as "software as a service" (SaaS). These processes and algorithms can be implemented in part or in whole in a dedicated circuit. The various features and processes described above can be used independently or in combination in various ways. Different combinations and sub-combinations are intended to be included within the scope of the present disclosure, and certain methods or process blocks may be omitted in certain implementations. The methods and processes described here are also not limited to any particular order, and the related blocks or states can be executed in other appropriate orders, or can be executed in parallel, or in other ways. Blocks or states can be added to or deleted from the disclosed examples. The execution of certain operations or processes can be distributed between computer systems or computer processors, not only residing in a single machine, but can be deployed on multiple machines.

[0148] As used herein, a "circuit" may be implemented using any form of hardware, software, or a combination of the two. For example, one or more processors, controllers, ASICs, PLAs, PALs, CPLDs, FPGAs, logic components, software routines, or other mechanisms may be used to form a circuit. In implementation, the various circuits described herein may be implemented as independent circuits, or the described functions and features may be shared in part or in whole between one or more circuits. Although various features or functional elements may be described or claimed to be independent circuits separately, these features and functions may be shared between one or more general circuits, and such description should not require or imply that independent circuits must be used to implement these features and functions. When a circuit is implemented in whole or in part using software, the software may be run in conjunction with a computing or processing system capable of performing the associated functions, such as computer system 800.

[0149] In this document, the word "or" may be understood to have an inclusive or exclusive meaning. In addition, the singular description of resources, operations or structures should not be understood to exclude the plural case. Conditional language, such as but not limited to "may", "might", "might" or "can", unless otherwise expressly stated or otherwise understood from the context, is generally intended to indicate that some examples include and other examples do not include certain features, elements and / or steps.

[0150] The terms and phrases used herein, and variations thereof, unless expressly stated otherwise, should be construed as open ended, not restrictive. Adjectives such as "conventional," "traditional," "normal," "standard," "known," and terms of similar meaning should not be construed to limit the items described to those available during a particular time period or at a particular point in time, but should be construed to encompass conventional, traditional, normal, or standard technology that may be available or known at any time now or in the future. In certain cases, the presence of expansive words and phrases such as "one or more," "at least," "but not limited to," or other similar phrases should not be interpreted as meaning that a narrower situation is intended or required where such expansive phrases may not be present.

Claims

1. A system comprising: A computer-readable medium, wherein the computer-readable medium stores executable instructions; and a processor coupled to the computer-readable medium, the processor being configured to execute the executable instructions to: First data associated with a first rendering pipeline, second data associated with a second rendering pipeline, and scene data are stored in the computer-readable medium, wherein: The first rendering pipeline includes first instructions, the first instructions causing the processor to render an image at a first frame rate based on the scene data; The second rendering pipeline includes second instructions that cause the processor to render the image at a second frame rate based on the scene data; and The first frame rate is greater than the second frame rate; detecting activation of a snapshot flag, wherein the snapshot flag indicates that the second instruction associated with the second rendering pipeline should be executed instead of executing the first instruction associated with the first rendering pipeline; and The second instruction associated with the second rendering pipeline is executed instead of the first instruction associated with the first rendering pipeline.

2. The system according to claim 1, wherein: The executable instructions further cause the processor to: While executing the first instruction of the first rendering pipeline or the second rendering pipeline that causes the processor to render an image based on the scene data, the scene data is altered to simulate interactions between virtual objects represented within the scene data. The system according to claim 2 , further, the interaction is initiated by user input.

4. The system according to claim 1, wherein: The executable instructions further cause the processor to: The first data associated with the first rendering pipeline, the second data associated with the second rendering pipeline, and the scene data are stored in the processor cache storage medium.

5. The system according to claim 1, wherein: The executable instructions further cause the processor to: The first data associated with the first rendering pipeline, the second data associated with the second rendering pipeline, and the scene data are stored in the RAM storage medium.

6. The system according to claim 1, wherein: The executable instructions further cause the processor to: A unified set of optical characteristics is written to a frame buffer prior to executing the second instruction associated with the second rendering pipeline but prior to executing the first instruction associated with the first rendering pipeline.

7. The system according to claim 1, wherein: The executable instructions further cause the processor to: After executing the second instruction associated with the second rendering pipeline, executing an additional iteration of the second rendering pipeline instead of executing the first instruction associated with the first rendering pipeline; as well as The first instruction associated with the first rendering pipeline is executed instead of the second instruction associated with the second rendering pipeline.

8. The system according to claim 7, wherein: The processor deactivates the snapshot flag after a predetermined period of time.

9. A method comprising: Apply a first set of graphics operations to the scene data; applying an offset to vertices within the scene data; applying a second set of graphics operations to the scene data to obtain a set of pixel data in the frame buffer; as well as An element of a display device is updated using the set of pixel data in the frame buffer, wherein the element of the display device comprises a pixel of the display device.

10. The method according to claim 9, further comprising: After sending the set of pixel data to the display device, a progressively rendered graphics operation is applied to the scene data, wherein the progressively rendered graphics operation changes a first pixel data value in the set of pixel data based on a second pixel data value in a previous set of pixel data associated with a previous set of graphics operations.

11. The method according to claim 10, wherein: The progressive rendering graphics operation includes averaging the first pixel data value and the second pixel data value.

12. The method according to claim 9, wherein: The scene data is copied from original scene data before applying the offsets to the vertices within the scene data.

13. The method according to claim 9, wherein: At least one of the first set of graphics operations and the second set of graphics operations performed on the scene data includes: applying a vertex shader to the vertices; projecting a polygon formed by the vertices onto a raster grid; applying a fragment shader to fragments of the raster grid; and The virtual optical characteristics at the fragment are converted into a plurality of pixel data.

14. The method according to claim 9, wherein: At least one of the first set of graphics operations and the second set of graphics operations performed on the scene data includes: Following a first straight line from a virtual camera perspective within the scene data to a polygon formed by the vertices; following a second straight line from the polygon to a virtual light source within the scene data; and The virtual optical property of the polygon is changed based on the virtual optical property of the virtual light source.

15. The method according to claim 9, further comprising: The vertices to which the offset is applied are limited to vertices within a determined volume, wherein the determined volume is associated with a virtual camera viewing angle.

16. The method according to claim 9, wherein: The offset applied to the vertex is random such that the offset moves the given vertex to a random point within a determined volume around the original position of the given vertex.

17. The method according to claim 9, further comprising: The offset applied to the vertex is constrained to a plane such that the offset moves a given vertex to a point on the plane, wherein the plane is perpendicular to a direction associated with a virtual camera viewing angle.

18. The method according to claim 9, further comprising: The offsets applied to the vertices are scaled such that a magnitude of a given offset is variable, wherein the scaling process is based on a distance between the position of a given vertex and a focal point.

19. The method according to claim 18, wherein: The scaling process increases the magnitude of the given offset as the distance between the position of the given vertex and the focal point increases.

20. A system comprising: A computer-readable medium, wherein the computer-readable medium stores executable instructions; and a processor coupled to the computer-readable medium, the processor being configured to execute the executable instructions to: Apply a first set of graphics operations to the scene data; applying an offset to vertices within the scene data; applying a second set of graphics operations to the scene data to obtain a set of pixel data in a frame buffer; and An element of a display device is updated using the set of pixel data in the frame buffer, wherein the element of the display device comprises a pixel of the display device.

Images