Techniques for generating visualization of ray tracking images

Abstract

Examples described herein generally relate to generating a visualization of an image. A proprietary structure specifying ray tracing instructions for generating images using ray tracing is intercepted from a graphics processing unit (GPU) or graphics driver. The proprietary structure may be converted to a visualization structure for generating a visualization of the image based on the auxiliary information. A visualization of the image may be generated from the visualization structure.

Description

[0001] Related Applications

[0002] This application is a divisional application of the invention patent application with application number 201980016496.2 and invention name “Technology for generating visualization of ray tracing images”. Background Art

[0003] The use of computing devices is becoming more common day by day. Computing devices range from standard desktop computers to wearable computing technologies, etc. One area of ​​computing devices that has grown in recent years is in the context of image rendering, such as rendering of games, video streams, etc., which typically rely on a graphics processing unit (GPU) to render graphics from a computing device to a display device based on rendering instructions received from the computing device. Ray tracing of images is being developed as another solution for generating images via an application and requesting the display of the images via a GPU. In ray tracing, an application can define an image to be displayed, and an acceleration structure can be defined to support efficient generation of the image on a display. The acceleration structure helps to decide as quickly as possible which objects in the scene a particular ray may intersect with and reject one or more objects that the ray will not hit. Traditionally, performing ray tracing based on an acceleration structure is driver-specific to allow GPU manufacturers to define the ray tracing process for the corresponding GPU.

[0004] Tools have been developed for rasterized graphics that allow visualization of certain rasterized images, objects within images, etc. to allow developers to debug or otherwise view the rendered images / objects at a given point in time. Because ray tracing uses driver-specific acceleration structures to define images, conventional tools may not be able to process image data related to ray tracing to provide similar visualization features. Summary of the invention

[0005] A simplified overview of one or more examples is given below in order to provide a basic understanding of such examples. This overview is not an exhaustive overview of all contemplated examples, and is neither intended to identify key or important elements of all examples nor to define the scope of any or all examples. Its sole purpose is to present certain concepts of one or more examples in a simplified form as a prelude to a more detailed description presented later.

[0006] In one example, a method for generating a visualization of an image is provided. The method includes: intercepting a proprietary structure specifying ray tracing instructions for generating a ray image using ray tracing from a graphics processing unit (GPU) or a graphics driver, converting the proprietary structure into a visualization structure for generating a visualization of the image based on auxiliary information, and generating a visualization of the image from the visualization structure.

[0007] In another example, a computing device for generating a visualization of an image is provided. The computing device includes: a memory storing one or more parameters or instructions for executing an operating system and one or more applications including a visualization application, and at least one processor coupled to the memory. The at least one processor is configured to intercept a proprietary structure specifying ray tracing instructions for generating an image using ray tracing from a GPU or a graphics driver, convert the proprietary structure into a visualization structure for generating a visualization of the image based on auxiliary information, and generate a visualization of the image from the visualization structure.

[0008] In another example, a computer-readable medium including code executable by one or more processors for generating a visualization of an image is provided. The code includes code for intercepting a proprietary structure specifying ray tracing instructions for generating an image using ray tracing from a GPU or graphics driver, converting the proprietary structure to a visualization structure for generating a visualization of the image based on auxiliary information, and generating a visualization of the image from the visualization structure.

[0009] To accomplish the foregoing and related ends, one or more examples include features fully described below and particularly pointed out in the claims. The following description and the accompanying drawings set forth in detail certain illustrative features of one or more examples. However, these features are indicative of only a few of the various ways in which the principles of the various examples may be employed, and this description is intended to include all such examples and their equivalents. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a schematic diagram of an example of a computing device for generating a visualization of an image that can be ray traced by a graphics processing unit (GPU) according to examples described herein.

[0011] Figure 2 is a flow chart of an example of a method for generating a visualization of an image based on a proprietary ray tracing structure according to examples described herein.

[0012] Figure 3 is a flow chart of an example of a method for modifying a visualization of an image according to examples described herein.

[0013] Figure 4 An example of a data flow for intercepting image data to generate a visualization of an image according to examples described herein is shown.

[0014] Figure 5 An example of an interface for displaying a visualization of an image according to examples described herein is shown.

[0015] Figure 6is a schematic diagram of an example of a computing device for performing the functions described herein. DETAILED DESCRIPTION

[0016] The detailed description set forth below in conjunction with the accompanying drawings is intended as a description of various configurations and is not intended to represent the only configuration in which the concepts described herein may be practiced. In order to provide a thorough understanding of the various concepts, the detailed description includes specific details. However, it is clear to those skilled in the art that these concepts may be practiced without these specific details. In some cases, well-known components are shown in block diagram form to avoid obscuring these concepts.

[0017] Described herein are various examples related to providing visualization of an image defined by a structure used in ray tracing. For example, a proprietary structure for instructing a graphics processing unit (GPU) to perform ray tracing can be intercepted and converted into a visualization structure for providing visualization of an image. For example, the proprietary structure can be generated by a graphics driver, for example, based on a defined acceleration structure for instructing the GPU to perform ray tracing. In this regard, the proprietary structure can include efficiency and can be a proprietary format that is unknown without the assistance of the graphics driver. Therefore, auxiliary information can be used to generate a visualization structure from the proprietary structure, because an image based solely on the proprietary structure may not be completely intuitive. In one example, the auxiliary information can be information determined and applied by a graphics driver that generates a proprietary structure when instructing the GPU to perform ray tracing on an image, wherein the graphics driver can return a visualization structure and / or one or more parameters for generating the visualization structure. The visualization structure can be used to generate a visualization of an image or at least a portion of an image.

[0018] In one example, a graphics driver may receive an acceleration structure generated by an application, and may generate a proprietary structure based on the acceleration structure to introduce optimizations used by the GPU when ray tracing an image. Thus, for example, the graphics driver may be equipped with instructions for converting a proprietary structure into another structure, such as a visualization structure, which in one example may be similar to the original acceleration structure. In another example, an application may specify one or more bounding boxes in the acceleration structure for defining the placement of certain objects. In this example, the application may specify one or more cross shaders for defining the position, shape, rotation, etc. of the object geometry within the bounding box for ray tracing. In this example, the cross shader may be auxiliary information and may also be received or determined for use in generating a visualization structure, and the visualization structure may be generated based at least in part on applying the cross shader to the proprietary structure.

[0019] In either case, a visualization of an image can be generated based on a proprietary structure (e.g., in addition to the actual image processed by the GPU) without the mechanism generating the visualization knowing the format of the proprietary structure. Instead, in one example, a graphics driver can be utilized to provide the visualization structure based on the proprietary structure, and / or, in another example, a cross shader from an application can be applied to the proprietary structure to generate a representation of the image in the visualization structure. A visualization of an image or at least a portion of an image can be generated based on the visualization structure and displayed using a tool to allow viewing of a visualization of the rendered image. In one example, context parameters can also be applied to the visualization structure to modify its characteristics (e.g., camera viewpoint, position, shape, orientation, zoom level, color, etc.) in the visualization to support programmers or application developers in debugging defects in the rendering.

[0020] Now turn to Figure 1-6 , examples are depicted with reference to one or more components and one or more methods that can perform the actions or operations described herein, where the components and / or actions / operations in dashed lines may be optional. Figure 2 The operations described in the description are presented in a particular order and / or performed by example components, but in some examples, the order of actions and the components performing the actions may be changed depending on the implementation. In addition, in some examples, one or more of the following actions, functions, and / or described components may be performed by a specially programmed processor, a processor executing specially programmed software or computer-readable media, or by any other combination of hardware components and / or software components capable of performing the described actions or functions.

[0021] Figure 1 1 is a schematic diagram of an example of a computing device 100 and / or related components for generating a visualization of an image rendered using a GPU. For example, the computing device 100 may include, or may be otherwise coupled to, a processor 104 and / or a memory 106, wherein the processor 104 and / or the memory 106 may be configured to execute or store instructions or other parameters related to generating a visualization of an image rendered by a GPU, as described herein. The computing device 100 may execute an operating system 108 (e.g., via the processor 104 and / or the memory 106) for providing an environment for executing one or more applications. The computing device 100 may also include a GPU 114 for processing rendering instructions and / or transmitting associated commands to a display interface 116 to cause display of one or more images on a display, which may be coupled to the computing device 100 via a display port 118.

[0022] In one example, the display interface 116 can be communicatively coupled to the processor 104 and / or the memory 106 for communicating with a display (not shown) via a display port 118. The display port 118 mentioned herein can include one or more of various types of ports, including a High Definition Multimedia Interface (HDMI) port, a Display Serial Interface (DSI) port, a Mobile Industry Processor Interface (MIPI) DSI port, a Universal Serial Bus (USB) port, a FireWire port, or other embedded or external wired or wireless display ports that can enable communication between the computing device 100 and the display.

[0023] In one example, the GPU 114 may be part of the display interface 116 (e.g., a processor on a circuit board of the display interface 116). In another example, the GPU 114, the display interface 116, etc. may be integrated with the processor 104. Substantially any combination of hardware is possible such that the GPU 114, the display interface 116, etc. may communicate with the processor 104 via a bus to support providing rendering instructions to the GPU 114. The GPU 114 may process the rendering instructions to render an image, and may initiate display of at least a portion of the image on a display by transmitting associated signals to the display via a display port 118 of the display interface 116.

[0024] In one example, GPU 114 may support ray tracing and, therefore, may include ray tracing component 120 operable to receive proprietary structure 134, which may be generated by graphics driver 122, and perform ray tracing based on proprietary structure 134 to render an image via display interface 116. Graphics driver 122 may be specific to the manufacturer of GPU 114 and / or provided by the manufacturer of GPU 114 to generate proprietary structure 134 to instruct ray tracing component 120 to perform ray tracing to display an image.

[0025] In one example, the application 124 may operate (e.g., via the operating system 108) to generate an image for display via the display interface 116. The application 124 may generate an acceleration structure 130 that may define the geometry of the image related to ray tracing, which may include information about one or more objects in the image, locations corresponding to the one or more objects in the image, etc., so that a GPU with ray tracing capabilities (e.g., the GPU 114) may simulate photons emitted via the rays, determine where the rays reflect, trace the locations where the rays return from the scene, etc., to generate the image. In one example, the application 124 may generate the acceleration structure 130 in a standardized format that may be provided to the graphics driver 122. The graphics driver 122 may generate a proprietary structure 134 based on the acceleration structure 130 to optimize the acceleration structure 130 for ray tracing on the GPU 114, wherein the proprietary structure 134 may have a proprietary format, may include efficiencies, etc. that are known to the graphics driver 122 but not necessarily known to the application 124, the visualization application 126, etc. In one example, the graphics driver 122 may optionally include a visualization structure component 136 for generating a visualization structure from the proprietary structure 134 for use by the visualization application 126 , 156 in generating a visualization of an image.

[0026] In addition, in one example, the application 124 can generate an acceleration structure 130 to define one or more bounding boxes (e.g., axis-aligned bounding boxes (AABBs), etc.) associated with objects to be rendered in an image. In this example, the application 124 can also define one or more cross shaders 132 that specify additional shape, orientation, etc. information of the objects in the bounding box, which can be used when running (e.g., by the GPU 114) to determine the actual position of the object within the bounding box when generating the image. The application 124 can provide the cross shader 132 to the graphics driver 122, or can execute the cross shader 132 in other ways when generating the acceleration structure 130 to define the position of the corresponding object within the image.

[0027] The operating system 108 may also include a visualization application 126 for generating visualizations of images to be displayed by the display interface 116, which may occur in a separate application to allow developers of the application 124 to view certain visualizations and verify or debug the images being rendered. In one example, the visualization application 126 may be executed on the same computing device 100, or as a visualization application 156 operating on a different computing device 150 (where the different computing device 150 may have a separate processor, memory, operating system, etc. to execute the visualization application 156), as long as the visualization application 126, 156 can at least communicate with the GPU 114 (or related components) to obtain the proprietary structure 134 sent to the GPU 114 for rendering the image. The visualization application 126, 156 can operate to convert the proprietary structure 134 to a visualization structure, or otherwise receive the visualization structure as further described herein, and display the visualization structure as a visualization of at least a portion of the image. When displaying the visualization structure, the visualization application 126 , 156 may provide some additional context to allow interaction with the visualization of the image, more simplified identification of possible errors represented in the visualization of the image, and so on.

[0028] In this regard, the visualization application 126, 156 may optionally include a proprietary structure interception component 138 for obtaining the proprietary structure 134 from the GPU 114 (or from the graphics driver 122 or from communication interception between the GPU 114 and the graphics driver 122), a visualization structure component 140 for generating or receiving a visualization structure based on the proprietary structure (e.g., based on applying auxiliary information to the proprietary structure), and / or a context application component 144 for applying additional context information to the visualization structure when generating or displaying a visualization of at least a portion of an image represented by the proprietary structure 134. Although these components are only used in conjunction with the proprietary structure 134, the context application component 144 may be used to generate or receive the visualization structure based on the proprietary structure (e.g., based on applying auxiliary information to the proprietary structure). Figure 1 1, but the visualization application 156, when configured to operate on a different computing device 150, may include similar or identical components to implement similar or identical functionality as described herein for the visualization application 126. In addition, where interactions with the graphics driver 122 are described below (e.g., to provide proprietary structures and / or receive visualization structures), the visualization application 156 in the different computing device 150 may communicate with a local version of the graphics driver 122, or may otherwise communicate remotely with the graphics driver 122 on the computing device.

[0029] Figure 2 is a flowchart of an example of a method 200 for generating a visualization of an image or at least a portion of an image rendered by a GPU. For example, the method 200 may be performed by the computing device 100, and thus reference is made to Figure 1 The description is provided as a non-limiting example of an environment for performing method 200 .

[0030] In method 200, a proprietary structure specifying ray tracing instructions for generating an image using ray tracing may be intercepted from a GPU or graphics driver at act 202. In one example, a proprietary structure interception component 138 (e.g., in conjunction with processor 104, memory 106, visualization application 126, 156, etc.) may intercept a proprietary structure 134 specifying ray tracing instructions for generating an image using ray tracing from a GPU 114 or graphics driver 122 (e.g., or communications therebetween). For example, proprietary structure interception component 138 may intercept proprietary structure 134 based on at least one of: retrieving proprietary structure 134 from graphics driver 122 (e.g., based on graphics driver 122 calling a registered callback function indicated by visualization application 126, 156 or other request), retrieving proprietary structure 134 from GPU 114 based on GPU 114 reporting memory accesses (including storage of proprietary structure 134) to visualization application 126, 156 (e.g., based on GPU 114 calling a registered callback function indicated by visualization application 126, 156 or other request), and the like.

[0031] For example, as described above, the application 124 may generate an acceleration structure 130 that defines the geometry of an image, where the image may be one of the images in an image stream. For example, the acceleration structure 130 may define one or more objects, the location of one or more objects in a scene, etc., and may be generated based on a standard format that the graphics driver 122 (e.g., various graphics drivers for various GPUs) may receive and generate a proprietary structure 134 for instructing the GPU 114 to ray trace the image via the ray tracing component 120. Because the proprietary structure 134 is specific to the graphics driver 122 and / or GPU 114 implementation, the visualization application 126, 156 may not be able to interpret the proprietary structure 134 without some auxiliary information from the graphics driver 122 (e.g., where the auxiliary information may be defined at the graphics driver 122 and used to convert the proprietary structure 134 to the acceleration structure 130, a simplified version of the acceleration structure 130, or substantially any visualization structure that the visualization application 126 may understand and interpret for displaying visualization). Because the graphics driver 122 converts the acceleration structure 130 to the proprietary structure 134, the graphics driver 122 may be able to convert the proprietary structure 134 back to the acceleration structure 130 (or a similar structure), which may be similar to a visualization structure that the visualization application 126, 156 may interpret and use to generate a visualization of an image.

[0032] In another example, the acceleration structure 130 may additionally or alternatively define one or more bounding boxes (e.g., AABBs) that may define boundaries for rendering objects in an image. In this example, the application 124 may also define or otherwise use one or more cross shaders 132 to specify properties of objects within the associated bounding boxes, such as the shape or orientation of the object, the intersection of the object, etc., and the graphics driver 122 may execute the cross shader 132 or use the output of the cross shader 132 when generating the proprietary structure 134 or during instructing the GPU 114 to perform ray tracing based on the proprietary structure 134. For example, during ray tracing performed by the GPU 114, the application 124 may apply one or more cross shaders 132 to define the shape / orientation of the object to be ray traced. In this example, the cross shader 132 may also be obtained and utilized by the visualization application 126, 156 to generate a visualization structure, as further described herein.

[0033] In the method 200, optionally, in act 204, auxiliary information including one or more cross shaders defined based on the acceleration structure can be obtained. In one example, the visualization structure component 140 (e.g., in combination with the processor 104, the memory 106, the visualization application 126, 156, etc.) can obtain auxiliary information including one or more cross shaders 132 defined based on the acceleration structure 130. For example, in one example, the acceleration structure 130 can define one or more bounding boxes including objects to be rendered in ray tracing, as described above, and one or more cross shaders 132 can be defined accordingly to indicate parameters for rendering objects within the bounding boxes, such as the shape and / or orientation of the objects, intersections of objects within the bounding boxes and / or with respect to other bounding boxes, etc. In one example, the visualization structure component 140 can obtain the cross shader 132 from the application 124 (e.g., via a request sent to the application 124), obtain the cross shader 132 from the graphics driver 122 based on a call from the application 124 to execute the one or more cross shaders 132 when performing ray tracing, and so on. In any case, as described herein, one or more crossing shaders 132 may assist in rendering images by providing shape / orientation information, crossing information, etc. for objects defined by bounding boxes in the acceleration structures 130 and / or the proprietary structures 134. Thus, the crossing shaders 132 may also assist in generating visualization structures by providing shape / orientation information, crossing information, etc. for objects defined by bounding boxes in the acceleration structures 130 and / or the corresponding proprietary structures 134, as further described herein.

[0034] In method 200, at act 206, the proprietary structure may be converted to a visualization structure for generating a visualization of an image based on the auxiliary information. In one example, a visualization structure component 136 or 140 (e.g., in conjunction with a processor 104, a memory 106, a graphics driver 122, and / or a visualization application 126, 156, etc.) may convert the proprietary structure to a visualization structure for generating a visualization of an image based on the auxiliary information. For example, the visualization structure component 136 may generate a visualization structure based on applying the auxiliary information to a proprietary structure 134, which may be received from a visualization application 126, 156 based on the auxiliary information, which may also be graphics driver-specific. For example, as described above, the graphics driver 122 (and / or a native version of a graphics driver on a different computing device 150, or other software provided by a GPU manufacturer) may include instructions for generating the proprietary structure 134 from the acceleration structure 130, and may therefore include instructions for converting the proprietary structure 134 back to the acceleration structure 130 and / or a visualization structure (e.g., the auxiliary information). In one example, the visualization structure may be a simplified version of the acceleration structure that may or may not include some / all layers of ray tracing instructions, shaders, etc., such as color or lighting layers applied to the image. Additionally, for example, the visualization structure component 136 may provide the visualization structure to the visualization application 126, 156 as a rasterized image or a set of rasterization instructions (e.g., to which a context may be applied using one or more shaders, as further described herein).

[0035] In this example, converting the proprietary structure at act 206 may optionally include providing the proprietary structure to the graphics driver at act 208 for obtaining the visualization structure. In this example, the visualization structure component 140 (e.g., in combination with the processor 104, the memory 106, and / or the visualization application 126, 156, etc.) may provide the proprietary structure 134 intercepted from the graphics driver 122 or the GPU 114 to the graphics driver 122 for obtaining the visualization structure. For example, as described above, the obtained visualization structure may be similar to the original acceleration structure 130 or another structure that may be processed by the visualization application 126, 156 for displaying a visualization of at least a portion thereof. In this example, the graphics driver 122 applies its proprietary instructions to the proprietary structure 134 to generate the visualization structure for the visualization application 126, 156. In one example, the graphics driver 122 may optimize one or more features of the image when generating the visualization structure, such as object layers that may not be displayed in the image.

[0036] In another example, the auxiliary information may include one or more cross shaders 132, and converting the proprietary structure in act 206 may optionally include converting the proprietary structure based on the one or more cross shaders in act 210. In this example, the visualization structure component 140 (e.g., in conjunction with the processor 104, the memory 106, and / or the visualization application 126, 156, etc.) may provide for converting the proprietary structure based on the one or more cross shaders 132. For example, when rendering the visualization structure, the visualization structure component 140 may convert the proprietary structure 134 based at least in part on applying the one or more cross shaders 132 to the proprietary structure 134, based at least in part on applying the one or more cross shaders 132 to an acceleration structure derived from the proprietary structure (e.g., by the graphics driver 122), etc. In this example, the visualization application 126, 156 may use the GPU 114 to render a visualization of an image as the proprietary structure 134 with the one or more cross shaders 132 applied. In one example, as described herein, the visualization application 126, 156 may allow selection of one or more cross shaders 132 to be applied when generating a visualization of an image. This can allow shapes / orientations to be applied to objects, intersections, etc. within bounding boxes to create a visualization of the image rather than just a list of bounding boxes that can be stored in a proprietary structure.

[0037] In any case, the visualization application 126, 156 can obtain the visualization structure from the proprietary structure 134 for generating a visualization of the corresponding image and / or a portion of the image (e.g., an object in the image, an object within a bounding box of the image, etc.), which visualization of the corresponding image and / or a portion of the image can be displayed via the visualization application 126, 156 to allow a developer to view the image, debug errors in the rendering of the image, and so on.

[0038] In method 200, optionally, in action 212, additional context parameters for generating a visualization of an image may be determined. In one example, a context application component 144 (e.g., in combination with a processor 104, a memory 106, and / or a visualization application 126, 156, etc.) may determine and apply additional context parameters for generating a visualization of an image. In one example, the context application component 144 may determine additional context parameters to provide a modified visualization of an image and / or a portion of an image. For example, the additional context parameters may involve modifying a camera viewpoint associated with an image and / or an associated object in the image to provide a view of the image and / or the object for visualizing the acceleration structure below. In another example, where a camera viewpoint is set, the additional context parameters may additionally or alternatively involve a rotation of an image or a portion thereof, a zoom level of an image or a portion thereof, lighting or color parameters for modifying an image or a portion thereof, and the like. For example, the visualization application 126, 156 can allow the context of the visualization to be modified to change the camera viewpoint, highlight certain areas of the image, zoom in or out of the image, rotate the image, etc., which can be features provided on the user interface to allow developers to view the image in order to debug errors in the rendering. In one example, the context application component 144 can apply one or more shaders to the visualization to enable application of one or more additional context parameters. In another example, the context application component 144 can obtain and apply one or more other context parameters that are applied when generating image geometry to be displayed by the application 124.

[0039] In method 200, at act 214, a visualization of an image may be generated from the visualization structure and / or based on additional context parameters. In one example, a visualization application 126, 156 (e.g., in combination with a processor 104, a memory 106, etc.) may generate a visualization of an image from the visualization structure and / or based on additional context parameters. For example, the visualization structure may be a format that the visualization application 126, 156 may interpret, which may include an acceleration structure 130 format from which a visualization may be ray traced, a format indicating rasterized geometry for rasterizing the visualization, and the like. In any case, the visualization application 126, 156 may generate and display the visualization (e.g., via a display interface 116), and / or may allow interaction therewith, which may include changing a camera viewpoint for viewing the image, rotating the image, zooming in or out of the image, and the like, as described above. In this regard, in one example, the interaction may cause the context application component 144 to apply the context to the visualization, as described above, and implement the interaction at act 212.

[0040] Figure 3is a flowchart of an example of a method 300 for generating a visualization of an image or at least a portion of an image rendered by a GPU. For example, the method 300 may be performed by the computing device 100, and accordingly, reference is made to Figure 1 300 is described as a non-limiting example of an environment for performing method 300. In addition, in some examples, method 300 may be used with reference to Figure 2 The visualization generated in method 200 may be modified using the actions of method 300.

[0041] In method 300, at act 214, a visualization of the image may be generated from the visualization structure and / or based on additional contextual parameters, such as with respect to Figure 2 As described. In one example, the visualization application 126, 156 (e.g., in combination with the processor 104, the memory 106, etc.) can generate a visualization of the image from the visualization structure and / or based on additional context parameters, as described above. In one example, the visualization application 126, 156 can generate a visualization of the image and can allow interaction with it to determine to move the image or a portion thereof to view from different angles, add color to the image or a portion thereof, determine one or more shaders used to generate the image, etc., as described herein. For example, the visualization application 126, 156 can provide an interface (e.g., a user interface, as further described herein) to display the visualization of the image and allow the developer to perform such actions on the image or a portion thereof. In one example, the visualization application 126, 156 can allow selection of a portion of the visualization of the image to provide more information about the selection, such as one or more shaders used when generating the selected portion of the visualization, the visualization of a portion of the image (e.g., an object to be carefully examined in the image) itself, etc.

[0042] Thus, in method 300, optionally at act 302, a selection of a portion of a visualization of an image may be determined. In one example, a visualization application 126, 156 (e.g., in conjunction with a processor 104, a memory 106, etc.) may determine a selection of a portion of a visualization of an image. For example, the visualization application 126, 156 may allow selection by moving a cursor to a portion of the visualization, using a selection box or other shape to select the portion, and so on. In any case, the visualization application 126, 156 may determine a portion of the visualization corresponding to the selection, which may include one or more objects in the image. In one example, the visualization application 126, 156 may receive or determine a visualization structure as a collection of objects (e.g., rather than a single structure of an image), which may allow selection of a portion of the visualization of an image (e.g., an object).

[0043] In addition, in method 300, optionally in action 304, at least one of the one or more shaders associated with a portion of the visualization of the rendered image may be determined and indicated based on the selection of a portion of the visualization of the image. In one example, the visualization application 126, 156 (e.g., in combination with the processor 104, the memory 106, etc.) may determine and indicate at least one of the one or more shaders associated with a portion of the visualization of the rendered image based on the selection of a portion of the visualization of the image. For example, the visualization application 126, 156 may determine and / or indicate one or more cross shaders determined to correspond to the selected portion (e.g., where the proprietary structure is converted to a visualization structure including multiple portions that may have associated cross shaders). At this point, the visualization application 126, 156 may determine and indicate one or more shaders to allow debugging of a given shader in the event of a rendering error being detected. In addition, in one example, real-time modification of the shader (e.g., via the context application component 144) may be allowed and / or reapplied to allow the developer to correct a rendering error detected in a portion of the image.

[0044] In addition, in method 300, optionally in action 306, another visualization structure of an object corresponding to a portion of the visualization of the image can be generated based on the selection. In one example, the visualization application 126, 156 (e.g., in combination with the processor 104, the memory 106, etc.) can generate another visualization structure of an object corresponding to a portion of the visualization of the image based on the selection. For example, the visualization application 126, 156 can present an additional interface that only shows the selected portion (e.g., an object in the image, where the proprietary structure is converted to a visualization structure including multiple portions or objects). The visualization application 126, 156 can allow similar modifications to the object, such as adding context via the context application component 144, which can execute one or more shaders to modify one or more parameters associated with the object, such as camera viewpoint, shape / orientation, size, magnification, reduction, color, lighting, etc., as described above. This can allow for more sophisticated debugging of objects within the rendered image.

[0045] Figure 4An example of a data flow 400 of creating image data from an image geometry 402 to a rendered scene 410 by an application (e.g., application 124) is shown. In this example, an application such as application 124 can generate image geometry 402 to be rendered on a display. Application 124 can combine image geometry 402 with context information 404 to build an acceleration structure 130 representing image geometry 402. For example, context information 404 can include the location of an object within the image, the color of an object within the image, one or more shaders used to generate the image, etc. In any case, application 124 can build acceleration structure 130, which can include calling an application programming interface (API) or other mechanism to generate a standardized acceleration structure 130. In examples where acceleration structure 130 is generated to include a bounding box, one or more intersection shaders 132 can also be created or otherwise determined or employed to define certain parameters of the image, such as object shape / orientation, intersections, etc., as described above.

[0046] The acceleration structure 130 may be used to generate a proprietary structure 134 (e.g., by a graphics driver) to instruct a GPU to perform ray tracing on an image of a rendered scene 410. For example, the proprietary structure 134 may be used to dispatch rays to a GPU (e.g., GPU 114) to perform ray tracing to generate the rendered scene 410. In another example, a cross shader 132 may also be provided for generating the rendered scene 410 based on the proprietary structure 134. Further, for example, the proprietary structure 134 and / or related parameters may be intercepted by the visualization application 126, 156, which may also include intercepting the existing cross shader 132. As described above, the visualization application 126, 156 may provide the intercepted proprietary structure 134 to the graphics driver for receiving a visualization structure, which the visualization application 126, 156 may use for generating a visualization of the image (e.g., in conjunction with a cross shader or otherwise).

[0047] Figure 5An example of an interface for displaying a visualization of an image 500 on a display device is shown (e.g., via a visualization application 126 that interacts with the GPU 114 to display the interface on a display device coupled to the display interface 116). As described above, the visualization of the image 500 may include a simplified rendering of an image generated from a proprietary structure (e.g., with auxiliary information in the form of proprietary instructions executed by a graphics driver and / or a cross shader). For example, the simplified rendering shows multiple objects, such as an object 502 that may be a three-dimensionally rendered building in the visualization of the image 500, but may not include all of the colors, lighting, or other parameters / shaders of the original proprietary structure that would otherwise be ray traced by the GPU. The visualization of the image 500 may include additional context parameters for the image, which may allow the camera viewpoint of the image to be changed, the image to be rotated, the image to be zoomed in or out, one or more portions of the image to be colored, lighting parameters to be applied to the image, and the like. In any case, for example, the context may be specified by the visualization application to allow the developer to modify the image viewpoint, add / remove colors, lighting, etc., rotate the image, zoom in or out, and the like, which may involve applying one or more shaders to the image to achieve these effects. As described above, this can support debugging rendering errors by providing different visualization views requested via the user interface. In addition, the interface can allow selection of a portion of the visualization of the image 500 for additional information, such as one or more shaders used in generating the portion of the image, another visualization used to generate the selected portion of the image, etc. In one example, the selected portion of the image can include one or more objects 502, which can have separate definitions of geometry in a proprietary structure and / or an associated acceleration structure (and therefore also in a visualization structure, as described above).

[0048] Figure 6 Shown include Figure 1 Examples of computing devices 100 with other optional component details shown. In one example, computing device 100 may include a processor 104 for performing processing functions associated with one or more components and functions described herein. Processor 104 may include a single or multiple processors or a multi-core processor. In addition, processor 104 may be implemented as an integrated processing system and / or a distributed processing system.

[0049] The computing device 100 may also include a memory 106, such as for storing local versions of applications executed by the processor 104, related instructions, parameters, etc. The memory 106 may include computer-usable memory types such as random access memory (RAM), read-only memory (ROM), tape, disk, optical disk, volatile memory, non-volatile memory, and any combination thereof. In addition, the processor 104 and the memory 106 may include and execute an operating system, one or more applications (such as the graphics driver 122, the visualization application 126 and / or components thereof, the display driver, etc., as described herein), and / or other components of the computing device 100 that execute on the processor 104.

[0050] In addition, the computing device 100 may include a communication component 602 for providing for establishing and maintaining communications with one or more other devices, parties, entities, etc. using the hardware, software, and services described herein. The communication component 602 may communicate between components on the computing device 100 and between the computing device 100 and external devices (such as devices located on a communication network and / or devices connected serially or locally to the computing device 100). For example, the communication component 602 may include one or more buses, and may also include a transmit chain component and a receive chain component associated with a wireless or wired transmitter and receiver, respectively, operable to interface with the external device. For example, the communication component 602 may communicate between the visualization application 126 and a GPU on another device, between distributed components of the visualization application 126, and the like.

[0051] In addition, the computing device 100 may include a data repository 604, which may be any suitable combination of hardware and / or software, that provides mass storage of information, databases, and programs used in conjunction with the examples described herein. For example, the data repository 604 may be or may include a data repository for applications and / or related parameters that are not currently being executed by the processor 104. Furthermore, the data repository 604 may be a data repository for an operating system, an application (such as a graphics driver 122, a visualization application 126, and / or components thereof, a display driver, etc., executed on the processor 104), and / or one or more other components of the computing device 100.

[0052] The computing device 100 may also include a user interface component 606 that is operable to receive input from a user of the computing device 100 and may also be operable to generate output for presentation to the user (e.g., via a display interface 116 to a display device). The user interface component 606 may include one or more input devices, including, but not limited to, a keyboard, a numeric keypad, a mouse, a touch-sensitive display, navigation keys, function keys, a microphone, a speech recognition component, a gesture recognition component, a depth sensor, a gaze tracking sensor, any other mechanism capable of receiving input from a user, or any combination thereof. In addition, the user interface component 606 may include one or more output devices, including, but not limited to, the display interface 116, a speaker, a tactile feedback mechanism, a printer, any other mechanism capable of presenting output to a user, or any combination thereof. In one example, the user interface component 606 may display or may include an interface 500 or a similar interface to display memory usage information, as described herein.

[0053] The computing device 100 may also include a GPU 114 for rendering an image based on ray tracing instructions received from the processor 104, as described herein. The GPU 114 may additionally send a signal via the display interface 116 to cause the rendered image to be displayed on a display (not shown). Additionally, the computing device 100 may include a graphics driver 122 as described herein to provide the GPU 114 with ray tracing instructions (e.g., in a proprietary structure) and / or convert the proprietary structure to a visualization structure, as described herein. Additionally, the computing device 100 may include a visualization application 126 as described herein to generate a visualization of an image to be ray traced based on the proprietary structure without knowing the details of the proprietary structure.

[0054] For example, an element or any part of an element or any combination of elements can be implemented with a "processing system" including one or more processors. Examples of processors include microprocessors, microcontrollers, digital signal processors (DSPs), field programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gated logic, discrete hardware circuits, and other suitable hardware configured to perform the various functions described throughout this disclosure. One or more processors in a processing system can execute software. Software should be broadly interpreted as referring to instructions, instruction sets, codes, code segments, program codes, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, processes, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description languages, or other forms.

[0055] Therefore, in one or more examples, one or more functions described can be implemented in hardware, software, firmware or any combination thereof. If implemented in software, the function can be stored or encoded as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media. Storage media can be any available media that a computer can access. As an example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program codes in the form of instructions or data structures and can be accessed by a computer. Disks and optical disks used in this article include compact disks (CDs), laser optical disks, optical disks, digital versatile disks (DVDs) and floppy disks, wherein disks usually copy data magnetically, while optical disks copy data optically by lasers. The above combination should also be included in the scope of computer-readable media.

[0056] The previous description is provided to enable any person skilled in the art to practice the various examples described herein. Various modifications to these examples will be clear to those skilled in the art, and the general principles defined herein can be applied to other examples. Therefore, the claims are not intended to be limited to the examples shown herein, but to be consistent with the full scope consistent with the language claims, wherein unless explicitly stated, reference to an element in the singular is not intended to mean "one and only one", but to mean "one or more". Unless otherwise specifically stated, the term "some" refers to one or more. All structural and functional equivalents of the elements of the various examples described herein that are known or will be known to those of ordinary skill in the art are intended to be covered by the claims. Moreover, whether or not the content disclosed herein is explicitly recorded in the claims, it is not intended to be dedicated to the public. Unless the element is explicitly recorded using the phrase "device for...", no claim element should be interpreted as a device plus function.

Claims

1. A method for providing structure to a visualization application, comprising: obtaining, by a graphics processing unit GPU, a structure specifying ray tracing instructions for generating an image for a different application executed on a central processing unit CPU based on a request from the different application; as well as The structure is provided by the GPU to the visualization application based on a request from the visualization application, wherein the visualization application is an application separate from the different application and is executed on the CPU or a different CPU to generate the image alone. 2 . The method of claim 1 , wherein providing the structure to the visualization application is based on accessing a memory by the GPU to store the structure for generating the image.

3. The method of claim 1 , wherein providing the structure to the visualization application comprises: A function registered by the visualization application for receiving the structure is called.

4. The method according to claim 1, further comprising: Ray tracing of the image is performed by the GPU based on receiving the structure from a graphics driver.

5. A method for providing a visualization structure to a visualization application, comprising: receiving, by a graphics driver from the visualization application, a structure specific to the graphics driver, the structure specifying ray tracing instructions for generating an image by a graphics processing unit GPU based on requests from a different application executing on a central processing unit CPU; converting the structure into the visualization structure for use by the visualization application; as well as The visualization structure is provided to the visualization application, wherein the visualization application is an application separate from the different application and is executed on the CPU or a different CPU to generate the image alone.

6. The method according to claim 5, further comprising: generating, by the graphics driver, the structure specific to the graphics driver according to the acceleration structure received from the different applications; as well as providing the structure to the GPU for ray tracing the image, The converting the structure into the visualization structure comprises generating the visualization structure as at least a part of the acceleration structure.

7. The method of claim 6, wherein converting the structure into the visualization structure comprises: One or more features of the acceleration structure are optimized.

8. The method of claim 7, wherein the one or more features include one or more object layers defined by the acceleration structure, the object layers not being displayed when ray tracing the image.

9. The method according to claim 5, further comprising: Auxiliary information of one or more cross shaders for the structure is received, wherein converting the structure to the visualization structure includes applying the one or more cross shaders to the structure to generate the visualization structure. 10 . The method of claim 9 , wherein applying the one or more intersection shaders comprises applying a shape or orientation to objects of the image within a bounding box.

11. The method of claim 9, wherein receiving the auxiliary information comprises: The auxiliary information is received from the visualization application.

12. A computing device for providing structure to a visualization application, comprising: a memory for storing a structure specifying ray tracing instructions for generating an image; as well as at least one graphics processing unit (GPU) coupled to the memory, wherein the at least one GPU is configured to: obtaining, based on a request from a different application executing on a central processing unit CPU, a structure specifying ray tracing instructions for generating an image for the different application; as well as The structure is provided to the visualization application based on a request from the visualization application, wherein the visualization application is an application separate from the different application and is executed on the CPU or a different CPU to generate the image alone.

13. The computing device of claim 12, wherein the at least one GPU is configured to provide the structure to the visualization application based on accessing a memory by the GPU to store the structure for generating the image. 14 . The computing device of claim 12 , wherein the at least one GPU is configured to provide the structure to the visualization application at least in part by calling a function registered by the visualization application to receive the structure.

15. The computing device of claim 12, wherein the at least one GPU is further configured to provide for performing ray tracing of the image based on receiving the structure from a graphics driver.

16. A computing device for providing a visualization structure to a visualization application, comprising: a memory storing one or more parameters or instructions for executing an operating system and a graphics driver to communicate with a graphics processing unit GPU; as well as at least one processor coupled to the memory, wherein the at least one processor is configured to: receiving from the visualization application a structure specific to the graphics driver, the structure specifying ray tracing instructions for generating an image by the GPU based on requests from a different application executing on a central processing unit (CPU); converting the structure into the visualization structure for use by the visualization application; as well as The visualization structure is provided to the visualization application, wherein the visualization application is an application separate from the different application and is executed on the CPU or a different CPU to generate the image alone.

17. The computing device of claim 16, wherein the at least one processor is further configured to: generating the structure specific to the graphics driver based on the acceleration structure received from the different application; and providing the structure to the GPU for ray tracing the image, Wherein the at least one processor is configured to convert the structure to the visualization structure at least in part by generating the visualization structure as at least a portion of the acceleration structure.

18. The computing device of claim 17, wherein the at least one processor is configured to convert the structure to the visualization structure at least in part by optimizing away one or more features of the acceleration structure.

19. The computing device of claim 18, wherein the one or more features include one or more object layers defined by the acceleration structure, the object layers not being displayed when ray tracing the image.

20. The computing device of claim 16, wherein the at least one processor is further configured to receive auxiliary information for one or more cross shaders for the structure, wherein the at least one processor is configured to convert the structure into the visualization structure at least in part by applying the one or more cross shaders to the structure to generate the visualization structure.

21. The computing device of claim 20, wherein the at least one processor is configured to apply the one or more intersection shaders at least in part by applying a shape or an orientation to objects of the image within a bounding box.

22. The computing device of claim 20, wherein the at least one processor is configured to receive the auxiliary information from the visualization application.

Images