A method of converting a vulkan ray tracing pipeline to a compute pipeline

Abstract

The application discloses a method for converting a Vulkan ray tracing pipeline into a computing pipeline. The method is designed based on the nature of the easy parallelism of ray tracing. The application replaces all API related to ray tracing extension with a self-implementation version by hijacking the instructions in the Vulkan native driver and the shader. The application analyzes the shader provided by the user at the SPIR-V level, combines all the shaders in a way to enable the user-provided shader to be loaded into the computing pipeline, uses a recursive expansion method to solve the problem of mutual calling of different shaders, uses a series of custom types to replace the opaque types in the RTX extension, and uses a series of custom functions to replace the ray tracing instructions in the RTX extension. The application successfully solves the problem that integrated graphics cards and edge computing devices cannot run Vulkan RTX software or need to be repeatedly coded, has a huge application space, and has high universality.

Description

Technical Field

[0001] This invention relates to the field of rendering in computer graphics, and more particularly to a method for converting a Vulkan ray tracing pipeline into a computational pipeline. Technical Background

[0002] While domestically produced GPU hardware has become increasingly mature in rasterization rendering, it remains in the exploratory stage in ray tracing technology. With the increasing computing power of foreign manufacturers like Nvidia and the growing demand from the entertainment industry, the industry has been continuously experimenting with ray tracing technology in games and digital content tools to achieve more realistic rendering effects in recent years.

[0003] Currently, ray tracing applications that run directly in the compute shaders using a compute pipeline on GPUs have many limitations. Each application vendor needs to encode aspects such as the creation of accelerated structures and the intersection of rays with these structures, resulting in low coding efficiency and inconsistent runtime performance. Therefore, more and more application vendors are adopting the Vulkan RTX standard for standardized coding and to fully utilize the acceleration capabilities of ray tracing hardware. Applications using direct pipeline coding for ray tracing acceleration are gradually being phased out. Furthermore, domestically produced GPUs, due to the immaturity of ray tracing hardware design and manufacturing, do not support the Vulkan RTX standard or the operation of such applications. Summary of the Invention

[0004] To overcome the shortcomings of the prior art, this invention provides a method for converting a Vulkan ray tracing pipeline into a computational pipeline. As a runtime intermediate layer, it converts the Vulkan RTX ray tracing pipeline into a computational pipeline, enabling GPUs without RTX hardware to run Vulkan RTX applications.

[0005] This invention is achieved through the following technical solution: a method for transforming a Vulkan ray tracing pipeline into a computational pipeline, which loads and hijacks the Vulkan API at runtime for replacement, merges and replaces the instructions and types related to ray tracing extensions in the shaders passed by the user, and achieves indirect calls between shaders through recursive unrolling; including: a Vulkan RTX resource preparation stage, a shader merging stage, and a ray tracing execution stage, wherein:

[0006] (1) During the Vulkan RTX resource preparation phase, the following sub-steps are performed:

[0007] (1.1) Replace the core APIs in the Vulkan driver, including replacing the APIs related to creating Vulkan instances and Vulkan devices, and pre-allocating storage buffers for the six push constants of the simulated ray tracing pipeline for devices without Vulkan RTX functionality; replace the APIs related to obtaining device characteristics and attributes in the Vulkan driver, add corresponding device characteristics and attributes for devices without Vulkan RTX functionality, replace the APIs related to creating caches in the Vulkan driver, remove the flags related to acceleration structures in the user-input parameters and replace them with: VK_BUFFER_USAGE_STORAGE_BUFFER_BIT; replace the APIs related to creating and updating descriptors in the Vulkan driver, remove the flags related to acceleration structures in the user-input parameters and replace them with:

[0008] VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER; Replaces the API related to creating shader modules in the Vulkan driver, removes the Vulkan RTX-related flags from user-input parameters and replaces them with:

[0009] VK_PIPELINE_STAGE_COMPUTE_SHADER_BIT;

[0010] (1.2) Implement Vulkan extension APIs related to the ray tracing pipeline, including using the surface area heuristic algorithm to build a BVH acceleration structure from bottom to top in the computation shader based on the scene data passed by the user. The BVH acceleration structure is divided into a bottom acceleration structure and a top acceleration structure. The leaf nodes of the bottom acceleration structure are the triangle input passed by the user, and the leaf nodes of the top acceleration structure are an instance of the bottom acceleration structure. The data passed by the two are different, but the algorithm is the same. Replace the push constant passed by the user with ordinary shader storage buffer resources, and create a new push constant to record the GPU memory address of the above shader buffer resources.

[0011] (2) During the shader merging phase, the following sub-steps are executed:

[0012] (2.1) Merge the shaders passed in by the user into an empty shader, record the symbol IDs for matching and deduplication, record the function ID corresponding to each shader passed in by the user and use it as the shader handle required to build the shader binding table;

[0013] (2.2) Replace the instructions and opacity types related to ray tracing extension in the shader with custom functions and types, remove the OpCapacity and OpExtension instructions related to ray tracing, and use the custom function to call the user-input ray generation shader Ray Generation in the entry function;

[0014] (2.3) Replace the instructions and opacity types related to ray query extension in the shader with custom functions and types, and remove the OpCapacity and OpExtension instructions related to ray query.

[0015] (2.4) Expand the indirect recursive calls generated by the replacement in steps (2.2) and (2.3); copy the functions in the user-provided shader that generate the recursive call chain according to the maximum recursion depth specified by the user when establishing the ray tracing pipeline, copy the global variables used therein, and replace the global variables used in the call chain with the newly copied global variables;

[0016] (3) During the ray tracing execution phase, the following sub-steps are executed:

[0017] (3.1) Replace the pipeline layout created by the user with the above resources, and use the newly created pipeline layout to build a computational pipeline to realize the API related to the creation of ray tracing pipeline, and replace the flag bit used by the user when creating and binding the ray tracing pipeline with the computational pipeline flag bit.

[0018] (3.2) When the user calls the vkCmdPushConstants function, the push constants provided by the user with the ray tracing shader flags are copied to the shader storage buffer described in step (1.2). The vkCmdTraceRayKHR function is implemented using the vkDispatch function, and the addresses of each SBT passed by the user are stored in the newly created push constants for the shader to use. The vkDispatch function and the vkCmdTraceRayKHR function are the names of two functions extended by Vulkan RTX.

[0019] (3.3) The dynamic link library of the Vulkan core API and the functions mentioned above are loaded using the runtime loading method.

[0020] Furthermore, Vulkan resources not required by the ray tracing pipeline are replaced with Vulkan resources from non-ray tracing extensions; user-provided ray tracing shader code is merged into compute shaders, and custom functions and types are used to replace instructions and opacity types in ray tracing extensions; and possible recursive calls between shaders in the ray tracing pipeline are resolved by unrolling functions that cannot be recursively called in compute shaders.

[0021] Furthermore, the six buffers used to simulate the push constant in step (1.1) specifically include: Ray Generation Shader, Miss Shader, Closest Hit Shader, Any Hit Shader, Intersection Shader, and Callable Shader, with each shader using a separate storage buffer.

[0022] Furthermore, the leaf node of the top-level acceleration structure in step (1.2) is an instance of the bottom-level acceleration structure; this instance is a reference to the bottom-level acceleration structure. By referencing, it is possible to avoid storing multiple copies of the same bottom-level acceleration structure for multiple identical objects in a scene. Only after referencing the same bottom-level acceleration structure is it necessary to apply different transformation matrices to each object in the scene.

[0023] Furthermore, in step (3.2), the vkDispatch function is used to call the computation pipeline, and the vkCmdTraceRayKHR function is used to call the ray tracing pipeline. That is, both are functions that call the ray tracing pipeline by using the function that calls the computation pipeline.

[0024] The beneficial effects of this invention are:

[0025] This invention provides a method for transforming a Vulkan ray tracing pipeline into a computational pipeline. By adding a runtime intermediate layer to Vulkan, the method converts the Vulkan ray tracing pipeline into a computational pipeline, enabling the running of Vulkan RTX applications that require ray tracing hardware. This enhances the versatility of GPUs without ray tracing hardware and provides a unified framework, reducing the need for repetitive coding in various aspects such as the establishment of acceleration structures and the intersection of rays with acceleration structures for individual applications to execute ray tracing algorithms, resulting in inconsistent runtime efficiency. Attached Figure Description

[0026] Figure 1 This is a flowchart of the method of the present invention.

[0027] Figure 2 This is a schematic diagram of the shader replacement method. Detailed Implementation

[0028] like Figure 1 As shown:

[0029] S1, replace the core APIs in the Vulkan driver, including replacing the APIs related to creating Vulkan instances and Vulkan devices, pre-allocating storage buffers for the six push constants of the simulated ray tracing pipeline for devices without Vulkan RTX functionality; replacing the APIs related to obtaining device characteristics and attributes in the Vulkan driver, adding corresponding device characteristics and attributes for devices without Vulkan RTX functionality; replacing the APIs related to creating caches in the Vulkan driver, removing the flags related to acceleration structures in the user-input parameters and replacing them with: VK_BUFFER_USAGE_STORAGE_BUFFER_BIT; replacing the APIs related to allocating descriptor space and creating descriptors in the Vulkan driver, removing the flags related to acceleration structures in the user-input parameters and replacing them with: VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER; replacing the APIs related to building shader modules in the Vulkan driver, removing the flags related to Vulkan RTX in the user-input parameters and replacing them with:

[0030] VK_PIPELINE_STAGE_COMPUTE_SHADER_BIT.

[0031] S2 implements the Vulkan extension API related to the ray tracing pipeline. This includes using a surface area heuristic algorithm to build a bottom-up BVH acceleration structure in the compute shader based on the user-input scene data. The acceleration structure is divided into a bottom-level acceleration structure and a top-level acceleration structure. The leaf nodes of the bottom-level acceleration structure are the triangle data input by the user, and the leaf nodes of the top-level acceleration structure are an instance of the bottom-level acceleration structure. The input data for the two structures are different, but the algorithm is the same. The acceleration structure is saved to a buffer accessible by the compute shader. The six possible push constants input by the user are replaced with six compute shader buffer resources. New push constants are created to record the GPU memory addresses of the aforementioned shader buffer resources, and the corresponding resource mappings are recorded for use when building the pipeline layout later.

[0032] S3, at the SPIR-V layer, merges the user-input ray tracing shaders into an empty compute shader, records the IDs of the merged symbols for matching and deduplication, and records the main function ID corresponding to each user-input shader, using it as the shader handle required to build the shader binding table. The shader binding table is a buffer for recording shader addresses required by Vulkan for ray tracing; the address of this binding table needs to be passed when calling the ray tracing API.

[0033] S4 replaces the ray tracing extension-related instructions and opacity types in the shaders with custom functions and types. It creates corresponding global variables for the new built-in variables added by the ray tracing extension, replaces ray tracing instructions with functions consisting of ray query instructions, and specifically replaces the OpTraceRayKHR instruction with a call to the custom CustomTraceRay function. Except for the last parameter, the function parameters are the same as those of the OpTraceRayKHR instruction, and the final payload parameter is modified to the payload's ID in SPIR-V.

[0034] The `CustomTraceRay` function calls the `OpTypeRayQuery`, `OpRayQueryInitializeKHR`, and `OpRayQueryProceed` instructions from the ray query extension to determine the relationship between the ray and the accelerator structure. The user-provided shader entry function ID is placed in two `switch` statements, one containing all shaders called when there is intersection and the other containing shaders called when there is no intersection. During ray tracing, the corresponding function is called based on the intersection result of the ray and the accelerator structure, as well as the address of the next shader to be called in the `OpTraceRayKHR` instruction parameter passed by the user.

[0035] Remove ray tracing-related directives such as OpCapacity and OpExtension, as well as ray tracing shader flags like Execution Mode. In the merged shader entry function, use a custom `CustomCallRayGen` function and a shader binding table to call the user-specified Ray Generation shader.

[0036] S5 replaces the instructions and opacity types related to ray query expansion in the merged shader with custom functions and types. (See attached image) Figure 2As shown, the OpTypeAccelerationStructureKHR instruction is replaced with the OpTypePointer instruction and a structure containing the address of the custom acceleration structure created in S2. The OpTypeRayQueryKHR instruction is replaced with a structure containing information related to the intersection of rays and the scene. The OpRayQueryProceed instruction is replaced with a custom RayQueryProceed function for the intersection of rays and the scene, with the same function parameters as the OpRayQueryProceed instruction.

[0037] Remove the OpCapacity and OpExtension directives related to ray queries.

[0038] S6 expands the indirect recursive calls generated by the replacements in S4 and S5. It copies and links the functions generating the recursive call chain in the user-provided shaders and variables of the ray tracing extension's newly added storage type, such as RayPayload, used to pass ray tracing results, according to the maximum recursion depth specified by the user when establishing the ray tracing pipeline. This includes:

[0039] a. Record new storage types such as RayPayload in the Ray Generation shader.

[0040] b. Link the newly added storage type in the first CustomTraceRay function and the Ray Generation shader; the CustomTraceRay function is a custom version of the GLSL built-in function TraceRayKHR, used to replace the built-in function; the function's purpose is to find the intersection of the ray and the scene, deciding whether to call the Intersection Shader based on the primitives, and then deciding whether to call the Miss Shader, Closest Hit Shader, or AnyHit Shader based on the intersection point. After the call, the result of the called shader is returned.

[0041] c. Copy the new CustomTraceRay function, replace the CustomTraceRay function in the merged non-Ray Generation type shader with the newly copied CustomTraceRay function, and record the newly copied RayPayloadid (ray payload); in Vulkan RTX, inter-shader calls use RayPayload to pass information; each time the TraceRayKHR function is called, the id of the RayPayload defined in the shader needs to be passed to indicate which RayPayload to pass information and return the result.

[0042] d. Copy a new non-Ray Generation type shader, call the newly copied non-Ray Generation type shader in the newly copied CustomTraceRay function, and record the newly copied RayPayload id.

[0043] e. Repeat S3 and S4 at the maximum recursion depth, and in the last loop, change CustomTraceRay in S3 to OpNop; OpNop is a set of instructions in the SPIR-V language, and it is an empty instruction.

[0044] f. Remove all OpNop instructions and copy and restore RayPayload data before and after the CustomTraceRay function according to the call chain.

[0045] S7, using the aforementioned Vulkan resources and merged compute shaders to create a pipeline layout to replace the user-created pipeline layout, using the aforementioned pipeline layout to establish a compute pipeline to replace the API that implements the functions related to establishing a ray tracing pipeline, and replacing the flags used by the user when establishing and binding the ray tracing pipeline with compute pipeline flags.

[0046] S8, when the user calls the vkCmdPushConstants function, the push constants provided by the user with the ray tracing shader flags are copied to the shader's storage buffer as described in S3. The vkDispatch function is used to implement the vkCmdTraceRayKHR function, storing the addresses of each SBT passed by the user into the newly created push constants for the shader to use.

[0047] S9 uses a runtime loading method to load the dynamic link library of the Vulkan core API and the function implementations that were replaced above, and encapsulates them into a new dynamic link library for direct use by applications.

[0048] Based on the above three stages of operation, the Vulkan ray tracing pipeline is transformed into a computational pipeline.

[0049] Using the above method for converting a Vulkan ray tracing pipeline into a computational pipeline, this invention implements a corresponding intermediate layer for the ray tracing pipeline to computational pipeline conversion, including a Vulkan API implementation module and a shader merging module:

[0050] The Vulkan API implementation module is a wrapper around S1, S2, and S7-S9 of this invention, and the shader merging module is a wrapper around S3-S6 of this invention. That is, the Vulkan API implementation module can handle the native core APIs in the Vulkan driver, increasing its ability to handle ray tracing related applications; it implements the RTX extended APIs not present in the Vulkan driver, converting ray tracing pipeline-related APIs into implementations using computation pipeline-related APIs.

[0051] The shader merging module combines user-provided ray tracing shaders into a single compute shader. It replaces the opacity types and instructions originally belonging to the ray tracing shader with custom-implemented types and functions. It also expands any indirect calls resulting from the replacement.

[0052] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0053] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A method for converting a Vulkan ray tracing pipeline into a computational pipeline, characterized in that, This process involves loading and hijacking the Vulkan API at runtime, replacing user-provided shaders with instructions and types related to ray tracing extensions, and implementing indirect calls between shaders through recursive unrolling. The process includes: Vulkan RTX resource preparation, shader merging, and ray tracing execution. (1) During the Vulkan RTX resource preparation phase, the following sub-steps are performed: (1.1) Replace the core APIs in the Vulkan driver, including replacing the APIs related to creating Vulkan instances and Vulkan devices, and pre-allocating storage buffers for the six push constants required for simulating the ray tracing pipeline for devices without Vulkan RTX functionality; replace the APIs related to obtaining device characteristics and attributes in the Vulkan driver, add corresponding device characteristics and attributes for devices without Vulkan RTX functionality, replace the APIs related to creating caches in the Vulkan driver, remove the flags related to the acceleration structure from the user-input parameters, and replace them with: VK_BUFFER_USAGE_STORAGE_BUFFER_BIT replaces the API related to creating and updating descriptors in the Vulkan driver, removing the flags related to the acceleration structure from the user-input parameters and replacing them with: VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER; Replaces the API related to creating shader modules in the Vulkan driver, removes the Vulkan RTX-related flags from user-input parameters and replaces them with: VK_PIPELINE_STAGE_COMPUTE_SHADER_BIT; (1.2) Implement Vulkan extension APIs related to the ray tracing pipeline, including using the surface area heuristic algorithm to build a BVH acceleration structure from bottom to top in the computation shader based on the scene data passed by the user. The BVH acceleration structure is divided into a bottom acceleration structure and a top acceleration structure. The leaf nodes of the bottom acceleration structure are the triangle input passed by the user, and the leaf nodes of the top acceleration structure are an instance of the bottom acceleration structure. The data passed by the two are different, but the algorithm is the same. Replace the push constant passed by the user with ordinary shader storage buffer resources, and create a new push constant to record the GPU memory address of the above shader buffer resources. (2) During the shader merging phase, the following sub-steps are executed: (2.1) Merge the shaders passed in by the user into an empty shader, record the symbol IDs for matching and deduplication, record the function ID corresponding to each shader passed in by the user and use it as the shader handle required to build the shader binding table; (2.2) Replace the instructions and opacity types related to ray tracing extension in the shader with custom functions and types, remove the OpCapacity and OpExtension instructions related to ray tracing, and use the custom function to call the user-input ray generation shader Ray Generation in the entry function; (2.3) Replace the instructions and opacity types related to ray query extension in the shader with custom functions and types, and remove the OpCapacity and OpExtension instructions related to ray query; (2.4) Expand the indirect recursive calls generated by the replacement in steps (2.2) and (2.3); copy the functions in the user-provided shader that generate the recursive call chain according to the maximum recursion depth specified by the user when establishing the ray tracing pipeline, copy the global variables used therein, and replace the global variables used in the call chain with the newly copied global variables; (3) During the ray tracing execution phase, the following sub-steps are executed: (3.1) Replace the pipeline layout created by the user with the above resources, and use the newly created pipeline layout to build a computational pipeline to realize the API related to the creation of ray tracing pipeline, and replace the flag bit used by the user when creating and binding the ray tracing pipeline with the computational pipeline flag bit. (3.2) When the user calls the vkCmdPushConstants function, the push constants provided by the user with the ray tracing shader flags are copied to the shader storage buffer described in step (1.2). The vkCmdTraceRayKHR function is implemented using the vkDispatch function, and the addresses of each SBT passed by the user are stored in the newly created push constants for the shader to use. The vkDispatch function and the vkCmdTraceRayKHR function are the names of two functions extended by Vulkan RTX. (3.3) The dynamic link library of the Vulkan core API and the functions that were replaced above are loaded using the runtime loading method.

2. The method for converting a Vulkan ray tracing pipeline into a computational pipeline according to claim 1, characterized in that, Use non-ray-tracing extended Vulkan resources instead of the Vulkan resources required for ray-tracing streamlines; merge user-provided ray-tracing shader code into compute shaders, and replace instructions and opacity types in ray-tracing extensions with custom functions and types; The problem of non-recursive function calls in computation shaders is solved by expanding the possible recursive calls between shaders in the ray tracing pipeline.

3. The method for converting a Vulkan ray tracing pipeline into a computational pipeline according to claim 1, characterized in that, The six buffers used in step (1.1) to simulate the push constant specifically include: RayGeneration Shader, Miss Shader, Closest Hit Shader, Any Hit Shader, Intersection Shader, and Callable Shader, with each shader using one buffer.

4. The method for converting a Vulkan ray tracing pipeline into a computational pipeline according to claim 1, characterized in that, The leaf node of the top-level acceleration structure in step (1.2) is an instance of the bottom-level acceleration structure. This instance is a reference to the bottom-level acceleration structure. By referencing, it is possible to avoid storing multiple copies of the same bottom-level acceleration structure for multiple identical objects in a scene. Only after referencing the same bottom-level acceleration structure, different transformation matrices can be applied to each object in the scene.

5. The method for converting a Vulkan ray tracing pipeline into a computational pipeline according to claim 1, characterized in that, In step (3.2), the vkDispatch function is used to call the computation pipeline, and the vkCmdTraceRayKHR function is used to call the ray tracing pipeline. That is, both are functions that call the ray tracing pipeline by using the function that calls the computation pipeline.

Images