A mobile terminal vulkan drawing process optimization method

Abstract

The application discloses a mobile terminal Vulkan drawing process optimization method, which is designed based on the rendering API features of Vulkan and the characteristics of a mobile terminal GPU. Through a series of methods, the number of rendering channel activation is reduced, and unnecessary frame buffers are discarded as much as possible, so that the read-write overhead of frame buffers when using the rendering channel is reduced. The purpose of the application is to reduce the data bandwidth required when the mobile terminal draws, and a series of methods for encapsulating the Vulkan graphics API are designed to optimize the performance of the mobile terminal without increasing the complexity of the upper layer call.

Description

Technical Field

[0001] This invention belongs to the field of rendering in computer graphics, and particularly relates to an optimization method for the Vulkan rendering process on mobile devices. Background Technology

[0002] The IMR (Immediate Mode Rendering) workflow on PCs requires reading and writing to the frame buffer and depth buffer for each triangle during rendering. This architecture demands significant bandwidth or a large cache. High bandwidth leads to high power consumption and demanding heat dissipation requirements. Furthermore, mobile devices, limited by size constraints, cannot accommodate large caches, making IMR unsuitable for mobile applications. Therefore, TBR (Tile-Based Rendering) was developed for mobile devices to reduce the external memory accesses required by the GPU during shading. TBR divides the screen into small tiles, such as 16x16 or 32x32 pixels. Primitives are assigned to each tile for computation, each with its own high-speed cache. All calculations are performed locally before the tile's frame buffer is written back to main memory. Once all tiles have completed their work, the final frame buffer is obtained. Therefore, it's clear that implementing mobile rendering requires minimizing read and write operations between the GPU and external memory throughout the rendering process.

[0003] Frame Graph is a new rendering framework that uses a technique to perform optimization operations by obtaining all the information in a complete frame and analyzing the dependencies between nodes in each rendering pass. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides an optimization method for Vulkan rendering on mobile devices. This method reduces the number of render passes within a single frame, automatically merging compatible render passes into a single one. For the same render target, it reduces the number of operations where it is written back to video memory and then read back into the tile by the render pass. Then, it automatically determines whether the texture corresponding to each render target in the render pass needs to save calculation results and whether it needs to read previously existing results; textures that do not need to be accessed are automatically set to not be accessed, thus saving bandwidth. Finally, for render targets that require MSAA, it automatically determines whether its texture in the Multiple Sample state is needed as a render target; otherwise, it specifies that the texture should be resolved to Single Sample at the end of the render pass.

[0005] This eliminates the multiple storage overhead caused by MSAA and optimizes the low performance issues caused by the overhead of reading and writing frame buffers in the Vulkan rendering process on mobile devices.

[0006] The main technical solution of this invention is as follows:

[0007] A method for optimizing the Vulkan rendering process on mobile devices encapsulates the VkRenderPass calls provided by Vulkan and optimizes the operations of reading and writing the frame buffer within the encapsulated function. This method includes the following steps:

[0008] (1) Encapsulate the three function interfaces: BeginRenderPass (start rendering channel function), Draw (drawing function), and EndRenderPass (end rendering channel function);

[0009] The BeginRenderPass function obtains the Vulkan rendering pass VkRenderPass and frame buffer object VkFrameBuffer by passing in a set of rendering targets RenderTarget and a bound depth map, and saves them. It then checks whether the objects saved when BeginRenderPass was last called are compatible with the rendering pass and frame buffer obtained in the current call.

[0010] If incompatible, the corresponding vkCmdEndRenderPass function provided by Vulkan is called to terminate the previously enabled rendering pass and save the new object for actual use.

[0011] If compatible, there is no need to terminate the already enabled rendering channels; the rendering channels will be automatically merged.

[0012] After the BeginRenderPass function obtains and saves the rendering channel and frame buffer, the Draw function is used to determine whether it is necessary to call the Vulkan API to start a new rendering channel and begin; the EndRenderPass function interface is used to check the semantic clarity when the upper layer calls this set of functions, and does not execute the Vulkan-provided function to terminate the rendering channel.

[0013] (2) The premise for calling the Start Rendering Channel function to perform compatibility judgment and thus form a merge in step (1) is that, before calling the Start Rendering Channel function, the homogeneous rendering channel object enabled by the previous drawing is not terminated due to other circumstances; specifically:

[0014] If a group of rendering passes writes the same framebuffer object, they can be merged. If each rendering pass requires different resources when drawing, those resources need to be updated before the rendering pass begins.

[0015] Therefore, in a set of homogeneous rendering channels, in order to prevent the update operation of the later-executed rendering channel from interrupting the already enabled homogeneous rendering channel, the update operation corresponding to the resource required by the later-executed rendering channel should be brought forward to the front of this set of rendering channels. This goal needs to be achieved using Frame Graph. (3) When determining the dependency relationship between rendering channels by resource connection in step (2), for resources that do not have a dependency, redundant write-back operations can be prevented by discarding the resources at the end of the rendering channel. Specifically, each rendering channel is abstracted into a rendering node using Frame Graph, and the rendering channels merged in step (1) are regarded as the same rendering node. For each rendering node, Frame Graph is used to determine whether each rendering target in its bound frame buffer is dependent on subsequent nodes or depends on previous nodes. For rendering targets that do not have a corresponding dependency relationship, redundant read and write operations are avoided by setting the corresponding flag bit when creating the rendering channel.

[0016] (4) For rendering channels with multisampling anti-aliasing (MSAA) enabled, it is necessary to parse the target and convert it into a normal texture without multisampling to produce the final result. However, writing the multisampling target back to video memory and then performing the parsing operation at the end of the rendering channel is too costly. Therefore, special processing is required when determining the rendering target in step (3) to avoid redundant read and write operations. Specifically, when creating the rendering channel, the corresponding parsed rendering target is specified for the multisampling target. That is, an additional rendering target is bound to the rendering channel so that the parsing operation can be completed in place at the end of the rendering channel, thereby discarding the multisampling target and only writing the parsed target back to video memory.

[0017] Further, step (1) includes the following sub-steps:

[0018] (1.1) Maintain the cache of rendering channels and frame buffers. Each time the function to start rendering channels is called, the corresponding rendering channels and frame buffers are searched in the cache according to the rendering target and depth map in the input parameters. If the search fails, new rendering channels and frame buffers are generated and saved in their respective caches.

[0019] (1.2) In the function that starts rendering channels, the compatibility of rendering channels is checked. At this time, the keywords for finding and creating rendering channels and framebuffers are only related to the parameters of the rendering target and the depth template, and are not related to the operation of the rendering channels themselves.

[0020] (1.3) In the drawing function, determine whether it is necessary to call the Vulkan API to start a new rendering pass and then start it;

[0021] (1.4) In the function to end the rendering channel, ensure the integrity of the rendering hardware interface semantics and ensure that each start call has a corresponding end call.

[0022] Furthermore, step (2) includes the following sub-steps:

[0023] (2.1) Utilizing the Frame Graph architecture design, most of the resource update operations required for rendering pipelines are placed in the Frame.

[0024] The graph is in the construction phase, while the actual drawing process occurs in the execution phase after construction.

[0025] (2.2) In the case where the result of a certain rendering node is dependent on subsequent rendering nodes, in order to avoid the interruption of merging homogeneous rendering channels caused by the state switching of the node occurring between subsequent homogeneous rendering channels, the state switching required for the resource is recorded in advance through the analysis of the Frame Graph. The execution time of the state switching operation is placed after the actual execution of Vulkan's termination rendering channel function, that is, between different rendering nodes, thereby preventing the result from being used as a resource to perform state switching between homogeneous rendering channels that should be merged.

[0026] The beneficial effects of this invention are as follows:

[0027] This invention provides a method for optimizing the Vulkan rendering process on mobile devices. By adding judgment logic to the Frame Graph layer to sort the rendering channel nodes, and determining the read and write operations of each color and depth accessory in each rendering channel node based on the sorted results, and performing special judgment on texture objects with MSAA enabled, the overhead of frame buffer read and write is saved when rendering with Vulkan on mobile devices. This reduces the performance bottleneck caused by bandwidth limitations on mobile devices to a certain extent and improves the overall performance.

[0028] As can be seen from the above embodiments, it should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and do not limit this application. Attached Figure Description

[0029] Figure 1 This is the underlying implementation of the merged Render Pass logic;

[0030] Figure 2 The flowchart for determining the Load Store Flag in the Frame Graph;

[0031] Figure 3 Example graph after calculating the Load Store Flag for the Frame Graph. Detailed Implementation

[0032] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0033] The purpose of this application is to provide a method to optimize the low performance problem caused by the overhead of reading and writing Frame Buffer in the Vulkan rendering process on mobile devices.

[0034] Step 1: First, encapsulate the function interfaces for the BeginRenderPass, Draw, and EndRenderPass. The BeginRenderPass function obtains VkRenderPass and VkFrameBuffer objects by passing in a set of rendering targets and bound depth maps.

[0035] like Figure 1 As shown, at this point, it checks whether the object set when BeginRenderPass was last called is compatible with the currently obtained VkRenderPass and VkFrameBuffer. If they are not compatible, vkCmdEndRenderPass is called to terminate the previously enabled VkRenderPass, and the new object is saved for actual use. If they are compatible, there is no need to terminate the already enabled VkRenderPass, thereby reducing the calls to vkCmdBeginRenderPass and vkCmdEndRenderPass, achieving the effect of automatically merging VkRenderPass.

[0036] (a) Maintain the cache of VkFrameBuffer and VkRenderPass. Each time the BeginRenderPass function is called, search for the corresponding VkFrameBuffer and VkRenderPass in the cache according to the Render Targets in the input parameter. If the search fails, generate new VkFrameBuffer and VkRenderPass and save them in their respective caches.

[0037] (b) The VkRenderPass obtained from the BeginRenderPass function will not start immediately, but will be called before the vkCmdDraw function is called within the encapsulated Draw function;

[0038] (c) The EndRenderPass function does not call vkCmdEndRenderPass to terminate the rendering pass. In order to merge the rendering passes as much as possible, a marker is added here. The actual vkCmdEndRenderPass call will occur when the BeginRenderPass function detects the rendering target switch.

[0039] (d) The keywords for finding and creating VkFrameBuffer and VkRenderPass are only related to the parameters of Render Targets, and are not related to the operations of Render Pass itself (such as whether to read or clear Render Targets). This ensures the judgment of VkRenderPass compatibility.

[0040] Step 2: In addition to the automatic merging logic in Step 1, some API calls, because they need to be made outside the scope of VkRenderPass, may interrupt VkRenderPasses that should be merged. There are two main scenarios:

[0041] (a) Some operations update the data or state of resources required for rendering (such as vkCmdCopyBuffer, vkCmdCopyImage, VkImageLayout conversion, etc.). These operations need to be performed as early as possible before all VkRenderPasses begin, while minimizing unnecessary state updates to resources. This can be achieved through FrameGraph. In FrameGraph, the dependencies between resources such as Buffer and Image and Render Pass nodes are determined. For resources that do not depend on all Render Passes in the current frame, the update operations are performed at the beginning of the frame drawing stage. For resources with dependencies (such as the resource result output by a certain node being used as the resource input by a subsequent node), the position where the VkRenderPass will be switched between the two nodes is searched and the operation is inserted.

[0042] (b) Since the operation of Compute Pass will inevitably interrupt VkRenderPass, the Frame Graph needs to merge as many Render Pass nodes as possible based on the dependency relationship between Compute Pass and Render Pass. (For example, if Render Pass C depends on both Compute Pass A and Render Pass B, but Compute Pass A and Render Pass B are independent of each other, the Frame Graph can place the execution of Compute Pass A before Render Pass B.)

[0043] The third step involves determining in the Frame Graph whether each rendering result of each Render node is dependent on subsequent Render or Compute nodes. It's important to consider that since the actual merging operation of the Render nodes occurs at a lower level and is invisible to the Frame Graph, the Frame Graph itself needs to determine whether adjacent Render nodes will be merged. This is done by checking if the output results of two nodes are identical (i.e., the rendering targets are the same). If a merge is possible, the merged whole is considered as a single node. This requires checking each output resource associated with each Render node (i.e., each Attachment in the Frame Buffer). The determination logic is as follows: Figure 2 As shown, the specific process is as follows:

[0044] (a) For each Render node, for each Attachment of that Render node, iterate through and determine whether subsequent nodes (excluding other Render nodes that will merge with the current Render node) depend on its result. If no other subsequent Render nodes need the current Attachment, the rendering result can be discarded after the current Render node ends without being written back to the video memory. This is done by specifying VkAttachmentStoreOp as VK_ATTACHMENT_STORE_OP_DONT_CARE in VkAttachmentDescription. Otherwise, set it to VK_ATTACHMENT_STORE_OP_STORE.

[0045] (b) Similarly, each Render node also needs to determine whether any previous Render nodes (excluding other Render nodes that will merge with the current Render node) have written to each bound Attachment. If a previous Render node has written to and saved the current Attachment, then VkAttachmentLoadOp needs to be set to VK_ATTACHMENT_LOAD_OP_LOAD in VkAttachmentDescription; otherwise, it needs to be set to VK_ATTACHMENT_LOAD_OP_DONT_CARE.

[0046] Fourth, for texture resources with MSAA enabled, manage them and their corresponding resolved textures as the same object. In the third step, when using the Frame Graph to determine the rendering target of a Render node, it distinguishes whether Multisampling Anti-aliasing (MSAA) is enabled for that rendering target object. For objects with MSAA enabled, by default, when used as input resources for a node, the resolved result is required; when used as output results for a node, it is required as a Multiple Sample object. Based on this logic, if no subsequent node uses the current MSAA rendering target again as a rendering target, but only as an input resource needed by subsequent nodes, the Multiple Sample object can be discarded, thus greatly reducing storage overhead. See details... Figure 3 The diagram shows a simple example of the effect achieved through this logical calculation.

[0047] (a) When using Frame Graph to determine the texture of MSAA in the third step, a special judgment is made when determining whether to write back to video memory. If there are no nodes to write to again, the VkAttachmentStoreOp of the MSAA object can be set to VK_ATTACHMENT_STORE_OP_DONT_CARE.

[0048] (b) The corresponding read and write operations for the Resolve texture are fixed as VK_ATTACHMENT_LOAD_OP_DONT_CARE and VK_ATTACHMENT_STORE_OP_STORE.

[0049] (c) For MSAA's Depth Stencil Attachment, the in-situ Resolve operation can also be performed using the extended structure VkSubpassDescriptionDepthStencilResolve named VK_KHR_depth_stencil_resolve.

[0050] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0051] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A method for optimizing the Vulkan rendering process on mobile devices, characterized in that, The VkRenderPass calls provided by Vulkan are encapsulated, and the operations for reading and writing frame buffers are optimized within the encapsulated function body. This method includes the following steps: (1) Encapsulate three functions: BeginRenderPass (start rendering channel function), Draw (drawing function), and EndRenderPass (end rendering channel function) for upper-level calls. Within the encapsulated function body, implement calls to Vulkan-related APIs. The BeginRenderPass function obtains the Vulkan rendering pass VkRenderPass and frame buffer object VkFrameBuffer by passing in a set of rendering targets and a bound depth map, and saves them. It also checks whether the objects saved when BeginRenderPass was last called are compatible with the rendering pass and frame buffer obtained in the current call. If incompatible, the corresponding vkCmdEndRenderPass function provided by Vulkan is called to terminate the previously enabled rendering pass and save the new object for actual use. If compatible, there is no need to terminate the already enabled rendering channels; the rendering channels will be automatically merged. After the BeginRenderPass function obtains and saves the rendering channel and frame buffer, the Draw function is used to determine whether it is necessary to call the Vulkan API to start a new rendering channel and begin; the EndRenderPass function interface is used to check the semantic clarity when the upper layer calls this set of functions, and does not execute the Vulkan-provided function to terminate the rendering channel. (2) The premise for calling the Start Rendering Channel function to perform compatibility judgment and thus form a merge in step (1) is that, before calling the Start Rendering Channel function, the homogeneous rendering channel object enabled by the previous drawing is not terminated due to other circumstances; specifically: If a group of rendering passes writes the same framebuffer object, they can be merged. If each rendering pass requires different resources when drawing, those resources need to be updated before the rendering pass begins. Therefore, in a set of homogeneous rendering pipelines, in order to prevent the update operations of the later-executed rendering pipeline from interrupting the already enabled homogeneous rendering pipelines, the update operations corresponding to the resources required by the later-executed rendering pipelines should be moved to before this set of rendering pipelines. This goal needs to be achieved using Frame Graph. (3) When determining the resource connection dependency between rendering channels in step (2), for resources that do not have a dependency, redundant write-back operations can be prevented by discarding the resources at the end of the rendering channel; specifically, each rendering channel is abstracted into a rendering node using Frame Graph, and the rendering channels merged in step (1) are regarded as the same rendering node; for each rendering node, Frame Graph is used to determine whether each rendering target in its bound frame buffer is dependent on subsequent nodes or depends on previous nodes; for rendering targets that do not have a corresponding dependency, redundant read and write operations are avoided by setting the corresponding flag bit when creating the rendering channel; (4) For rendering channels with multisampling anti-aliasing (MSAA) enabled, it is necessary to parse the target and convert it into a normal texture without multisampling to produce the final result. However, writing the multisampling target back to video memory and then performing the parsing operation at the end of the rendering channel is too costly. Therefore, special processing is required when determining the rendering target in step (3) to avoid redundant read and write operations. Specifically, when creating the rendering channel, the corresponding parsed rendering target is specified for the multisampling target. That is, an additional rendering target is bound to the rendering channel so that the parsing operation can be completed in place at the end of the rendering channel, thereby discarding the multisampling target and only writing the parsed target back to video memory.

2. The method for optimizing the Vulkan rendering process on a mobile device according to claim 1, characterized in that, Step (1) includes the following sub-steps: (1.1) Maintain the cache of rendering channels and frame buffers. Each time the function to start rendering channels is called, the corresponding rendering channels and frame buffers are searched in the cache according to the rendering target and depth map in the input parameters. If the search fails, new rendering channels and frame buffers are generated and saved in their respective caches. (1.2) In the function that starts rendering channels, the compatibility of rendering channels is checked. At this time, the keywords for finding and creating rendering channels and framebuffers are only related to the parameters of the rendering target and the depth template, and are not related to the operation of the rendering channels themselves. (1.3) In the drawing function, determine whether it is necessary to call the Vulkan API to start a new rendering pass and then start it; (1.4) In the function to end the rendering channel, ensure the integrity of the rendering hardware interface semantics and ensure that each start call has a corresponding end call.

3. The method for optimizing the Vulkan rendering process on a mobile device according to claim 1, characterized in that, Step (2) includes the following sub-steps: (2.1) Utilizing the architecture design of Frame Graph, most of the resource update operations required for rendering channels are placed in the Frame Graph construction phase, while the actual drawing process occurs in the execution phase after construction. (2.2) In the case where the result of a certain rendering node is dependent on subsequent rendering nodes, in order to avoid the interruption of merging homogeneous rendering channels caused by the state switching of the node occurring between subsequent homogeneous rendering channels, the state switching required for the resource is recorded in advance through the analysis of the Frame Graph. The execution time of the state switching operation is placed after the actual execution of Vulkan's termination rendering channel function, that is, between different rendering nodes, thereby preventing the result from being used as a resource to perform state switching between homogeneous rendering channels that should be merged.

Images