Image rendering method and device, electronic equipment and storage medium
By managing shader module resources through a tree structure and generating a descriptor set layout, the problem of shader mismatch with pipeline in the prior art is solved, which improves image rendering efficiency and simplifies management, and supports high-frequency shader switching.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- XIAMEN YAJI SOFTWARE
- Filing Date
- 2022-09-15
- Publication Date
- 2026-05-19
AI Technical Summary
In the existing technology, when the shader system manages resources individually through header file references, it is easy to introduce human error that leads to mismatch between the shader and the pipeline, causing image rendering problems. In addition, there is a lack of a systematic resource numbering and grouping management scheme.
The tree structure is used to organize the reference relationships between shader modules. By using a preset descriptor set layout, descriptor sets are created and bound to the current rendering pipeline in order of frequency from low to high, generating the shader pipeline layout. Resources are merged and overridden to generate the descriptor set layout.
It effectively avoids shader and pipeline mismatch caused by human error, improves image rendering efficiency, simplifies shader management, reduces development costs, and supports high-frequency shader switching.
Smart Images
Figure CN115311399B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and more specifically, to an image rendering method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology
[0002] Shaders are editable programs used to implement image rendering, replacing fixed rendering pipelines. Shader technology is widely used in game development; a visually realistic game often contains hundreds of shader effects.
[0003] When shaders use rendering resources, there is a need for grouped management. In typical shader systems, resources are usually managed individually through header file references. If users organize resources themselves, they need to manually number these resources, which can easily introduce human error, causing shader mismatches with the pipeline and resulting in image rendering problems. Summary of the Invention
[0004] The purpose of this application is to provide an image rendering method, apparatus, electronic device, and storage medium that can improve query efficiency. To achieve the above objective, the technical solutions provided by this application are as follows:
[0005] On one hand, embodiments of this application provide an image rendering method, the method comprising:
[0006] Receive image rendering instructions;
[0007] In response to the image rendering command, based on the preset descriptor set layout, corresponding descriptor sets are created sequentially in order of frequency from low to high, and bound to the current rendering pipeline to generate the current shader pipeline layout.
[0008] The descriptor set layout is a tree structure; each leaf node of the tree structure includes at least one shader; the descriptor set layout is generated based on the resources of each module of at least one shader.
[0009] The image is rendered based on the current shader pipeline layout.
[0010] In some possible implementations, it also includes:
[0011] By merging the resources of each shader module in multiple leaf nodes layer by layer in the direction from the leaf nodes to the root node of the tree structure, a preset descriptor set layout is generated.
[0012] In some possible implementations, a preset descriptor set layout is generated by merging the resources of each shader module in multiple leaf nodes layer by layer in the direction from the leaf nodes to the root node of the tree structure, specifically including:
[0013] Obtain the resources corresponding to each module of multiple shaders in multiple leaf nodes of a preset tree structure, and determine the update frequency of each resource in each leaf node.
[0014] Based on the merging frequency corresponding to each level and the update frequency of each resource in each leaf node, the resources of the shaders in multiple leaf nodes are merged level by level in the direction from the leaf nodes of the tree structure to the root node to obtain the resources of the root node.
[0015] The descriptor set layout is generated by covering the resources of the root node layer by layer, following the direction from the root node to the leaf node of the tree structure.
[0016] In some possible implementations, the process of merging the shader resources in multiple leaf nodes according to the merging frequency corresponding to each level and the update frequency of each resource in each leaf node, in a hierarchical manner from the leaf nodes of the tree structure to the root node, to obtain the resources of the root node, includes:
[0017] Leaf nodes are treated as child nodes, and the resources of the child nodes are merged into the corresponding parent nodes based on the merging frequency corresponding to the child nodes and the update frequency of the resources in the child nodes.
[0018] The parent node is treated as a new child node, and the steps of merging the resources of the child node into the corresponding parent node are repeated based on the merging frequency corresponding to the child node and the update frequency of the resources in the child node, until the resources of the root node are obtained.
[0019] In some possible implementations, the merging frequency is the update frequency of the parent node corresponding to the child node; the update frequency of each child node is greater than the update frequency of its corresponding parent node.
[0020] In some possible implementations, merging the resources of a child node into its corresponding parent node based on the merging frequency corresponding to the child node and the update frequency of the resources in the child node includes:
[0021] Identify a first resource in a child node whose update frequency is less than or equal to the merging frequency, and merge the first resource of at least one child node into the corresponding parent node;
[0022] Identify a second resource in the child node whose update frequency is greater than the merging frequency, determine the maximum quantity of the second resource, and update the maximum quantity to the parent node.
[0023] In some possible implementations, the descriptor set layout is obtained by covering the resources of the root node hierarchically from the root node to the leaf node of the tree structure, including:
[0024] By taking the root node as the parent node, the resources of the parent node are propagated to the corresponding child nodes, and the resources of the child nodes are overwritten.
[0025] The child node is used as the new parent node, and the steps of propagating the resources of the parent node to the corresponding child node and overwriting the resources of the child node are repeated until the resources of the leaf node are obtained, thus obtaining the layout of the descriptor set.
[0026] In some possible implementations, the step of propagating the resources of the parent node to the corresponding child nodes and overwriting the resources of the child nodes includes:
[0027] If the update frequency of the resources contained in the parent node is less than the update frequency of the corresponding child node, then the resources of the parent node are propagated to the corresponding child node, overwriting the resources of the child node.
[0028] In some possible implementations, generating the descriptor set layout by covering the resources of the root node hierarchically from the root node to the leaf node of the tree structure includes:
[0029] Following the direction from the root node to the leaf node of the tree structure, the updated resources of each node in the tree structure are obtained by covering the resources of the root node layer by layer.
[0030] The updated resources of each node are sorted based on a preset descriptor index to obtain the layout of the descriptor set.
[0031] In some possible implementations, the descriptor index includes at least one of the following: resource update frequency, parameter binding type, descriptor type, and shader stage.
[0032] On the other hand, embodiments of this application provide an image rendering apparatus, the apparatus comprising:
[0033] The receiving module is used to receive image rendering instructions;
[0034] The first generation module is used to respond to the image rendering instruction by creating corresponding descriptor sets in order of frequency from low to high based on a preset descriptor set layout, and binding them to the current rendering pipeline to generate the current shader pipeline layout.
[0035] The descriptor set layout is a tree structure; each leaf node of the tree structure includes at least one shader; the descriptor set layout is generated based on the resources of each module of at least one shader.
[0036] The rendering module is used to render the image based on the current shader pipeline layout.
[0037] In some possible implementations, a second generation module is also included, which generates the descriptor set layout by merging the resources of each shader module in the multiple leaf nodes in a hierarchical manner from the leaf nodes to the root node of the tree structure.
[0038] In some possible implementations, when the second generation module generates the descriptor set layout by merging the resources of each shader module in multiple leaf nodes hierarchically from the leaf nodes to the root node of the tree structure, it is specifically used for:
[0039] Obtain the resources corresponding to each module of multiple shaders in multiple leaf nodes of a preset tree structure, and determine the update frequency of each resource in each leaf node.
[0040] Based on the merging frequency corresponding to each level and the update frequency of each resource in each leaf node, the resources of the shaders in multiple leaf nodes are merged level by level in the direction from the leaf nodes of the tree structure to the root node to obtain the resources of the root node.
[0041] The descriptor set layout is generated by covering the resources of the root node layer by layer, following the direction from the root node to the leaf node of the tree structure.
[0042] In some possible implementations, when the second generation module merges the shader resources in multiple leaf nodes level by level according to the merging frequency corresponding to each level and the update frequency of each resource in each leaf node to obtain the resources of the root node, it is specifically used for:
[0043] Leaf nodes are treated as child nodes, and the resources of the child nodes are merged into the corresponding parent nodes based on the merging frequency corresponding to the child nodes and the update frequency of the resources in the child nodes.
[0044] The parent node is treated as a new child node, and the steps of merging the resources of the child node into the corresponding parent node are repeated based on the merging frequency corresponding to the child node and the update frequency of the resources in the child node, until the resources of the root node are obtained.
[0045] In some possible implementations, the merging frequency is the update frequency of the parent node corresponding to the child node; the update frequency of each child node is greater than the update frequency of its corresponding parent node.
[0046] In some possible implementations, when the second generation module merges the resources of the child node into the corresponding parent node based on the merging frequency corresponding to the child node and the update frequency of the resources in the child node, it is specifically used for:
[0047] Identify a first resource in a child node whose update frequency is less than or equal to the merging frequency, and merge the first resource of at least one child node into the corresponding parent node;
[0048] Identify a second resource in the child node whose update frequency is greater than the merging frequency, determine the maximum quantity of the second resource, and update the maximum quantity to the parent node.
[0049] In some possible implementations, when the second generation module obtains the descriptor set layout by covering the resources of the root node layer by layer in the direction from the root node to the leaf node of the tree structure, it is specifically used for:
[0050] By taking the root node as the parent node, the resources of the parent node are propagated to the corresponding child nodes, and the resources of the child nodes are overwritten.
[0051] The child node is used as the new parent node, and the steps of propagating the resources of the parent node to the corresponding child node and overwriting the resources of the child node are repeated until the resources of the leaf node are obtained, thus obtaining the layout of the descriptor set.
[0052] In some possible implementations, when the second generation module propagates the resources of the parent node to the corresponding child nodes and overwrites the resources of the child nodes, it is specifically used for:
[0053] If the update frequency of the resources contained in the parent node is less than the update frequency of the corresponding child node, then the resources of the parent node are propagated to the corresponding child node, overwriting the resources of the child node.
[0054] In some possible implementations, when the second generation module generates the descriptor set layout by covering the resources of the root node layer by layer in the direction from the root node to the leaf node of the tree structure, it is specifically used for:
[0055] Following the direction from the root node to the leaf node of the tree structure, the updated resources of each node in the tree structure are obtained by covering the resources of the root node layer by layer.
[0056] The updated resources of each node are sorted based on a preset descriptor index to obtain the layout of the descriptor set.
[0057] In some possible implementations, the descriptor index includes at least one of the following: resource update frequency, parameter binding type, descriptor type, and shader stage.
[0058] On the other hand, embodiments of this application also provide an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method provided in any optional embodiment of this application.
[0059] On the other hand, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in any optional embodiment of this application.
[0060] On the other hand, embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the methods provided in any optional embodiment of this application.
[0061] The beneficial effects of the technical solution provided in this application are as follows:
[0062] By pre-generating a tree-structured descriptor set layout, which is obtained by merging and laying out the resources of each shader module in multiple leaf nodes in a hierarchical manner from the leaf nodes to the root node of the tree structure, each leaf node of the tree structure includes at least one shader. Based on the created descriptor set layout, corresponding descriptor sets can be created sequentially in order of frequency from low to high and bound to the current rendering pipeline to generate the current shader pipeline layout. This effectively avoids shader and pipeline mismatch caused by human error and effectively improves image rendering efficiency.
[0063] Furthermore, by dividing the shader into multiple modules, users can write shaders according to the modules, making development easier and reducing management costs.
[0064] Furthermore, shaders within the same leaf node share the same pipeline layout. Therefore, when switching shaders, it's unnecessary to rebind the descriptor set, effectively reducing data state transitions and data uploads, enabling high-frequency shader switching. Resource layout is automatically completed during resource merging, reducing the user's development workload. Attached Figure Description
[0065] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.
[0066] Figure 1 This is a schematic diagram illustrating the application environment of the image rendering method provided in the embodiments of this application;
[0067] Figure 2 A flowchart illustrating an image rendering method provided in an embodiment of this application;
[0068] Figure 3 A flowchart illustrating a method for generating a descriptor set layout, provided in an embodiment of this application;
[0069] Figure 4 A schematic diagram of the resources for the shader of the leaf node in a tree structure provided in this application;
[0070] Figure 5 A schematic diagram illustrating a scheme for merging resources from leaf nodes to root nodes, provided as an example in this application;
[0071] Figure 6 A schematic diagram illustrating a scheme for merging resources from leaf nodes to root nodes, provided as an example in this application;
[0072] Figure 7 A schematic diagram illustrating a scheme for resource propagation from the root node to the leaf node, provided as an example in this application;
[0073] Figure 8 A schematic diagram illustrating a scheme for resource propagation from the root node to the leaf node, provided as an example in this application;
[0074] Figure 9 This is a schematic diagram of the structure of an image rendering apparatus provided in an embodiment of this application;
[0075] Figure 10 This is a schematic diagram of the structure of an electronic device to which this application applies. Detailed Implementation
[0076] The embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the embodiments described below with reference to the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions of the embodiments of this application.
[0077] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the terms “comprising” and “including” as used in embodiments of this application mean that the corresponding feature can be implemented as the presented feature, information, data, step, operation, element, and / or component, but do not exclude implementation as other features, information, data, step, operation, element, component, and / or combinations thereof supported by the art. It should be understood that when we say that an element is “connected” or “coupled” to another element, the one element can be directly connected or coupled to the other element, or it can mean that the one element and the other element establish a connection relationship through an intermediate element. Furthermore, “connected” or “coupled” as used herein can include wireless connection or wireless coupling. The term “and / or” as used herein indicates at least one of the items defined by the term; for example, “A and / or B” can be implemented as “A,” or as “B,” or as “A and B.” When describing multiple (two or more) items, if the relationship between the multiple items is not explicitly defined, the multiple items can refer to one, several or all of the multiple items. For example, the description of "parameter A includes A1, A2, A3" can be implemented as parameter A includes A1 or A2 or A3, or it can be implemented as parameter A includes at least two of the three items A1, A2 and A3.
[0078] The advent of programmable shader pipelines has revolutionized modern graphics rendering technology. Shader technology is widely used in game development; a visually stunning game can easily contain hundreds of shader effects. Traditional shader systems typically manage shaders individually through header file references. This individual management approach has several problems: when shaders use rendering resources, there's a need for grouping them, such as by update frequency, resource type, or usage method. If users organize resources themselves, they need to manually number these resources, a process highly susceptible to human error, leading to shader-pipeline mismatches and incorrect resource usage. Fixing these issues consumes significant development manpower and resources. Currently, there is no systematic solution for managing resource numbering and grouping.
[0079] This application introduces a tree-like structure to organize the reference relationships between shader modules based on their update frequency and other characteristics. This structured relationship simplifies management, improves shader update efficiency, and makes user customization easier.
[0080] The following description of several optional embodiments illustrates the technical solutions provided in this application and the technical effects produced by these solutions. It should be noted that the following embodiments can be referenced, borrowed from, or combined with each other. Identical terms, similar features, and similar implementation steps in different embodiments will not be repeated.
[0081] Figure 1 This is a schematic diagram illustrating the application environment of the image rendering method provided in this embodiment. The application environment may include a rendering terminal 10, which includes a shader management system 101 and a rendering module 102. Specifically, the shader management system 101 acquires resources corresponding to each module of multiple shaders and determines the update frequency of each resource in each leaf node, generating a descriptor set layout. The shader management system 101 sends the generated tree-structured descriptor set layout to the rendering module 102. The rendering module 102 receives an image rendering instruction; in response to the image rendering instruction, based on the preset descriptor set layout, it sequentially creates corresponding descriptor sets in ascending order of frequency and binds them to the current rendering pipeline, generating the current shader pipeline layout, and renders the image based on the current shader pipeline layout.
[0082] In the above scenario, the shader management system pre-generates the descriptor set layout. In other application scenarios, the server may pre-generate the descriptor set layout and then send it to the rendering terminal. Those skilled in the art will know that the above scenario is only an example and does not limit the application scenarios of the image rendering method of this application.
[0083] Those skilled in the art will understand that a server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server or server cluster that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.
[0084] The terminal (also referred to as a user terminal or user equipment) can be a smartphone, tablet, laptop, desktop computer, intelligent voice interaction device (such as a smart speaker), wearable electronic device (such as a smartwatch), in-vehicle terminal, smart home appliance (such as a smart TV), AR / VR device, etc., but is not limited thereto. The terminal and the server can be connected directly or indirectly through wired or wireless communication, which is not limited herein.
[0085] Figure 2The illustration shows a flowchart of an image rendering method provided in an embodiment of this application, which can be executed by a rendering terminal.
[0086] like Figure 2 As shown, taking the rendering terminal as the execution subject as an example, the image rendering method provided in this application may include the following steps:
[0087] Step S201: Receive image rendering instructions;
[0088] Step S202: In response to the image rendering instruction, based on the preset descriptor set layout, corresponding descriptor sets are created sequentially in order of frequency from low to high, and bound to the current rendering pipeline to generate the current shader pipeline layout.
[0089] Step S203: Render the image based on the current shader pipeline layout.
[0090] Here, a descriptor is a special opaque shader variable that the shader uses to access buffers and image resources indirectly; a descriptor set is called a "set" because it can reference a group of isomorphic resources that can be described using the same layout binding; the descriptor set layout is used to describe the contents of the descriptor list.
[0091] The descriptor set layout is a tree structure; each leaf node of the tree structure includes at least one shader; the descriptor set layout is generated based on the resources of each module of at least one shader.
[0092] Specifically, the descriptor set layout is generated by merging and overlaying the resources of each shader module in multiple leaf nodes in a hierarchical manner from the leaf nodes to the root node of the tree structure.
[0093] The tree structure of the descriptor set layout is divided into multiple levels based on the update frequency. Each node in each level has its own update frequency. Nodes in the same level have the same update frequency. The lower the update frequency of the top-level node, the higher the update frequency of the bottom-level node. The specific generation process of the descriptor combination layout will be further explained in detail below.
[0094] In practical implementation, taking the lighting pass in the rendering process as an example, the resource layout for the lighting stage (per-rendering pass) can be obtained from the descriptor set layout. Based on this resource layout, a corresponding descriptor set layout (e.g., Vk Descriptor Set Layout) is created. Afterward, data is uploaded and resources are located according to the layout requirements. Finally, the corresponding descriptor set (e.g., Vk Descriptor Set) is created and populated, and bound to the current rendering pipeline. After binding the low-frequency descriptor sets, this process is repeated, binding higher-frequency descriptor sets according to the rendering order, ultimately resulting in the current shader pipeline layout.
[0095] In the above embodiments, a tree-structured descriptor set layout is pre-generated. The descriptor set layout is obtained by merging and laying out the resources of each shader module in multiple leaf nodes in a hierarchical manner from the leaf node to the root node of the tree structure. Each leaf node of the tree structure includes at least one shader. Based on the created descriptor set layout, corresponding descriptor sets can be created sequentially in order of frequency from low to high and bound to the current rendering pipeline to generate the current shader pipeline layout. This effectively avoids shader and pipeline mismatch caused by human error and effectively improves image rendering efficiency.
[0096] The following will further elaborate on the process of generating the descriptor set layout with reference to the embodiments.
[0097] In some possible implementations, such as Figure 3 As shown, the method further includes:
[0098] The preset descriptor set layout is generated by merging the resources of each module of the shader in multiple leaf nodes in a hierarchical manner from the leaf nodes to the root node of the tree structure.
[0099] Specifically, by merging the resources of each shader module in multiple leaf nodes layer by layer according to the direction from the leaf nodes to the root node of the tree structure, and then overwriting to generate a preset descriptor set layout, it can include:
[0100] Step S301: Obtain the resources corresponding to each module of the multiple shaders in the multiple leaf nodes of the preset tree structure, and determine the update frequency of each resource in each leaf node.
[0101] Specifically, multiple shaders can be obtained, and for each shader, it can be divided into multiple chunks, which are stored in text format on the file system. Users will reference these chunks when writing shaders.
[0102] Specifically, the resources used within a module are grouped according to their update frequency and are labeled and maintained by the user.
[0103] For example, if a user declares resources A (uniform block A), B (sampler2DB), and C (image3D C) in a module, the user needs to specify the update frequency for resources A, B, and C. For instance, resource A might be updated per-pass, resource B per-material, and resource C per-instance. The shader system records the frequency information for resources A, B, and C and stores it in an internal tree structure.
[0104] Step S302: Based on the merging frequency corresponding to each level and the update frequency of each resource in each leaf node, the resources of the shaders in multiple leaf nodes are merged level by level in the direction from the leaf nodes of the tree structure to the root node to obtain the resources of the root node.
[0105] Specifically, starting from the leaf nodes, for each node, resources are recursively merged upwards to the parent node, with the merging frequency being the update frequency of the parent node.
[0106] It is important to note that the tree structure is pre-set, meaning that the node relationship between each node and its corresponding parent node is already set. In the initial state of the tree structure, the content of each internal node is empty, each leaf node contains the resources of the corresponding shader, and each node is set with its own update frequency.
[0107] In the specific implementation process, step S302, based on the merging frequency corresponding to each level and the update frequency of each resource in each leaf node, merges the shader resources in multiple leaf nodes level by level in the direction from the leaf nodes of the tree structure to the root node to obtain the resources of the root node, which may include:
[0108] (1) Take the leaf node as a child node, and merge the resources of the child node into the corresponding parent node based on the merging frequency corresponding to the child node and the update frequency of the resources in the child node.
[0109] In practice, the merging frequency is the update frequency of the parent node corresponding to the child node; the update frequency of each child node is greater than the update frequency of its corresponding parent node.
[0110] In other words, the update frequency decreases sequentially from the leaf nodes upwards.
[0111] For example, when merging uniform block A (per-pass) in shader A with uniform block C (per-pass) in shader B, if the merging frequency of the parent node is per-pass, then the resource merged into the parent node is uniform block [A, B], which means it contains two uniform blocks.
[0112] Specifically, merging the resources of the child node into the corresponding parent node based on the merging frequency corresponding to the child node and the update frequency of the resources in the child node may include:
[0113] a. Determine the first resource in the child node whose update frequency is less than or equal to the merging frequency, and merge the first resource of at least one child node into the corresponding parent node;
[0114] b. Determine the second resource in the child node whose update frequency is greater than the merging frequency, determine the maximum quantity of the second resource, and update the maximum quantity to the parent node.
[0115] In other words, resources with a merging frequency of less than or equal to that of merging are considered public resources and merged together; resources with a merging frequency higher than that of merging are considered private resources, and the maximum number of such resources is calculated as the resource budget.
[0116] For example, the user uses uniform block A (per-pass) and sampler2D B (per-material) in shader (A), and uniform block C (per-pass) and sampler2D D (per-material) in shader (B). When merging is performed at per-pass, uniform block A and uniform block C will be merged according to step a to form uniform block [A,B], which contains two uniform blocks. Since B and D have a higher update frequency, the maximum number will be reserved according to step b, which is 1 sampler2D in this case, forming sampler2D[1].
[0117] (2) Take the parent node as the new child node, and repeatedly execute the step of merging the resources of the child node into the corresponding parent node based on the merging frequency corresponding to the child node and the update frequency of the resources in the child node, until the resources of the root node are obtained.
[0118] Specifically, the process involves repeatedly updating the resources recursively upwards until the resources of the root node are obtained.
[0119] For example, the node between the leaf node and the root node includes the parent node of the leaf node, which is the first node; and the node including the parent node of the first node, which is the second node. Resources in the leaf nodes are then merged into the corresponding first node; resources in the first node are then merged into the corresponding second node; and finally, resources in the second node are merged into the root node. Each merge follows the principle described above: resources with a merging frequency lower than or equal to the merging frequency are considered public resources and merged together; resources with a merging frequency higher than the merging frequency are considered private resources, and the maximum number of such resources is calculated as the resource budget.
[0120] Step S303: Following the direction from the root node to the leaf node of the tree structure, the descriptor set layout is generated by covering the resources of the root node layer by layer.
[0121] Specifically, the resources merged from the parent node are propagated downwards and overwrite the resources of the child nodes.
[0122] Specifically, overriding the resources of a child node can be achieved by replacing the resources in the child node with the resources in the parent node.
[0123] In the specific implementation process, step S303, following the direction from the root node to the leaf node of the tree structure, covers the resources of the root node layer by layer to obtain the descriptor set layout, which may include:
[0124] (1) Take the root node as the parent node, propagate the resources of the parent node to the corresponding child nodes, and overwrite the resources of the child nodes.
[0125] Specifically, the condition that can be covered is that the update frequency of the current resource is lower than the update frequency of the child nodes.
[0126] In other words, taking the root node as the parent node, propagating the resources of the parent node to the corresponding child nodes, and overwriting the resources of the child nodes, can include:
[0127] If the update frequency of the resources contained in the parent node is less than the update frequency of the corresponding child node, then the resources of the parent node are propagated to the corresponding child node, overwriting the resources of the child node.
[0128] Understandably, during this process, the resources merged by the parent node will be shared by all child nodes. This means that the layout of the parent node's resources is relatively fixed compared to the child nodes, and has a lower update frequency.
[0129] (2) Take the child node as the new parent node, and repeat the steps of propagating the resources of the parent node to the corresponding child node and overwriting the resources of the child node until the resources of the leaf node are obtained, thus obtaining the layout of the descriptor set.
[0130] Specifically, the process involves recursively updating from the root node to the leaf node until the resources of the leaf node are obtained.
[0131] For example, from the root node to the leaf node, including the parent node of the leaf node, which is the first node; and including the parent node of the first node, which is the second node; then the resources in the root node are overwritten to the corresponding second node; then the resources in the second node are overwritten to the corresponding first node; finally, the resources in the first node are overwritten to the leaf node. Each overwriting is based on the above principle: the overwriting condition is that the update frequency of the current resource is lower than the update frequency of the child node.
[0132] In some possible implementations, step S303, which generates the descriptor set layout by covering the resources of the root node hierarchically from the root node to the leaf node of the tree structure, may include:
[0133] (1) Following the direction from the root node to the leaf node of the tree structure, the updated resources of each node of the tree structure are obtained by covering the resources of the root node layer by layer.
[0134] (2) Sort the updated resources of each node based on the preset descriptor index to obtain the layout of the descriptor set.
[0135] The descriptor index includes at least one of the following: resource update frequency, parameter binding type, descriptor type, and shader stage.
[0136] In the implementation process, for the updated resources in each node, the resources are first grouped according to the descriptor index. The sorting rule between groups is the lexicographic order of the descriptor index tuples. For resources within a group, they can be sorted according to certain rules, such as the lexicographic order of the resource names. After sorting, the layout of all resources is completed, resulting in the descriptor set layout.
[0137] In the above embodiments, the shader is divided into multiple modules, which allows users to write shaders according to the modules, making development easier and reducing management costs.
[0138] Furthermore, shaders within the same leaf node share the same pipeline layout. Therefore, when switching shaders, it's unnecessary to rebind the descriptor set, effectively reducing data state transitions and data uploads, and enabling high-frequency shader switching. Resource layout is automatically completed during resource merging, further reducing the user's development workload.
[0139] To more clearly illustrate the image rendering method of this application, the following will further explain the image rendering method of this application with examples.
[0140] In one example, the image rendering method of this application may include the following steps;
[0141] 1) Obtain the resources corresponding to each module of multiple shaders in multiple leaf nodes of a preset tree structure, and determine the update frequency of each resource in each leaf node.
[0142] like Figure 4 As shown, the preset tree structure in the diagram includes leaf nodes "Shader 0", "Shader 1", and "Shader 2". The parent node of "Shader 0" and "Shader 1" is "Queue 0", the parent node of "Shader 2" is "Queue 1", and the parent node of "Queue 1" and "Queue 0" is "Pass 0". Figure 4 The "Shader 0", "Shader 1" and "Shader 2" are each divided into several different modules.
[0143] 2) Taking leaf nodes as child nodes, and based on the merging frequency corresponding to the child nodes and the update frequency of resources in the child nodes, merge the resources of the child nodes into the corresponding parent nodes; wherein, the specific merging method is as follows:
[0144] Identify a first resource in a child node whose update frequency is less than or equal to the merging frequency, and merge the first resource of at least one child node into the corresponding parent node;
[0145] Identify a second resource in the child node whose update frequency is greater than the merging frequency, determine the maximum quantity of the second resource, and update the maximum quantity to the parent node.
[0146] like Figure 5 As shown, the resources corresponding to each module in "Shader 0" and "Shader 1" are merged into "Queue 0", and the resources of each module in "Shader 2" are merged into "Queue 1".
[0147] 3) Take the parent node as the new child node, and repeatedly execute the step of merging the resources of the child node into the corresponding parent node based on the merging frequency corresponding to the child node and the update frequency of the resources in the child node, until the resources of the root node are obtained.
[0148] like Figure 6 As shown, the resources in "Queue 1" and "Queue 0" have been merged into "Pass 0".
[0149] 4) Using the root node as the parent node, propagate the resources of the parent node to the corresponding child nodes, and overwrite the resources of the child nodes;
[0150] The specific coverage process is as follows:
[0151] If the update frequency of the resources contained in the parent node is less than the update frequency of the corresponding child node, then the resources of the parent node are propagated to the corresponding child node, overwriting the resources of the child node.
[0152] like Figure 7 As shown, the resources in "Pass 0" are propagated to "Queue 1" and "Queue 0" respectively.
[0153] 5) Take the child node as the new parent node, and repeat the steps of propagating the resources of the parent node to the corresponding child node and overwriting the resources of the child node until the resources of the leaf node are obtained, thus obtaining the layout of the descriptor set.
[0154] like Figure 8 As shown, the resources in "Queue 0" are propagated to "Shader 0" and "Shader 1" respectively, and the resources in "Queue 1" are propagated to "Shader 2", resulting in the final descriptor set layout.
[0155] It's important to note that above a certain frequency, only the size is reserved during collection; the specific content is determined by each shader. For example... Figure 8 The resources for frequency 3 of "Shader 0" and "Shader 1" are different. The frequency 3 of "Shader 0" corresponds to (null), which means "empty".
[0156] 6) Receive image rendering instructions;
[0157] 7) In response to the image rendering instruction, based on the descriptor set layout constructed in the above process, the corresponding descriptor sets are created sequentially in order of frequency from low to high and bound to the current rendering pipeline to generate the current shader pipeline layout.
[0158] 8) Render the image based on the current shader pipeline layout.
[0159] The image rendering method described above pre-generates a tree-structured descriptor set layout. This layout is achieved by merging and laying out the resources of each shader module in multiple leaf nodes in a hierarchical manner from the leaf nodes to the root node of the tree structure. Each leaf node in the tree structure includes at least one shader. Based on the pre-created descriptor set layout, corresponding descriptor sets can be created sequentially in ascending order of frequency and bound to the current rendering pipeline to generate the current shader pipeline layout. This effectively avoids shader-pipeline mismatch caused by human error and significantly improves image rendering efficiency.
[0160] Furthermore, by dividing the shader into multiple modules, users can write shaders according to the modules, making development easier and reducing management costs.
[0161] Furthermore, shaders within the same leaf node share the same pipeline layout. Therefore, when switching shaders, it's unnecessary to rebind the descriptor set, effectively reducing data state transitions and data uploads, enabling high-frequency shader switching. Resource layout is automatically completed during resource merging, reducing the user's development workload.
[0162] like Figure 9 As shown, in some possible implementations, an image rendering apparatus is provided, comprising:
[0163] Receiver module 901 is used to receive image rendering instructions;
[0164] The first generation module 902 is used to respond to the image rendering instruction, based on a preset descriptor set layout, sequentially create corresponding descriptor sets in order of frequency from low to high, and bind them to the current rendering pipeline to generate the current shader pipeline layout.
[0165] The descriptor set layout is a tree structure; each leaf node of the tree structure includes at least one shader; the descriptor set layout is generated based on the resources of each module of at least one shader.
[0166] Rendering module 903 is used to render an image based on the current shader pipeline layout.
[0167] In some possible implementations, a second generation module is also included, for:
[0168] By merging the resources of each module of the shaders in multiple leaf nodes layer by layer in the direction from the leaf nodes to the root node of the tree structure, the descriptor set layout is generated.
[0169] In some possible implementations, when the second generation module generates the descriptor set layout by merging the resources of each shader module in multiple leaf nodes hierarchically from the leaf nodes to the root node in a tree structure, the specific steps include:
[0170] Obtain the resources corresponding to each module of multiple shaders in multiple leaf nodes of a preset tree structure, and determine the update frequency of each resource in each leaf node.
[0171] Based on the merging frequency corresponding to each level and the update frequency of each resource in each leaf node, the resources of the shaders in multiple leaf nodes are merged level by level in the direction from the leaf nodes of the tree structure to the root node to obtain the resources of the root node.
[0172] The descriptor set layout is generated by covering the resources of the root node layer by layer, following the direction from the root node to the leaf node of the tree structure.
[0173] In some possible implementations, when the second generation module merges the shader resources in multiple leaf nodes level by level according to the merging frequency corresponding to each level and the update frequency of each resource in each leaf node to obtain the resources of the root node, it is specifically used for:
[0174] Leaf nodes are treated as child nodes, and the resources of the child nodes are merged into the corresponding parent nodes based on the merging frequency corresponding to the child nodes and the update frequency of the resources in the child nodes.
[0175] The parent node is treated as a new child node, and the steps of merging the resources of the child node into the corresponding parent node are repeated based on the merging frequency corresponding to the child node and the update frequency of the resources in the child node, until the resources of the root node are obtained.
[0176] In some possible implementations, the merging frequency is the update frequency of the parent node corresponding to the child node; the update frequency of each child node is greater than the update frequency of its corresponding parent node.
[0177] In some possible implementations, when the second generation module merges the resources of the child node into the corresponding parent node based on the merging frequency corresponding to the child node and the update frequency of the resources in the child node, it is specifically used for:
[0178] Identify a first resource in a child node whose update frequency is less than or equal to the merging frequency, and merge the first resource of at least one child node into the corresponding parent node;
[0179] Identify a second resource in the child node whose update frequency is greater than the merging frequency, determine the maximum quantity of the second resource, and update the maximum quantity to the parent node.
[0180] In some possible implementations, when the second generation module obtains the descriptor set layout by covering the resources of the root node layer by layer in the direction from the root node to the leaf node of the tree structure, it is specifically used for:
[0181] By taking the root node as the parent node, the resources of the parent node are propagated to the corresponding child nodes, and the resources of the child nodes are overwritten.
[0182] The child node is used as the new parent node, and the steps of propagating the resources of the parent node to the corresponding child node and overwriting the resources of the child node are repeated until the resources of the leaf node are obtained, thus obtaining the layout of the descriptor set.
[0183] In some possible implementations, when the second generation module propagates the resources of the parent node to the corresponding child nodes and overwrites the resources of the child nodes, it is specifically used for:
[0184] If the update frequency of the resources contained in the parent node is less than the update frequency of the corresponding child node, then the resources of the parent node are propagated to the corresponding child node, overwriting the resources of the child node.
[0185] In some possible implementations, when the second generation module generates the descriptor set layout by covering the resources of the root node layer by layer in the direction from the root node to the leaf node of the tree structure, it is specifically used for:
[0186] Following the direction from the root node to the leaf node of the tree structure, the updated resources of each node in the tree structure are obtained by covering the resources of the root node layer by layer.
[0187] The updated resources of each node are sorted based on a preset descriptor index to obtain the layout of the descriptor set.
[0188] In some possible implementations, the descriptor index includes at least one of the following: resource update frequency, parameter binding type, descriptor type, and shader stage.
[0189] The image rendering apparatus of this application pre-generates a tree-structured descriptor set layout. The descriptor set layout is obtained by merging and laying out the resources of each shader module in multiple leaf nodes in a hierarchical manner from the leaf nodes to the root node of the tree structure. Each leaf node of the tree structure includes at least one shader. Based on the pre-created descriptor set layout, corresponding descriptor sets can be created sequentially in order of frequency from low to high and bound to the current rendering pipeline to generate the current shader pipeline layout. This effectively avoids shader-pipeline mismatch caused by human error and effectively improves image rendering efficiency.
[0190] Furthermore, by dividing the shader into multiple modules, users can write shaders according to the modules, making development easier and reducing management costs.
[0191] Furthermore, shaders within the same leaf node share the same pipeline layout. Therefore, when switching shaders, it's unnecessary to rebind the descriptor set, effectively reducing data state transitions and data uploads, enabling high-frequency shader switching. Resource layout is automatically completed during resource merging, reducing the user's development workload.
[0192] The apparatus in this application embodiment can execute the method provided in this application embodiment, and the implementation principle is similar. The actions performed by each module in the apparatus of each embodiment of this application correspond to the steps in the method of each embodiment of this application. For detailed functional descriptions of each module of the apparatus, please refer to the descriptions in the corresponding methods shown above, which will not be repeated here.
[0193] This application provides an electronic device, including a memory, a processor, and a computer program stored in the memory. When the processor executes the computer program stored in the memory, it can implement the method in any optional embodiment of this application.
[0194] Figure 10 A schematic diagram of the structure of an electronic device to which an embodiment of the present invention applies is shown, such as... Figure 10 As shown, the electronic device can be a server or a user terminal, and it can be used to implement the methods provided in any embodiment of the present invention.
[0195] like Figure 10 As shown, the electronic device 1000 may primarily include at least one processor 1001. Figure 10 The diagram shows components such as a memory 1002, a communication module 1003, and an input / output interface 1004. Optionally, these components can be connected and communicate with each other via a bus 1005. It should be noted that... Figure 10The structure of the electronic device 1000 shown is merely illustrative and does not constitute a limitation on the electronic devices to which the methods provided in the embodiments of this application are applicable.
[0196] The memory 1002 can be used to store operating systems and applications, etc. The applications can include computer programs that implement the methods shown in the embodiments of the present invention when invoked by the processor 1001, and can also include programs for implementing other functions or services. The memory 1002 can be ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices that can store information and computer programs, or it can be EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disk storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto.
[0197] Processor 1001 is connected to memory 1002 via bus 1005 and implements corresponding functions by calling application programs stored in memory 1002. Processor 1001 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 1001 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.
[0198] Electronic device 1000 can connect to a network via communication module 1003 (which may include, but is not limited to, components such as a network interface) to communicate with other devices (such as user terminals or servers) through the network and achieve data interaction, such as sending data to or receiving data from other devices. Communication module 1003 may include wired network interfaces and / or wireless network interfaces, meaning the communication module may include at least one of wired or wireless communication modules.
[0199] Electronic device 1000 can connect to required input / output devices, such as keyboards and display devices, via input / output interface 1004. Electronic device 1000 itself may have a display device, and other display devices can also be connected externally via interface 1004. Optionally, storage devices, such as hard drives, can also be connected via interface 1004 to store data from electronic device 1000, retrieve data from storage devices, or store data from storage devices into memory 1002. It is understood that input / output interface 1004 can be a wired interface or a wireless interface. Depending on the actual application scenario, the device connected to input / output interface 1004 can be a component of electronic device 1000 or an external device connected to electronic device 1000 when needed.
[0200] The bus 1005 used to connect the components may include a path for transmitting information between the components. The bus 1005 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Depending on its function, the bus 1005 may be divided into an address bus, a data bus, a control bus, etc.
[0201] Optionally, for the solution provided in the embodiments of the present invention, the memory 1002 can be used to store a computer program that executes the solution of the present invention, and the processor 1001 runs the computer program. When the processor 1001 runs the computer program, it implements the operation of the method or apparatus provided in the embodiments of the present invention.
[0202] Based on the same principle as the method provided in the embodiments of this application, the embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the corresponding content of the aforementioned method embodiments.
[0203] This application also provides a computer program product, which includes a computer program that, when executed by a processor, can implement the corresponding content of the aforementioned method embodiments.
[0204] It should be noted that the terms "first," "second," "third," "fourth," "1," "2," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown in the figures or text.
[0205] It should be understood that although arrows indicate various operation steps in the flowcharts of this application's embodiments, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of this application's embodiments, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all steps in each flowchart, based on the actual implementation scenario, may include multiple sub-steps or multiple stages. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured according to requirements, and this application's embodiments do not limit this.
[0206] The above description is only an optional implementation method for some implementation scenarios of this application. It should be noted that for those skilled in the art, other similar implementation methods based on the technical concept of this application without departing from the technical concept of this application also fall within the protection scope of the embodiments of this application.
Claims
1. An image rendering method, characterized in that, include: Receive image rendering instructions; In response to the image rendering command, based on the preset descriptor set layout, corresponding descriptor sets are created sequentially in order of frequency from low to high, and bound to the current rendering pipeline to generate the current shader pipeline layout. The descriptor set layout is a tree structure; each leaf node of the tree structure includes at least one shader; the descriptor set layout is generated based on the resources of each module of at least one shader. Render the image based on the current shader pipeline layout; The method further includes: after merging the resources of each module of the shader in multiple leaf nodes in a hierarchical manner from the leaf nodes to the root node of the tree structure, the preset descriptor set layout is generated. The process of merging and overwriting the resources of each shader module in multiple leaf nodes in a hierarchical manner from the leaf nodes to the root node of the tree structure to generate the preset descriptor set layout includes: Obtain the resources corresponding to each module of multiple shaders in multiple leaf nodes of a preset tree structure, and determine the update frequency of each resource in each leaf node. Based on the merging frequency corresponding to each level and the update frequency of each resource in each leaf node, the resources of the shaders in multiple leaf nodes are merged level by level in the direction from the leaf nodes of the tree structure to the root node to obtain the resources of the root node. The descriptor set layout is generated by covering the resources of the root node layer by layer, following the direction from the root node to the leaf node of the tree structure.
2. The image rendering method according to claim 1, characterized in that, Based on the merging frequency corresponding to each level and the update frequency of each resource in each leaf node, the resources of the shaders in multiple leaf nodes are merged level by level in the direction from the leaf nodes to the root node of the tree structure to obtain the resources of the root node, including: Leaf nodes are treated as child nodes, and the resources of the child nodes are merged into the corresponding parent nodes based on the merging frequency corresponding to the child nodes and the update frequency of the resources in the child nodes. The parent node is treated as a new child node, and the steps of merging the resources of the child node into the corresponding parent node are repeated based on the merging frequency corresponding to the child node and the update frequency of the resources in the child node, until the resources of the root node are obtained.
3. The image rendering method according to claim 2, characterized in that, The merging frequency is the update frequency of the parent node corresponding to the child node; the update frequency of each child node is greater than the update frequency of its corresponding parent node.
4. The image rendering method according to claim 2, characterized in that, The step of merging the resources of a child node into its corresponding parent node based on the merging frequency corresponding to the child node and the update frequency of the resources in the child node includes: Identify a first resource in a child node whose update frequency is less than or equal to the merging frequency, and merge the first resource of at least one child node into the corresponding parent node; Identify a second resource in the child node whose update frequency is greater than the merging frequency, determine the maximum quantity of the second resource, and update the maximum quantity to the parent node.
5. The image rendering method according to claim 1, characterized in that, The step of generating the descriptor set layout by covering the resources of the root node hierarchically from the root node to the leaf node of the tree structure includes: By taking the root node as the parent node, the resources of the parent node are propagated to the corresponding child nodes, and the resources of the child nodes are overwritten. The child node is used as the new parent node, and the steps of propagating the resources of the parent node to the corresponding child node and overwriting the resources of the child node are repeated until the resources of the leaf node are obtained, thus obtaining the layout of the descriptor set.
6. The image rendering method according to claim 5, characterized in that, The step of propagating the resources of the parent node to the corresponding child nodes and overwriting the resources of the child nodes includes: If the update frequency of all resources contained in the parent node is less than the update frequency of the corresponding child node, then the resources of the parent node are propagated to the corresponding child node, overwriting the resources of the child node.
7. The image rendering method according to claim 1, characterized in that, The step of generating the descriptor set layout by covering the resources of the root node hierarchically from the root node to the leaf node of the tree structure includes: Following the direction from the root node to the leaf node of the tree structure, the updated resources of each node in the tree structure are obtained by covering the resources of the root node layer by layer. The updated resources of each node are sorted based on a preset descriptor index to obtain the layout of the descriptor set.
8. The image rendering method according to claim 7, characterized in that, The descriptor index includes at least one of the following: resource update frequency, parameter binding type, descriptor type, and shader stage.
9. An image rendering apparatus, characterized in that, include: The receiving module is used to receive image rendering instructions; The first generation module is used to respond to the image rendering instruction by creating corresponding descriptor sets in order of frequency from low to high based on a preset descriptor set layout, and binding them to the current rendering pipeline to generate the current shader pipeline layout. The descriptor set layout is a tree structure; each leaf node of the tree structure includes at least one shader; the descriptor set layout is generated based on the resources of each module of at least one shader. A rendering module for rendering an image based on the current shader pipeline layout; The device further includes a second generation module, which is used to generate the preset descriptor set layout by merging the resources of each shader module in multiple leaf nodes in a hierarchical manner from the leaf nodes to the root node of the tree structure. The second generation module is specifically used for: Obtain the resources corresponding to each module of multiple shaders in multiple leaf nodes of a preset tree structure, and determine the update frequency of each resource in each leaf node. Based on the merging frequency corresponding to each level and the update frequency of each resource in each leaf node, the resources of the shaders in multiple leaf nodes are merged level by level in the direction from the leaf nodes of the tree structure to the root node to obtain the resources of the root node. The descriptor set layout is generated by covering the resources of the root node layer by layer, following the direction from the root node to the leaf node of the tree structure.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the image rendering method according to any one of claims 1-9.
11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the image rendering method according to any one of claims 1-9.
12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the image rendering method according to any one of claims 1-9.