A multi-objective optimization method for real-time rendering pipeline

Abstract

The present invention discloses a multi-objective optimization method for a real-time rendering pipeline. The present invention changes the power consumption target to the frame delay target, starts from the load cause of the frame delay, and uses a linear model and a neural network model to make more accurate numerical predictions of the frame delay and graphics quality of the real-time rendering program based on the relevant data of the drawing command, vertex shader, and fragment shader. On the basis of the original method, the linear model is improved as necessary for the time-sensitive characteristics of the frame delay, and the relevant parameters of the shader are introduced to optimize the neural network model. The method can better complete the task of balancing frame delay and graphics quality, and is suitable for high-performance rendering pipelines with multiple scenes, multiple perspectives, multiple budgets, and multiple dynamics.

Description

Technical Field

[0001] The present invention relates to the field of computer graphics, and in particular to a multi-objective optimization method for a real-time rendering pipeline. Background Art

[0002] The progress of computer graphics has spawned highly complex real-time rendering algorithms, and the huge computing power required to render a single frame with ideal effects. How to make the rendering program ensure the real-time performance of graphics generation (measured by frame delay in this invention) while maximizing the quality of graphics is a topic worth exploring in the field of computer graphics. The research task of this invention is to automatically optimize the multi-objective frame delay and graphics quality of the real-time rendering pipeline.

[0003] For a long time, the related work of multi-objective automatic optimization has focused on the optimization of indicators such as performance (compilation time, execution time), quality (correctness), energy consumption, code size, etc. of general computer programs. The optimization target of the present invention is the frame delay and graphics quality of real-time rendering programs. The optimization focus is on the coupling relationship of multiple rendering algorithms, rather than the working effect of a single rendering algorithm. Many optimization algorithms are used to balance the frame delay and graphics quality of rendering applications, such as rendering algorithms that use temporal coherence, shader simplification algorithms, etc. Some of the work has achieved good results. However, they either perform optimization in offline solutions, and the corresponding preprocessing is usually time-consuming, especially when the scene has a large range of uniform parameters; or they have poor flexibility and are only suitable for the optimization of specific rendering algorithms, but cannot handle the coupling relationship of multiple algorithms well. Or they are only applicable to a single perspective of a single scene, and the evaluation of frame delay is based on relative values ​​rather than accurate numerical analysis, resulting in unclear optimal configuration and inaccurate parameter adjustment. Therefore, they cannot be directly applied to the research objectives of this article.

[0004] On the other hand, research on multi-objective automatic optimization of device power consumption and graphics quality has also made good progress in recent years. Some specific pipeline designs and hardware implementations have been developed to optimize resources and reduce power consumption during rendering. Power consumption models and quality models for specific pipelines have been proposed, which can quantify device power consumption and graphics quality during real-time rendering. Based on these quantitative models, the automatic parameter adjustment framework can achieve real-time optimization tasks for both objectives. Summary of the invention

[0005] The purpose of the present invention is to provide a multi-objective optimization method for a real-time rendering pipeline to address the deficiencies of the prior art, so as to achieve the task of balancing frame delay and graphics quality in a multi-parameter space. The present invention introduces the multi-objective optimization frontier method of power consumption and graphics quality into the multi-objective optimization problem of frame delay and graphics quality, and changes the power consumption target to the frame delay target, and makes necessary improvements.

[0006] The objective of the present invention is achieved through the following technical solution: A multi-objective optimization method for a real-time rendering pipeline comprises the following steps:

[0007] Step 1: Introduce the cutting-edge multi-objective optimization methods for power consumption and graphics quality of real-time rendering devices in recent years into the multi-objective optimization problem of frame delay and graphics quality. Based on the relevant indicators of drawing commands, vertex shaders, and fragment shaders, use linear models to accurately predict the frame delay and graphics quality of multi-algorithm coupled real-time rendering programs. Based on the predicted values, dynamically select the rendering configuration with the best graphics quality within a given frame delay budget in real time.

[0008] Step 2: Improve the results of the linear model in step 1 and add new modules using a greedy strategy to solve the problem of linear model failure.

[0009] Step 3: Add a neural network model based on step 2 to solve problems such as additional errors generated by the linear model and uneven optimization work.

[0010] Furthermore, step one is implemented by the following sub-steps:

[0011] (1.1) Design a multi-algorithm coupled real-time rendering pipeline to form a sufficiently large parameter space for optimization; these algorithms include: ambient light map, shadow algorithm, anti-aliasing algorithm, ambient light occlusion, screen space reflection, etc. Each algorithm corresponds to a visual effect and provides a control parameter for adjusting the visual effect, plus the luminous effect of self-luminous objects and the resolution of the generated image, a total of 7 variable parameters; except for the ambient light map configuration with or without two values, the other 6 variable parameters have 3 values, so the total parameter space contains 1458 values;

[0012] (1.2) Based on the existing research on multi-objective optimization problems of device power consumption and graphics quality, a linear model for frame delay prediction and graphics quality evaluation is established to search for the rendering configuration corresponding to the best graphics quality that meets the frame delay budget at runtime; this method is based on two key components: a real-time frame delay prediction model and a graphics quality evaluation model, both of which are expressed in the form of linear equations; the frame delay prediction model is based on relevant data indicators of drawing commands, vertex shaders, fragment shaders, etc., and exponential models are established for 7 control parameters respectively, and then the exponential models are linearly added to calculate the final frame delay; the graphics quality evaluation model is based on the following observation: under different values ​​of the same control parameter, the graphics quality loss value is linearly related, so it can also be evaluated using a linear model;

[0013] Furthermore, the specific problems and solutions of step 2 are:

[0014] Since the time cost of estimating the graphics quality in step (1.2) is relatively high, the optimization framework can only work at a certain frequency and cannot achieve real-time optimization. On the other hand, the frame delay prediction model needs to be fitted in real time. If the accuracy is inaccurate, a relatively long refitting phase needs to be started, which will also introduce additional calculation errors. Due to the above two problems, if the frame delay prediction is inaccurate during the optimization interval or due to changes in the viewing angle, frame drops, continuous low graphics quality, and graphics quality jumps may occur, making the rendering picture not smooth.

[0015] In this regard, the improvement plan for step two is as follows: when the viewpoint changes within the optimization interval or during the model correction window, the initial frame uses a medium rendering configuration, and then based on the correlation coefficient of the graphics quality evaluation model, a greedy strategy is used to adjust the parameters according to the relative impact of each control parameter on the graphics quality. Only the parameters with the greatest impact are adjusted for each frame until the frame delay exceeds the predetermined value. Experimental results verify that this method can solve the above problems and can generate graphics close to the highest quality.

[0016] Furthermore, the specific problems and solutions of step three are:

[0017] Based on the existing research on multi-objective optimization problems of device power consumption and graphics quality, a neural network-based optimization method is introduced to meet the needs of smooth and stable real-time optimization in multiple scenarios and multiple perspectives; this framework uses two independent neural networks trained entirely by synthetic datasets to predict frame latency and graphics quality under various parameter configurations, and calculates the frame latency and graphics quality estimates under all configurations in real time during the rendering program, thereby maximizing the graphics quality of each frame under a given frame latency budget.

[0018] The research object of the relevant prior art is GPU power consumption. The GPU operating frequency is considered as an intermediate result in the neural network model to enhance the prediction accuracy. The experimental configuration of the present invention is simpler and does not require real-time acquisition of the GPU operating frequency. Instead, this factor is implicitly included through the relevant data of the vertex shader and the fragment shader. The present invention modifies the network structure and adds these relevant data to the model input. The experimental results show that compared with the linear model, this method can achieve real-time optimization, and the prediction model obtained after one training can be applicable to different rendering scenarios. There is no long model correction stage, and no additional calculation errors will be introduced. The target problem can be solved simply and effectively.

[0019] The beneficial effect of the present invention is that the automatic parameter adjustment framework of the present invention can dynamically select the rendering configuration with the best graphics quality within a given frame delay budget, and can better complete the task of balancing frame delay and graphics quality in a multi-parameter space. The final result can be applicable to multi-scene, multi-perspective, multi-budget, and multi-dynamic high-performance rendering pipelines. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a framework diagram of a multi-objective optimization method for a real-time rendering pipeline;

[0021] Figure 2 It is a diagram of the multi-algorithm coupling relationship of the rendering pipeline;

[0022] Figure 3 It is the workflow diagram of the optimization framework;

[0023] Figure 4 It is a network structure diagram of the frame delay and graphics quality neural network prediction model. DETAILED DESCRIPTION

[0024] The present invention will be described in detail below with reference to the accompanying drawings.

[0025] like Figure 1 As shown, the multi-objective optimization method of the real-time rendering pipeline of the present invention comprises the following steps:

[0026] Step 1: Introduce the cutting-edge multi-objective optimization methods for power consumption and graphics quality of real-time rendering devices in recent years into the multi-objective optimization problem of frame latency and graphics quality. Based on the relevant indicators of drawing commands, vertex shaders, and fragment shaders, use linear models to accurately predict the frame latency and graphics quality of multi-algorithm coupled real-time rendering programs. Based on the predicted values, dynamically select the rendering configuration with the best graphics quality in real time within a given frame latency budget.

[0027] This step is the technical basis of the present invention and is divided into the following sub-steps.

[0028] 1) Design a real-time rendering pipeline with multiple algorithms coupled to form a large enough parameter space for optimization; Figure 2 , these algorithms include: ambient light map, shadow algorithm, anti-aliasing algorithm, ambient light occlusion, screen space reflection, etc. Each algorithm corresponds to a visual effect and provides a control parameter to adjust the visual effect, plus the luminous effect of self-luminous objects and the resolution of the generated image, a total of 7 variable parameters; except for the ambient light map configuration with or without two values, the other 6 variable parameters have 3 values, so the total parameter space contains 1458 values;

[0029] 2) Based on the existing research on multi-objective optimization of device power consumption and graphics quality, a linear model for frame delay prediction and graphics quality evaluation is established to search for the rendering configuration corresponding to the best graphics quality that meets the frame delay budget at runtime. The workflow is as follows: Figure 3As shown in FIG. 1 , the method is based on two key components: a real-time frame delay prediction model and a graphics quality assessment model. Both models are expressed in the form of linear equations. The frame delay prediction model is based on the relevant data indicators of the drawing command, vertex shader, fragment shader, etc., and an exponential model is established for each of the seven control parameters. The exponential models are then linearly added to calculate the final frame delay, as shown in formula (1). The graphics quality assessment model is based on the following observation: under different values ​​of the same control parameter, the graphics quality loss value is linearly related, so it can also be evaluated using a linear model, as shown in formula (2).

[0030] P(s,c)=P min +(P max -P min )(1-e -α )

[0031]

[0032] Where b, v, and f represent the number of executed instructions, the number of vertex shader executions, and the number of fragment shader executions, respectively. B, V, and F are the b, v, and f corresponding to the maximum frame delay, respectively. b, v, and f are respectively calculated by the coefficient k b , k v , k f The weighting is done to take into account the relative impact of each coefficient on the total latency. All parameters depend on the rendering configuration s; in addition, the camera parameters c are implicit in the primitives b, v, f.

[0033]

[0034]

[0035] Among them, s 0 Defined as the rendering configuration where each parameter takes the highest value (l0), This means that except for the xth control parameter value being l (l>0), the rest are the highest parameter configuration l0.

[0036] Step 2: Improve the results of the linear model in step 1 and add new modules using a greedy strategy to solve the problem of linear model failure.

[0037] This step is one of the core technologies of the present invention, and the specific problems and methods to be solved are:

[0038] Since the time cost of estimating the graphics quality in step 1 (2) is high, the optimization framework can only work at a certain frequency and cannot achieve real-time optimization. On the other hand, the frame delay prediction model needs to be fitted in real time. If the accuracy is inaccurate, a relatively long re-fitting phase needs to be started, which will also introduce additional calculation errors. Due to the above two problems, if the frame delay prediction is inaccurate during the optimization interval or due to changes in the viewing angle, frame drops, continuous low graphics quality, and graphics quality jumps may occur, making the rendering picture not smooth.

[0039] In this regard, the improvement plan for step two is as follows: when the viewpoint changes within the optimization interval or during the model correction window, the initial frame uses a medium rendering configuration, and then based on the correlation coefficient of the graphics quality evaluation model, a greedy strategy is used to adjust the parameters according to the relative impact of each control parameter on the graphics quality. Only the parameters with the greatest impact are adjusted for each frame until the frame delay exceeds the predetermined value. Experimental results verify that this method can solve the above problems and can generate graphics close to the highest quality.

[0040] Step 3: Add a neural network model based on step 2 to solve problems such as additional errors generated by the linear model and uneven optimization work.

[0041] Step three is also one of the core technologies of the present invention, and the specific problems and solutions solved are:

[0042] Based on the existing research on multi-objective optimization problems of device power consumption and graphics quality, a neural network-based optimization method is introduced to meet the requirements of smooth and stable real-time optimization in multiple scenes and multiple perspectives; this framework uses two independent neural networks trained entirely by synthetic datasets to predict frame delay and graphics quality under various parameter configurations, and calculates the estimated frame delay and graphics quality under all configurations in real time during the rendering program, thereby maximizing the graphics quality of each frame under a given frame delay budget. The implicit function of frame delay is expressed as formula (3).

[0043] P(c,s)=Θ(b,v,f,s).............(3)

[0044] The function of image quality is expressed as formula (4):

[0045] e(c,s j )=φ(a,s j ,s cur , e(c, s cur )).......(4)

[0046] This function takes scene information a, rendering settings s j , Current Rendering Settings cur and its image quality loss e(c, scur ) is the input and the prediction error is e(c, sj).

[0047] The research object of the related prior art is GPU power consumption. The GPU operating frequency is considered as an intermediate result in the neural network model to enhance the prediction accuracy. However, the experimental configuration of the present invention is simpler and does not need to obtain the GPU operating frequency in real time. Instead, this factor is implicitly included through the relevant data of the vertex shader and the fragment shader. The present invention modifies the network structure and adds these relevant data to the model input; accordingly, formula (3) is adjusted to formula (5):

[0048] P(c,s)=Θ(b,v,f,s,m1,m2,n2).....(5)

[0049] Among them, m1, m2, and n2 are respectively related to the execution costs of vertex shader statements, fragment shader statements, and fragment shader texel access instructions, which can well reflect the working status of the GPU.

[0050] The network structure of the frame delay and image quality neural network prediction model is as follows Figure 4 .

[0051] Experimental results show that compared with linear models, this method can achieve real-time optimization. The prediction model obtained after one training can be applied to different rendering scenarios. There is no long model correction stage, no additional calculation errors are introduced, and the target problem can be solved simply and effectively.

Claims

1. A multi-objective optimization method for a real-time rendering pipeline, characterized in that: The following steps are involved: Step 1: (1.1) Design a real-time rendering pipeline with multiple algorithms coupled to form a parameter space for optimization; the algorithms include: ambient light map, shadow algorithm, anti-aliasing algorithm, ambient light occlusion, screen space reflection, each algorithm provides a control parameter for adjusting a visual effect, plus the luminous effect of self-luminous objects and the resolution of generated images, a total of 7 control parameters; the ambient light map configuration has two values, with or without, and the other 6 control parameters have 3 values, the total parameter space contains 1458 values; (1.2) Based on the multi-objective optimization problem of device power consumption and graphics quality, a linear model for frame delay prediction and graphics quality evaluation is established, and the rendering configuration corresponding to the best graphics quality that meets the frame delay budget is searched at runtime; this method is based on two key components: a real-time frame delay prediction model and a graphics quality evaluation model, both of which are expressed in the form of linear equations; the frame delay prediction model is based on the relevant data indicators of the drawing command, vertex shader, and fragment shader, and an exponential model is established for each of the seven control parameters, and then the exponential models are linearly added to calculate the final frame delay; the graphics quality evaluation model is based on the following observation: the graphics quality loss value is linearly related to different values ​​of the same control parameter, and is evaluated using a linear model; Step 2: Use a greedy strategy to solve the linear model failure problem; this step is as follows: when the viewpoint changes during the optimization interval or during the model correction window, the initial frame uses a medium rendering configuration, and then based on the correlation coefficient of the graphics quality evaluation model, the greedy strategy is used to adjust the parameters according to the relative impact of each control parameter on the graphics quality, and only the parameters with the greatest impact are adjusted for each frame until the frame delay exceeds the predetermined value; Step 3: Add a neural network model. Based on the multi-objective optimization problem of device power consumption and graphics quality, a neural network-based optimization method is introduced. The framework uses two independent neural networks trained entirely by synthetic data sets to predict frame delay and graphics quality under various parameter configurations. The estimated frame delay and graphics quality under all configurations are calculated in real time during the rendering program. The implicit function of frame delay is expressed as formula (5). P(c,s)=Θ(b,v,f,s,m1,m2,n2).....(5) Wherein, b, v, and f represent the number of executed instructions, the number of vertex shader executions, and the number of fragment shader executions, respectively, and c represents the camera parameters; the function of the graphics quality is expressed as shown in formula (4): e(c,s j )=Φ(a,s j ,s cur ,e(c,s cur )).......(4) This function takes scene information a, rendering settings s j , Current Rendering Settings cur and its image quality loss e(c,s cur ) is the input, the prediction error e(c,sj); m1, m2, n2 are related to the execution cost of the vertex shader statement, fragment shader statement, and fragment shader texel access instruction, respectively.

Images