An online cost model calibration method and system based on shader compilation feedback

Abstract

The application provides an online cost model calibration method and system based on Shader compilation feedback, the method comprising: obtaining a plurality of preprocessed programs corresponding to shader compilation tasks; extracting program structure features representing compilation complexity from the preprocessed programs, and determining the estimated compilation workload of each shader compilation task according to the program structure features; sending the shader compilation tasks to a remote compilation node for execution, and obtaining the actual compilation time consumption; normalizing the actual compilation time consumption according to the performance parameters of the remote compilation node to obtain the standardized compilation cost; and updating the model parameters of the compilation cost prediction model according to the deviation value between the standardized compilation cost and the estimated compilation workload. The application replaces the original file attributes with the preprocessed program structure features as the prediction input, combines node performance normalization and actual compilation feedback online calibration, eliminates the difference pollution of heterogeneous nodes, and makes the model continuously converge.

Description

Technical Field

[0001] This application relates to the field of distributed compilation technology, and in particular to an online cost model calibration method and system based on shader compilation feedback. Background Technology

[0002] Modern game engines generate numerous shader compilation tasks during the build process, taking into account different platforms, graphics interfaces, shader stages, and macro definition combinations. The actual compilation time of a single task is significantly affected by the preprocessed program structure, and different variants may exhibit orders of magnitude differences in compilation time under the same original shader file.

[0003] Existing distributed compilation schemes typically estimate compilation costs based on original file attributes or fixed historical averages, and cannot distinguish the structural differences of the final program under different macro definitions and conditional compilation combinations for the same shader file. Summary of the Invention

[0004] To address the shortcomings of existing shader compilation cost prediction technologies that rely on original file attributes or fixed historical statistics, which cannot adapt to differences in preprocessed program structure, heterogeneous node performance differences, and continuous model drift caused by toolchain version changes, this application provides an online cost model calibration method and system based on shader compilation feedback. This method uses the structural characteristics of the preprocessed shader program as prediction input, combined with remote compilation node performance normalization and actual compilation time feedback to calibrate the compilation cost prediction model online. This eliminates the contamination of the model by heterogeneous node performance differences and ensures that the model parameters continuously converge with actual compilation feedback. Specifically, this application provides the following technical solutions: This application provides an online cost model calibration method based on shader compilation feedback, including: Obtain the first program corresponding to multiple preprocessed shader compilation tasks; Extract first program structure features that characterize compilation complexity from the first program, and determine the first estimated workload of each shader compilation task based on the first program structure features; The shader compilation tasks are sent to a remote compilation node for execution to obtain the first actual compilation time of each shader compilation task; The first actual compilation time is normalized based on the first performance parameter of the remote compilation node that performs the shader compilation task to obtain the first standardized compilation cost. The first model parameters of the compilation cost prediction model are updated based on the first deviation between the first standardized compilation cost and the corresponding first estimated workload.

[0005] Optionally, after determining the first estimated workload of each shader compilation task, the method further includes: aggregating multiple shader compilation tasks into a remote compilation batch based on the first estimated workload, wherein the remote compilation batch meets the range of the first target workload.

[0006] Optionally, it also includes: Based on the first remaining workload before tasks are dispatched, the first workload rate, and the first consumption capacity of the remote compilation node, determine whether there is insufficient parallelism in the remote compilation batch; If the parallelism is insufficient, adjust the first target workload and re-aggregate the shader compilation tasks that have not yet been dispatched.

[0007] Optionally, the first performance parameter includes a first performance factor for each remote compilation node, and the normalization process includes updating the first performance factor according to a preset period or the cumulative number of samples.

[0008] Optionally, updating the first model parameters of the compilation cost prediction model includes: The shader compilation tasks are grouped according to shader type or target platform, and the first model parameters of the corresponding group are updated respectively.

[0009] Optionally, after obtaining the first actual compilation time for each shader compilation task, the following is also included: The shader compilation task that hits the cache is marked as the first non-training sample, and the first non-training sample is excluded from the input of the first model parameter of the updated compilation cost prediction model.

[0010] Optionally, the first program structure features include lexical complexity features and control flow complexity features.

[0011] Optionally, when adjusting the first target workload, the oscillation can be suppressed by setting a first minimum target value, a first maximum target value, and a first hysteresis interval.

[0012] This application also provides an online cost model calibration system based on shader compilation feedback, including: The acquisition module is used to acquire the first program corresponding to multiple preprocessed shader compilation tasks; The feature extraction module is used to extract first program structure features that characterize compilation complexity from the first program; The prediction module is used to determine the first estimated workload of each shader compilation task based on the first program structure characteristics. An execution module is used to send the shader compilation tasks to a remote compilation node for execution and to obtain the first actual compilation time of each shader compilation task; The normalization module is used to normalize the first actual compilation time based on the first performance parameter of the remote compilation node that performs the shader compilation task, so as to obtain the first normalized compilation cost. The calibration module is used to update the first model parameters of the compilation cost prediction model based on the first deviation between the first standardized compilation cost and the corresponding first estimated workload.

[0013] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0014] The beneficial effects of this application are as follows: 1. By replacing the original file attributes with the structural features of the preprocessed shader program as the input for cost prediction, the prediction model can distinguish the structural differences of the final program under different macro definitions and conditional compilation combinations of the same shader file, thereby accurately reflecting the actual compilation complexity of the current variant. 2. By normalizing the performance parameters of remote compilation nodes, the pollution of historical compilation time data by the performance differences of heterogeneous nodes is eliminated, so that the compilation feedback generated by different nodes can be uniformly used for model calibration. 3. By updating the model parameters online based on the deviation between the standardized compilation cost and the expected compilation workload, the compilation cost prediction model continuously converges with the actual compilation feedback, adapting to model drift caused by toolchain version changes and shader variant structure changes. Attached Figure Description

[0015] Figure 1 A schematic diagram of the overall process of the online cost model calibration method provided in the embodiments of this application.

[0016] Figure 2 This is a schematic diagram of the normalization processing and online calibration sub-process provided in the embodiments of this application.

[0017] Figure 3 The prediction error convergence curve with calibration rounds provided in the embodiments of this application is shown.

[0018] Figure 4 A comparison chart of prediction stability before and after normalization provided for embodiments of this application.

[0019] Figure 5 A schematic diagram of the overall architecture of the online cost model calibration system provided in this application embodiment. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0022] The online cost model calibration method and system provided in this application can be applied to scenarios that require distributed compilation of a large number of shader variants, such as game engine build systems, graphics rendering pipeline compilation tools, cloud shader compilation services, continuous integration build platforms, and graphics driver compiler optimization systems. The method can be deployed on a build control node with distributed compilation capabilities, and the system can run in a server cluster, workstation cluster, or hybrid cloud compilation environment.

[0023] The execution environment of the method described in this embodiment includes a local build control node and multiple remote compilation nodes. The local build control node is responsible for the preprocessing, feature extraction, cost prediction, batch aggregation, and model calibration of the shader compilation task; the remote compilation nodes are responsible for receiving compilation batches and executing the actual shader compilation work, and returning the compilation results and actual compilation time to the local build control node after completion. The remote compilation nodes can be homogeneous or heterogeneous computing nodes, and their processing capabilities may vary depending on the CPU model, number of cores, and toolchain version.

[0024] like Figure 1 As shown in the embodiments of this application, the online cost model calibration method based on shader compilation feedback includes the following steps: S100: Obtain the first program corresponding to multiple preprocessed shader compilation tasks.

[0025] In a game engine's build pipeline, shader source files undergo a preprocessing stage before entering the platform compiler. This preprocessing stage includes macro expansion, conditional compilation branch evaluation, recursive inclusion of include files, and keyword and permutation combination expansion. For the same shader source file, different combinations of macro definitions, different keyword enable states, and different permutation arrangements can lead to final shader programs with completely different structures and sizes after preprocessing.

[0026] After the above preprocessing is completed, the Shader compilation task (i.e., the shader compilation task) produces a final preprocessed Shader program (i.e., the first program), which serves as the input for the platform's shader compiler. This step obtains the complete text content of the first program from the preprocessed shader compilation task and uses it as input for subsequent feature extraction.

[0027] The specific execution flow of the preprocessing stage is as follows: the shader source file is recursively included by the include file to form a complete source code text. Then, the macro preprocessor evaluates the conditional branches according to the global macros and conditional compilation macros defined in the current build configuration, removes inactive code branches, and expands all macro definitions.

[0028] In a typical game engine build scenario, a single shader source file may be associated with tens to thousands of macro definition combinations, each of which produces an independent shader variant.

[0029] For example, a PBR material shader source file defines 8 Boolean keywords and 3 multiple-choice permutations, theoretically resulting in 2^8 factorial combinations. After filtering by the engine's variant pruning strategy, approximately 340 valid variants are actually generated. The code size, control flow complexity, and number of resource bindings for each variant may differ after preprocessing. This is the fundamental technical basis for this method to use the preprocessed program rather than the original source file as the object of complexity analysis.

[0030] The first program is acquired during the preprocessing stage of the engine's shader compilation pipeline, before the platform compiler is invoked. The system intercepts the complete preprocessed program text. This interception is implemented using memory references, without incurring additional disk read / write overhead. For engine architectures employing asynchronous preprocessing, the acquisition module obtains the program text reference during the preprocessing task completion callback, binds it to the compilation task identifier, and passes it to the subsequent feature extraction process.

[0031] The first program is transmitted internally within the system as a plain text string encoded in UTF-8. Each shader compilation task is stored in the task queue of the local build control node as a task descriptor. The task descriptor includes a unique task identifier, the shader source file path, a variant macro definition combination identifier, a preprocessed program text reference, a shader stage identifier, and a target platform identifier. The unique task identifier is generated by concatenating the shader source file hash value and the variant macro definition combination hash value, ensuring that different variants of the same source file have different task identifiers. When the shader compilation pipeline uses incremental compilation mode, the system compares the unique task identifier with historical build records, skipping tasks whose preprocessed programs have not changed, and only performing subsequent feature extraction and compilation processes on newly added or modified variants.

[0032] For example, a vertex shader source file, with skeletal animation macros and GPU skinning macros enabled, has a preprocessed program size of 2400 lines and 18500 tokens; while the same source file, with only static mesh macros enabled, has a preprocessed program size of 680 lines and 4200 tokens. The actual compilation time can differ by 3 to 8 times. Therefore, this step uses the preprocessed final program as the object of complexity analysis, rather than using the original source file attributes.

[0033] S200, extract the first program structure features that characterize the compilation complexity from the first program, and determine the first estimated workload of each shader compilation task based on the first program structure features.

[0034] The extraction of the program structure feature vector (i.e., the first program structure feature) uses the complete text of the first program as input. The feature extraction employs a lightweight lexical scanning method to perform a single-pass scan of the preprocessed program, without the need to construct a complete abstract syntax tree. The first program structure feature includes two dimensions: lexical complexity features and control flow complexity features.

[0035] The lexical complexity features include the number of bytes in the preprocessed source code, the number of lines of code, the total number of tokens, the number of function definitions, and the number of function calls. The control flow complexity features include the number of branch statements, switch statements, loop statements, static loops, texture sampling and loading operations, atomic operations, and wavefront operations. Additionally, auxiliary features such as shader stage identifiers, target platform identifiers, and optimization levels can be extracted.

[0036] The lightweight lexical scanner is implemented by performing a single-pass sequential scan of the preprocessed program's byte stream using a finite state automaton. During the scan, a set of counters is maintained, and the corresponding counter is incremented each time a specific lexical pattern is identified. The finite state automaton includes four states: normal code state, line comment state, block comment state, and string literal state, ensuring that the statistical process excludes interference content from comments and strings.

[0037] The extraction rules for the lexical complexity features are as follows: the number of bytes in the source code is taken as the total number of bytes in the UTF-8 encoding of the preprocessed program; the number of lines of code is counted by the number of non-empty lines separated by newline characters; the number of tokens is counted by the number of valid tokens separated by whitespace characters and operators; the number of function definitions is counted by the number of times the function signature pattern appears; and the number of function calls is counted by the number of times the pattern of the identifier followed by a left parenthesis appears.

[0038] The extraction rules for the control flow complexity features are as follows: For branch statements, the sum of the occurrences of the `if` and `else` keywords is counted; for switch statements, the sum of the occurrences of the `switch` keyword is counted; for loop statements, the sum of the occurrences of the `for` and `while` keywords is counted; for static loops, the number of loops marked with the `UNROLL` annotation is counted; for texture sampling, the number of calls to texture operation functions such as `texture`, `textureLod`, and `texelFetch` is counted; for atomic operations, the number of calls to atomic operation functions such as `atomicAdd` and `atomicMax` is counted; and for wavefront operations, the number of calls to wavefront-level built-in functions such as `WaveActiveSum` and `WaveReadLaneFirst` is counted.

[0039] The feature vector has a fixed dimension of 13 plus an auxiliary feature dimension. The fixed 13 dimensions consist of the aforementioned 5 lexical complexity features, 7 control flow complexity features, and the number of resource bindings. The number of resource bindings is calculated based on the total number of declarations for Uniform Buffer, Storage Buffer, Sampler, and Image resources. Among the auxiliary features, the shader stage identifier uses one-hot encoding to represent the vertex, fragment, geometry, tessellation, or computation shader stage to which the current shader belongs; the target platform identifier uses one-hot encoding to represent the target compilation format such as Vulkan SPIR-V, Metal MSL, DirectX DXIL, or GLSL.

[0040] For example, performing a lexical scan on a preprocessed vertex shader program of 2400 lines yields the following feature vector: 76800 bytes of source code, 2400 lines of code, 18500 tokens, 12 function definitions, 48 ​​function calls, 36 branches, 2 switch statements, 15 loops, 8 static loops, 0 texture samples, 0 atomic operations, 0 wavefront operations, and 6 resource bindings. The scan typically takes between 0.5 and 3 milliseconds to execute in a single thread, far less than the time required to compile the shader itself.

[0041] When determining the estimated shader compilation workload (i.e., the first estimated workload) based on the first program structure features, the feature vector is input into the compilation cost prediction model. The compilation cost prediction model takes the first program structure features as input and outputs the estimated compilation time of the shader compilation task on the standardized reference node, i.e., the first estimated workload. The specific form of the model can be, for example, a bucketed median model, an online statistical model, or a gradient boosting tree-based regression model, etc. This embodiment uses a bucketed median model as the initial model.

[0042] The bucketing median model divides historical compilation records into buckets based on shader stage and token count intervals, maintaining a sliding median of normalized compilation time within each bucket. For a newly arriving shader compilation task, it falls into the corresponding bucket based on its shader stage and token count. The sliding median of that bucket is multiplied by a correction coefficient based on other features to obtain the first estimated workload.

[0043] The compilation cost prediction model selects the bucket median model as the initial model because the distribution of shader compilation time is affected by the shader stage and size range, exhibiting obvious multi-peak characteristics. The statistical differences in compilation time of different stages and size ranges are greater than the individual differences within the same range.

[0044] The bucketed median model, by dividing the data into buckets based on shader stages and token count intervals, results in an approximately unimodal distribution of compilation time within each bucket, and the median statistic is more robust to outliers than the mean. Furthermore, the parameter updates of this bucketed median model only involve the sorting statistics of samples within the sliding window, requiring no gradient calculations or matrix operations. The computational complexity of a single update is O(W), where W is the window length. The bucket partitioning strategy of the bucketed median model is as follows: The shader stages are divided into first-level buckets, and each first-level bucket is further divided into second-level buckets based on the number of tokens. The division of first-level buckets is determined by the shader stages supported by the target platform's shader compiler. For example, it may include five first-level buckets: vertex shader bucket, fragment shader bucket, computation shader bucket, geometry shader bucket, and tessellation shader bucket.

[0045] The boundaries of the secondary buckets are determined using an equal-ratio bucketing method. Specifically, the minimum token count of 500 and the maximum token count of 100,000 are used as boundaries, and the buckets are divided into 14 secondary buckets with a common ratio of 1.5. These buckets are 500 to 750, 750 to 1125, 1125 to 1687, 1687 to 2531, 2531 to 3796, and so on, until the upper limit of 100,000 is reached. This equal-ratio bucketing method results in narrower bucket widths for small-scale programs and wider bucket widths for large-scale programs, which matches the characteristic that shader compilation time increases approximately logarithmically with program size.

[0046] It should be noted that each secondary bucket maintains a fixed-length sliding window and the current median value. The sliding window is implemented using a circular buffer; when a new sample arrives and the buffer is full, the oldest sample is overwritten. The median is updated using an incremental sorting method: an ordered index array is maintained. When a new sample is inserted, a binary search is used to locate the insertion position and update the median pointer. The time complexity of a single update is O(W), where the time complexity of the binary search for locating the median is O(log W), and the time complexity of inserting and shifting the ordered index array is O(W), where W is the window length.

[0047] In an alternative implementation, an online statistical model can be used instead of the bucketed median model. The online statistical model updates the mean and variance using an exponential moving average method, with a parameter update complexity of O(1). It has a faster response speed but is less robust to outliers.

[0048] For example, the feature vector of the aforementioned 2400-line vertex shader program falls into the "Vertex Shader, Token Count 15000 to 20000" bucket, which currently has a sliding median of 4.2 seconds. Based on the correction factor of 1.15 corresponding to the program's 36 branches and 15 loops, the first estimated workload is 4.2 × 1.15 = 4.83 seconds. The 680-line static mesh version falls into the "Vertex Shader, Token Count 3000 to 5000" bucket, which has a sliding median of 1.1 seconds and a correction factor of 0.95, resulting in a first estimated workload of 1.05 seconds.

[0049] It should be noted that the correction coefficient is determined as follows: for the historical compilation records within each bucket, other feature dimensions besides the token count are used as auxiliary variables, and an auxiliary correction relationship is established by segmenting and statistically analyzing the median ratio.

[0050] For example, within the "Vertex Shader, Token Count 15000 to 20000" bucket, the historical records are divided into a low-branch group (less than 20 branches) and a high-branch group (20 or more branches) based on the number of branches. The median normalized compilation time for each group is calculated. If the median of the high-branch group is 1.15 times that of the low-branch group, the branch correction coefficient is set to 1.15 if the current task has 20 or more branches, and 1.0 otherwise. The correction coefficients for other feature dimensions such as loop count and texture sampling count are calculated independently using the same method. The final correction coefficient is the product of the correction coefficients for each dimension. These correction coefficients are recalculated synchronously as the sliding window updates, ensuring consistency with the overall online calibration rhythm of the model.

[0051] S300, the shader compilation task is sent to a remote compilation node for execution, and the first actual compilation time of each shader compilation task is obtained.

[0052] The local build control node sends the shader compilation task to a remote compilation node for execution. Upon receiving the shader compilation task, the remote compilation node invokes the shader compiler for the corresponding platform to compile the first program. After compilation, it records the actual time elapsed from entering the compiler to the compiler outputting the result, i.e., the actual CompileTime (i.e., the first actual compilation time). The first actual compilation time only includes the platform shader compiler's own processing time and does not include network transmission time, process startup time, or task queuing time.

[0053] After the remote compilation node completes compilation, it returns the compilation result along with the first actual compilation time to the local build control node. For shader compilation tasks with cache hits, the remote compilation node directly returns the cached result, and no valid first actual compilation time record is generated in this case.

[0054] The remote compilation node is equipped with a content-addressable cache. This cache uses a combination of the hash value of the preprocessed program and the hash values ​​of the compilation parameters as the cache key, and the binary output of the compilation result as the cache value. When the remote compilation node receives a shader compilation task, it first calculates the cache key for that task and then searches for it in the content-addressable cache. If the cache is hit, the compiled result in the cache is returned directly, and the completion status is marked as a cache hit, without recording the effective first actual compilation time. If the cache is missed, the platform shader compiler is called to perform actual compilation. After compilation, the result is written to the cache and returned. The cache hit rate is typically between 60% and 85% in incremental build scenarios and close to 0% in full clean build scenarios. Since cache-hit tasks do not undergo the actual compilation process, their return time only reflects the cache lookup time rather than the compiler's actual working time. If used indiscriminately for model training, it will cause the model parameters to be severely dragged down by the extremely short cache lookup time.

[0055] For example, the local build control node sends the vertex shader compilation task, which is expected to take 4.83 seconds, to remote compilation node A for execution. The platform shader compiler on remote compilation node A actually takes 5.6 seconds to compile the task. At the same time, it sends the static mesh version, which is expected to take 1.05 seconds, to remote compilation node B for execution. The actual compilation time on remote compilation node B is 1.35 seconds.

[0056] The execution process of the remote compilation node after receiving the compilation batch is as follows: the remote compilation node extracts each shader compilation task sequentially from the received compilation batch, and calls the shader compiler process of the corresponding platform for each task. The shader compiler process uses the text content of the first program, the target platform identifier, the optimization level, and the output format as input parameters. The remote compilation node records a high-precision timestamp before the compiler process starts and records the timestamp again after the compiler process exits. The difference between the two timestamps is the first actual compilation time.

[0057] When a remote compilation node is equipped with multiple CPU cores, multiple shader compilation tasks within the same batch can be executed in parallel on the same node. Each task has its own dedicated compiler process, and the initial actual compilation time of each task is measured independently without affecting each other. After all tasks within the batch are completed, the remote compilation node packages the compilation results, initial actual compilation time, and completion status flags of each task into a batch response and returns it to the local build control node via the network. The completion status flags include three states: compilation successful, compilation failed, and cache hit. Tasks that fail to compile do not generate a valid initial actual compilation time record, and the system marks them as an exception and excludes them.

[0058] S400, the first actual compilation time is normalized according to the first performance parameter of the remote compilation node that performs the shader compilation task to obtain the first normalized compilation cost.

[0059] like Figure 2As shown, due to differences in hardware configurations across different remote compilation nodes, the actual compilation time for the same shader compilation task may vary on different nodes. To eliminate the impact of heterogeneous node performance differences on model calibration, the first actual compilation time needs to be normalized.

[0060] The WorkerSpeedFactor (i.e., the first performance parameter) characterizes the ratio of the compilation processing speed of each remote compilation node to the standardized reference node. The first performance parameter includes the WorkerSpeedFactor (i.e., the first performance factor) for each remote compilation node. When the first performance factor is 1.0, it indicates that the compilation processing speed of that node is the same as that of the standardized reference node; when the first performance factor is greater than 1.0, it indicates that the compilation processing speed of that node is faster than that of the standardized reference node.

[0061] The method for normalizing the first actual compilation time is as follows: the first actual compilation time is correlated with the first performance factor of the remote compilation node executing the task to obtain NormalizedCost (i.e., the first normalized compilation cost). The first normalized compilation cost is the compilation cost metric of the shader compilation task after being normalized by the performance factor of the remote compilation node. It is used to eliminate the performance differences between different remote compilation nodes and to make the compilation feedback data from different nodes usable for model calibration under a unified scale.

[0062] The initial value of the first performance factor can be determined based on the CPU benchmark test results of the remote compilation node, and continuously corrected based on actual compilation feedback data during system operation. The normalization process includes updating the first performance factor according to a preset period or the cumulative number of samples.

[0063] For example, the preset period is typically in the range of 50 to 200 samples; in this embodiment, 100 samples are used as an update period. Whenever a remote compilation node has completed a total of 100 shader compilation tasks, the system updates the first performance factor of that node based on the statistical relationship between the actual compilation time of the node's nearly 100 tasks and the standardized estimated compilation workload of the corresponding tasks.

[0064] For example, the first performance factor of remote compilation node A is 1.2, indicating that the compilation processing speed of node A is 1.2 times that of the normalized reference node. The first actual compilation time of the above vertex shader compilation task on node A is 5.6 seconds, and after normalization, the first normalized compilation cost is 5.6 / 1.2 = 4.67 seconds. The first performance factor of remote compilation node B is 0.85, and the first actual compilation time of the above static mesh version on node B is 1.35 seconds, and after normalization, the first normalized compilation cost is 1.35 / 0.85 = 1.59 seconds.

[0065] The algorithm for updating the first performance factor is as follows: Within each update cycle, collect the first actual compilation time and corresponding first estimated workload for all shader compilation tasks completed by the remote compilation node. For each sample, calculate the actual speed ratio, i.e., the first estimated workload divided by the first actual compilation time, and take the median of all actual speed ratios within the current update cycle as the instantaneous performance factor for that cycle. The update of the first performance factor uses an exponential moving average method to fuse historical values ​​and instantaneous values.

[0066] In the formula, As the new first performance factor, For the old first performance factor, Smoothing coefficient ( ), The smoothing coefficient is the instantaneous performance factor, and it is typically in the range of 0.7 to 0.95. In this embodiment, it is 0.85.

[0067] For example, remote compilation node A completed 100 shader compilation tasks in the 5th update cycle (samples 401 to 500), with a median actual speedup ratio of 1.18 for each task. Node A's current first performance factor is 1.2. The updated first performance factor is 1.2 × 0.85 + 1.18 × 0.15 = 1.02 + 0.177 = 1.197. The exponential moving average method smooths out short-term fluctuations in the first performance factor while tracking the long-term drift trend of node performance.

[0068] It should be noted that the actual speed ratio reflects the actual processing speed multiple of the remote compilation node relative to the current scale of the compilation cost prediction model. It, together with the division operation in the normalization process, constitutes a calibration mechanism: the normalization process converts the actual compilation time of different nodes to a unified standardized cost scale for model parameter updates, while the performance factor update continuously corrects the standardization coefficient of each node based on the ratio of the model prediction value to the actual time. The two work together to make the system tend to a steady state after multiple iterations.

[0069] When a remote compilation node undergoes hardware changes or toolchain upgrades, its compilation performance may experience a step change. The system identifies this step change by detecting the deviation between the instantaneous performance factor and the current first performance factor: when the absolute value of the deviation exceeds 25% of the current first performance factor, the system directly resets the first performance factor of that node to the instantaneous performance factor, skipping the exponential moving average smoothing process, in order to accelerate the adaptation to the performance step change.

[0070] In another embodiment, the first performance parameter can also be implemented through a performance level classification method. This performance level classification divides remote compilation nodes into several performance levels based on their hardware configurations, with nodes within the same performance level sharing the same first performance factor. The performance level classification is based on factors including CPU base frequency, number of cores, and compiler cache size. For example, performance level A represents a high-performance node with a first performance factor of 1.5; performance level B represents a standard node with a first performance factor of 1.0; and performance level C represents a low-performance node with a first performance factor of 0.7. Compared to maintaining the first performance factor on a node-by-node basis, the performance level classification method reduces the number of parameters that need to be maintained, making it suitable for cluster environments with a large number of nodes but limited hardware configurations, and so on.

[0071] In another embodiment, the normalization process can also be implemented using a tiered mapping method. This tiered mapping method converts the actual compilation time of each remote compilation node into a standardized compilation cost according to a preset performance tier mapping table, where the performance tier mapping table replaces continuous factor operations with discrete correspondences.

[0072] For example, for a node with performance level A, the actual compilation time in the range of 0 to 2 seconds is mapped to a normalized cost of 0 to 3 seconds, and the actual compilation time in the range of 2 to 5 seconds is mapped to a normalized cost of 3 to 7.5 seconds. The mapping coefficients for each level are determined based on piecewise linear regression of historical data. Compared with linear normalization of a single factor, this level-based mapping method can adapt to the non-linear relationship between compilation time and node performance.

[0073] S500, based on the first deviation between the first standardized compilation cost and the corresponding first estimated workload, update the first model parameters of the compilation cost prediction model.

[0074] like Figure 2 As shown, the online calibration involves comparing the first standardized compilation cost as the actual label value with the first estimated workload output by the compilation cost prediction model, and updating the cost prediction model parameters (i.e., the first model parameters) based on the first deviation value between the two, so that the model prediction value gradually converges to the actual compilation cost.

[0075] The first deviation value is the difference between the first standardized compilation cost and the first estimated workload. When the first deviation value is positive, it indicates that the model underestimates the actual compilation cost of the task; when the first deviation value is negative, it indicates that the model overestimates the actual compilation cost of the task.

[0076] For the bucketed median model used in this embodiment, the first model parameter of the updated compilation cost prediction model specifically includes: inserting the first standardized compilation cost into the sliding window of the bucket corresponding to the task, recalculating the sliding median of the bucket, and adjusting the correction coefficient according to the statistical distribution of the first deviation value. The length of the sliding window is usually in the range of 30 to 100 samples; in this embodiment, 50 samples are used. When the sliding window is full, the earliest sample is removed, and the new first standardized compilation cost enters the end of the window.

[0077] The internal data structure of the sliding window consists of three components: a sample circular buffer, an ordered index array, and metadata records.

[0078] The sample circular buffer is implemented using a fixed-length array, with the write pointer incrementing cyclically and new samples overwriting the oldest. The ordered index array maintains an index sequence of valid samples in the buffer, arranged in ascending order of standardized compilation cost, with the median being the sample value corresponding to the middle position of the ordered index array. The metadata record includes the current number of valid samples, the write pointer position, the timestamp of the most recent median update, and the exponential moving average of the prediction error within the bucket.

[0079] The exponential moving average of the prediction error within a bucket is used to monitor the prediction quality of the model within that bucket. When the exponential moving average of the prediction error consistently exceeds a preset quality warning threshold, the system marks the bucket as unstable. Buckets in an unstable state are given an uncertainty flag when providing the estimated compilation workload. The batch aggregation scheduling submodule reserves additional workload margin for tasks with uncertainty flags to avoid significant deviations in batch workload from the target due to inaccurate predictions. The quality warning threshold is typically set to 30% to 50% of the median within the bucket; in this embodiment, it is set to 40%.

[0080] The first model parameter of the updated compilation cost prediction model may also include grouping the shader compilation tasks by shader type or target platform, and updating the first model parameter of the corresponding group respectively.

[0081] For example, the vertex shader, fragment shader, and compute shader each maintain independent bucketed model parameters; for different target platforms, Vulkan and Metal each maintain independent correction coefficients. This grouped update enables the model to adapt to differences in compilation characteristics between different shader types and target platforms.

[0082] The grouping and updating by shader type or target platform specifically involves maintaining an independent set of bucket median model parameters for each combination of shader stage and target platform.

[0083] For example, the model parameters of the vertex shader on the Vulkan platform are independent of those on the Metal platform, and the model parameters of the fragment shader on the Vulkan platform are also independent of those of the vertex shader. This inter-group parameter isolation ensures that differences in compilation time between shader types with different compilation characteristics and target platforms with different optimization strategies do not interfere with each other.

[0084] During the cold start phase, when the cumulative number of samples for a particular shader stage combined with the target platform is insufficient, the system employs a hierarchical backoff strategy: it prioritizes using the aggregated statistical data of the same shader stage across all platforms as the backoff prediction value; if the aggregated data is still insufficient, it uses the global median. This hierarchical backoff strategy enables the system to provide reasonable initial predictions even with sparse combinations, while ensuring that combinations with sufficient samples are not affected by sparse combinations.

[0085] The grouping update also supports dynamic adjustment of grouping granularity. When the cumulative number of samples in a fine-grained group (such as "Fragment Shader on Metal Platform") remains below 30% of the window length, the system automatically merges it into the next higher-level coarse-grained group (such as "Fragment Shader on All Platforms"), reducing parameter maintenance overhead. When the cumulative number of samples recovers to more than 60% of the window length, the fine-grained grouping is re-enabled.

[0086] For example, the first estimated workload for the vertex shader task is 4.83 seconds, the first normalized compilation cost is 4.67 seconds, and the first bias is 4.67 minus 4.83, which is -0.16 seconds, indicating that the model slightly overestimates the task. This normalized compilation cost is inserted into a sliding window of the "vertex shader, token count 15000 to 20000" bucket, and the sliding median is updated from 4.2 seconds to 4.18 seconds. For the static mesh version, the first estimated workload is 1.05 seconds, the first normalized compilation cost is 1.59 seconds, and the first bias is positive 0.54 seconds, indicating that the model underestimates the task. After this normalized compilation cost is inserted into the corresponding bucket, the sliding median is adjusted upwards accordingly.

[0087] The compilation cost prediction model experiences a cold start phase during the initial system operation. During this phase, the sliding windows of each bucket have not yet accumulated sufficient samples, resulting in low confidence levels for the median statistic. The system handles the cold start phase as follows: when the cumulative sample count of a bucket is less than 50% of the window length, the first estimated workload is taken as the global median for that shader stage as a backoff estimate, and a cold start flag is added to the task. Shader compilation tasks marked with a cold start flag do not participate in workload balancing calculations during batch aggregation; instead, they are added to existing batches as additional components.

[0088] For example, when the cumulative sample count of the "vertex shader, token count 15000 to 20000" bucket is 12, which is less than 50% of the window length 50 (i.e., 25), the system takes the global median of the vertex shader (2.8 seconds) as the first estimated workload for new tasks falling into this bucket. When the cumulative sample count of this bucket reaches 25, the system switches to using the sliding median within the bucket as the prediction value, and the cold start state ends. The applicable boundary of the compilation cost prediction model is: when the toolchain version of the remote compilation node undergoes a major version upgrade, the historical compilation time and the actual time of the new version compiler may show a systematic deviation. At this time, the system triggers a model reset by detecting that the first deviation value of multiple consecutive tasks exceeds a preset offset threshold in the same direction, clears the sliding window of the affected bucket, and re-enters the cold start process. The preset offset threshold is usually in the range of 30% to 50% of the estimated workload; in this embodiment, it is taken as 40%.

[0089] like Figure 3 As shown, with the increase in the number of build rounds, the compilation cost prediction model undergoes online calibration by continuously receiving actual compilation feedback. The mean absolute value of the prediction error gradually decreases and tends to converge with the number of calibration rounds. Figure 4 As shown, the prediction stability after normalization is significantly improved compared to the actual compilation time when using the unnormalized method, and the coefficient of variation for cross-node prediction is reduced.

[0090] After obtaining the first actual compilation time for each shader compilation task, for shader compilation tasks with cache hits, the system marks them as CacheHit exclusion markers (i.e., the first non-training sample) and excludes the first non-training sample from the input of the first model parameters of the updated compilation cost prediction model. A cache hit means that the compilation result of the task already exists in the content addressing cache, and the remote compilation node directly returns the cached result without actually performing compilation. The time returned is the cache lookup time, not the compiler's working time. If such data is included in model training, extremely short cache lookup times will be misjudged as extremely fast compiler completion, thus seriously polluting the model parameters.

[0091] S600, based on the first estimated workload, aggregate multiple shader compilation tasks into a remote compilation batch that satisfies the range of the first target workload.

[0092] After determining the initial estimated workload for each shader compilation task, the local build control node aggregates multiple compatible shader compilation tasks into a remote compilation batch based on their estimated compilation workload. The compatibility criteria include identical shader formats, identical target platforms, identical toolchain versions, and compatible remote execution environments.

[0093] The aggregation uses TargetBatchCost (i.e., the first target workload) as the batch granularity control parameter. The sum of the first estimated workloads of all shader compilation tasks within each remote compilation batch should satisfy the first target workload range. The first target workload range is typically between 0.8 and 1.2 times the first target workload; in this embodiment, the first target workload is taken as 30 seconds.

[0094] The aggregation adopts a greedy minimum load strategy, that is, the shader compilation tasks to be aggregated are sorted from largest to smallest according to the first expected workload, and the current task is placed into the batch with the smallest cumulative workload in turn, until the cumulative workload of the batch reaches the lower bound of the first target workload, or all tasks have been allocated.

[0095] For example, the current shader compilation task queue contains 12 tasks with estimated workloads of 8.2 seconds, 7.5 seconds, 6.8 seconds, 5.9 seconds, 5.1 seconds, 4.83 seconds, 4.2 seconds, 3.6 seconds, 2.8 seconds, 2.1 seconds, 1.5 seconds, and 1.05 seconds, respectively, for a total workload of 53.58 seconds. Based on a target workload of 30 seconds, the system aggregates these 12 tasks into two remote compilation batches: Batch 1 contains 6 tasks with estimated workloads of 8.2 seconds, 6.8 seconds, 5.1 seconds, 4.83 seconds, 3.6 seconds, and 2.1 seconds, for a total workload of 30.63 seconds; Batch 2 contains 6 tasks with estimated workloads of 7.5 seconds, 5.9 seconds, 4.2 seconds, 2.8 seconds, 1.5 seconds, and 1.05 seconds, for a total workload of 22.95 seconds. The workload difference between the two batches is significantly more balanced compared to allocating tasks with a fixed number.

[0096] The boundary handling of the greedy minimum load aggregation strategy includes the following scenarios: When the first estimated workload of a single task in the aggregation queue exceeds the first target workload, the task is grouped into a separate batch, and the batch workload is the first estimated workload of that task. When the sum of the first estimated workloads of all tasks in the aggregation queue is less than the lower bound of the first target workload, all tasks are aggregated into one batch, and no new tasks are waited for. As new shader compilation tasks continue to arrive, the system performs batch aggregation at a fixed aggregation trigger interval, which is typically between 100 milliseconds and 500 milliseconds; in this embodiment, it is 200 milliseconds. Each time a trigger occurs, the system performs a complete greedy minimum load aggregation on all tasks to be dispatched in the current aggregation buffer, submitting batches that meet the first target workload range to the remote compilation node, while tasks that do not meet the target workload range are kept in the buffer awaiting the next trigger.

[0097] For example, the aggregation buffer contains three tasks with initial estimated workloads of 12.5 seconds, 9.8 seconds, and 6.2 seconds, respectively, totaling 28.5 seconds. The initial target workload is 30 seconds, with a lower bound of 24 seconds. Since 28.5 seconds falls within the target range of 24 to 36 seconds, the system aggregates these three tasks into a batch and submits it immediately. If, when aggregation is triggered, the buffer contains only two tasks (12.5 seconds and 9.8 seconds), with a total of 22.3 seconds less than the lower bound of 24 seconds, then these tasks are retained and will continue to accumulate upon the next trigger.

[0098] S700: Based on the first remaining workload of tasks that have not yet been dispatched, the first workload rate, and the first consumption capacity of the remote compilation node, determine whether there is insufficient parallelism in the remote compilation batch. If there is insufficient parallelism, adjust the first target workload and re-aggregate the shader compilation tasks that have not yet been dispatched.

[0099] During the game engine build process, shader compilation tasks are continuously generated and consumed by remote compilation nodes. As the build enters the later stages, the total number of tasks to be compiled gradually decreases. If the large batch granularity of the early stage is maintained, the remaining tasks may not be able to form a sufficient number of batches to fully utilize the parallel capabilities of all remote compilation nodes, resulting in some nodes being idle and creating a structural long tail.

[0100] The system continuously monitors the following three metrics: RemainingWork (i.e., the first remaining workload), which is the sum of the first estimated workloads of all undispatched shader compilation tasks; ArrivalWorkRate (i.e., the first workload rate), which is the sum of the estimated workloads of newly arriving shader compilation tasks per unit time; and DrainWorkRate (i.e., the first consumption capacity) of remote compilation nodes, which is the sum of the normalized compilation workloads completed by remote compilation nodes per unit time.

[0101] The system estimates the number of equivalent batches that can be formed in the future based on the ratio of the first remaining workload to the current first target workload. When the number of equivalent batches is less than the effective remote parallelism multiplied by the parallelism margin parameter, it is determined that there is insufficient parallelism. The parallelism margin parameter is usually in the range of 1.2 to 2.0, and is taken as 1.5 in this embodiment.

[0102] For example, if the current remaining workload is 45 seconds and the target workload is 30 seconds, the number of equivalent batches that can be formed is 45 divided by 30, which is 1.5. The effective remote parallelism is 4 nodes, multiplied by the parallelism margin parameter 1.5, resulting in a threshold of 6.0. Since 1.5 is less than 6.0, the system determines that there is insufficient parallelism.

[0103] When insufficient parallelism exists, the system reduces the first target workload. When adjusting the first target workload, adjustment oscillations are suppressed by setting a first minimum target value, a first maximum target value, and a first hysteresis interval. The first minimum target value is the maximum estimated compilation workload for a single task, ensuring that each batch contains at least one task; the first maximum target value is twice the initial first target workload, preventing excessively large batches from causing excessively long execution times for single nodes. The first hysteresis interval is defined as follows: actual adjustment is only performed when the target adjustment ratio deviates from 1.0 by more than half the hysteresis width, avoiding frequent reorganizations triggered by minor fluctuations.

[0104] The system continuously monitors three indicators, including: The first remaining workload is obtained by summing the first estimated workload of all tasks in the task queue to be dispatched maintained by the local build control node, and is updated in real time whenever a new task enters the queue or an existing task is dispatched to a remote node.

[0105] The first workload rate is calculated using a sliding time window statistical method. The length of the sliding time window is usually in the range of 5 to 30 seconds. In this embodiment, it is 10 seconds. The statistical value is the sum of the first estimated workload of newly arrived tasks within the window divided by the window duration.

[0106] The first consumption capacity is also statistically analyzed using a sliding time window. The statistical value is the sum of the first standardized compilation costs of the completed tasks returned by remote compilation nodes within the window, divided by the window duration.

[0107] The effective remote parallelism is calculated as follows: the number of remote compilation nodes that actually received and started executing batches within the most recent monitoring window is counted. When a remote compilation node fails to return any completion result within two consecutive monitoring windows, it is removed from the effective parallelism count until the node returns a completion result again.

[0108] The adjustment of the first target workload includes: Calculate the target adjustment ratio :

[0109] In the formula, For effective long-range parallelism, This is the parallelism margin parameter. This represents the number of equivalent batches that can be formed at present. When the target adjustment ratio is greater than 1.0 plus the hysteresis half-width, the new target value is obtained by multiplying the first target workload by the reciprocal of the target adjustment ratio. Here, the hysteresis half-width... , Here, the width of the first hysteresis interval is 0.1, and the half-width of the hysteresis interval is 0.05. The constraint adjustment rule for the hysteresis interval is as follows:

[0110] In the formula, The adjusted first target workload, The initial target workload before the adjustment. Adjust the ratio to the target. This is a hysteresis half-width. That is, when... When the number of equivalent batches that can be formed at the current batch granularity is insufficient to fully utilize the remote parallel capability, the first target workload is multiplied by the reciprocal of the target adjustment ratio. At this point, the first target workload decreases, allowing the remaining tasks to form a larger number of smaller batches. When the current batch granularity is too fine, resulting in an excessive number of batches and a high proportion of remote scheduling overhead, the first target workload is multiplied by the reciprocal of the target adjustment ratio. This increases the first target workload, causing subsequent tasks to be aggregated into smaller, larger batches to improve throughput efficiency. When the particle size of the current batch is within a reasonable range, no adjustment will be performed.

[0111] For example, the target adjustment ratio is the effective remote parallelism of 4 multiplied by the parallelism margin parameter of 1.5, divided by the current number of equivalent batches that can be formed, which is 4. This is greater than 1.0, plus the hysteresis half-width. Therefore, the system multiplies the first target workload of 30 seconds by the reciprocal of the target adjustment ratio to obtain a new target value of 7.5 seconds, and only re-executes aggregation on shader compilation tasks that have not yet been dispatched. Batches already submitted to remote compilation nodes and currently being executed are unaffected. After the adjustment, the first remaining workload of 45 seconds can form 6 batches, which can fully utilize the parallelism of 4 remote compilation nodes. In this example, the maximum expected compilation workload of the remaining tasks is less than 7.5 seconds, so the new target value does not trigger the truncation constraint of the first minimum target value.

[0112] After adjusting the first target workload, the system also performs an adjustment effect verification, that is, within two monitoring periods after the adjustment, it calculates the ratio of the actual number of batches formed to the effective remote parallelism. If this ratio is still lower than 80% of the parallelism margin parameter, the system further reduces the first target workload until it reaches the first minimum target value. When the build enters a new round of task peak, the first remaining workload rebounds, and the system automatically detects that the number of equivalent batches that can be formed has exceeded 1.5 times the effective remote parallelism multiplied by the parallelism margin parameter, triggering the first target workload adjustment recovery process. The recovery is also constrained by the first hysteresis interval, and the actual adjustment is only executed when the target adjustment ratio deviates from 1.0 by more than half the hysteresis width, preventing frequent adjustments when the task rate fluctuates. The adjustment and recovery process is executed automatically throughout, without manual intervention, enabling the system to adaptively balance batch granularity and parallel utilization throughout the entire build lifecycle.

[0113] like Figure 5 As shown in the illustration, this application also provides an online cost model calibration system based on shader compilation feedback. The system includes a data acquisition module, a feature extraction module, a prediction module, an execution module, a normalization module, and a calibration module.

[0114] The acquisition module is used to acquire the first program corresponding to multiple shader compilation tasks that have completed preprocessing. The acquisition module interfaces with the shader preprocessing pipeline of the game engine, and extracts the complete text content of the final preprocessed program after the shader compilation task completes macro expansion, conditional compilation, and include preprocessing.

[0115] The feature extraction module is used to extract first program structure features representing compilation complexity from the first program. The feature extraction module has a built-in lightweight lexical scanner that performs a single scan on the preprocessed program and outputs a feature vector containing lexical complexity features and control flow complexity features.

[0116] The prediction module is used to determine the first estimated workload of each shader compilation task based on the first program structure characteristics. The prediction module maintains the current parameters of the compilation cost prediction model, receives feature vectors as input, and outputs the estimated compilation time at the standardized reference node.

[0117] The execution module is used to send the shader compilation tasks to remote compilation nodes for execution and to obtain the first actual compilation time of each shader compilation task. The execution module communicates with the remote compilation nodes through a distributed build interface, serializes the compilation tasks and sends them to the target node, and receives feedback on the compilation results and actual compilation time.

[0118] The normalization module is used to normalize the first actual compilation time based on the first performance parameter of the remote compilation node executing the shader compilation task, thereby obtaining a first standardized compilation cost. The normalization module maintains a first performance factor for each remote compilation node and corrects the first performance factor according to accumulated compilation feedback data at a preset update cycle.

[0119] The calibration module is used to update the first model parameters of the compilation cost prediction model based on the first deviation between the first standardized compilation cost and the corresponding first estimated workload. The calibration module receives the standardized compilation cost output by the normalization module, calculates the deviation between it and the first estimated workload output by the prediction module, and feeds back the deviation signal to the model parameter storage of the prediction module to complete the online calibration closed loop.

[0120] In a distributed deployment scenario, the acquisition module, feature extraction module, prediction module, calibration module, and the communication component of the execution module are all deployed on the local build control node, the remote compilation executor is deployed on multiple remote compilation nodes, and the normalization module is deployed on the local build control node and obtains the actual compilation time returned by each remote compilation node through the network.

[0121] In one implementation, the acquisition module, feature extraction module, prediction module, normalization module, and calibration module are implemented as software modules, with each module's function performed by the processor of the locally built control node executing the corresponding program instructions stored in memory. The execution module includes a task serialization component and a network communication component. The task serialization component packages the shader compilation task and its associated preprocessed program into a transmittable binary format, while the network communication component maintains communication channels with each remote compilation node via a TCP long connection or RPC protocol.

[0122] The prediction module internally maintains a model parameter storage area, which contains sliding window data for each bucket, the current median value, and correction coefficients. The calibration module updates the model parameters by writing to the model parameter storage area, and the prediction module reads the latest parameters to perform prediction when the next prediction request arrives. The two modules are asynchronously decoupled through a shared storage area.

[0123] The execution module further includes a batch aggregation scheduling submodule. This submodule receives the first estimated workload of each task output by the prediction module, executes a greedy minimum load aggregation strategy according to the first target workload, and generates a remote compilation batch. The batch aggregation scheduling submodule also receives real-time monitoring data on the first remaining workload, the first workload rate, and the first consumption capacity. When insufficient parallelism is detected, the first target workload is adjusted, and aggregation is re-executed for tasks that have not yet been dispatched. The batch aggregation scheduling submodule operates in an event-driven manner, triggering an aggregation operation when the aggregation trigger timer expires or the parallelism monitoring index exceeds the limit.

[0124] In a distributed deployment scenario, when the processor load of the local build control node is high, the feature extraction module and prediction module can be deployed on independent coprocessing threads, running asynchronously with the main control thread. The main control thread is responsible for task scheduling and network communication, while the coprocessing thread is responsible for feature extraction and prediction calculations. Data is transferred between the two via a lock-free queue. This asynchronous deployment method avoids the delays in feature extraction and prediction calculations from blocking the compilation task dispatch process.

[0125] The online cost model calibration system supports the dynamic addition and removal of remote compilation nodes. When a new remote compilation node joins the cluster, the normalization module initializes its default first performance factor to 1.0, and automatically corrects it to the actual value after accumulating sufficient samples. When a remote compilation node leaves the cluster, the system removes it from the effective parallelism statistics, and any unfinished tasks assigned to that node that time out are re-added to the dispatch queue. This dynamic node management enables the system to adaptively utilize elastic computing resources in a cloud compilation environment without requiring manual reconfiguration of model parameters or aggregation strategies.

[0126] The normalization module internally maintains a node performance parameter table, which uses the remote compilation node identifier as an index and stores the current first performance factor, cumulative sample count, and most recent update timestamp of each node. When the cumulative sample count of a node reaches a preset update cycle, the normalization module recalculates its first performance factor based on the node's recent sample data and updates the parameter table.

[0127] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements all the steps of the aforementioned online cost model calibration method based on shader compilation feedback. The computer-readable storage medium can be, for example, a non-transient storage medium such as a read-only memory, random access memory, flash memory, hard disk, or optical disk. In one embodiment, the computer-readable storage medium is a local solid-state drive of a server, and the computer program is deployed as a build system plugin, loaded into memory when the build system starts, and executed by the processor of the build control node.

[0128] In another embodiment, the compilation cost prediction model can also be implemented using an online statistical model. This involves maintaining an exponential moving average and standard deviation for each shader stage and feature bucket, and updating the mean and variance at a fixed learning rate based on the first standardized compilation cost of newly arriving samples. The first estimated workload is the result of calculating the bucket mean and feature correction coefficients. This online statistical model offers a faster parameter update response time compared to the bucket median model, making it suitable for build environments with frequent toolchain version changes.

[0129] In another embodiment, the compilation cost prediction model can also be implemented using a gradient boosting tree-based regression model. This model uses all dimensions of the first program structure features as input features and the historical first standardized compilation cost as the training label. New samples are added and the model is updated after each build cycle through incremental learning. The gradient boosting tree model can capture the non-linear interactions between features and is suitable for projects with significant differences in shader variant structures.

[0130] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit described above can be implemented in hardware.

[0131] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An online cost model calibration method based on shader compilation feedback, characterized in that, include: Obtain the first program corresponding to multiple preprocessed shader compilation tasks; Extract first program structure features that characterize compilation complexity from the first program, and determine the first estimated workload of each shader compilation task based on the first program structure features; The shader compilation tasks are sent to a remote compilation node for execution to obtain the first actual compilation time of each shader compilation task; The first actual compilation time is normalized based on the first performance parameter of the remote compilation node that performs the shader compilation task to obtain the first standardized compilation cost. The first model parameters of the compilation cost prediction model are updated based on the first deviation between the first standardized compilation cost and the corresponding first estimated workload.

2. The method according to claim 1, characterized in that, After determining the initial estimated workload for each shader compilation task, the following is also included: Based on the first estimated workload, multiple shader compilation tasks are aggregated into remote compilation batches, wherein the remote compilation batches meet the range of the first target workload.

3. The method according to claim 2, characterized in that, Also includes: Based on the first remaining workload before tasks are dispatched, the first workload rate, and the first consumption capacity of the remote compilation node, determine whether there is insufficient parallelism in the remote compilation batch; If the parallelism is insufficient, adjust the first target workload and re-aggregate the shader compilation tasks that have not yet been dispatched.

4. The method according to claim 1, characterized in that, The first performance parameter includes the first performance factor of each remote compilation node, and the normalization process includes updating the first performance factor according to a preset period or the cumulative number of samples.

5. The method according to claim 1, characterized in that, The first model parameters for updating the compiler cost prediction model include: The shader compilation tasks are grouped according to shader type or target platform, and the first model parameters of the corresponding group are updated respectively.

6. The method according to claim 1, characterized in that, After obtaining the first actual compilation time for each shader compilation task, the following is also included: The shader compilation task that hits the cache is marked as the first non-training sample, and the first non-training sample is excluded from the input of the first model parameter of the updated compilation cost prediction model.

7. The method according to claim 1, characterized in that, The first program structure features include lexical complexity features and control flow complexity features.

8. The method according to claim 3, characterized in that, When adjusting the first target workload, the oscillation is suppressed by setting a first minimum target value, a first maximum target value, and a first hysteresis interval.

9. An online cost model calibration system based on shader compilation feedback, characterized in that, include: The acquisition module is used to acquire the first program corresponding to multiple preprocessed shader compilation tasks; The feature extraction module is used to extract first program structure features that characterize compilation complexity from the first program; The prediction module is used to determine the first estimated workload of each shader compilation task based on the first program structure characteristics. An execution module is used to send the shader compilation tasks to a remote compilation node for execution and to obtain the first actual compilation time of each shader compilation task; The normalization module is used to normalize the first actual compilation time based on the first performance parameter of the remote compilation node that performs the shader compilation task, so as to obtain the first normalized compilation cost. The calibration module is used to update the first model parameters of the compilation cost prediction model based on the first deviation between the first standardized compilation cost and the corresponding first estimated workload.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

Images