Parameter tuning method and device of calculation engine and storage medium

Abstract

The invention discloses a parameter tuning method and device for a calculation engine and a storage medium, and the method comprises the steps: carrying out the feature extraction of an initial configuration parameter of the calculation engine, and obtaining a parameter feature of the initial configuration parameter; performing data prediction processing according to the parameter characteristics to obtain predicted operation data when the calculation engine operates according to the initial configuration parameters; and determining a corresponding target configuration parameter according to the predicted operation data. According to the scheme, the optimization efficiency of the calculation engine can be improved.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a method, device, and storage medium for parameter tuning of a computing engine. Background Technology

[0002] In recent years, the rapid development of big data technology has become one of the most important technologies in the scientific community. Traditional data processing methods can no longer meet the needs of processing massive amounts of data. Against this backdrop, big data computing engines have emerged and have long played a crucial role in the field of big data technology.

[0003] The core technologies of big data computing engines include distributed computing and stream processing frameworks. Common computing engines include MapReduce, Apache Spark, and Apache Flink, which can support complex data processing tasks. Taking Spark as an example, it supports multiple data processing modes, including batch processing, stream processing, and interactive queries, and is widely used in data engineering, data science, and machine learning scenarios.

[0004] The performance of a computing engine largely depends on the selection of its configuration parameters. Incorrect configuration parameters can lead to poor performance, often manifested as long runtimes, resource contention, and low resource utilization. However, in current computing engine applications, tuning configuration parameters is very difficult, time-consuming, and labor-intensive, and it is also difficult to extend to other tasks on the data platform. Summary of the Invention

[0005] This application provides at least one method, apparatus, device, and computer-readable storage medium for parameter tuning of a computing engine.

[0006] The first aspect of this application provides a parameter tuning method for a computing engine, comprising: performing feature extraction processing on the initial configuration parameters of the computing engine to obtain parameter features of the initial configuration parameters; performing data prediction processing based on the parameter features to obtain predicted running data of the computing engine running with the initial configuration parameters; and determining corresponding target configuration parameters based on the predicted running data.

[0007] In one embodiment, the step of performing feature extraction processing on the initial configuration parameters of the computing engine to obtain the parameter features of the initial configuration parameters includes: performing word vector conversion processing on the initial configuration parameters according to a preset data format to obtain the configuration parameter vector corresponding to the initial configuration parameters; and inputting the configuration parameter vector into a pre-trained natural language model for feature extraction processing to obtain the parameter features output by the natural language model.

[0008] In one embodiment, the step of performing data prediction processing based on the parameter features to obtain the predicted running data of the computing engine running with the initial configuration parameters includes: performing dimensionality reduction processing on the parameter features to obtain low-dimensional parameter features; inputting the low-dimensional parameter features into a pre-trained running data prediction head for prediction processing to obtain the predicted running data output by the running data prediction head.

[0009] In one embodiment, the runtime data prediction head includes a time prediction head, a memory prediction head, and a security prediction head. The predicted runtime data includes predicted runtime, predicted memory usage, and predicted security probability. The step of inputting the low-dimensional parameter features into the pre-trained runtime data prediction head for prediction processing to obtain the predicted runtime data output by the runtime data prediction head includes: inputting the low-dimensional parameter features into the time prediction head, the memory prediction head, and the security prediction head respectively to obtain the predicted runtime output by the time prediction head, the predicted memory usage output by the memory prediction head, and the predicted security probability output by the security prediction head.

[0010] In one embodiment, after inputting the low-dimensional parameter features into a pre-trained runtime data prediction head for prediction processing to obtain the predicted runtime data output by the runtime data prediction head, the method further includes: obtaining a current security threshold; comparing the predicted security probability with the current security threshold to obtain a security comparison result; and determining the corresponding target configuration parameter based on the predicted runtime data, including: in response to the security comparison result indicating that the predicted security probability is less than or equal to the current security threshold, determining the target configuration parameter based on the predicted runtime data.

[0011] In one embodiment, obtaining the current security threshold includes: obtaining a historical security threshold, wherein the historical security threshold is earlier in time than the current security threshold; and performing a weighted summation of the historical security threshold and the predicted security probability to obtain the current security threshold.

[0012] In one embodiment, before performing feature extraction processing on the initial configuration parameters of the computing engine, the method further includes: inputting the acquired configuration parameter samples into the initial model for feature extraction processing to obtain sample features of the configuration parameter samples; inputting each sample feature into the corresponding running data prediction head for data prediction processing to obtain the sample prediction loss output by each running data prediction head; determining the target prediction loss based on each sample prediction loss; and adjusting the initial model based on the target prediction loss to obtain the target model.

[0013] In one embodiment, the target model includes an encoder and a data prediction head, and the method includes: inputting the initial configuration parameters into the target model; performing feature extraction processing on the initial configuration parameters according to the encoder to obtain the parameter features; and performing data prediction processing on the parameter features according to the data prediction head to obtain the prediction running data.

[0014] A second aspect of this application provides a parameter tuning apparatus for a computing engine, comprising: a feature extraction module for performing feature extraction processing on initial configuration parameters of the computing engine to obtain parameter features of the initial configuration parameters; a data prediction module for performing data prediction processing based on the parameter features to obtain predicted running data of the computing engine running with the initial configuration parameters; and a parameter determination module for determining corresponding target configuration parameters based on the predicted running data.

[0015] A third aspect of this application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the parameter tuning method of the computing engine described above.

[0016] The fourth aspect of this application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement the parameter tuning method of the aforementioned computing engine.

[0017] The proposed solution uses a multi-task regression Transformer encoder to learn the complex nonlinear relationship between Spark configuration parameters and runtime data, thereby predicting runtime data under different configuration parameters. Therefore, this solution can input the initial configuration parameters of the computing engine into a trained neural network model for feature extraction to obtain the parameter features of the initial configuration parameters; then, based on the prediction head in the neural network model, data prediction processing is performed on the parameter features to obtain the predicted runtime data of the computing engine running with the initial configuration parameters; finally, a regression task is executed based on the predicted runtime data to determine the target configuration parameters that the predicted runtime data should correspond to, i.e., the recommended configuration parameters after optimization based on the initial configuration parameters. This improves the accuracy and efficiency of configuration parameter optimization.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0020] Figure 1This is a flowchart illustrating an exemplary embodiment of the parameter tuning method for the computing engine of this application;

[0021] Figure 2 This is a schematic diagram of the structure of an exemplary DMTE-Spark model in the parameter tuning method of the computing engine of this application;

[0022] Figure 3 This is an exemplary flowchart of the prediction data detection process in the parameter tuning method of the computing engine of this application;

[0023] Figure 4 This is a block diagram illustrating a parameter tuning apparatus for a computing engine, as shown in an exemplary embodiment of this application.

[0024] Figure 5 This is a schematic diagram of the structure of an embodiment of the electronic device of this application;

[0025] Figure 6 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0026] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0027] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0028] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0029] To facilitate understanding, some applicable scenarios related to this application are illustrated with examples.

[0030] In the field of big data technology, a computing engine is the underlying framework or component that drives the execution of computing tasks. It encompasses core functions such as data processing, resource management, and parallel computing, providing computing power for various applications. There are various existing big data computing engines, such as MapReduce, Storm, Spark, and Spark Streaming. This application primarily uses Spark (Apache Spark) as an example for illustration.

[0031] Apache Spark is an open-source distributed computing framework designed for large-scale data processing. It provides highly efficient in-memory computing capabilities, enabling data processing speeds far exceeding traditional disk-based computing frameworks. Spark supports various data processing modes, including batch processing, stream processing, and interactive queries, and is widely used in data engineering, data science, and machine learning.

[0032] The performance of a Spark job largely depends on the selection of its configuration parameters. Incorrect configuration can lead to poor performance, such as long runtime, resource contention, and low resource utilization. Scheduled tasks account for the vast majority of Spark tasks, and for these scheduled tasks, tuning configuration parameters can effectively improve their running efficiency.

[0033] However, during parameter configuration, users need to determine many key parameters to ensure that scheduled tasks achieve near-optimal performance. Manually adjusting Spark task configuration parameters is extremely difficult, typically time-consuming and laborious, and difficult to scale to other tasks on the data platform. Furthermore, Spark configuration parameters can have hundreds of dimensions (such as Spark.executor.instances, Spark.driver.memory, etc.). As the dimensionality of configuration parameters increases, the number of samples required for model learning also grows exponentially, significantly negatively impacting the parameter tuning process.

[0034] To address this issue, this application proposes a Spark intelligent tuning system (hereinafter referred to as the DMTE-Spark model) that integrates a multi-task Transformer encoder and an EMA (Exponential Moving Average) dynamic safety threshold. This system learns the complex nonlinear relationship between Spark configuration parameters and runtime data (such as runtime and memory usage) through a multi-task regression Transformer encoder, thereby predicting runtime and memory usage under different configurations. The EMA safety threshold mechanism detects and prompts the user whether the entered configuration parameters are safe. Furthermore, the system can adjust Spark configurations based on user-issued parameter adjustment commands based on these prediction results to optimize task performance. This application's method, through the combination of deep learning and dynamic thresholds, solves problems such as static threshold dependence, human experience dominance, and multi-objective optimization conflicts in Spark tuning, providing an intelligent and adaptive solution for Spark tuning, improving the optimization efficiency of computing engine configuration parameters, and enhancing the execution efficiency and stability of distributed computing tasks.

[0035] Please see Figure 1 , Figure 1 This is a flowchart illustrating an exemplary embodiment of the parameter tuning method for the computing engine of this application. Specifically, it may include the following steps:

[0036] Step S110: Perform feature extraction processing on the initial configuration parameters of the computing engine to obtain the parameter features of the initial configuration parameters.

[0037] Initial configuration parameters refer to Spark configuration parameters before tuning. These parameters can be obtained from user input or from other connected devices; the method of acquisition is not limited here. Parameter features are the results of feature extraction from the initial configuration parameters. The feature extraction process can be implemented based on the DMTE-Spark model described in the previous example.

[0038] For example, this application mainly uses some configuration parameters (configuration items) that are important to Spark jobs for illustration, but does not limit the configuration parameters that can be tuned in this application. For example, the configuration parameters can be shown in the following table:

[0039]

[0040] After obtaining the configuration parameters, these parameters can be input into the pre-trained DMTE-Spark model to extract features from these configuration parameters and obtain parameter features.

[0041] The DMTE-Spark model in this application may include at least a learning module and a multi-task regression module. The learning module may be implemented based on a Transformer encoder. The Transformer is a deep learning model based on a self-attention mechanism, and it consists of three modules: encoder, decoder, and output layer. Therefore, this application may use the encoder module from the Transformer for feature learning.

[0042] In this application, in addition to using the Transformer encoder to extract parametric features, the BERT (Bidirectional Encoder Representation from Transformer) model can also be used to extract parametric features. The BERT model is a pre-trained natural language model whose network structure references the encoder part of the Transformer. Therefore, in this application, BERT can also achieve the effect of the Transformer encoder. The main difference between the two lies in the data processing part. BERT proposes a Segment Embedding layer, which has a slightly different processing method from the Transformer's Input Embedding layer. The Segment Embedding layer can be used to segment the information of each word in a sentence. Taking "I am a Lakersfan" as an example, the Segment Embedding layer will segment this sentence into "[CLS]I[SEG]am[SEG]a[SEG]Lakers[SEG]fan[SEG]".

[0043] Since BERT has already been pre-trained on large-scale natural language datasets and achieved excellent results, this application can use a pre-trained BERT model to fine-tune Spark configuration parameters to adapt it to this task. For example, refer to... Figure 2 As shown, Figure 2 This is a schematic diagram of an exemplary DMTE-Spark model in the parameter tuning method of the computing engine of this application. For a description of this structure, please refer to the examples in the preceding and following embodiments of this application; further details will not be provided here.

[0044] Step S120: Perform data prediction processing based on parameter characteristics to obtain the predicted running data of the computing engine running with the initial configuration parameters.

[0045] The multi-task regression module of the DMTE-Spark model in this application is described in conjunction with the aforementioned steps. The multi-task regression module includes a running data prediction head, which can be used to predict data based on received parameter features, obtaining the running data (i.e., predicted running data) of the running data prediction calculation engine running with the initial configuration parameters.

[0046] For example, these predicted runtime data may include, but are not limited to, continuous metrics such as predicted Spark job runtime (execution time), memory usage, and safety probability.

[0047] Step S130: Determine the corresponding target configuration parameters based on the predicted running data.

[0048] Based on the steps described above, in the multi-task regression module of the DMTE-Spark model of this application, after obtaining the prediction running data output by the prediction head, the corresponding target configuration parameters can be determined based on the prediction running data.

[0049] It should be noted that if the predicted running data meets the preset running requirements, the initial configuration parameters can be used as the target configuration parameters, or the initial configuration parameters can be adjusted before being used as the target configuration parameters; there is no limitation here. If the predicted running data does not meet the preset running requirements, the initial configuration parameters need to be adjusted before being used as the target configuration parameters, or the initial configuration parameters need to be adjusted and then re-entered into the DMTE-Spark model to perform the above example operations; this will not be elaborated here.

[0050] For example, the predicted running data can be displayed to the user so that the user can analyze the predicted running data and determine whether it meets the requirements; if it meets the requirements, the user's confirmation instruction can be received so that Spark runs according to the initial configuration parameters; if it does not meet the requirements, the user's adjustment instruction can be received so that Spark runs according to the target configuration parameters corresponding to the adjustment instruction.

[0051] Another example is that the DMTE-Spark model itself analyzes the prediction runtime data to determine whether the data meets the requirements. If it does, the initial configuration parameters can be used as the target configuration parameters for execution; if it does not meet the requirements, the DMTE-Spark model can output recommended configuration parameters as the target configuration parameters for execution. These recommended configuration parameters can be preset or regenerated after each data prediction. For example, if the current prediction memory usage is greater than a first memory threshold, the `Spark.driver.memory` configuration item can be increased based on the difference or ratio between the prediction memory usage and the first memory threshold; if the prediction memory usage is less than a second memory threshold, the `Spark.driver.memory` configuration item can be decreased based on the difference or ratio between the prediction memory usage and the second memory threshold, where the second memory threshold is less than the first memory threshold. Specifically, various parameter adjustment methods can be included (e.g., adjusting `Spark.executor.instances` and / or `Spark.sql.shuffle.partitions` based on the prediction runtime). These adjustment methods can be flexibly set according to the actual application scenario, and will not be elaborated here. It should be noted that in practical applications, configuration parameters usually cannot be adjusted indefinitely. Therefore, during the adjustment process, some adjustment limits can be set to restrict the adjustable range of the configuration parameters. If the adjusted configuration parameters exceed the adjustment limits, the adjusted configuration parameters can be considered invalid.

[0052] Furthermore, the DMTE-Spark model of this application can also predict the safety probability of Spark running based on the initial configuration parameters. Regardless of whether other predicted runtime data (e.g., predicted runtime and predicted memory usage) meet the requirements, if the predicted safety probability is less than the safety threshold, Spark is not allowed to run with the initial configuration parameters. This improves the stability of Spark operation by constraining high-risk configurations through predicted safety probabilities.

[0053] As can be seen, this application uses a multi-task regression Transformer encoder to learn the complex nonlinear relationship between Spark configuration parameters and runtime data, thereby predicting runtime data under different configuration parameters. Then, the initial configuration parameters of the computing engine can be input into a trained neural network model for feature extraction to obtain the parameter features of the initial configuration parameters. Next, data prediction processing is performed on these parameter features using the prediction head in the neural network model to obtain the predicted runtime data of the computing engine running with the initial configuration parameters. Finally, a regression task is executed based on the predicted runtime data to determine the target configuration parameters that should correspond to the predicted runtime data, i.e., the recommended configuration parameters after optimization based on the initial configuration parameters. This improves the accuracy and efficiency of configuration parameter optimization.

[0054] For ease of understanding, based on the above embodiments, this embodiment briefly describes the training process of the DMTE-Spark model of this application.

[0055] For example, a dataset can be obtained from historical Spark jobs. As shown in the initial configuration parameters of the above embodiment, a training set can be constructed from the configuration parameters of each Spark scheduled task and the runtime, memory usage, and other runtime data corresponding to those parameters. For ease of explanation, the configuration parameter Spark.executor.instances will be abbreviated as instances, Spark.driver.memory as memory, and Spark.sql.shuffle.partitions as partitions, as shown in the table below:

[0056]

[0057] Each row of data constitutes a training set of data samples. The sample labels for the learning objective can be extracted from the historical data of Spark jobs. "Execution_time" is the time label, in seconds; "Memory_usage" is the memory label, in gigabytes (GB); "Is_safe" is the safety label, marked based on historical execution results, with "1" defining safe and "0" defining unsafe. Therefore, by training the model using training sets including but not limited to those shown above, the trained DMTE-Spark model can predict the received configuration parameters and obtain the corresponding predicted execution data.

[0058] The Input+Segment Embedding layer is primarily used to process word vectors from the input Spark configuration parameters, including but not limited to segmentation and embedding. For example, it converts the configuration parameters into strings, segments them to obtain sub-tokens, and then adjusts the token sequence to obtain the token sequence X that the model can process. InputIDs and X AttentionMask Then X can be... InputIDs After being mapped to word vectors X by the Input Embedding layer embedding The output shape can be [batch_size, 512, 768], and can be flexibly set according to the actual application scenario; no limitation is made here. See the table below for examples:

[0059]

[0060] X embedding Overlay position code X pos For each X embedding The location information is marked. Its mathematical expression can be:

[0061] X pos =positionalEmbedding(X embedding )

[0062] X embedding =X embedding +X pos

[0063] At this point, the word vectorization of the configuration parameters is complete, which means that the parameter vectors of the configuration parameters can be obtained.

[0064] It should be noted that the training process of the DMTE-Spark model can include more steps, which can be seen in the following examples and will not be elaborated here.

[0065] Based on the above embodiments, this application embodiment exemplarily describes the steps of performing feature extraction processing on the initial configuration parameters of the computing engine to obtain the parameter features of the initial configuration parameters. Specifically, the method of this embodiment includes the following steps:

[0066] The initial configuration parameters are converted into word vectors according to the preset data format to obtain the configuration parameter vectors corresponding to the initial configuration parameters; the configuration parameter vectors are then input into a pre-trained natural language model for feature extraction to obtain the parameter features output by the natural language model.

[0067] Referring to the aforementioned embodiments, the Input+Segment Embedding layer in the DMTE-Spark model can be used to process word vectors from the initial configuration information of the input. For the initial configuration parameter X, X can be converted into a string format (e.g., "Spark.executor.instances:100", "Spark.driver.memory:8g"). Then, the string-formatted initial configuration parameter can be segmented into sub-tokens, and their sequence length can be set to a fixed value (e.g., 512). Any length exceeding the limit can be truncated, and any length below the limit can be padded with [PAD], and [CLS] and [SEP] delimiters are added for segmentation, converting it into a token sequence that the model can process. This sequence, after passing through the Input Embedding layer, can then be mapped to word vector X. embedding Unlike RNN models, Transformers cannot remember the order of all input sentences. Therefore, positional encoding can be used to assign a unique sequence number to each input sentence. Common positional encoding uses sine and cosine functions, which will not be elaborated upon here. Positional encoding assigns a unique sequence number to each X... embedding Get a position X pos Then, its positional information is superimposed on the original word vector to obtain a new word vector. Therefore, this application can also superimpose positional encoding X on word vectors. pos By labeling the positional information of each word vector, a configuration parameter vector can be obtained. This configuration parameter vector is then input into the pre-trained BERT Encoder for feature extraction, yielding the parameter features.

[0068] For example, the output dimensions of the BERT Encoder model can be shown in the following table:

[0069]

[0070] Understandably, in Spark tuning scenarios, there are complex non-linear dependencies between configuration parameters. Traditional linear models (such as logistic regression) struggle to explicitly model these high-dimensional representations. However, the Transformer-based multi-head attention mechanism, through adaptive weight allocation and context-aware representation learning, can effectively capture the dynamic relationships between parameters.

[0071] Since the original Transformer Encoder requires a large amount of data and complex training processes and hyperparameter tuning to achieve good performance, this application can choose to use BERT-BaseEncoder with pre-trained model weights loaded. By loading pre-trained model weights and fine-tuning them in conjunction with the configuration parameters of this task, the model can quickly learn and converge to the optimal effect, thereby improving the training efficiency and performance of the DMTE-Spark model.

[0072] Based on the above embodiments, this application embodiment describes the steps of performing data prediction processing based on parameter features to obtain predicted running data for the computing engine running with initial configuration parameters. Specifically, the method of this embodiment includes the following steps:

[0073] The parameter features are dimensionality reduced to obtain low-dimensional parameter features; these low-dimensional parameter features are then input into the pre-trained running data prediction head for prediction processing, resulting in the predicted running data output by the running data prediction head.

[0074] Referring to the foregoing embodiments, in the Spark tuning scenario of this application, the core objective of the regression module is to accurately predict continuous metrics such as Spark job runtime and memory usage from the context-aware features extracted by the encoder, while simultaneously satisfying safety constraints. This module adopts a multi-task joint prediction structure, and through the design of independent task heads for multiple tasks, it can achieve high-performance prediction and dynamic loss balance.

[0075] Since the original feature dimension of the encoder output is 768, directly inputting it into a separate task head could lead to an explosion in computational complexity. Therefore, to reduce model complexity, this application reduces the feature dimension from 768 to 128 through linear transformation of a fully connected MLP layer and the nonlinear activation function ReLU. This operation also benefits feature decoupling and enhances generalization ability.

[0076] Therefore, in the specific implementation process, the parameter features can be dimensionality reduced to obtain low-dimensional parameter features, as shown in the table below:

[0077]

[0078] Furthermore, X Outputs1 By inputting the corresponding running data prediction header for prediction processing, the predicted running data output by the running data prediction header can be obtained.

[0079] Based on the above embodiments, this application embodiment describes the steps of inputting low-dimensional parameter features into a pre-trained runtime data prediction head for prediction processing to obtain the predicted runtime data output by the runtime data prediction head. The runtime data prediction head includes a time prediction head, a memory prediction head, and a security prediction head, and the predicted runtime data includes the predicted runtime, predicted memory usage, and predicted security probability. Specifically, the method of this embodiment includes the following steps:

[0080] The low-dimensional parameter features are input into the time prediction head, memory prediction head, and security prediction head respectively to obtain the prediction runtime output by the time prediction head, the prediction memory usage output by the memory prediction head, and the prediction security probability output by the security prediction head.

[0081] Referring to the foregoing embodiments, the runtime data prediction head of this application can be one or more types, and is not limited here. To improve the accuracy of parameter tuning, multiple prediction heads can be selected. For example, the runtime data prediction head can include a time prediction head, a memory prediction head, and a safety prediction head. The time prediction head can consist of two linear layers plus a ReLU activation function. The memory prediction head is symmetrical to the time prediction head. The safety prediction head can output the predicted safety probability through the Sigmoid function, which is used to constrain high-risk configurations.

[0082] By inputting the low-dimensional parameter features into the time prediction head, memory prediction head, and security prediction head respectively, the predicted runtime y output by the time prediction head can be obtained respectively. time The memory usage of the prediction output from the memory prediction header (y) memory And the predicted safety probability y output by the safety prediction head isSafe For specific examples, please refer to the following representation:

[0083] Data types Dimension <![CDATA[y time ]]> [batch_size,1] <![CDATA[y memory ]]> [batch_size,1] <![CDATA[y isSafe ]]> [batch_size,1]

[0084] Based on the above embodiments, this application embodiment describes the steps after performing prediction processing on the pre-trained running data prediction head by inputting low-dimensional parameter features to obtain the predicted running data output by the running data prediction head, and the steps of determining the corresponding target configuration parameters based on the predicted running data. Specifically, the method of this embodiment includes the following steps:

[0085] Obtain the current security threshold; compare the predicted security probability with the current security threshold to obtain a security comparison result; in response to the security comparison result indicating that the predicted security probability is less than or equal to the current security threshold, determine the target configuration parameters based on the predicted running data.

[0086] Referring to the foregoing embodiments, these embodiments describe situations where initial configuration parameters can be used as target configuration parameters to run Spark. However, this requires ensuring that the initial configuration parameters are safe and compliant, and that Spark can run normally with the initial configuration parameters. Therefore, this embodiment can perform a safety judgment by predicting the predicted running data of Spark when it runs with the initial configuration parameters, thereby deciding whether Spark should run with the initial configuration parameters or whether the initial configuration parameters should be tuned.

[0087] For example, the current security threshold is a threshold used to measure whether the initial configuration parameters are safe and compliant. The current security threshold can be preset and fixed, or it can be dynamically adjusted; there is no limitation here.

[0088] After making a prediction based on the initial configuration parameters, the predicted security probability is output by the security prediction header. This predicted security probability characterizes the security of the initial configuration parameters used in this prediction. The predicted security probability is compared with the current security threshold to obtain a security comparison result. If the security comparison result indicates that the predicted security probability is less than or equal to the current security threshold, it means the predicted security probability is low, and the current initial configuration parameters are considered high-risk. Therefore, the Spark job is not run using the initial configuration parameters, and the target configuration parameters are determined based on the prediction data. If the security comparison result indicates that the predicted security probability is greater than the current security threshold, it means the predicted security probability is high, and the current initial configuration parameters are considered low-risk. Therefore, the Spark job can be run using the initial configuration parameters, or the target configuration parameters can be adjusted based on the prediction data and then used to run the Spark job. No specific restriction is imposed here.

[0089] Based on the above embodiments, this application embodiment describes the steps for obtaining the current security threshold. Specifically, the method of this embodiment includes the following steps:

[0090] Obtain historical safety thresholds, whose time series are earlier than that of the current safety threshold; then perform a weighted summation of the historical safety thresholds and the predicted safety probability to obtain the current safety threshold.

[0091] As illustrated in the foregoing embodiments, the security threshold of this application can be fixed or dynamically adjusted. However, a fixed threshold (e.g., setting a threshold when memory usage is 80%) may not adapt to the dynamic changes in Spark job load in certain scenarios. For example, a brief memory overrun during a sudden surge in traffic might be reasonable, but a fixed threshold might determine that the configuration parameters are unsafe at this time. Therefore, this embodiment provides a dynamic threshold optimization method that can dynamically adjust the security threshold and solve the problem of rigid security constraints.

[0092] For example, in a Spark tuning scenario, this embodiment uses the EMA (Exponential Moving Average) strategy to enhance the stability of dynamic safety threshold modeling and optimization strategies. This addresses issues such as rigidity in safety constraints with fixed thresholds by adaptively adjusting safety boundaries and optimizing parameter update processes.

[0093] Specifically, the EMA strategy can be used to dynamically adjust the safety threshold τ. t The confidence level, mathematically expressed as:

[0094] τ t =α·τ t-1 +(1-α)·violation_rate t

[0095] Where α is the decay factor, which can be set to 0.95, and is used to control the weight of the historical safety threshold. `violation_rate` t This represents the current predicted security probability. The decay factor α can be adjusted as needed. A larger decay factor indicates that the DMTE-Spark model has a stronger memory of historical security states and is less responsive to new illegal configurations; conversely, a smaller decay factor indicates that the DMTE-Spark model is more sensitive to new illegal configurations.

[0096] It's understandable that Spark parameter tuning may involve one or more iterations, so the safety threshold used in each tuning process may not be the same. The historical safety threshold refers to the safety threshold determined in a previous (e.g., the last) parameter tuning. It's important to note that there's a preset initial safety threshold when initially tuning parameters, and this threshold (τ) can be reduced using a decay factor after each tuning iteration. t-1 The predicted safety probability violation_rate obtained by tuning the current parameters t By performing a weighted summation, we obtain the safety threshold τ for the current parameter tuning. t .

[0097] In summary, during the t-th parameter tuning process, the safety threshold (historical safety threshold) of the (t-1)-th time and the predicted safety probability of the t-th time are weighted and summed according to the decay factor to obtain the safety threshold (current safety threshold) of the t-th time. If the predicted safety probability of the t-th time is less than or equal to the safety threshold of the t-th time, the target configuration parameters available for Spark to run need to be redefined.

[0098] Optionally, based on the above embodiments, a queue of a preset length (e.g., length 100) can be designed using the sliding window method to store the predicted safety probabilities of the parameter tuning process over a recent period. This facilitates the acquisition of the changing trend of the predicted safety probabilities during subsequent data analysis, significantly aiding in efficiently identifying optimization points. For example, when the decreasing trend of the predicted safety probability in the sliding window is greater than the preset trend, and / or when the predicted safety probability in the sliding window is monotonically decreasing, a prompt message can be generated to inform users that the change in the current safety probability needs to be closely monitored. Furthermore, the recent predicted safety probabilities can be used for subsequent expansion to visualize and detect the system status. More divergent execution steps can also be included, which can be set as needed and will not be elaborated here.

[0099] For example, Prometheus's time-series data processing capabilities can be used to detect the prediction runtime data (including but not limited to prediction safety probabilities, prediction runtime, and prediction memory usage) of the DMTE-Spark model's output for real-time detection and analysis of prediction Spark tasks. Examples can be found in... Figure 3 As shown, Figure 3 This is an exemplary flowchart of the prediction data detection process in the parameter tuning method of the computing engine of this application.

[0100] The custom Exporter (data export module) integrates the metrics (prediction execution data) predicted by the DMTE-Spark model and transmits them to Prometheus (data inspection module). For example, the Exporter can define the Prometheus collection metrics shown in the table below, where Gauge is a built-in Prometheus library function. The metrics are then sent to Prometheus through a pre-defined port, allowing Prometheus to perform real-time or periodic inspections of the collected prediction execution data. The inspection information is then displayed and / or sent to the corresponding users, improving the user experience and inspection efficiency during Spark parameter tuning.

[0101]

[0102] Based on the above embodiments, this application embodiment describes the steps prior to feature extraction processing of the initial configuration parameters of the computing engine. Specifically, the method of this embodiment includes the following steps:

[0103] The acquired configuration parameter samples are input into the initial model for feature extraction to obtain the sample features of the configuration parameter samples; each sample feature is input into the corresponding running data prediction head for data prediction to obtain the sample prediction loss output by each running data prediction head; the target prediction loss is determined based on the prediction loss of each sample; the initial model is adjusted based on the target prediction loss to obtain the target model.

[0104] Referring to the foregoing embodiments, the initial model refers to the DMTE-Spark model that has not yet been fully trained. Its model structure can be found in the previous examples and will not be repeated here. The configuration parameter samples can be found in the description of the training set in the foregoing embodiments and will not be repeated here. This embodiment mainly illustrates the main process of training the DMTE-Spark model.

[0105] For example, DMTE-Spark mainly consists of two parts: a learning module and a multi-task regression module. The learning module mainly consists of an input layer and two sub-modules based on a BERT pre-trained model; the multi-task regression module mainly consists of one or more running data prediction heads. These multiple running data prediction heads can respectively perform three tasks: runtime prediction, memory consumption prediction, and safety probability prediction, and each is supervised learning using the corresponding loss function to finally obtain the trained DMTE-Spark model (i.e., the target model).

[0106] Specifically, the training set from the aforementioned embodiment will still be used as an example for explanation. After feature extraction and dimensionality reduction by the BERT Encoder, the sample features are input into each prediction head for data prediction processing, resulting in the sample prediction loss output by each prediction head. Time and memory usage can be supervised using the MAE loss function, while the safety probability can be supervised using the binary cross-entropy loss. For example, the prediction time loss L output by the time prediction head... time The mathematical expression can be:

[0107]

[0108] The predicted memory loss L output by the memory prediction header mem The mathematical expression can be:

[0109]

[0110] Predicted safety loss I output by the safety prediction head safe The mathematical expression can be:

[0111]

[0112] In summary, based on the above method of determining the target prediction loss according to the prediction loss of each sample, the mathematical expression of the target prediction loss can be:

[0113] L total =αL time +βL mem +γL safe

[0114] The task weights α, β, and γ can be dynamically adjusted during model training. Their initial values ​​can be set to 0.6, 0.3, and 0.1 respectively, without limitation here. Based on existing model training principles, the model parameters of the initial model are iteratively adjusted based on the target prediction loss of each training iteration to obtain the target model.

[0115] Based on the above embodiments, this application embodiment describes the steps for performing model inference using a trained target model. The target model includes an encoder and a data prediction head. Specifically, the method of this embodiment includes the following steps:

[0116] The initial configuration parameters are input into the target model. The encoder performs feature extraction on the initial configuration parameters to obtain parameter features. The data prediction head then performs data prediction on the parameter features to obtain the prediction running data.

[0117] As illustrated in the foregoing embodiments, based on the model structure provided in the foregoing embodiments, after obtaining the trained target model, the target model can be used in the parameter tuning method.

[0118] Specifically, the initial configuration parameters are input into the target model. The BERT encoder performs feature extraction on the initial configuration parameters to obtain parameter features. Then, the data prediction heads in the regression module perform data prediction processing on the parameter features to obtain the prediction running data. The target configuration parameters can then be determined based on the prediction running data, which will not be elaborated here.

[0119] As can be seen from the above, the parameter tuning method of this application mainly includes:

[0120] 1. Use a tuning framework based on multi-task Transformer.

[0121] DMTE-Spark model architecture design:

[0122] Input layer: Combines configuration parameter features with cluster operating status to construct a set of data inputs.

[0123] Encoder: Multi-head attention modeling of multi-dimensional feature interactions.

[0124] Regression layer: Independent multi-task output heads can predict the runtime, memory usage, and security constraint mechanisms for three tasks respectively.

[0125] 2. Integration of security constraints with online learning.

[0126] Constraint mechanism: Safety rules are embedded in the output layer of the Transformer encoder, and the strength of safety constraints is dynamically adjusted through the EMA threshold.

[0127] Online update: Based on a hybrid optimization strategy of gradient descent and EMA, the model parameters and safety thresholds are updated.

[0128] 3. Dynamic threshold calculation method.

[0129] Core mathematical expression: τ t =α·τ t-1 +(1-α)·violation_rate t

[0130] A larger decay factor α indicates a stronger memory of historical security states and a slower response to new violations. In other words, the DMTE-Spark model will focus on using previously determined historical security thresholds to measure whether the current predicted security probability meets the standard. Conversely, a smaller decay factor α makes it easier for the DMTE-Spark model to account for data fluctuations caused by special circumstances (such as the impact of short-term large traffic surges), thus allowing the security threshold to be adjusted upwards based on the current predicted security probability.

[0131] In summary, the multi-task encoder of this application can simultaneously optimize runtime data such as task execution time and resource utilization, realizing a multi-task system through shared feature representations and independent output heads. Meanwhile, the multi-head attention mechanism within the Transformer can capture the complex nonlinear interaction between task features and task runtime states, improving the accuracy of parameter tuning. The dynamic safety threshold mechanism can track model prediction errors in real time through EMA and dynamically adjust the safety threshold, avoiding the time-consuming and laborious process of obtaining manual experience thresholds and the safety misjudgment caused by fixed thresholds. Regarding lightweight design and compatibility, the method of this application does not require modification of the Spark core code, achieving integration through dynamic loading of scheduling strategies, supporting CPU / GPU hybrid clusters and scenarios with mixed batch and stream processing workloads. This application solves problems such as static threshold dependence, manual experience dominance, and multi-objective optimization conflicts in Spark tuning, providing an intelligent and adaptive solution for Spark tuning, significantly improving the execution efficiency and stability of distributed computing tasks.

[0132] It should be further noted that the execution entity of the computing engine parameter tuning method can be a computing engine parameter tuning device. For example, the computing engine parameter tuning method can be executed by a terminal device, server, or other processing device. The terminal device can be a user equipment (UE), computer, mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. In some possible implementations, the computing engine parameter tuning method can be implemented by the processor calling computer-readable instructions stored in memory.

[0133] Figure 4 This is a block diagram illustrating a parameter tuning apparatus for a computing engine, as shown in an exemplary embodiment of this application. Figure 4 As shown, the exemplary computing engine parameter tuning device 400 includes: a feature extraction module 410, a data prediction module 420, and a parameter determination module 430. Specifically:

[0134] The feature extraction module 410 is used to perform feature extraction processing on the initial configuration parameters of the computing engine to obtain the parameter features of the initial configuration parameters.

[0135] The data prediction module 420 is used to perform data prediction processing based on parameter characteristics to obtain the prediction running data when the computing engine runs with the initial configuration parameters.

[0136] The parameter determination module 430 is used to determine the corresponding target configuration parameters based on the predicted running data.

[0137] In this exemplary parameter tuning device for the computing engine, a Transformer encoder using multi-task regression learns the complex nonlinear relationship between Spark configuration parameters and runtime data, thereby predicting runtime data under different configuration parameters. The initial configuration parameters of the computing engine can then be input into a trained neural network model for feature extraction, yielding parameter features of the initial configuration parameters. Next, data prediction processing is performed on these parameter features using the prediction head in the neural network model to obtain predicted runtime data for the computing engine running with the initial configuration parameters. Finally, a regression task is executed based on the predicted runtime data to determine the target configuration parameters that should correspond to the predicted runtime data, i.e., the recommended configuration parameters after tuning based on the initial configuration parameters. This improves the accuracy and efficiency of configuration parameter tuning.

[0138] It should be noted that the apparatus and method provided in the above embodiments belong to the same concept, and the specific ways in which each module and unit performs operations have been described in detail in the method embodiments, and will not be repeated here. In practical applications, the apparatus provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above, and this is not a limitation.

[0139] The functions of each module can be found in the implementation examples of parameter tuning methods for the computing engine, and will not be repeated here.

[0140] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. The electronic device 100 includes a memory 101 and a processor 102. The processor 102 is used to execute program instructions stored in the memory 101 to implement the steps in the parameter tuning method embodiment of any of the above-described computing engines. In a specific implementation scenario, the electronic device 100 may include, but is not limited to, a microcomputer or a server. In addition, the electronic device 100 may also include mobile devices such as laptops and tablets, which are not limited here.

[0141] Specifically, processor 102 controls itself and memory 101 to implement the steps in the parameter tuning method embodiment of any of the above-described computing engines. Processor 102 can also be referred to as a CPU (Central Processing Unit). Processor 102 may be an integrated circuit chip with signal processing capabilities. Processor 102 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 102 can be implemented using integrated circuit chips.

[0142] In this exemplary electronic device, a Transformer encoder employing multi-task regression learns the complex nonlinear relationship between Spark configuration parameters and runtime data, thereby predicting runtime data under different configuration parameters. The initial configuration parameters of the computing engine are then input into a trained neural network model for feature extraction, yielding parameter features of the initial configuration parameters. Next, the prediction head in the neural network model performs data prediction processing on these parameter features to obtain predicted runtime data for the computing engine running with the initial configuration parameters. Finally, a regression task is executed based on the predicted runtime data to determine the target configuration parameters that should correspond to the predicted runtime data, i.e., the recommended configuration parameters after optimization based on the initial configuration parameters. This improves the accuracy and efficiency of configuration parameter optimization.

[0143] Please see Figure 6 , Figure 6 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 110 stores program instructions 111 that can be executed by a processor. The program instructions 111 are used to implement the steps in the parameter tuning method embodiment of any of the above-described computing engines.

[0144] In this exemplary storage medium, by running program instructions within the storage medium, the Transformer encoder of a multi-task regression system learns the complex nonlinear relationship between Spark configuration parameters and runtime data, thereby predicting runtime data under different configuration parameters. Then, the initial configuration parameters of the computing engine can be input into a trained neural network model for feature extraction, obtaining the parameter features of the initial configuration parameters. Next, data prediction processing is performed on these parameter features using the prediction head in the neural network model to obtain the predicted runtime data for the computing engine running with the initial configuration parameters. Finally, a regression task is executed based on the predicted runtime data to determine the target configuration parameters that should correspond to the predicted runtime data, i.e., the recommended configuration parameters after optimization based on the initial configuration parameters. This improves the accuracy and efficiency of configuration parameter optimization.

[0145] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0146] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0147] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0148] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for parameter tuning of a computing engine, characterized in that, The method includes: The initial configuration parameters of the computing engine are subjected to feature extraction processing to obtain the parameter features of the initial configuration parameters; Data prediction processing is performed based on the parameter characteristics to obtain the predicted running data of the computing engine running with the initial configuration parameters; The corresponding target configuration parameters are determined based on the predicted running data.

2. The method according to claim 1, characterized in that, The feature extraction process performed on the initial configuration parameters of the computing engine to obtain the parameter features of the initial configuration parameters includes: The initial configuration parameters are processed by word vector conversion according to a preset data format to obtain the configuration parameter vector corresponding to the initial configuration parameters. The configuration parameter vector is input into a pre-trained natural language model for feature extraction to obtain the parameter features output by the natural language model.

3. The method according to claim 1, characterized in that, The step of performing data prediction processing based on the parameter features to obtain the predicted running data of the computing engine running with the initial configuration parameters includes: The parameter features are then subjected to dimensionality reduction processing to obtain low-dimensional parameter features; The low-dimensional parameter features are input into a pre-trained runtime data prediction head for prediction processing to obtain the predicted runtime data output by the runtime data prediction head.

4. The method according to claim 3, characterized in that, The runtime data prediction head includes a time prediction head, a memory prediction head, and a security prediction head. The predicted runtime data includes predicted runtime, predicted memory usage, and predicted security probability. The low-dimensional parameter features are input into the pre-trained runtime data prediction head for prediction processing to obtain the predicted runtime data output by the runtime data prediction head, including: The low-dimensional parameter features are input into the time prediction head, the memory prediction head, and the security prediction head, respectively, to obtain the prediction running time output by the time prediction head, the prediction memory usage output by the memory prediction head, and the prediction security probability output by the security prediction head.

5. The method according to claim 4, characterized in that, After inputting the low-dimensional parameter features into the pre-trained running data prediction head for prediction processing to obtain the predicted running data output by the running data prediction head, the method further includes: Get the current security threshold; The predicted security probability is compared with the current security threshold to obtain a security comparison result; The step of determining the corresponding target configuration parameters based on the predicted operational data includes: in response to the security comparison result indicating that the predicted security probability is less than or equal to the current security threshold, determining the target configuration parameters based on the predicted operational data.

6. The method according to claim 5, characterized in that, The step of obtaining the current security threshold includes: Obtain historical security thresholds, wherein the time sequence of the historical security thresholds is earlier than that of the current security thresholds; The current security threshold is obtained by weighted summation of the historical security threshold and the predicted security probability.

7. The method according to claim 1, characterized in that, Before performing feature extraction processing on the initial configuration parameters of the computing engine, the method further includes: The obtained configuration parameter samples are input into the initial model for feature extraction processing to obtain the sample features of the configuration parameter samples; Each sample feature is input into the corresponding running data prediction head for data prediction processing, and the sample prediction loss output by each running data prediction head is obtained. Determine the target prediction loss based on the prediction loss of each sample; The initial model is adjusted based on the target prediction loss to obtain the target model.

8. The method according to claim 7, characterized in that, The target model includes an encoder and a data prediction head, and the method includes: The initial configuration parameters are input into the target model, and the encoder performs feature extraction processing on the initial configuration parameters to obtain the parameter features; The parameter features are processed by the data prediction head to obtain the prediction running data.

9. An electronic device, characterized in that, The method includes a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, they implement the method described in any one of claims 1 to 8.

Images