Data processing method and apparatus, device, and storage medium
By dynamically adjusting the number of iterations and runs for parameter tuning, the problems of low efficiency and resource waste in traditional methods are solved, achieving more efficient and accurate parameter tuning.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ALIBABA (CHINA) CO LTD
- Filing Date
- 2022-07-11
- Publication Date
- 2026-05-29
AI Technical Summary
In traditional parameter tuning, the number of iterations and runs are manually set, which leads to inaccurate results in fluctuating load environments, reducing efficiency and wasting resources.
By obtaining the performance score of the optimization task running M times in the Nth iteration, the number of runs in the next iteration is dynamically adjusted, and the iteration is terminated when preset conditions are met, thus avoiding resource waste.
It improves the efficiency and accuracy of parameter tuning, reduces resource waste, adapts to fluctuations in different environments, and shortens tuning time.
Smart Images
Figure CN115145897B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, device, and storage medium. Background Technology
[0002] There is a need for parameter tuning in scenarios such as operating system kernels, compilers, and databases. Effective parameter settings can optimize service performance and improve overall service quality, thus giving rise to parameter tuning.
[0003] Traditional parameter tuning processes require pre-defining the number of iterations and epochs per iteration. The parameter tuning algorithm then executes according to this pre-defined number of iterations and epochs. However, since these numbers are manually set before the algorithm runs, this process cannot provide accurate parameter tuning results in environments with highly volatile loads. This not only reduces tuning efficiency and affects tuning effectiveness but also wastes resources by continuing to execute the pre-defined number of iterations and epochs even after the parameters have converged. Summary of the Invention
[0004] This application provides a data processing method, apparatus, device, and storage medium that can improve parameter tuning efficiency and effect, and reduce resource waste.
[0005] According to a first aspect of the embodiments of this application, a data processing method is provided, comprising:
[0006] Obtain the performance score of the tuning task after running M times in the Nth iteration;
[0007] Based on the performance index scores of running M times in the Nth iteration, determine the number of times the optimization task will run in the N+1th iteration.
[0008] If the performance index score of the target number of iterations from the Nth iteration to the Yth iteration meets the first preset condition, then determine the target performance index score that meets the second preset condition from the Nth iteration to the Yth iteration.
[0009] The target round iteration to which the run number of the target performance index score belongs is determined as the parameter tuning result.
[0010] According to a second aspect of the embodiments of this application, a data processing method based on a relational database is provided, including:
[0011] Based on the parameter tuning task of the relational database, obtain multiple sets of parameters in the relational database. The parameter sets include multiple parameters to be tuned and the parameter values of the parameters to be tuned.
[0012] During the parameter tuning task, the performance score of the parameter set is calculated M times in the Nth iteration.
[0013] Based on the performance index scores of running M times in the Nth iteration, determine the number of times the optimization task will run in the N+1th iteration.
[0014] If the performance index score of the target number of iterations from the Nth iteration to the Yth iteration meets the first preset condition, then select the target performance index score that meets the second preset condition from the Nth iteration to the Yth iteration.
[0015] Based on the target round iteration to which the target performance index score belongs, the target parameter set including the target round iteration is determined as the parameter tuning result of the relational database. Multiple parameter sets include the target parameter set, and the target parameter set includes multiple tuned parameters and the parameter values of the tuned parameters.
[0016] According to a third aspect of the embodiments of this application, a data processing method based on network services is provided, comprising:
[0017] Receive tuning requests sent by users. Tuning requests include multiple sets of parameters, each set including multiple parameters to be tuned and the parameter values of the parameters to be tuned.
[0018] During the execution of the tuning request, obtain the performance index score of the parameter set running M times in the Nth iteration;
[0019] Based on the performance index scores of running M times in the Nth iteration, determine the number of times the optimization task will run in the N+1th iteration.
[0020] If the performance index score of the target number of iterations from the Nth iteration to the Yth iteration meets the first preset condition, then determine the target performance index score that meets the second preset condition from the Nth iteration to the Yth iteration.
[0021] Send the parameter tuning results to the user. The parameter tuning results include the target round iteration to which the target performance index score belongs and the target parameter set of the target round iteration. The target parameter set includes multiple tuned parameters and the parameter values of the tuned parameters.
[0022] According to a fourth aspect of the embodiments of this application, a data processing apparatus is provided, comprising:
[0023] The acquisition module is used to acquire the performance index score of the optimization task running M times in the Nth iteration; the determination module is used to determine the number of times the optimization task will run in the N+1th iteration based on the performance index score of the M times run in the Nth iteration.
[0024] The processing module is used to filter target performance index scores that meet the second preset condition from the Nth iteration to the Yth iteration if the performance index score of the target number of iterations meets the first preset condition.
[0025] The output module is used to determine the target round iteration to which the run number of the target performance index score belongs as the parameter tuning result.
[0026] According to a fifth aspect of the embodiments of this application, a data processing apparatus based on a relational database is provided, comprising:
[0027] The acquisition module is used to acquire multiple sets of parameters from the relational database based on the parameter tuning task of the relational database. The parameter sets include multiple parameters to be tuned and the parameter values of the parameters to be tuned.
[0028] The calculation module is used to calculate the performance index score of the parameter set in the Nth iteration M times during the parameter tuning task.
[0029] The determination module is used to determine the number of times the tuning task will run in the (N+1)th iteration based on the performance index score of running M times in the Nth iteration.
[0030] The filtering module is used to filter target performance index scores that meet the second preset condition from the Nth iteration to the Yth iteration, provided that the performance index scores of the target number of iterations meet the first preset condition.
[0031] The output module is used to determine the target parameter set of the target round iteration to which the target performance index score belongs, as the parameter tuning result of the relational database. Multiple parameter sets include the target parameter set, and the target parameter set includes multiple tuned parameters and the parameter values of the tuned parameters.
[0032] According to a sixth aspect of the embodiments of this application, a data processing apparatus based on network services is provided, comprising:
[0033] The receiving module is used to receive tuning requests sent by users. The tuning request includes multiple sets of parameters, and the parameter sets include multiple parameters to be tuned and the parameter values of the parameters to be tuned.
[0034] The acquisition module is used to acquire the performance metric scores of the parameter set running M times in the Nth iteration during the execution of the tuning request.
[0035] The determination module is used to determine the number of times the tuning task will run in the (N+1)th iteration based on the performance index score of running M times in the Nth iteration.
[0036] The filtering module is used to filter target performance index scores that meet the second preset condition from the Nth iteration to the Yth iteration, provided that the performance index scores of the target number of iterations meet the first preset condition.
[0037] The sending module is used to send parameter tuning results to the user. The parameter tuning results include the target round iteration to which the target performance index score belongs and the target parameter set of the target round iteration. The target parameter set includes multiple tuned parameters and the parameter values of the tuned parameters.
[0038] According to a seventh aspect of the embodiments of this application, a computer device is provided, including: a memory and a processor;
[0039] Memory, used to store computer programs;
[0040] A processor is used to execute a computer program stored in memory. When the computer program is run, it causes the processor to perform the steps of the data processing method as shown in the first aspect, the steps of the data processing method based on a relational database as shown in the second aspect, and the steps of the data processing method based on a network service as shown in the third aspect.
[0041] According to an eighth aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a program or instructions are stored. When the program or instructions are executed by a computer device, the computer device is caused to perform the steps of the data processing method as shown in the first aspect, the steps of the data processing method based on a relational database as shown in the second aspect, and the steps of the data processing method based on a network service as shown in the third aspect.
[0042] According to a ninth aspect of the present application, a computer program product is provided, comprising a computer program that, when executed by a computer device, causes the computer device to perform the steps of the data processing method as shown in the first aspect, the steps of the data processing method based on a relational database as shown in the second aspect, and the steps of the data processing method based on a network service as shown in the third aspect.
[0043] According to the data processing method, apparatus, device, and storage medium in the embodiments of this application, the performance index score of the optimization task running M times in the Nth iteration is obtained, and the number of times the optimization task runs in the N+1th iteration is determined based on the performance index score of the M times running in the Nth iteration. In this way, the running rounds and the number of runs in each round of the optimization task can be dynamically controlled for different environments and different effects, avoiding the influence of the performance index score of each run in each round due to the large fluctuation of the load environment, ensuring the reliability of the optimization effect and improving the accuracy of parameter optimization efficiency. Then, if the performance index score of the target number of runs in the Nth iteration to the Yth iteration meets the first preset condition, the target performance index score that meets the second preset condition is determined in the Nth iteration to the Yth iteration, and the target round iteration to which the run of the target performance index score belongs is determined as the parameter optimization result. In this way, if the parameter optimization effect has been achieved in a certain round of iteration, it is not necessary to continue running, avoiding resource waste and shortening the parameter optimization time. Attached Figure Description
[0044] This application can be better understood from the following description of specific embodiments in conjunction with the accompanying drawings, wherein the same or similar reference numerals denote the same or similar features.
[0045] Figure 1 This is a schematic diagram illustrating a data query process in related technologies;
[0046] Figure 2 This is a schematic diagram illustrating a data processing architecture according to one embodiment;
[0047] Figure 3 This is a schematic diagram illustrating a data processing flow according to one embodiment;
[0048] Figure 4 This is a flowchart illustrating a data processing method according to one embodiment;
[0049] Figure 5 This is a flowchart illustrating a data processing method based on a relational database according to one embodiment;
[0050] Figure 6 This is a flowchart illustrating a data processing method based on a network service according to one embodiment;
[0051] Figure 7 This is a schematic diagram illustrating the structure of a data processing apparatus according to one embodiment;
[0052] Figure 8 This is a schematic diagram illustrating the structure of a data processing apparatus based on a relational database according to one embodiment;
[0053] Figure 9 This is a schematic diagram illustrating the structure of a network service-based data processing apparatus according to one embodiment;
[0054] Figure 10 This is a schematic diagram illustrating the hardware structure of a computer device according to one embodiment. Detailed Implementation
[0055] The features and exemplary embodiments of various aspects of this application will now be described in detail. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only configured to explain this application and are not configured to limit this application. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples of this application.
[0056] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.
[0057] Parameter optimization is the process of adjusting system parameters to continuously improve the system's score on benchmark programs. System parameters refer to configuration items in the operating system that can be dynamically set and affect application performance, such as kernel configuration parameters and application configuration parameters. Benchmark programs are evaluation programs that estimate the performance of optimized applications and give performance scores.
[0058] In related technologies, the traditional parameter tuning process can be executed by a parameter tuning algorithm program. This program provides an interface that defines the number of iterations and the number of epochs required for each iteration. The algorithm then executes according to the preset number of iterations and epochs to obtain the tuning results. Here, the parameter tuning algorithm is an algorithmic strategy that continuously provides combinations of system parameters to find parameter configurations that improve application performance.
[0059] For example, such as Figure 1 As shown, when performing parameter tuning on a relational database tuning system (MySQL) using the Bayes algorithm, the number of iterations executed by the parameter tuning algorithm program can be defined as 500 rounds (N), with each round executed 3 times (M). In this case, the actual tuning task will generate 500 sets of parameters, and after each set of parameters is set, the benchmark will be executed 3 times, which means a total of 1500 benchmarks will be executed.
[0060] Based on this, during the execution of the parameter tuning algorithm, after each parameter set is set, the difference in performance index scores across three executions is considered. A smaller difference in performance index scores indicates a more stable environment during the execution or load testing of the parameter tuning algorithm, while a larger difference indicates an unstable environment. Traditional parameter tuning algorithms do not consider this; that is, regardless of whether the environment during execution or load testing is stable, or whether the tuning has reached an ineffective iteration stage, traditional parameter tuning algorithms will only execute according to the predefined number of iterations and runs, ultimately providing the parameter tuning results. This leads to two problems: First, if the environment is relatively stable, running the algorithm once per round is sufficient. However, if the environment is unstable, running it only once per round might result in a benchmark result that reflects environmental fluctuations rather than the actual parameter tuning results from the algorithm, thus reducing tuning efficiency and affecting accuracy. Second, once the algorithm reaches convergence, further iterations will not improve the accuracy of the tuning results. Since the number of iterations and runs is manually set before the algorithm runs, it cannot be modified during execution, leading to wasted resources. Here, convergence in this embodiment refers to the parameter tuning algorithm's result no longer undergoing significant changes.
[0061] Therefore, when faced with the challenge of adapting parameter tuning algorithms to different operating environments, such as the execution environment of the algorithm and the fluctuations in the environment during load operation, and automatically controlling the number of rounds of the tuning task based on the convergence status of the algorithm, minimizing the number of execution rounds while ensuring performance tuning effects, thereby reducing the time and resource consumption caused by benchmark execution, this application provides a data processing method. By adding the judgment of environmental stability and convergence effect during the operation of the parameter tuning algorithm, this method can dynamically control the number of rounds of the tuning task for different environments and effects. This solves the problems of manually determining the number of rounds before execution, which leads to the inability to provide accurate parameter tuning results in environments with large fluctuations, and the resource waste caused by long-term operation after convergence.
[0062] The following is combined with Figure 2 The data processing architecture provided in the embodiments of this application will be described in detail.
[0063] In one or more possible embodiments, such as Figure 2 As shown, the data processing architecture proposed in this application embodiment includes a first computing node 201 and a second computing node 202.
[0064] The first computing node 201 and the second computing node 202 will be described in detail below.
[0065] The first computing node 201 is used to judge the stability of the environment and control the number of iterations and the number of runs in each iteration of the optimization task. In this embodiment, the environment can refer to the system running environment, the parameter optimization algorithm program execution environment, and the load running environment.
[0066] First, the process of judging environmental stability can be carried out throughout the entire lifecycle of the parameter tuning algorithm. Its purpose is to address the situation where, during parameter tuning, environmental fluctuations can cause a sudden, exceptionally high value that the algorithm fails to recognize, resulting in a seemingly excellent tuning outcome that is actually inaccurate. Based on this, an environmental stability detection mechanism can be implemented to calculate indicators determining the stability of the current system. For example, using the same benchmark program, the performance index score of the number of runs in each iteration of the tuning task can be calculated. If the difference between the performance index scores of the runs in each iteration is small, the current system is considered stable; conversely, if the difference is large, the current system is considered unstable. This allows for the control of the number of iterations and the number of runs in each iteration based on the stability of the current system, thereby reducing errors and ensuring the accuracy of the parameter tuning results.
[0067] Then, based on the judgment of environmental stability, the number of iterations and the number of runs in each iteration of the control tuning task can be adjusted. In one example, if the current system is determined to be stable, the number of runs of the tuning task in the next iteration can be reduced. Conversely, if the current system is determined to be unstable, the number of runs of the tuning task in the next iteration can be increased or maintained.
[0068] In this way, the number of runs and the number of runs in each run can be dynamically controlled for different environments and effects. This avoids the impact of highly volatile load environments on the performance index scores of each run in each run, ensuring the reliability of the tuning effect and improving the accuracy of parameter tuning efficiency.
[0069] The second computing node 202 is used to determine whether to trigger the early stop mechanism and control the output of parameter tuning results based on the running results of the first computing node 201. The early stop mechanism is an optimization technique used during algorithm model training to reduce overfitting without affecting the model's accuracy. It effectively stops training before the model becomes overfitted. In other words, if the environment is damaged beyond repair, further training is meaningless; similarly, if parameter tuning results no longer improve, further training is also meaningless. Therefore, the early stop mechanism aims to reduce the actual number of iterations to minimize resource waste. Based on this, the stop mechanism can detect and terminate the tuning process early when the algorithm has already entered the convergence phase and subsequent iterations no longer contribute to the parameter tuning results, thus outputting the parameter tuning results. In one example, the second computing node 202 can be specifically used to determine the target performance index score that meets the second preset condition from the Nth iteration to the Yth iteration, provided that the performance index score of the target run from multiple iterations meets the first preset condition, and to determine the target iteration to which the run of the target performance index score belongs as the parameter tuning result. Here, N and Y are both integers greater than or equal to 1.
[0070] Based on such Figure 2 The data processing architecture shown below, in conjunction with the appendix Figure 3 The data processing method provided in the embodiments of this application will be described in detail.
[0071] like Figure 2 The data processing architecture 10 shown can perform the following two stages, namely, specific combination Figure 3 Please provide a detailed explanation.
[0072] like Figure 3As shown, if N is 1, then the first computing node 201, having obtained the performance index score of the optimization task running M times in the first iteration, determines the number of times the optimization task will run in the second iteration based on the performance index score of the M times run in the first iteration. Here, besides determining the number of times the optimization task will run in the second iteration based on the performance index score of the first run in the first M iterations, in some scenarios, the number of times the optimization task will run in the second iteration can also be determined based on the performance index score of the first run in the first M iterations.
[0073] Then, if the performance index score of the target run from the second iteration to the tenth iteration meets the first preset condition, the second computing node 202 determines the target performance index score that meets the second preset condition from the second iteration to the tenth iteration, ends the iteration, and outputs the target iteration to which the run of the target performance index score belongs as the parameter tuning result.
[0074] Therefore, the number of iterations of the parameter tuning algorithm will no longer be defined manually. The number of iterations in the next iteration can be automatically determined based on the actual environment and tuning effect, avoiding the impact of a single fluctuation in the load environment, ensuring the accuracy of parameter tuning efficiency, and effectively saving the time and resources consumed by the parameter tuning algorithm.
[0075] It should be noted that the data processing method provided in this application embodiment can be applied to the following scenarios, namely system load basic scenarios and application load scenarios. The system load basic scenarios include: processor (Central Processing Unit / Processor, CPU) intensive, memory intensive, input / output (IO) intensive, network intensive, etc., and the application load scenarios include: database, network services, big data, etc.
[0076] Based on the above architecture and application scenarios, the following will be combined with... Figure 4 The data processing method provided in the embodiments of this application will be described in detail.
[0077] Figure 4 This is a flowchart illustrating a data processing method according to one embodiment.
[0078] like Figure 4 As shown, the data processing method provided in this application embodiment can be applied to, for example... Figure 2 The data processing architecture shown may specifically include:
[0079] Step 410: Obtain the performance index score of the optimization task running M times in the Nth iteration; Step 420: Based on the performance index score of the optimization task running M times in the Nth iteration, determine the number of times the optimization task will run in the N+1th iteration; Step 430: If the performance index score of the target number of runs from the Nth iteration to the Yth iteration meets the first preset condition, determine the target performance index score that meets the second preset condition from the Nth iteration to the Yth iteration; Step 440: Determine the target iteration to which the number of runs of the target performance index score belongs as the parameter optimization result.
[0080] The steps described above are explained in detail below.
[0081] First, regarding step 410, in one or more possible examples, based on the tuning task, multiple sets of parameters are obtained. Here, the parameter set refers to the parameter to be tuned and the parameter values of the parameter to be tuned. For example, if there are three parameters to be tuned, A, B, and C, each with a value range of 1-100, then if four sets of parameter sets and parameter values of the parameter to be tuned are taken, they can be: the first set A=1, B=1, C=1; the second set A=1, B=100, C=2; the third set A=20, B=50, C=47; and the fourth set A=5, B=11, C=80.
[0082] Next, step 420 is involved, which may specifically include:
[0083] Step 4201: Based on the performance index scores from the M runs in the Nth iteration, calculate the first difference value between the performance index scores from the M runs.
[0084] Step 4202: If the first difference value is less than or equal to the first preset difference value, determine that the number of times the optimization task runs in the N+1th iteration is less than M times in the Nth iteration; otherwise, if the first difference value is greater than the first preset difference value, determine that the number of times the optimization task runs in the N+1th iteration is greater than or equal to M times in the Nth iteration.
[0085] For example, in one or more possible examples, when N takes a value such as N=1 and M=3, based on the performance index scores of the benchmarks run 3 times in the first iteration of the parameter tuning algorithm program, the variance or covariance between the performance index scores of the 3 benchmarks is calculated, and the variance or covariance is determined as the first difference value. If the first difference value is less than or equal to the first preset difference value, it can be characterized that the environment is stable, and the number of times the tuning task runs in the second iteration is determined to be less than 3 times in the first iteration, such as 1 time. Conversely, if the first difference value is greater than the first preset difference value, it can be characterized that the environment is unstable, and the number of times the tuning task runs in the second iteration is determined to be greater than or equal to 3 times in the first iteration, such as 3 times or 5 times. That is, if the environment is stable, the number of runs in the next iteration is reduced; otherwise, the number of runs in the next iteration is increased.
[0086] In another or more possible examples, where N takes at least two values (e.g., N = 1, 2, and 3) and M = 3, the variance or covariance between the three benchmark performance scores is calculated based on the performance metric scores of the three benchmark runs in each of the first, second, and third iterations of the parameter tuning algorithm. This variance or covariance is then determined as the first difference value. If the first difference value is less than or equal to a first preset difference value, indicating that the environment is stable, the number of runs of the tuning task in the fourth iteration is determined to be less than three times in the first, second, and third iterations (e.g., one time). Conversely, if the first difference value is greater than the first preset difference value, indicating that the environment is unstable, the number of runs of the tuning task in the fourth iteration is determined to be greater than or equal to three times in the first, second, and third iterations (e.g., three or five times).
[0087] It should be noted that if the first difference value in consecutive iterations indicates that the environment is stable, then unless there is a sudden and irrecoverable drop in the performance metric score, the current environment can still be considered stable, and the number of runs in subsequent iterations can be based on the number of runs when the environment is stable. If a sudden and significant drop in the performance metric score occurs in consecutive iterations, it can be categorized into the early stopping mechanism's judgment phase.
[0088] Based on this, before step 430, it is necessary to determine whether the current optimization task should be subject to the early stop mechanism judgment and control phase. Here, the early stop mechanism mainly takes effect under two conditions: (1) the parameter optimization results determined based on the performance index score of the target run have not improved for several consecutive rounds; (2) the performance index score of the target run has dropped significantly and cannot be recovered for several consecutive rounds. Thus, this application embodiment provides a process for determining whether the performance index score of the target run meets the first preset condition. Here, the process for determining whether the performance index score of the target run meets the first preset condition can be explained separately by considering the different meanings of the target run in different scenarios.
[0089] In one or more possible examples, where the target number of runs is the number of runs in at least two consecutive iterations from iteration N to iteration Y, step 430 may further include:
[0090] If the performance index score of the target run is lower than the preset performance index score, then the performance index score of the target run meets the first preset condition.
[0091] For example, if N is 1, Y is 10, and the target number of runs is the number of runs in 5 consecutive iterations from the 1st to the 10th iteration (e.g., the number of runs in the 5th to the 10th iterations), and none of them have a higher performance score than the 3 runs in the 4th iteration, then it can be said that the parameter tuning algorithm has entered the convergence stage and there is no need to continue iterating. In this way, the 4th iteration can be determined as the target iteration.
[0092] In another or more possible examples, where the target number of runs is the number of runs in each iteration from the Nth to the Yth iteration, the following may also be included before step 430:
[0093] If the second difference value between the performance index scores of the target run is less than or equal to the second preset difference value, the performance index scores of the target run are determined to meet the first preset condition.
[0094] For example, if N is 1, Y is 10, and the target number of runs is the number of runs in each round from the 1st to the 10th iteration, then it can be said that if the performance index score for 10 consecutive rounds is lower than the preset baseline value, it indicates that the tuning task and related parameters are abnormal, or the environment is abnormal. In this way, without continuing the iteration, an error reminder can be output, or the target round iteration to which the run number of the target performance index score that meets the second preset condition in the 1st to the 10th iteration belongs can be determined as the parameter tuning result.
[0095] Therefore, the judgment and control phase of the early stopping mechanism is designed to detect and terminate parameter tuning as early as possible when the algorithm has entered the convergence phase and subsequent iterations no longer contribute to the tuning results. Simultaneously, it addresses the issue of unrecoverable benchmark results due to sudden environmental problems, where subsequent iterations are useless and contribute nothing to the tuning outcome, effectively reducing the number of actual iterations and avoiding resource waste.
[0096] Then, regarding step 430, in one or more possible examples, step 430 may specifically include:
[0097] In the performance index scores of the target runs, the highest performance index score is determined as the target performance index score.
[0098] To better understand the data processing method provided in the embodiments of this application, the data processing method provided in the embodiments of this application will be described in detail below in conjunction with two application scenarios, such as the data processing scenario of relational databases and the data processing scenario based on network services.
[0099] Figure 5 This is a flowchart illustrating a data processing method based on a relational database according to one embodiment.
[0100] like Figure 5 As shown, the data processing method based on a relational database provided in this application embodiment can be applied to, for example... Figure 2 The data processing architecture shown may specifically include:
[0101] Step 510: Based on the parameter tuning task of the relational database, obtain multiple sets of parameters in the relational database. The parameter sets include multiple parameters to be tuned and the parameter values of the parameters to be tuned.
[0102] Step 520: During the parameter tuning task, calculate the performance score of the parameter set running M times in the Nth iteration.
[0103] Step 530: Based on the performance index scores of running M times in the Nth iteration, determine the number of times the tuning task will run in the N+1th iteration;
[0104] Step 540: If the performance index score of the target number of iterations from the Nth iteration to the Yth iteration meets the first preset condition, select the target performance index score that meets the second preset condition from the Nth iteration to the Yth iteration.
[0105] Step 550: Based on the target round iteration to which the target performance index score belongs, determine the target parameter set including the target round iteration as the parameter tuning result of the relational database. Multiple parameter sets include the target parameter set, and the target parameter set includes multiple tuned parameters and the parameter values of the tuned parameters.
[0106] In this way, the number of runs and the number of runs in each run can be dynamically controlled for different environments and effects. This avoids the impact of highly volatile load environments on the performance index scores of each run in each run, ensuring the reliability of the tuning effect, improving the accuracy of parameter tuning efficiency, and eliminating the need for continuous running when parameter tuning has been achieved in a certain iteration, thus avoiding resource waste and shortening the parameter tuning time.
[0107] Figure 6 This is a flowchart illustrating a data processing method based on a network service according to one embodiment.
[0108] like Figure 6 As shown, the data processing method based on network services provided in this application embodiment can be applied to, for example... Figure 2 The data processing architecture shown may specifically include:
[0109] Step 610: Receive the tuning request sent by the user. The tuning request includes multiple sets of parameters, and the parameter sets include multiple parameters to be tuned and the parameter values of the parameters to be tuned.
[0110] Step 620: During the execution of the tuning request, obtain the performance index score of the parameter set running M times in the Nth iteration;
[0111] Step 630: Based on the performance index scores of running M times in the Nth iteration, determine the number of times the tuning task will run in the N+1th iteration;
[0112] Step 640: If the performance index score of the target number of iterations from the Nth iteration to the Yth iteration meets the first preset condition, determine the target performance index score that meets the second preset condition from the Nth iteration to the Yth iteration.
[0113] Step 650: Send the parameter tuning results to the user. The parameter tuning results include the target round iteration to which the target performance index score belongs and the target parameter set of the target round iteration. The target parameter set includes multiple tuned parameters and the parameter values of the tuned parameters.
[0114] This allows for dynamic control of the number of runs and the number of runs in each run for different environments and desired effects. It avoids the impact of fluctuating load environments on the performance metrics scores of each run in each round, ensuring the reliability of the tuning effect, improving the accuracy of parameter tuning efficiency, and eliminating the need for continuous running when parameter tuning has already been achieved in a certain iteration, thus avoiding resource waste and shortening the parameter tuning time. For the user, it can also reduce user costs and improve parameter tuning efficiency.
[0115] It should be clarified that this application is not limited to the specific configurations and processes described in the above embodiments and shown in the figures. For the sake of convenience and brevity, detailed descriptions of known methods are omitted here, and the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, which will not be repeated here.
[0116] Based on the same inventive concept, embodiments of this application provide a data processing apparatus corresponding to the data processing method described above. Specifically, in conjunction with... Figure 7 Please provide a detailed explanation.
[0117] Figure 7 This is a schematic diagram illustrating the structure of a data processing apparatus according to one embodiment.
[0118] like Figure 7 As shown, the data processing device 70 is applied to, for example... Figure 2 The data processing architecture shown may include, specifically, the data processing device 70, which may include:
[0119] Module 701 is used to obtain the performance index score of the tuning task running M times in the Nth iteration.
[0120] Module 702 is used to determine the number of times the tuning task will run in the (N+1)th iteration based on the performance index score of running M times in the Nth iteration.
[0121] Processing module 703 is used to filter target performance index scores that meet the second preset condition from the Nth iteration to the Yth iteration if the performance index score of the target number of iterations meets the first preset condition.
[0122] Output module 704 is used to determine the target round iteration to which the run number of the target performance index score belongs as the parameter tuning result.
[0123] Based on this, the data processing apparatus 70 provided in the embodiments of this application will be described in detail below.
[0124] In one or more possible embodiments, the determining module 702 in this application embodiment can be specifically used to calculate a first difference value between the performance index scores of the M runs based on the performance index scores of the M runs in the Nth iteration.
[0125] If the first difference value is less than or equal to the first preset difference value, it is determined that the number of times the optimization task is run in the (N+1)th iteration is less than M times in the Nth iteration.
[0126] In another or more possible embodiments, the determining module 702 in this application embodiment can also be used to determine, when the first difference value is greater than the first preset difference value, that the number of times the optimization task runs in the N+1th iteration is greater than or equal to M times in the Nth iteration.
[0127] In another possible embodiment, the determining module 702 in this application embodiment can also be used to determine that the performance index score of the target run meets the first preset condition when the target run is the number of runs in at least two consecutive iterations from the Nth iteration to the Yth iteration and the performance index score of the target run is lower than the preset performance index score.
[0128] In another or more possible embodiments, the determining module 702 in this application embodiment can also be used to determine that the performance index score of the target run meets the first preset condition when the target run is the number of runs in each iteration from the Nth iteration to the Yth iteration and the second difference value between the performance index scores of the target run is less than or equal to the second preset difference value.
[0129] In another or more possible embodiments, the processing module 703 in this application embodiment can be specifically used to determine the highest performance index score as the target performance index score in the performance index score of the target run.
[0130] Based on the same inventive concept, embodiments of this application provide a data processing apparatus corresponding to the aforementioned data processing method based on a relational database. Specifically, in conjunction with... Figure 8 Please provide a detailed explanation.
[0131] Figure 8 This is a schematic diagram illustrating the structure of a data processing apparatus based on a relational database according to one embodiment.
[0132] like Figure 8 As shown, the data processing device 80 based on a relational database is applied to, for example... Figure 2 The data processing architecture shown, specifically the data processing device 80 based on a relational database, may include:
[0133] The acquisition module 801 is used to acquire multiple sets of parameters in the relational database according to the parameter tuning task of the relational database. The parameter sets include multiple parameters to be tuned and the parameter values of the parameters to be tuned.
[0134] Calculation module 802 is used to calculate the performance index score of the parameter set running M times in the Nth iteration during the parameter tuning task.
[0135] Module 803 is used to determine the number of times the tuning task will run in the (N+1)th iteration based on the performance index score of running M times in the Nth iteration.
[0136] The filtering module 804 is used to filter the target performance index scores that meet the second preset condition from the Nth iteration to the Yth iteration if the performance index scores of the target run meet the first preset condition.
[0137] The output module 805 is used to determine the target parameter set, which includes the target round iteration to which the target performance index score belongs, as the parameter tuning result of the relational database. Multiple parameter sets include the target parameter set, which includes multiple tuned parameters and the parameter values of the tuned parameters.
[0138] Based on the same inventive concept, embodiments of this application provide a data processing apparatus corresponding to the aforementioned data processing method based on network services. Specifically, in conjunction with... Figure 9 Please provide a detailed explanation.
[0139] Figure 9 This is a schematic diagram illustrating the structure of a network service-based data processing apparatus according to one embodiment.
[0140] like Figure 9 As shown, the network service-based data processing device 90 is applied to, for example... Figure 2 The data processing architecture shown, specifically the network service-based data processing device 90, may include:
[0141] The receiving module 901 is used to receive the tuning request sent by the user. The tuning request includes multiple sets of parameters, and the parameter sets include multiple parameters to be tuned and the parameter values of the parameters to be tuned.
[0142] Module 902 is used to obtain the performance index score of the parameter set running M times in the Nth iteration during the execution of the tuning request.
[0143] Module 903 is used to determine the number of times the tuning task will run in the (N+1)th iteration based on the performance index score of running M times in the Nth iteration.
[0144] The filtering module 904 is used to filter the target performance index scores that meet the second preset condition from the Nth iteration to the Yth iteration if the performance index scores of the target run meet the first preset condition.
[0145] The sending module 905 is used to send parameter tuning results to the user. The parameter tuning results include the target round iteration to which the target performance index score belongs and the target parameter set of the target round iteration. The target parameter set includes multiple tuned parameters and the parameter values of the tuned parameters.
[0146] Therefore, the performance index score of the tuning task is obtained after running M times in the Nth iteration. Based on the performance index score of the M times in the Nth iteration, the number of times the tuning task will run in the N+1th iteration is determined. This allows for dynamic control of the number of iterations and the number of runs in each iteration for different environments and effects, avoiding the impact of highly volatile load environments on the performance index score of each run in each iteration, ensuring the reliability of the tuning effect and improving the accuracy of parameter tuning efficiency. Then, if the performance index score of the target number of runs in the Nth to Yth iterations meets the first preset condition, the target performance index score that meets the second preset condition is determined in the Nth to Yth iterations. The target iteration to which the run of the target performance index score belongs is determined as the parameter tuning result. In this way, if the parameter tuning effect has been achieved in a certain iteration, it is not necessary to continue running, avoiding resource waste and shortening the parameter tuning time.
[0147] Figure 10 This is a schematic diagram illustrating the hardware structure of a computer device according to one embodiment.
[0148] like Figure 10 As shown, the computer device 1000 includes an input device 1001, an input interface 1002, a processor 1003, a memory 1004, an output interface 1005, and an output device 1006.
[0149] Input interface 1002, processor 1003, memory 1004, and output interface 1005 are interconnected via bus 1010. Input device 1001 and output device 1006 are connected to bus 1010 via input interface 1002 and output interface 1005 respectively, and thus connected to other components of computer device 1000. Specifically, input device 1001 receives input information from the outside and transmits the input information to processor 1003 via input interface 1002; processor 1003 processes the input information based on computer-executable instructions stored in memory 1004 to generate output information, temporarily or permanently stores the output information in memory 1004, and then transmits the output information to output device 1006 via output interface 1005; output device 1006 outputs the output information to the outside of computer device 1000 for user use.
[0150] In one embodiment, Figure 10 The computer device 1000 shown can be implemented as a data processing device, which may include: a memory configured to store a program; and a processor configured to run the program stored in the memory to perform the data processing method described in the above embodiments.
[0151] In one embodiment, the memory can also be used to store performance metric scores, target performance metric scores, and combinations thereof. Figures 3 to 6 The calculation results of each step in the described data processing process.
[0152] According to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as a computer-readable storage medium. For example, embodiments of this application include a computer-readable storage medium that stores a program or instructions on the medium, which, when executed by a computer device, causes the computer device to perform the steps of the above-described method.
[0153] According to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network, and / or installed from a removable storage medium.
[0154] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions that, when run on a computer, cause the computer to perform the methods described in the various embodiments above. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives), etc.
[0155] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0156] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.
Claims
1. A data processing method, comprising: Obtain the performance score of the tuning task after running M times in the Nth iteration; Based on the performance index scores run M times in the Nth iteration, determine the first difference value between the performance index scores of the M times; The number of times the optimization task is run in the N+1th iteration is determined based on the first difference value; If the performance index score of the target number of runs in the Nth iteration to the Yth iteration meets the first preset condition, the target performance index score is determined in the Nth iteration to the Yth iteration; the target number of runs is the number of runs in each iteration in the Nth iteration to the Yth iteration. The target performance index score is the highest performance index score among the performance index scores of the target run. The first preset condition includes: the difference between the performance index scores of the target run is less than or equal to a preset difference value, or, in at least two consecutive iterations, the performance index score of the target run is lower than the preset performance index score. The target round iteration to which the run number of the target performance index score belongs is determined as the parameter tuning result.
2. The method according to claim 1, wherein, If the first difference value is less than or equal to the first preset difference value, the number of times the optimization task is run in the (N+1)th iteration is less than the number of times it is run in the Nth iteration.
3. The method according to claim 2, wherein, If the first difference value is greater than the first preset difference value, the number of times the optimization task is run in the (N+1)th iteration is greater than or equal to M times in the Nth iteration.
4. A data processing method based on a relational database, comprising: According to the parameter tuning task of the relational database, obtain multiple sets of parameters in the relational database, the parameter sets include multiple parameters to be tuned and the parameter values of the parameters to be tuned; During the parameter tuning process, the performance score of the parameter set is calculated after running M times in the Nth iteration. Based on the performance index scores run M times in the Nth iteration, determine the first difference value between the performance index scores of the M times; The number of times the optimization task is run in the N+1th iteration is determined based on the first difference value; If the performance index score of the target number of runs in the Nth iteration to the Yth iteration meets the first preset condition, then the iterations from the Nth iteration to the Yth iteration will be selected to meet the target performance index score; the target number of runs is the number of runs in each iteration from the Nth iteration to the Yth iteration. The target performance index score is the highest performance index score among the performance index scores of the target run. The first preset condition includes: the difference between the performance index scores of the target runs is less than or equal to a preset difference value, or, in at least two consecutive iterations, the performance index score of the target runs is lower than the preset performance index score; based on the target iteration to which the run of the target performance index score belongs, the set of target parameters including the run of the target iteration is determined as the parameter tuning result of the relational database, the multiple parameter sets include the target parameter set, the target parameter set includes multiple tuned parameters and the parameter values of the tuned parameters.
5. A data processing method based on network services, comprising: Receive a tuning request sent by a user, the tuning request including multiple sets of parameters, the parameter sets including multiple parameters to be tuned and the parameter values of the parameters to be tuned; During the execution of the tuning request, the performance index score of the parameter set is obtained after running M times in the Nth iteration; Based on the performance index scores run M times in the Nth iteration, determine the first difference value between the performance index scores of the M times; The number of times the optimization task is run in the N+1th iteration is determined based on the first difference value. If the performance index score of the target number of runs in the Nth iteration to the Yth iteration meets the first preset condition, the target performance index score is determined in the Nth iteration to the Yth iteration; the target number of runs is the number of runs in each iteration in the Nth iteration to the Yth iteration. The target performance index score is the highest performance index score among the performance index scores of the target run. The first preset condition includes: the second difference value between the performance index scores of the target number of runs is less than or equal to the second preset difference value, or the target number of runs is the number of runs in at least two consecutive iterations from the Nth iteration to the Yth iteration, and the performance index score of the target number of runs is lower than the preset performance index score. The parameter tuning results are sent to the user. The parameter tuning results include the target round iteration to which the target performance index score belongs and the target parameter set of the target round iteration. The target parameter set includes multiple tuned parameters and the parameter values of the tuned parameters.
6. A data processing apparatus, comprising: The acquisition module is used to obtain the performance index scores of the tuning task after running M times in the Nth iteration. The determination module is used to determine the first difference value between the performance index scores of the M runs in the Nth iteration based on the performance index scores of the M runs in the Nth iteration. The number of times the optimization task is run in the N+1th iteration is determined based on the first difference value; The processing module is used to filter target performance index scores from the Nth to the Yth iterations if the performance index score of the target number of runs meets a first preset condition; the target number of runs is the number of runs in each iteration from the Nth to the Yth iterations. The target performance index score is the highest performance index score among the performance index scores of the target run. The first preset condition includes: the second difference value between the performance index scores of the target number of runs is less than or equal to the second preset difference value, or the target number of runs is the number of runs in at least two consecutive iterations from the Nth iteration to the Yth iteration, and the performance index score of the target number of runs is lower than the preset performance index score. The output module is used to determine the target round iteration to which the run number of the target performance index score belongs as the parameter tuning result.
7. A data processing apparatus based on a relational database, comprising: The acquisition module is used to acquire multiple sets of parameters in the relational database according to the parameter tuning task of the relational database. The parameter sets include multiple parameters to be tuned and the parameter values of the parameters to be tuned. The calculation module is used to calculate the performance index score of the parameter set running M times in the Nth iteration during the parameter tuning task. The determination module is used to determine the first difference value between the performance index scores of the M runs in the Nth iteration based on the performance index scores of the M runs in the Nth iteration. The number of times the optimization task is run in the N+1th iteration is determined based on the first difference value; The filtering module is used to filter target performance index scores from the Nth to the Yth iterations if the performance index score of the target number of runs meets a first preset condition; the target number of runs refers to the number of runs in each iteration from the Nth to the Yth iterations. The target performance index score is the highest performance index score among the performance index scores of the target run. The first preset condition includes: the second difference value between the performance index scores of the target number of runs is less than or equal to the second preset difference value, or the target number of runs is the number of runs in at least two consecutive iterations from the Nth iteration to the Yth iteration, and the performance index score of the target number of runs is lower than the preset performance index score. The output module is used to determine the target parameter set, which includes the target round iteration to which the target performance index score belongs, as the parameter tuning result of the relational database. The multiple parameter sets include the target parameter set, which includes multiple tuned parameters and the parameter values of the tuned parameters.
8. A data processing apparatus based on network services, comprising: The receiving module is used to receive the tuning request sent by the user. The tuning request includes multiple sets of parameters, and the parameter sets include multiple parameters to be tuned and the parameter values of the parameters to be tuned. The acquisition module is used to acquire the performance index scores of the parameter set running M times in the Nth iteration during the execution of the tuning request. The determination module is used to determine the first difference value between the performance index scores of the M runs in the Nth iteration based on the performance index scores of the M runs in the Nth iteration. The number of times the optimization task is run in the N+1th iteration is determined based on the first difference value. The filtering module is used to filter target performance index scores from the Nth to the Yth iterations if the performance index score of the target number of runs meets a first preset condition; the target number of runs refers to the number of runs in each iteration from the Nth to the Yth iterations. The target performance index score is the highest performance index score among the performance index scores of the target run. The first preset condition includes: the second difference value between the performance index scores of the target number of runs is less than or equal to the second preset difference value, or the target number of runs is the number of runs in at least two consecutive iterations from the Nth iteration to the Yth iteration, and the performance index score of the target number of runs is lower than the preset performance index score. The sending module is used to send parameter tuning results to the user. The parameter tuning results include the target round iteration to which the running number of the target performance index score belongs and the target parameter set of the target round iteration. The target parameter set includes multiple tuned parameters and the parameter values of the tuned parameters.
9. A computer device, comprising: Memory and processor The memory is used to store computer programs; The processor is configured to execute a computer program stored in the memory. When the computer program is executed, the processor performs the steps of the data processing method according to any one of claims 1 to 3, or the steps of the data processing method based on a relational database according to claim 4, or the steps of the data processing method based on a network service according to claim 5.
10. A computer-readable storage medium storing a program or instructions that, when executed by a computer device, cause the computer device to perform the steps of the data processing method as claimed in any one of claims 1 to 3, or the steps of the data processing method based on a relational database as claimed in claim 4, or the steps of the data processing method based on a network service as claimed in claim 5.
11. A computer program product comprising a computer program that, when executed by a computer device, causes the computer device to perform the steps of the data processing method as described in any one of claims 1 to 3, or the steps of the data processing method based on a relational database as described in claim 4, or the steps of the data processing method based on a network service as described in claim 5.