Data processing method and device, electronic equipment and computer readable medium

Abstract

The invention discloses a data processing method and device, electronic equipment and a computer readable medium, and relates to the technical field of big data processing.The method comprises the steps that received target parameter configuration is operated, and performance index data is obtained; in response to the condition that the iteration end condition is not met, updating the knowledge base according to the performance index data and obtaining updated configuration parameters; calling an interpretive framework to identify importance values of the updated configuration parameters to determine to-be-adjusted parameters and to-be-merged parameters, and merging the to-be-merged parameters to obtain merged parameters; and determining parameter types and discrete step sizes of the to-be-adjusted parameters and the merging parameters, carrying out tuning test on the to-be-adjusted parameters and the merging parameters in the database based on an iteration end condition to determine tuning parameter values of the to-be-adjusted parameters and the merging parameters, and outputting the tuning parameter values. The optimization efficiency and accuracy of database parameter configuration are improved, and the parameter configuration of the database meets the scale of the database and the complexity of an application program.

Description

Technical Field

[0001] The present application relates to the technical field of big data processing, and in particular to a data processing method, device, electronic device and computer-readable medium. Background Art

[0002] As one of the core service systems, the equity center needs to uniformly manage the creation and management of equity activities, the issuance, consumption and settlement of coupons, etc. Each equity promotion activity will require massive amounts of data to be stored, retrieved and modified, which results in database performance that not only affects user experience, but also directly affects the stable operation and cost-effectiveness of the business. Various parameters control the reading and writing, memory, cache, log files and network of the database system, but currently the configuration of database parameters is mostly performed by database administrators. The manual tuning of database parameter configuration alone is no longer able to meet the scale of the database and the complexity of the application. The data processing efficiency and accuracy related to the database parameter configuration are low. Summary of the invention

[0003] In view of this, the embodiments of the present application provide a data processing method, device, electronic device and computer-readable medium, which can solve the existing problems of low efficiency and low accuracy of data processing related to database parameter configuration.

[0004] To achieve the above objective, according to one aspect of an embodiment of the present application, a data processing method is provided, including:

[0005] Running the received inferred target parameter configuration to obtain performance indicator data;

[0006] In response to not meeting the iteration end condition, updating the knowledge base according to the performance indicator data, and obtaining the updated configuration parameters in the knowledge base;

[0007] invoking an interpretative framework to identify importance values ​​of updated configuration parameters;

[0008] Determine the parameters to be adjusted and the parameters to be merged according to the importance values, and merge the parameters to be merged to obtain the merged parameters;

[0009] Determine the parameter type and discrete step length of the parameters to be adjusted and the merged parameters, and then perform tuning tests on the parameters to be adjusted and the merged parameters in the database based on the iteration end condition to determine the tuning parameter values ​​of the parameters to be adjusted and the merged parameters, and output the tuning parameter values.

[0010] Optionally, before running the received inferred target parameter configuration, the method further includes:

[0011] The optimizer is called to infer a target parameter configuration for obtaining a performance above a preset threshold based on the existing data in the knowledge base.

[0012] Optionally, invoking an interpretive framework to identify updated importance values ​​of configuration parameters, including:

[0013] The updated configuration parameters are input into the explanatory framework to determine the importance value of each input updated configuration parameter in the decision making.

[0014] Optionally, the parameters to be adjusted and the parameters to be merged are determined according to the importance values, including:

[0015] The updated configuration parameters that are greater than the preset importance threshold are determined as parameters to be adjusted, and the updated configuration parameters that are less than the preset importance threshold are determined as parameters to be merged.

[0016] Optionally, determining tuning parameter values ​​of the parameters to be adjusted and the merged parameters includes:

[0017] Determine the value range of the parameter to be adjusted and the combined parameter, and determine the discrete set corresponding to each parameter to be adjusted and each combined parameter according to the value range and the discrete step length;

[0018] Performing numerical modification and tuning tests on the parameters to be adjusted and the merged parameters based on the discrete set, scaling the parameter values ​​obtained by this iterative tuning test to a first value range, wherein the minimum value of the first value range is a special value;

[0019] Obtaining a biased special value sampling probability associated with the optimizer, and obtaining a second value range based on the biased special value sampling probability, wherein a minimum value of the second value range is a special value;

[0020] Determine a third value range based on the value range and the first value range;

[0021] If the parameter value is within the second value range, the tuning parameter value of the parameter for this iterative tuning test is determined to be a special value. If the parameter value is not within the second value range, the tuning parameter value of the parameter for this iterative tuning test is determined based on the parameter value and the third value range.

[0022] Optionally, the method further comprises:

[0023] Determine the set deviation values ​​of the parameters to be adjusted and the merged parameters, and determine the parameters for improving database performance based on the set deviation values ​​and the tuning parameter values ​​of the corresponding parameters;

[0024] Output database performance improvement parameters.

[0025] In addition, the present application also provides a data processing device, including:

[0026] an operation unit configured to operate the received inferred target parameter configuration to obtain performance indicator data;

[0027] an acquisition unit, configured to update the knowledge base according to the performance indicator data in response to failure to meet the iteration end condition, and acquire the updated configuration parameters in the knowledge base;

[0028] an identification unit configured to call the interpretive framework to identify an importance value of the updated configuration parameter;

[0029] A parameter determination unit is configured to determine the parameters to be adjusted and the parameters to be merged according to the importance values, and merge the parameters to be merged to obtain the merged parameters;

[0030] The tuning test unit is configured to determine the parameter type and discrete step length of the parameters to be adjusted and the merged parameters, and then perform tuning tests on the parameters to be adjusted and the merged parameters in the database based on the iteration end condition to determine the tuning parameter values ​​of the parameters to be adjusted and the merged parameters, and output the tuning parameter values.

[0031] Optionally, the device further comprises a calculation unit configured to:

[0032] The optimizer is called to infer a target parameter configuration for obtaining a performance above a preset threshold based on the existing data in the knowledge base.

[0033] Optionally, the identification unit is further configured to:

[0034] The updated configuration parameters are input into the explanatory framework to determine the importance value of each input updated configuration parameter in the decision making.

[0035] Optionally, the parameter determination unit is further configured to:

[0036] The updated configuration parameters that are greater than the preset importance threshold are determined as parameters to be adjusted, and the updated configuration parameters that are less than the preset importance threshold are determined as parameters to be merged.

[0037] Optionally, the tuning test unit is further configured to:

[0038] Determine the value range of the parameter to be adjusted and the combined parameter, and determine the discrete set corresponding to each parameter to be adjusted and each combined parameter according to the value range and the discrete step length;

[0039] Performing numerical modification and tuning tests on the parameters to be adjusted and the merged parameters based on the discrete set, scaling the parameter values ​​obtained by this iterative tuning test to a first value range, wherein the minimum value of the first value range is a special value;

[0040] Obtaining a biased special value sampling probability associated with the optimizer, and obtaining a second value range based on the biased special value sampling probability, wherein a minimum value of the second value range is a special value;

[0041] Determine a third value range based on the value range and the first value range;

[0042] If the parameter value is within the second value range, the tuning parameter value of the parameter for this iterative tuning test is determined to be a special value. If the parameter value is not within the second value range, the tuning parameter value of the parameter for this iterative tuning test is determined based on the parameter value and the third value range.

[0043] Optionally, the device further includes a database performance improvement parameter determination unit configured to:

[0044] Determine the set deviation values ​​of the parameters to be adjusted and the merged parameters, and determine the parameters for improving database performance based on the set deviation values ​​and the tuning parameter values ​​of the corresponding parameters;

[0045] Output database performance improvement parameters.

[0046] In addition, the present application also provides a data processing electronic device, including: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by one or more processors, the one or more processors implement the data processing method as described above.

[0047] In addition, the present application also provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processor, the data processing method as described above is implemented.

[0048] To achieve the above objective, according to another aspect of the embodiments of the present application, a computer program product is provided.

[0049] A computer program product according to an embodiment of the present application includes a computer program, and when the program is executed by a processor, the data processing method provided by the embodiment of the present application is implemented.

[0050] One embodiment of the above invention has the following advantages or beneficial effects: the application obtains performance indicator data by running the received inferred target parameter configuration; in response to not meeting the iteration end condition, updates the knowledge base according to the performance indicator data, and obtains the updated configuration parameters in the knowledge base; calls the explanatory framework to identify the importance value of the updated configuration parameter; determines the parameters to be adjusted and the parameters to be merged according to the importance value, merges the parameters to be merged to obtain the merged parameters; determines the parameter type and discrete step length of the parameters to be adjusted and the merged parameters, and then performs tuning tests on the parameters to be adjusted and the merged parameters in the database based on the iteration end condition to determine the tuning parameter values ​​of the parameters to be adjusted and the merged parameters, and outputs the tuning parameter values. Improve the tuning efficiency and accuracy of database parameter configuration, and achieve that the database parameter configuration meets the scale of the database and the complexity of the application.

[0051] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific implementation examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings are used to better understand the present application and do not constitute an improper limitation on the present application.

[0053] Figure 1 is a schematic diagram of the main flow of a data processing method according to an embodiment of the present application;

[0054] Figure 2 is a schematic diagram of the main flow of a data processing method according to an embodiment of the present application;

[0055] Figure 3 It is a main flow chart of a data processing method according to an embodiment of the present application;

[0056] Figure 4 is a schematic diagram of main units of a data processing device according to an embodiment of the present application;

[0057] Figure 5 is an exemplary system architecture diagram to which the embodiments of the present application can be applied;

[0058] Figure 6 It is a structural diagram of a computer system of a terminal device or server suitable for implementing an embodiment of the present application. DETAILED DESCRIPTION

[0059] The following is a description of the exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, and they should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of known functions and structures is omitted in the following description. It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of the present application are in compliance with the relevant provisions of national laws and regulations. It should be noted that in the embodiments of the present application, certain software, components, models, and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or must have used the solution. In the technical solution of the present application, the collection, analysis, use, transmission, storage, and other aspects of the user's personal information involved are in compliance with the provisions of relevant laws and regulations, and are used for legal and reasonable purposes, and are not shared, leaked, or sold outside of these legal uses, and are subject to supervision and management by regulatory authorities. Take necessary measures to prevent illegal access to such personal information, ensure that persons with access to personal information comply with relevant laws and regulations, and ensure the security of personal information. Once such personal information is no longer needed, risks should be minimized by limiting or even prohibiting data collection and / or deleting data.

[0060] When used, including in certain related applications, user privacy is protected by de-identifying data, such as by removing specific identifiers when used, controlling the amount or specificity of data stored, controlling how data is stored, and / or other methods of de-identification.

[0061] Figure 1 is a schematic diagram of the main flow of a data processing method according to an embodiment of the present application, such as Figure 1 As shown, the data processing method mainly includes the following steps S101 to S105.

[0062] Step S101, running the received inferred target parameter configuration to obtain performance indicator data.

[0063] In this embodiment, after receiving the data processing request, the execution subject of the data processing method (for example, a controller) can receive the inferred target parameter configuration, and run the received inferred target parameter configuration in real time in a database management system (DBMS), and obtain performance indicator data obtained by the database management system based on the target parameter configuration. For example, the performance indicator data of the database management system may include: optimized configuration parameter values ​​and corresponding response time, throughput, CPU utilization, memory utilization, etc.

[0064] Specifically, before running the received inferred target parameter configuration, the method further includes: calling an optimizer to infer a target parameter configuration for obtaining a performance higher than a preset threshold based on existing data in the knowledge base.

[0065] The optimizer is called to infer a target parameter configuration for obtaining a performance higher than a preset threshold based on a large amount of configuration parameter data already in the knowledge base, that is, to infer a parameter configuration that may obtain a higher database management system performance.

[0066] Step S102: in response to the iteration end condition not being met, updating the knowledge base according to the performance indicator data, and acquiring the updated configuration parameters in the knowledge base.

[0067] The iteration end condition may include the number of iterations, iteration time or other cost limits. If the iteration end condition is not met, and the performance corresponding to the acquired performance indicator data is higher than the performance corresponding to the stored performance indicator data in the knowledge base, the execution subject may compare and update the configuration parameters and corresponding configuration parameter values ​​in the knowledge base based on the optimized configuration parameter values ​​in the acquired performance indicator data and the corresponding response time, throughput, CPU utilization, memory utilization, etc. Obtain the updated configuration parameters in the knowledge base.

[0068] Step S103 , calling the interpretive framework to identify the importance value of the updated configuration parameter.

[0069] Specifically, calling the interpretive framework to identify the importance value of the updated configuration parameter includes: inputting the updated configuration parameters into the interpretive framework to determine the importance value of each input updated configuration parameter in the decision.

[0070] Explanatory frameworks such as SHAP are an important concept in machine learning and data science, which stands for interpretability and model understanding. Using SHAP, you can accurately and quickly identify the importance of each input configuration parameter in the decision to improve database performance.

[0071] Step S104: determining the parameters to be adjusted and the parameters to be merged according to the importance values, and merging the parameters to be merged to obtain the merged parameters.

[0072] The execution subject can obtain the preset first importance value range corresponding to the parameter to be adjusted and the preset second importance value range corresponding to the parameter to be merged, determine the configuration parameters corresponding to the importance values ​​within the first importance value range as the parameter to be adjusted, determine the configuration parameters corresponding to the importance values ​​within the second importance value range as the parameter to be merged, merge the parameters to be merged to obtain the merged parameters, thereby avoiding missing parameters and improving the efficiency of database parameter configuration tuning. Any first importance value within the first importance value range is higher than any second importance value within the second importance value range, so as to ensure the accuracy of determining the parameters to be adjusted and the parameters to be merged, thereby improving the accuracy of corresponding data processing.

[0073] Step S105, determining the parameter type and discrete step length of the parameters to be adjusted and the combined parameters, and then performing tuning tests on the parameters to be adjusted and the combined parameters in the database based on the iteration end condition to determine the tuning parameter values ​​of the parameters to be adjusted and the combined parameters, and outputting the tuning parameter values.

[0074] Parameter types can include numeric and categorical types. Parameters of each parameter type have special values. A biased special value sampling probability p is associated with the optimizer. Assume that [0, max] is the value range of the parameter, where 0 is a special value. If the parameter value inferred by the optimizer in a certain iteration is v, we will scale this value to [0, 1]. If the scaled value is in the range [0, p], the value of the parameter in this iteration is 0, otherwise it will be uniformly enlarged to the value in the range [1, max]. The values ​​of different parameters of the database can be discrete or continuous, and the value ranges are also very different. For some parameters with a large value range, small changes are unlikely to significantly affect the performance of the database; some parameters require fine-tuning. To balance these effects, different k values, that is, discrete step lengths, are set for parameters with different value ranges. If the value range of a parameter is [0,1000] and the discrete step length k is set to 100, the new value range of the parameter is a discrete set [0,100,200,...,1000]. Value changes below k=100 will be considered to have no effect on database performance.

[0075] Specifically, the tuning parameter values ​​of the parameters to be adjusted and the parameters to be merged are determined, including: determining the value ranges of the parameters to be adjusted and the parameters to be merged, and determining the discrete sets corresponding to each parameter to be adjusted and each merged parameter according to the value ranges and discrete steps; performing numerical modification tuning tests on the parameters to be adjusted and the parameters to be merged based on the discrete sets in the database under the constraints of the iteration end conditions, and scaling the parameter values ​​obtained in this iterative tuning test to a first value range, wherein the minimum value of the first value range is a special value; obtaining a biased special value sampling probability associated with the optimizer, and obtaining a second value range based on the biased special value sampling probability, wherein the minimum value of the second value range is a special value; determining a third value range based on the value range and the first value range; if the parameter value is within the second value range, determining the tuning parameter value of the parameter for this iterative tuning test to be a special value, and if the parameter value is not within the second value range, determining the tuning parameter value of the parameter for this iterative tuning test based on the parameter value and the third value range.

[0076] For example, taking the value range [0, max] of one of the parameters to be adjusted and the merged parameters, for example, [0, 1000] as an example, according to the value range [0, 1000] and the discrete step length k = 100, determine the discrete set [0, 100, 200, ..., 1000] corresponding to each parameter to be adjusted and each merged parameter; based on the discrete set [0, 100, 200, ..., 1000], perform numerical modification tuning test on the parameter to be adjusted and the merged parameter, and scale the parameter value obtained by this iterative tuning test to the first value range, for example, [0, 1], wherein the minimum value 0 of the first value range [0, 1] is a special value; obtain the sampling probability p of the bias special value associated with the optimizer, based on the bias The special value sampling probability p obtains the second value range [0, p], wherein the minimum value 0 of the second value range [0, p] is a special value; the value range [0, 1000] of the parameter of this iterative tuning test is obtained, wherein the minimum value 0 of the value range [0, 1000] is a special value, and the third value range [1, 1000] is determined based on the value range [0, 1000] and the first value range [0, 1]; if the parameter value is within the second value range [0, p], the tuning parameter value of the parameter of this iterative tuning test is determined to be the special value 0; if the parameter value is not within the second value range [0, p], the tuning parameter value of the parameter of this iterative tuning test is determined based on the parameter value and the third value range [1, 1000].

[0077] This embodiment obtains performance indicator data by running the received inferred target parameter configuration; in response to not meeting the iteration end condition, updates the knowledge base according to the performance indicator data and obtains the updated configuration parameters in the knowledge base; calls the explanatory framework to identify the importance value of the updated configuration parameters; determines the parameters to be adjusted and the parameters to be merged according to the importance value, merges the parameters to be merged to obtain the merged parameters; determines the parameter type and discrete step length of the parameters to be adjusted and the merged parameters, and then performs tuning tests on the parameters to be adjusted and the merged parameters in the database based on the iteration end condition to determine the tuning parameter values ​​of the parameters to be adjusted and the merged parameters, and outputs the tuning parameter values. Improve the tuning efficiency and accuracy of database parameter configuration, and achieve that the database parameter configuration meets the scale of the database and the complexity of the application.

[0078] Figure 2 is a schematic diagram of the main flow of a data processing method according to an embodiment of the present application, such as Figure 2 As shown, the data processing method mainly includes the following steps S201 to S207.

[0079] Step S201, running the received inferred target parameter configuration to obtain performance indicator data.

[0080] For example, the controller can receive a target parameter configuration that may achieve higher performance, which is inferred by the optimizer based on the existing data in the knowledge base, and then use this target parameter configuration to run in the database management system. After the database management system executes for 5-10 minutes, the controller collects the performance indicator data of the database management system and sends the data to the knowledge base, and finally compares and updates the knowledge base.

[0081] Step S202: in response to the iteration end condition not being met, updating the knowledge base according to the performance indicator data, and obtaining the updated configuration parameters in the knowledge base.

[0082] The iteration end condition may be the number of iterations, iteration time, or other cost restrictions. If the performance corresponding to the performance indicator data obtained before the iteration end condition is met is higher than the performance corresponding to the performance indicator data stored in the knowledge base, the execution subject may compare and update the knowledge base based on the obtained performance indicator data to improve the accuracy of data processing.

[0083] Step S203 , calling the interpretive framework to identify the importance value of the updated configuration parameter.

[0084] Step S204: determine the updated configuration parameters greater than the preset importance threshold as parameters to be adjusted, determine the updated configuration parameters less than the preset importance threshold as parameters to be merged, and merge the parameters to be merged to obtain merged parameters.

[0085] The updated configuration parameters that are less than the preset importance threshold are determined as parameters to be merged, and the parameters to be merged are merged to obtain merged parameters, that is, several unimportant parameters are merged into one, which can significantly improve the efficiency of database parameter configuration tuning and avoid missing parameters.

[0086] Step S205, determining the parameter type and discrete step length of the parameters to be adjusted and the combined parameters, and then performing tuning tests on the parameters to be adjusted and the combined parameters in the database based on the iteration end condition to determine the tuning parameter values ​​of the parameters to be adjusted and the combined parameters, and outputting the tuning parameter values.

[0087] Based on the determined parameter types and discrete step lengths of the parameters to be adjusted and the parameters to be merged, iteration end conditions, and database, a tuning test of the parameters to be adjusted and the parameters to be merged is performed to accurately determine the tuning parameter values ​​of the parameters to be adjusted and the parameters to be merged, and output the tuning parameter values.

[0088] Step S206, determining the set deviation values ​​of the parameters to be adjusted and the merged parameters, and determining the database performance improvement parameters based on the set deviation values ​​and the tuning parameter values ​​of the corresponding parameters.

[0089] The configuration calculated by the optimizer based on the existing data may enable the database to achieve higher performance after DBMS testing, but the improvement in database performance cannot be attributed to specific values. That is, when the number of parameters is large, the optimizer cannot accurately find which parameter changes lead to improved database performance. For this reason, the deviation value (i.e., setting deviation) is set to α for the parameter, and the number of samples evaluating special values ​​follows a binomial distribution. , where α=20% can be set by default. For example, when the tuning parameter value (i.e., the parameter value obtained by iterative optimization) exceeds the set deviation value α (e.g., 20%) of the initial parameter value, it can be determined that the change in the parameter leads to the improvement of database performance, that is, the parameter is the parameter that improves database performance, and the parameter is the database performance improvement parameter.

[0090] Step S207: outputting database performance improvement parameters.

[0091] The execution entity may output the database performance improvement parameters in various forms, such as in the form of graphs or tables, so that the user can accurately determine which parameter changes lead to improved database performance, thereby facilitating user decision-making.

[0092] Figure 3The following is a schematic diagram of the main flow of a data processing method according to an embodiment of the present application. Database parameter configuration tuning is an optimization problem. Taking the parameter tuning method based on Bayesian optimization as an example, the Bayesian optimization algorithm is used to find the optimal configuration of database parameters in the current scenario. For example, suppose the database system has n configuration parameters , the threshold value corresponding to the parameter is , configuration space , the parameter value can be continuous, discrete, or classified. Suppose the performance index evaluation function is , then the goal is to obtain a set of optimal configurations ,in The model framework of parameter configuration tuning mainly consists of three parts: knowledge base, optimizer and controller. The knowledge base saves the evaluated records. , and each evaluation optimization will be updated; the optimizer is based on the existing records in the knowledge base, and it is possible to obtain a higher performance configuration (Suggest new Configuration) The controller controls the execution of the entire tuning process, records parameters and outputs results. The optimization process is iterative. The optimization process is as follows: First, the optimizer infers a parameter configuration that may achieve higher performance based on the existing data in the knowledge base, and then uses this configuration (i.e., uses the set parameters) to run in the database management system (DBMS). After the DBMS executes for 5-10 minutes, the parameters are passed to the controller. The controller collects the performance indicator data of the DBMS and sends the collected performance indicator data to the knowledge base. Finally, the knowledge base is compared and updated. Repeat the above steps until the iteration ends (the conditions can be the number of iterations, iteration time, or other cost restrictions).

[0093] Database management systems have a large number of configuration parameters, many of which have little impact on database performance, so adjusting some (important) parameter configurations can make database performance close to the optimal situation. Reducing parameters can significantly improve the tuning process.

[0094] Based on previous work, we first use SHAP to rank the importance of database parameters and then define a parameter configuration space , the threshold value corresponding to the parameter is , where d is much smaller than n. Combine several "unimportant" parameters with similar values ​​into one and define and , and This method can significantly improve the efficiency of database parameter configuration tuning and avoid missing parameters.

[0095] The configuration calculated by the optimizer based on the existing data may achieve higher database performance after DBMS testing, but the improvement in database performance cannot be attributed to specific values. That is, when the number of parameters is large, the optimizer cannot accurately find which parameter changes lead to improved database performance. To this end, set the deviation value for the parameter to , the number of samples evaluating a particular value follows a binomial distribution , where the default setting =20%. For example, when the parameter value obtained by iterative optimization exceeds the set deviation value of the initial parameter value When the change in this parameter is greater than 20%, for example, it can be determined that the improvement in database performance is caused by the change in this parameter.

[0096] Database parameter types can be divided into numerical and categorical parameters. Parameters of categorical types: such as skip_name_resolve, query_cache_type, autocommit, etc., control the enablement / disability of certain functions. Choosing different values ​​will lead to substantial changes in database performance. Many numerical parameters also have special values. When these parameters are set to special values, some functions will be enabled / disabled; when they are set to other values, they are similar to other common numerical parameters, such as setting the size of the buffer or cache, the speed of index processing, refresh delay, etc. For this purpose, a biased special value sampling probability p is associated with the optimizer. Assume that [0, max] is the value range of the parameter, where 0 is a special value. If the parameter value calculated by the optimizer in a certain iteration is v, this value is scaled to [0, 1]. If the scaled value is within the range of [0, p], the value of the parameter in this iteration is 0, otherwise it will be uniformly enlarged to the value in the range of [1, max].

[0097] The values ​​of different database parameters can be discrete or continuous, and the range of values ​​can also vary greatly. For some parameters with a large range of values, small changes are unlikely to significantly affect the performance of the database; some parameters need to be fine-tuned, such as innodb_flush_log_at_trx_commit, which affects the frequency of log flushing to disk. Small changes in parameter values ​​will affect database performance. To balance these effects, different k values ​​(i.e., discrete steps) are set for parameters with different ranges. If the range of a parameter is [0,1000], and k=100 is set, the new range of the parameter is a discrete set [0,100,200,...,1000]. Numerical changes below k=100 will be considered to have no effect on database performance.

[0098] For example, a parameter tuning test was performed on the MySQL database. Multiple configured database settings were used as the initial knowledge base. The optimizer was built using Python based on SAMC (based on the sequential model algorithm). The optimization was iterated 100 times. The DBMS was evaluated after running for 5 minutes each time. After the tuning was completed, the final throughput of the DBMS increased by an average of 11.16%.

[0099] In an embodiment of the present application, the database management system has a large number of configuration parameters. By sorting the importance of the database parameters and merging "unimportant" or similarly valued parameters, the efficiency of database parameter configuration tuning can be significantly improved, and at the same time, missing parameters can be avoided; setting deviation values ​​for the parameters can effectively increase the probability of the optimizer inferring better parameter values; for parameters with special values, associating the biased special value sampling probability with the optimizer can effectively improve the efficiency of optimizer parameter tuning; adjusting the parameter values ​​with a larger value range can improve the efficiency and stability of iterative tuning of database parameters and reduce the tuning time.

[0100] Figure 4 Schematic diagram of the main units of the data processing device according to an embodiment of the present application. Figure 4 As shown, the data processing device 400 includes an operating unit 401 , an acquiring unit 402 , an identifying unit 403 , a parameter determining unit 404 and a tuning and testing unit 405 .

[0101] The operation unit 401 is configured to operate the received inferred target parameter configuration, thereby obtaining performance indicator data.

[0102] The acquisition unit 402 is configured to update the knowledge base according to the performance indicator data in response to the iteration end condition not being met, and acquire the updated configuration parameters in the knowledge base.

[0103] The identification unit 403 is configured to call the interpretive framework to identify the importance value of the updated configuration parameter.

[0104] The parameter determination unit 404 is configured to determine the parameters to be adjusted and the parameters to be merged according to the importance values, and merge the parameters to be merged to obtain the merged parameters.

[0105] The tuning test unit 405 is configured to determine the parameter type and discrete step length of the parameters to be adjusted and the merged parameters, and then perform tuning tests on the parameters to be adjusted and the merged parameters in the database based on the iteration end condition to determine the tuning parameter values ​​of the parameters to be adjusted and the merged parameters, and output the tuning parameter values.

[0106] In some embodiments, the data processing device further includes Figure 4The inference unit not shown in the figure is configured to call the optimizer to infer a target parameter configuration for obtaining a performance higher than a preset threshold based on the existing data in the knowledge base.

[0107] In some embodiments, the identification unit 403 is further configured to: input the updated configuration parameters into the interpretative framework to determine the importance value of each input updated configuration parameter in the decision making.

[0108] In some embodiments, the parameter determination unit 404 is further configured to: determine updated configuration parameters greater than a preset importance threshold as parameters to be adjusted, and determine updated configuration parameters less than the preset importance threshold as parameters to be merged.

[0109] In some embodiments, the tuning test unit 405 is further configured to: determine the value range of the parameters to be adjusted and the merged parameters, and determine the discrete sets corresponding to each parameter to be adjusted and each merged parameter based on the value range and the discrete step length; perform numerical modification tuning tests on the parameters to be adjusted and the merged parameters based on the discrete sets, and scale the parameter values ​​obtained in this iterative tuning test to a first value range, wherein the minimum value of the first value range is a special value; obtain a biased special value sampling probability associated with the optimizer, and obtain a second value range based on the biased special value sampling probability, wherein the minimum value of the second value range is a special value; determine a third value range based on the value range and the first value range; if the parameter value is within the second value range, determine that the tuning parameter value of the parameter of this iterative tuning test is a special value; if the parameter value is not within the second value range, determine the tuning parameter value of the parameter of this iterative tuning test based on the parameter value and the third value range.

[0110] In some embodiments, the data processing device further includes Figure 4 The database performance improvement parameter determination unit not shown in the figure is configured to: determine the set deviation values ​​of the parameters to be adjusted and the merged parameters, determine the database performance improvement parameters based on the set deviation values ​​and the tuning parameter values ​​of the corresponding parameters; and output the database performance improvement parameters.

[0111] It should be noted that the data processing method and the data processing device of the present application have a corresponding relationship in terms of specific implementation contents, so the repeated contents will not be described again.

[0112] Figure 5 An exemplary system architecture 500 to which the data processing method or data processing device according to the embodiment of the present application can be applied is shown.

[0113] like Figure 5As shown, system architecture 500 may include terminal devices 501, 502, 503, a network 504 and a server 505. Network 504 is used to provide a medium for communication links between terminal devices 501, 502, 503 and server 505. Network 504 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0114] Users can use terminal devices 501, 502, 503 to interact with server 505 through network 504 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 501, 502, 503, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0115] The terminal devices 501 , 502 , and 503 may be various electronic devices having a data processing screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0116] Server 505 can be a server that provides various services, such as a background management server that provides support for data processing requests submitted by users using terminal devices 501, 502, and 503 (only as an example). The background management server can run the received inferred target parameter configuration to obtain performance indicator data; in response to the iteration end condition not being met, update the knowledge base according to the performance indicator data to obtain the updated configuration parameters in the knowledge base; call the explanatory framework to identify the importance value of the updated configuration parameters; according to the importance value, determine the parameters to be adjusted and the parameters to be merged, merge the parameters to be merged to obtain the merged parameters; determine the parameter type and discrete step length of the parameters to be adjusted and the merged parameters, and then perform tuning tests on the parameters to be adjusted and the merged parameters in the database based on the iteration end condition to determine the tuning parameter values ​​of the parameters to be adjusted and the merged parameters, and output the tuning parameter values. Improve the efficiency and accuracy of database parameter configuration tuning, and achieve that the database parameter configuration meets the scale of the database and the complexity of the application.

[0117] It should be noted that the data processing method provided in the embodiment of the present application is generally executed by the server 505 , and accordingly, the data processing device is generally set in the server 505 .

[0118] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0119] Reference below Figure 6, which shows a schematic diagram of the structure of a computer system 600 of a terminal device suitable for implementing an embodiment of the present application. Figure 6 The terminal device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0120] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the computer system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0121] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0122] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present application are executed.

[0123] It should be noted that the computer-readable medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. Computer-readable storage media may include, for example, but are not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0124] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0125] The units involved in the embodiments described in the present application may be implemented by software or hardware. The units described may also be set in a processor, for example, it may be described as: a processor includes an operation unit, an acquisition unit, an identification unit, a parameter determination unit, and a tuning and testing unit. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0126] As another aspect, the present application also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device runs the received inferred target parameter configuration, thereby obtaining performance indicator data; in response to the failure to reach the iteration end condition, the knowledge base is updated according to the performance indicator data, and the updated configuration parameters in the knowledge base are obtained; the explanatory framework is called to identify the importance value of the updated configuration parameter; according to the importance value, the parameters to be adjusted and the parameters to be merged are determined, and the parameters to be merged are merged to obtain the merged parameters; the parameter type and discrete step length of the parameters to be adjusted and the merged parameters are determined, and then the tuning test of the parameters to be adjusted and the merged parameters is performed in the database based on the iteration end condition to determine the tuning parameter values ​​of the parameters to be adjusted and the merged parameters, and output the tuning parameter values.

[0127] The computer program product of the present application includes a computer program, and the computer program implements the data processing method in the embodiment of the present application when executed by a processor.

[0128] According to the technical solution of the embodiment of the present application, the efficiency and accuracy of database parameter configuration tuning can be improved, and the database parameter configuration can be made to meet the scale of the database and the complexity of the application.

[0129] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A data processing method, characterized in that: include: Running the received inferred target parameter configuration to obtain performance indicator data; In response to not meeting the iteration end condition, updating the knowledge base according to the performance indicator data, and obtaining updated configuration parameters in the knowledge base; invoking an interpretive framework to identify a significance value of the updated configuration parameter; Determining parameters to be adjusted and parameters to be merged according to the importance values, and merging the parameters to be merged to obtain merged parameters; Determine the parameter type and discrete step length of the parameter to be adjusted and the combined parameter, and then perform a tuning test on the parameter to be adjusted and the combined parameter in a database based on the iteration end condition to determine the tuning parameter values ​​of the parameter to be adjusted and the combined parameter, and output the tuning parameter values.

2. The method according to claim 1, characterized in that Prior to running the received inferred target parameter configuration, the method further includes: The optimizer is called to infer a target parameter configuration for obtaining a performance above a preset threshold based on the existing data in the knowledge base.

3. The method according to claim 1, characterized in that The invoking an interpretive framework to identify the importance value of the updated configuration parameter comprises: The updated configuration parameters are input into an interpretive framework to determine an importance value of each input updated configuration parameter in the decision making.

4. The method according to claim 1, characterized in that: The step of determining the parameters to be adjusted and the parameters to be merged according to the importance values ​​includes: The updated configuration parameters greater than the preset importance threshold are determined as parameters to be adjusted, and the updated configuration parameters less than the preset importance threshold are determined as parameters to be merged.

5. The method according to claim 1, characterized in that The determining of tuning parameter values ​​of the to-be-adjusted parameter and the combined parameter includes: Determine the value ranges of the parameter to be adjusted and the combined parameter, and determine the discrete set corresponding to each parameter to be adjusted and each combined parameter according to the value range and the discrete step length; Performing a numerical modification and tuning test on the parameter to be adjusted and the combined parameter based on the discrete set, scaling the parameter value obtained by this iterative tuning test to a first value range, wherein the minimum value of the first value range is a special value; Obtaining a biased special value sampling probability associated with the optimizer, and obtaining a second value range based on the biased special value sampling probability, wherein a minimum value of the second value range is the special value; Determine a third value range based on the value range and the first value range; If the parameter value is within the second value range, the tuning parameter value of the parameter for this iterative tuning test is determined to be the special value; if the parameter value is not within the second value range, the tuning parameter value of the parameter for this iterative tuning test is determined based on the parameter value and the third value range.

6. The method according to claim 1, characterized in that The method further comprises: Determine the setting deviation values ​​of the parameter to be adjusted and the combined parameter, and determine the database performance improvement effect parameter based on the setting deviation value and the tuning parameter value of the corresponding parameter; Output the database performance improvement effect parameters.

7. A data processing device, characterized in that: include: an operation unit configured to operate the received inferred target parameter configuration to obtain performance indicator data; an acquisition unit, configured to update a knowledge base according to the performance indicator data in response to failure to meet an iteration end condition, and acquire updated configuration parameters in the knowledge base; an identification unit configured to call an interpretative framework to identify a significance value of the updated configuration parameter; a parameter determination unit, configured to determine the parameters to be adjusted and the parameters to be merged according to the importance value, and merge the parameters to be merged to obtain a merged parameter; The tuning test unit is configured to determine the parameter type and discrete step length of the parameter to be adjusted and the combined parameter, and then perform a tuning test on the parameter to be adjusted and the combined parameter in a database based on the iteration end condition to determine the tuning parameter values ​​of the parameter to be adjusted and the combined parameter, and output the tuning parameter values.

8. The device according to claim 7, characterized in that The device also includes a calculation unit configured to: The optimizer is called to infer a target parameter configuration for obtaining a performance above a preset threshold based on the existing data in the knowledge base.

9. The device according to claim 7, characterized in that The identification unit is further configured to: The updated configuration parameters are input into an interpretive framework to determine an importance value of each input updated configuration parameter in the decision making.

10. The device according to claim 7, characterized in that The parameter determination unit is further configured to: The updated configuration parameters greater than the preset importance threshold are determined as parameters to be adjusted, and the updated configuration parameters less than the preset importance threshold are determined as parameters to be merged.

11. The device according to claim 7, characterized in that The tuning test unit is further configured to: Determine the value ranges of the parameter to be adjusted and the combined parameter, and determine the discrete set corresponding to each parameter to be adjusted and each combined parameter according to the value range and the discrete step length; Performing a numerical modification and tuning test on the parameter to be adjusted and the combined parameter based on the discrete set, scaling the parameter value obtained by this iterative tuning test to a first value range, wherein the minimum value of the first value range is a special value; Obtaining a biased special value sampling probability associated with the optimizer, and obtaining a second value range based on the biased special value sampling probability, wherein a minimum value of the second value range is the special value; Determine a third value range based on the value range and the first value range; If the parameter value is within the second value range, the tuning parameter value of the parameter for this iterative tuning test is determined to be the special value; if the parameter value is not within the second value range, the tuning parameter value of the parameter for this iterative tuning test is determined based on the parameter value and the third value range.

12. The device according to claim 7, characterized in that The device also includes a database performance improvement parameter determination unit, which is configured to: Determine the setting deviation values ​​of the parameter to be adjusted and the combined parameter, and determine the database performance improvement effect parameter based on the setting deviation value and the tuning parameter value of the corresponding parameter; Output the database performance improvement effect parameters.

13. A data processing electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

14. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images