A performance optimization method and device for data synchronization software

Abstract

The present invention relates to the technical field of data synchronization optimization, and specifically to a performance optimization method and device for data synchronization software, wherein the method comprises: dividing multiple metrics of the data synchronization software into n groups according to correlation through factor analysis, and constructing corresponding n valid metrics based on the n groups of metrics; screening out m parameters with the greatest overall impact on the n valid metrics from multiple parameters of the data synchronization software through linear regression and regularization; configuring the m parameters based on historical data and Gaussian process regression until the obtained n valid metrics meet the optimization requirements, and saving the m parameters configured each time and the corresponding n valid metrics in a mapping form. The above-mentioned performance optimization scheme provided by the present invention can perform automated parameter tuning throughout the entire process, reduce the complexity of manual optimization, and reduce the workload of manual optimization.

Description

[Technical field]

[0001] The present invention relates to the technical field of data synchronization optimization, and in particular to a performance optimization method and device for data synchronization software. [Background technology]

[0002] The function of data synchronization software is relatively simple, that is, to synchronize data from the source database to the target database. However, in various application environments, in order to achieve the best synchronization efficiency of data synchronization software, this is usually a difficult task. The main reasons are:

[0003] (1) Parameter dependency: Many parameters are not independent, but rather dependent on each other. Changing the value of one parameter may affect the benefits of another parameter. Therefore, traditional parameter optimization usually only changes one parameter at a time, but when a large number of parameters are involved, the optimization process becomes very slow.

[0004] (2) Continuous parameter settings: Some parameters have continuous values, that is, there are many possible parameter settings. For example, the size of the memory buffer pool can be any value from 0 to the system upper limit. In some ranges, increasing this parameter by 0.1GB may not matter, while in other ranges, increasing it by 0.1GB may cause a sharp drop in performance.

[0005] (3) Parameters are not reusable: Usually, the same parameter configuration cannot be reused from one application to the next. Because different application systems have different requirements and environments, when manual effort is spent on optimizing one application scenario, the optimal configuration of the software may not be the best for another application scenario.

[0006] (4) Parameter complexity: With the release of new versions and new features of data synchronization software, new parameters will continue to increase, making it difficult for data synchronization software performance tuning to keep up with these changes.

[0007] In traditional solutions, data synchronization software is generally optimized through manual optimization. However, manual optimization is complex and labor-intensive. In order to achieve the optimal synchronization efficiency of data synchronization software, it is usually time-consuming and labor-intensive.

[0008] In view of this, overcoming the defects of the above-mentioned prior art is an urgent problem to be solved in the field of this technology. [Summary of the invention]

[0009] The technical problems to be solved by the present invention are:

[0010] The tuning of data synchronization software is difficult due to the dependency, continuous setting, and non-reusability of parameters. Traditional solutions generally achieve this through manual optimization, but manual optimization is complex and labor-intensive. In order to achieve the optimal synchronization efficiency of the data synchronization software, it is usually time-consuming and labor-intensive. To solve the above problems, the present invention proposes an automated parameter tuning method for achieving performance optimization of the data synchronization software.

[0011] The present invention achieves the above object through the following technical solutions:

[0012] In a first aspect, the present invention provides a performance optimization method for data synchronization software, characterized in that the data synchronization software is provided with a plurality of parameters and a plurality of metric indicators, and the parameter configuration determines the metric indicator; the performance optimization method comprises:

[0013] Through factor analysis, multiple metrics of the data synchronization software are divided into n groups according to their relevance, and corresponding n effective indicators are constructed based on the n groups of metrics;

[0014] Through linear regression and regularization, m parameters with the greatest overall impact on the n effective indicators are selected from multiple parameters of the data synchronization software;

[0015] The m parameters are configured based on historical data and Gaussian process regression until the obtained n effective indicators meet the optimization requirements, and the m parameters configured each time and the corresponding n effective indicators are saved in a mapping form;

[0016] The historical data includes m parameters configured each time when the performance optimization is performed in history and n corresponding valid indicators.

[0017] Preferably, the multiple metrics of the data synchronization software are divided into n groups according to their relevance by factor analysis, and corresponding n effective indicators are constructed based on the n groups of metrics, specifically:

[0018] Obtain the correlation coefficients between various metrics through factor analysis, and divide the metrics with correlation coefficients greater than the preset coefficients into one group, and finally divide the multiple metrics of the data synchronization software into n groups;

[0019] For each group, an effective index is constructed according to the commonality of one or more metrics in the group to uniformly represent the metrics in the group, so n groups will eventually obtain n effective indicators.

[0020] Preferably, the m parameters having the greatest overall impact on the n effective indicators are screened out from the multiple parameters of the data synchronization software through linear regression and regularization, specifically:

[0021] Analyzing the relationship between the n effective indicators and multiple parameters of the data synchronization software by using the Lasso linear regression method, and obtaining a linear mapping function from the multiple parameters to the n effective indicators;

[0022] Regularization is adopted to adjust the order of magnitude of each parameter in the linear mapping function according to the degree of influence on the n effective indicators, and m parameters whose order of magnitude is greater than a preset threshold are selected from the multiple parameters.

[0023] Preferably, when the performance optimization is performed for the first time, the historical data is empty, then the m parameters are configured based on the historical data and Gaussian process regression until the corresponding n valid indicators meet the optimization requirements, and the m parameters configured each time and the corresponding n valid indicators are saved in a mapping form, specifically:

[0024] The data synchronization software collects multiple corresponding metrics after running for a preset time according to the initial default m parameters, calculates n valid metrics based on the collected multiple metrics, and saves the m parameters and n valid metrics corresponding to this operation in a mapping form;

[0025] Based on the currently saved m parameters and n valid indicators, the m parameters are reconfigured through Gaussian process regression, the corresponding n valid indicators are obtained through collection and calculation, and the configured m parameters and the corresponding n valid indicators are saved in a mapping form;

[0026] Determine whether the latest n valid indicators meet the optimization requirements. If not, continue to reconfigure the m parameters through Gaussian process regression until the latest n valid indicators meet the optimization requirements; wherein, after each reconfiguration, the corresponding m parameters and n valid indicators are saved in a mapping form.

[0027] Preferably, when it is not the first time to perform performance optimization, the historical data already contains multiple mapping relationships between the m parameters and the n effective indicators, then the m parameters are configured based on the historical data and Gaussian process regression until the corresponding n effective indicators meet the optimization requirements, and the m parameters configured each time and the corresponding n effective indicators are saved in a mapping form, specifically:

[0028] By comparing the n expected effective indicators given by the user with the existing effective indicators in the historical data, find the n effective indicators that are most similar to the n expected effective indicators and obtain the corresponding m parameters;

[0029] It is determined whether the n valid indicators meet the optimization requirements. If not, the m parameters are reconfigured through Gaussian process regression until the latest n valid indicators meet the optimization requirements. After each reconfiguration, the corresponding m parameters and n valid indicators are saved in a mapping form.

[0030] Preferably, the method for setting the n expected effective indicators is specifically as follows:

[0031] Used to give n expected effective indicators according to optimization requirements; or,

[0032] The user provides multiple expected metrics of the data synchronization software according to the optimization requirements, divides the multiple expected metrics into n groups according to the correlation, and constructs corresponding n expected effective metrics based on the n groups of expected metrics.

[0033] Preferably, the method of comparing the n expected effective indicators given by the user with the existing effective indicators in the historical data to find the n effective indicators that are most similar to the n expected effective indicators is specifically:

[0034] Each set of n valid indicators in the historical data is taken as an n-dimensional vector, and the n expected valid indicators given by the user are taken as n-dimensional reference vectors;

[0035] The distance between each n-dimensional vector and the n-dimensional reference vector is calculated, and the n valid indicators corresponding to the n-dimensional vector closest to the n-dimensional reference vector are the n most similar valid indicators to be found.

[0036] Preferably, the parameters include at least two of IO idle time, number of log analysis threads, log sorting buffer size, log analysis linked list size, log memory heap size, number of worker threads, transaction cache size, number of batch execution rows and commit compression ratio.

[0037] Preferably, the metric indicators include at least two of log cache volume, log parsing volume, log delivery number, log sending queue, log receiving queue, number of submitted transactions, number of active transactions and network synchronization delay.

[0038] In a second aspect, the present invention further provides a performance optimization device for data synchronization software, which is used to implement the performance optimization method described in the first aspect, and includes a client controller and a tuning manager;

[0039] The client controller is used to interact with the data synchronization software that needs to be optimized for performance, so as to collect parameters and metrics from the data synchronization software and transmit them to the tuning manager;

[0040] The tuning manager is used to store the received data in an internal database, and obtain n valid indicators and m parameters that have the greatest overall impact on the n valid indicators after processing. When performing performance optimization, the m parameters are configured based on historical data and Gaussian process regression until the corresponding n valid indicators meet the optimization requirements, and the m parameters configured each time and the corresponding n valid indicators are saved in the database in a mapping form.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] In the performance optimization method for data synchronization software provided by the present invention, multiple parameters and multiple measurement indicators of the data synchronization software can be simplified through machine learning methods such as factor analysis, linear regression and regularization, and some parameters that have the greatest impact on the overall performance can be selected; when performing performance optimization, based on past historical data experience, these parameters are continuously configured and optimized through machine learning methods until the measured indicators meet the optimization requirements. The entire process can perform automated parameter tuning, reducing the complexity of manual optimization and reducing the workload of manual optimization.

Brief Description of the Drawings

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 A flow chart of a method for optimizing the performance of data synchronization software provided by an embodiment of the present invention;

[0045] Figure 2 A simplified method flow chart of a measurement indicator provided by an embodiment of the present invention;

[0046] Figure 3 A simplified method flow chart of a parameter provided by an embodiment of the present invention;

[0047] Figure 4 A parameter configuration flow chart for the first performance optimization provided by an embodiment of the present invention;

[0048] Figure 5 A parameter configuration flow chart for non-first-time performance optimization provided by an embodiment of the present invention;

[0049] Figure 6 An architecture diagram of a performance optimization device for data synchronization software provided in an embodiment of the present invention. [Specific implementation method]

[0050] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0051] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other. The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0052] Embodiment 1:

[0053] In order to solve the technical problems of high complexity and heavy workload in traditional manual performance optimization of data synchronization software, an embodiment of the present invention provides a performance optimization method for data synchronization software, which can be used to realize automatic performance optimization of data synchronization software.

[0054] Among them, it is first necessary to clarify that the data synchronization software is usually equipped with multiple parameters and multiple metrics, and the parameter information and metric information can be obtained by collecting the memory of the data synchronization software. Under different parameter configurations, the metrics collected are also different, that is, the parameter configuration determines the metrics. The parameters usually include IO idle time, number of log analysis threads, log sorting buffer size, log analysis linked list size, log memory heap size, number of working threads, transaction cache size, batch execution line number and submission compression ratio, etc.; the metrics usually include log cache volume, log parsing volume, number of log delivery, log sending queue, log receiving queue, number of submitted transactions, number of active transactions and network synchronization delay, etc., which can be directly viewed through the operating system and the data synchronization software itself. The goal of the present invention is to find a set of parameter configurations through a machine learning method, so that under this parameter configuration, the overall metric of the data synchronization software can meet the optimization requirements, that is, at this time, the overall performance of the data synchronization software in all aspects reaches the user's optimization goal.

[0055] like Figure 1 As shown, the performance optimization method provided by the embodiment of the present invention mainly includes the following steps:

[0056] Step 10: divide the multiple metrics of the data synchronization software into n groups according to their relevance through factor analysis, and construct corresponding n valid indicators based on the n groups of metrics.

[0057] There are many metrics within the data synchronization software. Some metrics actually have the same measurement function, but may only have different units, such as data_read and page_read. Therefore, there is no need to consider them at the same time. This is because if all metrics are considered at the same time, the search space of the machine learning algorithm will be too large during subsequent performance optimization, and the overall optimization efficiency will be reduced. For this reason, the factor analysis method is used here to simplify multiple metrics, that is, to reduce the number of metrics, thereby reducing the search space of the machine learning algorithm during subsequent performance optimization, accelerating the entire learning process, and quickly configuring the parameters that meet the optimization requirements. For the specific indicator simplification process, please refer to Figure 2 :

[0058] Step 101, obtain the correlation coefficient between each metric through factor analysis, and divide the metric with a correlation coefficient greater than a preset coefficient into one group, and finally divide the multiple metric of the data synchronization software into n groups. Here, all the metric is actually grouped according to the size of the correlation, the purpose is to find some metric with commonality or the same essence and divide them into one group, so that the correlation between the metrics in the same group is high, and the correlation between the metrics in different groups is unrelated or low. For example, io_data_read and io_page_read both express io read, so they can be divided into one group.

[0059] Step 102, for each group, construct an effective indicator based on the commonality of one or more metrics in the group to uniformly represent the metrics in the group, so that n groups finally obtain n effective indicators; wherein the value of the effective indicator can be the average value of one or more metrics in the corresponding group. For example, the two metrics io_data_read and io_page_read both express io reads, so an effective indicator "io read" can be constructed to uniformly represent it, and the value of "io read" can be the average value of the two metrics, that is, io read = (io_data_read + io_page_read) / 2.

[0060] Based on the above factor analysis method, assuming that there are N metrics in the data synchronization software, the N metrics can be reduced to n valid metrics after simplification. Obviously, n is much smaller than N. This can greatly simplify the subsequent performance optimization process and improve optimization efficiency.

[0061] Step 20, through linear regression and regularization, screen out m parameters that have the greatest overall impact on the n effective indicators from multiple parameters of the data synchronization software.

[0062] There are also many parameters inside the data synchronization software. Some parameters actually have little impact on the overall performance of the software and can be ignored. If all of them are taken into account when optimizing performance, the search space of the subsequent machine learning algorithm will be too large, and the overall optimization efficiency will be reduced; moreover, considering all parameters may also cause overfitting. Therefore, after simplifying the metrics in step 10, it is necessary to further identify which parameters have the greatest impact on the metrics of the data synchronization software, that is, to simplify the parameters. Here, Lasso's linear regression method and regularization are used to find the parameters that are most strongly correlated with the overall metrics of the system. For the specific simplification process of the metrics, please refer to Figure 3 :

[0063] Step 201, the relationship between the n effective indicators and the multiple parameters of the data synchronization software is analyzed by the Lasso linear regression method, and a linear mapping function from the multiple parameters to the n effective indicators is obtained. Linear regression is a method for analyzing the strength of the relationship between the result variable and the independent variable, and the relationship between the independent variable and the result variable is linear; here, the n effective indicators are used as the result variable, and the multiple parameters are used as independent variables to find the relationship between the n effective indicators and the multiple parameters, that is, to find an optimal linear mapping function from the feature space X (i.e., the multiple parameters) to the output space Y (i.e., the n effective indicators). Here, such a straight line can be found by the least squares method.

[0064] Step 202, regularization is used to adjust the order of magnitude of each parameter in the linear mapping function according to the degree of influence on the n effective indicators, and m parameters whose order of magnitude is greater than a preset threshold are selected from the multiple parameters. If the influence of all parameters on the performance is considered, it may cause overfitting, that is, the obtained linear mapping function (i.e., straight line) will be at a loss when facing new parameter data; for example, considering parameters a, b, c and effective indicator d, when the straight line obtained by 5 sets of parameters a, b, c encounters a new set of parameter a, b, c data, it is found that this set of parameter data is very far from the straight line, then this straight line has an overfitting problem; and the actual cause of this problem may be that parameter a should not be taken into account. Therefore, it is necessary to perform regularization operation at this time, only focusing on useful data (i.e., parameters that have a greater impact on the overall performance) and ignoring interference items (i.e., parameters that have a small impact on the overall performance).

[0065] Here, by using lasso regularization, the order of magnitude of unimportant parameters can be reduced. The smaller the order of magnitude, the smaller the impact of the parameter on the overall performance. In this way, parameters with an order of magnitude less than the preset threshold can be automatically deleted, and only parameters with an order of magnitude greater than the preset threshold are retained. Assuming that there are M parameters in the data synchronization software, after simplification, M parameters can be deleted to the remaining m parameters. Obviously, m is much smaller than M. In this way, overfitting can be prevented, and the subsequent performance optimization process can be greatly simplified, improving optimization efficiency.

[0066] Step 30, configuring the m parameters based on historical data and Gaussian process regression until the obtained n effective indicators meet the optimization requirements, and saving the m parameters configured each time and the corresponding n effective indicators in a mapping form.

[0067] The historical data is stored in a database, including the m parameters configured each time the performance optimization is performed and the corresponding n valid indicators. For ease of description, the m parameters can be recorded as the minimum parameter set, and the n valid indicators can be recorded as the minimum indicator set. Then, the database stores the mapping relationship of minimum parameter set 1---> minimum indicator set 1, minimum parameter set 2---> minimum indicator set 2, ..., minimum parameter set i---> minimum indicator set i. The minimum parameter set is actually an m-dimensional vector, and the minimum indicator set is actually an n-dimensional vector. Therefore, what is stored in the database is actually the mapping relationship between multidimensional vectors and multidimensional vectors.

[0068] When the performance optimization is performed for the first time, the historical data is empty. You need to run it once with the initial default parameters to obtain the corresponding valid indicators, save the data, and then configure the parameters through Gaussian process regression based on the saved data. For the specific process, please refer to Figure 4 , including the following steps:

[0069] Step 301, the data synchronization software collects multiple corresponding measurement indicators after running the preset time according to the initial default m parameters, calculates n valid indicators based on the collected multiple measurement indicators, and saves the m parameters and n valid indicators corresponding to this operation in a mapping form.

[0070] Among them, the calculation method of each effective indicator has been introduced above, and the average value of each metric indicator in the corresponding group can be directly taken; after completing data collection and saving, what is saved in the database is the mapping relationship of minimum parameter set 1--->minimum indicator set 1.

[0071] Step 302, based on the currently saved m parameters and n valid indicators, reconfigure the m parameters through Gaussian process regression, obtain the corresponding n valid indicators through collection and calculation, and save the configured m parameters and the corresponding n valid indicators in a mapping form.

[0072] That is, first, according to the mapping relationship of minimum parameter set 1--->minimum indicator set 1 saved in the database at this time, Gaussian process regression is used to obtain the latest parameter configuration that can further improve the minimum indicator set, that is, reconfigure the minimum parameter set; secondly, the data synchronization software is run according to the reconfigured minimum parameter set for a preset time to collect the corresponding multiple measurement indicators, and then n valid indicators are calculated based on the collected multiple measurement indicators to obtain the latest minimum indicator set; finally, the latest minimum indicator set and the minimum parameter set are saved in the database in a mapping form, that is, the mapping relationship of minimum parameter set 2--->minimum indicator set 2 is saved.

[0073] Step 303, determine whether the latest n valid indicators meet the optimization requirements. If not, continue to reconfigure the m parameters through Gaussian process regression until the latest n valid indicators meet the optimization requirements; wherein, after each reconfiguration, the corresponding m parameters and n valid indicators are saved in a mapping form.

[0074] Here, the user can judge whether the optimization requirements are met based on the latest minimum indicator set. If the optimization requirements are met, the performance optimization ends and the data synchronization software can be directly run according to the latest configured m parameters. If the optimization requirements are not met, the minimum parameter set is reconfigured through Gaussian process regression according to the mapping relationships stored in the current database. The data synchronization software collects multiple corresponding metrics after running the reconfigured minimum parameter set for a preset time, and then calculates n valid metrics based on the collected multiple metrics to obtain the latest minimum indicator set until the user judges that the optimization requirements are met based on the latest minimum indicator set. Among them, after each operation according to the reconfigured minimum parameter set, the latest minimum indicator set and the minimum parameter set are saved in the database in a mapping form, and multiple mapping relationships of minimum parameter set i---> minimum indicator set i are saved in the database.

[0075] When the performance optimization is performed later, that is, not the first time, the historical data already contains multiple mapping relationships between the m parameters and the n valid indicators, that is, multiple mapping relationships of minimum parameter set i---> minimum indicator set i. Then, the minimum indicator set that is most similar to the expected indicator given by the user can be found based on the historical data, and then the corresponding minimum parameter set can be found for parameter configuration. For the specific process, please refer to Figure 5 , including the following steps:

[0076] Step 301 ′: by comparing the n expected effective indicators given by the user with the existing effective indicators in the historical data, find the n effective indicators that are most similar to the n expected effective indicators, and obtain the corresponding m parameters.

[0077] Here, n expected effective indicators can be given according to the optimization requirements, that is, the expected optimization values ​​corresponding to the n effective indicators; or the user can give multiple expected metrics of the data synchronization software according to the optimization requirements, that is, the expected optimization values ​​corresponding to the multiple metrics, and then simplify them into n expected effective indicators for comparison; the specific simplification process is also to divide the multiple expected metrics into n groups according to the correlation through factor analysis, and then construct the corresponding n expected effective indicators based on the n groups of expected metrics, which will not be described in detail. Among them, each group of n effective indicators in the historical data can be used as an n-dimensional vector, and the n expected effective indicators given by the user can be used as an n-dimensional reference vector; then when looking for the most similar indicators, the distance between each n-dimensional vector and the n-dimensional reference vector can be calculated, and the n effective indicators corresponding to the n-dimensional vector closest to the n-dimensional reference vector are the n most similar effective indicators to be found.

[0078] Step 302', determine whether the n valid indicators meet the optimization requirements. If not, reconfigure the m parameters through Gaussian process regression until the latest n valid indicators meet the optimization requirements; wherein, after each reconfiguration, the corresponding m parameters and n valid indicators are saved in a mapping form.

[0079] Here, the user can first determine whether the optimization requirements are met based on the most similar n effective indicators. If the optimization requirements are met, the parameters can be directly configured according to the m parameters corresponding to the most similar n effective indicators, so that the data synchronization software runs according to the m parameters corresponding to the most similar n effective indicators, and the performance optimization is completed. If the optimization requirements are not met, the minimum parameter set is reconfigured through Gaussian process regression based on the mapping relationships stored in the current database and the m parameters corresponding to the most similar n effective indicators. The data synchronization software runs according to the reconfigured minimum parameter set for a preset time and then collects the corresponding multiple metrics. Then, the n effective indicators are calculated based on the collected multiple metrics to obtain the latest minimum indicator set, until the user determines that the optimization requirements have been met based on the latest minimum indicator set. Among them, after each operation according to the reconfigured minimum parameter set, the latest minimum indicator set and the minimum parameter set are saved in the database in the form of mapping for continued use in subsequent performance optimization.

[0080] Furthermore, when optimizing, users may not pay attention to all metrics or effective metrics in most cases, but only have optimization requirements for some of them. Therefore, when judging whether the latest minimum metric set meets the optimization requirements, users only need to confirm whether the part of the minimum metric set that they pay attention to has achieved the optimization purpose. As long as the part of the metric that they pay attention to is close to the corresponding set standard optimization value, it can be judged that the optimization requirements have been met.

[0081] Among them, Gaussian process is used to represent the distribution of function, and regression is to find a function that best fits the data distribution. The function here is the mapping relationship between the minimum parameter set and the minimum indicator set. For the minimum parameter set, since the value of each parameter usually has a range, each parameter can be uniformly valued within the corresponding interval, and finally a function distribution with the input as the minimum parameter set (i.e., X vector, including m parameters) and the output as the minimum indicator set (i.e., Y vector, including n valid indicators) is obtained. Based on this function, the X value that makes the Y value optimal can be found, that is, the minimum parameter set that makes the minimum indicator set optimal can be found.

[0082] The performance optimization method of the above-mentioned data synchronization software provided by the present invention can simplify multiple parameters and multiple measurement indicators of the data synchronization software through machine learning methods such as factor analysis, linear regression and regularization, and select some parameters that have the greatest impact on the overall performance; when performing performance optimization, based on past historical data experience, these parameters are continuously configured and optimized through the machine learning method of Gaussian process regression until the measured indicators meet the optimization requirements. The entire process can perform automated parameter tuning, reducing the complexity of manual optimization and reducing the workload of manual optimization.

[0083] Embodiment 2:

[0084] Based on the performance optimization method of data synchronization software provided in the above embodiment 1, the present invention also provides a performance optimization device for data synchronization software that can be used to implement the above method, such as Figure 6 As shown, it includes a client controller and a tuning manager.

[0085] After being started by the user, the client controller can be used to interact with the data synchronization software that needs to be optimized for performance, so as to collect parameters and metrics from the data synchronization software and transmit them to the tuning manager.

[0086] The tuning manager is used to store the received data in an internal database, and obtain n valid indicators and m parameters that have the greatest overall impact on the n valid indicators after processing. When performing performance optimization, the m parameters are configured based on Gaussian process regression in the machine learning method until the corresponding n valid indicators meet the optimization requirements, and the m parameters configured each time and the corresponding n valid indicators are saved in the database in a mapping form. Among them, the machine learning method mainly includes factor analysis, linear regression, and Gaussian process regression methods.

[0087] The tuning manager includes an analysis module and a decision module. The analysis module can call the factor analysis method in the machine learning method to divide multiple measurement indicators of the data synchronization software into n groups according to correlation, and construct corresponding n effective indicators based on the n groups of measurement indicators; it can also call the linear regression method in the machine learning method to screen out the m parameters with the greatest overall impact on the n effective indicators from the multiple parameters of the data synchronization software.

[0088] The decision module can call the Gaussian process regression method in the machine learning method, configure the m parameters based on historical data and Gaussian process regression until the obtained n effective indicators meet the optimization requirements, and save the m parameters configured each time and the corresponding n effective indicators in a mapping form.

[0089] Among them, after each reconfiguration of parameters and running for a preset time, the client controller can collect the m parameters and multiple metrics from the data synchronization software and transmit them to the tuning manager. The decision module can calculate n valid indicators based on the received multiple metrics, and then save the m parameters and n valid indicators in the database in a mapping form, and can return the mapping relationship to the client controller, and the user can judge whether the optimization requirements are met. The user feeds back the judgment result to the tuning manager through the client controller. If the optimization requirements are met, the optimization ends; if the optimization requirements are not met, the decision module continues to configure parameters based on Gaussian process regression, and continuously interacts with the client controller until the user returns a result that meets the optimization requirements, and the optimization ends.

[0090] Those skilled in the art will appreciate that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A performance optimization method for data synchronization software, characterized in that: The data synchronization software has multiple parameters and multiple metrics. The parameter configuration determines the metric. The performance optimization methods include: Through factor analysis, multiple metrics of the data synchronization software are divided into n groups according to their correlation, and corresponding n valid indicators are constructed based on the n groups of metrics; specifically: the correlation coefficients between the metrics are obtained through factor analysis, and the metrics with correlation coefficients greater than the preset coefficients are divided into one group, and finally the multiple metrics of the data synchronization software are divided into n groups; for each group, an effective indicator is constructed according to the commonality of one or more metrics in the group to uniformly represent the metrics in the group, and n groups finally obtain n valid indicators; the value of the effective indicator is the average value of one or more metrics in the corresponding group; Through linear regression and regularization, m parameters with the greatest overall impact on the n effective indicators are selected from multiple parameters of the data synchronization software; The m parameters are configured based on historical data and Gaussian process regression until the obtained n effective indicators meet the optimization requirements, and the m parameters configured each time and the corresponding n effective indicators are saved in a mapping form; The historical data includes m parameters configured each time when the performance optimization is performed in history and n corresponding valid indicators; When the performance optimization is performed for the first time, the historical data is empty, and the m parameters are configured based on the historical data and Gaussian process regression until the corresponding n valid indicators meet the optimization requirements, and the m parameters configured each time and the corresponding n valid indicators are saved in a mapping form, specifically: the data synchronization software collects the corresponding multiple metrics after running the preset time according to the initial default m parameters, calculates n valid indicators based on the collected multiple metrics, and saves the m parameters and n valid indicators corresponding to this operation in a mapping form; on the basis of the currently saved m parameters and n valid indicators, reconfigure the m parameters through Gaussian process regression, obtain the corresponding n valid indicators through collection and calculation, and save the m parameters configured this time and the corresponding n valid indicators in a mapping form; judge whether the latest obtained n valid indicators meet the optimization requirements, if not, continue to reconfigure the m parameters through Gaussian process regression until the latest obtained n valid indicators meet the optimization requirements; wherein, after each reconfiguration, the corresponding m parameters and n valid indicators are saved in a mapping form; When it is not the first time to perform performance optimization, the historical data already contains multiple mapping relationships between the m parameters and the n effective indicators, then the m parameters are configured based on the historical data and Gaussian process regression until the corresponding n effective indicators meet the optimization requirements, and the m parameters configured each time and the corresponding n effective indicators are saved in a mapping form, specifically: Calculate the distance between each n-dimensional vector and the n-dimensional reference vector. The n valid indicators corresponding to the n-dimensional vector closest to the n-dimensional reference vector are the n most similar valid indicators to be found, and obtain the corresponding m parameters. Take each group of n valid indicators in the historical data as an n-dimensional vector, and take the n expected valid indicators given by the user as the n-dimensional reference vector. It is determined whether the n valid indicators meet the optimization requirements. If not, the m parameters are reconfigured through Gaussian process regression until the latest n valid indicators meet the optimization requirements. After each reconfiguration, the corresponding m parameters and n valid indicators are saved in a mapping form.

2. The performance optimization method of data synchronization software according to claim 1, characterized in that: The m parameters with the greatest overall impact on the n effective indicators are selected from multiple parameters of the data synchronization software through linear regression and regularization, specifically: Analyzing the relationship between the n effective indicators and multiple parameters of the data synchronization software by using the Lasso linear regression method, and obtaining a linear mapping function from the multiple parameters to the n effective indicators; Regularization is adopted to adjust the order of magnitude of each parameter in the linear mapping function according to the degree of influence on the n effective indicators, and m parameters whose order of magnitude is greater than a preset threshold are selected from the multiple parameters.

3. The performance optimization method of data synchronization software according to claim 1, characterized in that: The method for setting the n expected effective indicators is specifically as follows: Used to give n expected effective indicators according to optimization requirements; or, The user provides multiple expected metrics of the data synchronization software according to the optimization requirements, divides the multiple expected metrics into n groups according to the correlation, and constructs corresponding n expected effective metrics based on the n groups of expected metrics.

4. The performance optimization method of data synchronization software according to any one of claims 1 to 3, characterized in that: The parameters include at least two of IO idle time, number of log analysis threads, log sorting buffer size, log analysis linked list size, log memory heap size, number of worker threads, transaction cache size, number of batch execution rows and commit compression ratio.

5. The performance optimization method of data synchronization software according to any one of claims 1 to 3, characterized in that: The metric indicators include at least two of the log cache volume, log parsing volume, log delivery number, log sending queue, log receiving queue, number of submitted transactions, number of active transactions and network synchronization delay.

6. A performance optimization device for data synchronization software, characterized in that: Used to implement the performance optimization method described in any one of claims 1 to 5, including a client controller and a tuning manager; The client controller is used to interact with the data synchronization software that needs to be optimized for performance, so as to collect parameters and metrics from the data synchronization software and transmit them to the tuning manager; The tuning manager is used to store the received data in an internal database, and obtain n valid indicators and m parameters that have the greatest overall impact on the n valid indicators after processing. When performing performance optimization, the m parameters are configured based on historical data and Gaussian process regression until the corresponding n valid indicators meet the optimization requirements, and the m parameters configured each time and the corresponding n valid indicators are saved in the database in a mapping form.

Images