Big data processing framework configuration tuning method and system based on causal reasoning

Abstract

The invention relates to the technical field of big data processing, in particular to a big data processing framework configuration tuning method and system based on causal reasoning. According to the method, multiple sample data sets are collected, multiple causal structure learning algorithms are adopted to analyze the sample data sets so as to construct a hierarchical causal structure model related to the data scale, then target scale data are taken into consideration, and a data scale sensing causal structure model is constructed. And a configuration parameter set adapted to the target new data is given, so that adaptive configuration adjustment and optimization aiming at the target scale are realized. According to the invention, the problem of poor tuning effect of the existing automatic configuration tuning method based on causal reasoning is solved.

Description

Technical Field

[0001] The present invention relates to the field of big data processing technology, and in particular to automatic configuration optimization technology for big data processing jobs, and more specifically to: 1. a big data processing framework configuration tuning method based on causal reasoning; 2. a big data processing configuration tuning system based on causal reasoning. Background Art

[0002] Existing big data processing systems often have difficulty automatically selecting key performance parameters and adjusting configurations to achieve optimal performance when faced with data volumes of different sizes. Configuration parameters in big data processing frameworks are critical to system performance, but due to the complexity of the configuration space, professional developers also find it difficult to fully utilize its configurability.

[0003] Traditional big data configuration optimization usually relies on manual experience to select and set parameters and configure the system, which has many drawbacks. Subsequently, automatic configuration tuning methods have been developed, which are generally based on search and machine learning methods, but they still have shortcomings when dealing with recurring big data processing jobs, such as difficulty in identifying key parameters and inability to adapt to changes in data set size. In recent years, automatic configuration tuning methods based on causal reasoning have emerged. Although this method has certain advantages, it also faces parameter selection problems such as the accuracy of structural learning depends on the number of samples, the effects of different structural learning algorithms vary greatly, and it is difficult to cope with changes in data set size, resulting in poor tuning results. Summary of the invention

[0004] Based on this, it is necessary to provide a big data processing framework configuration tuning method and system based on causal reasoning to address the problem of poor tuning effect of existing automatic configuration tuning methods based on causal reasoning.

[0005] The present invention is implemented by the following technical solutions:

[0006] In a first aspect, the present invention discloses a method for optimizing a big data processing framework configuration based on causal reasoning, comprising:

[0007] Step 1: The big data processing framework is used for N data of different sizes Num1~Num N Parameter samples are collected during operation to obtain N sample data sets A1~A N ;

[0008] Among them, the nth sample data set A n Includes: Big data processing framework for n-th scale data Num n X configuration parameters, Z intermediate performance indicators, and Y performance goals at runtime; n∈[1,N];

[0009] Step 2: Use M causal structure learning algorithms to learn A n Analyze and get A n M hierarchical causal structure models B 1,n ~B M,n ;

[0010] Among them, the mth causal structure learning algorithm is n Analyze and get A n The mth hierarchical causal structure model B m,n ; m∈[1,M];

[0011] Among them, B m,n Includes: A n Several causal paths obtained by analyzing the mth causal structure learning algorithm; B m,n There is X m,n Configuration parameters; X m,n ≤X;

[0012] Step 3: When the big data processing framework needs to run the target size data Num new , then combined with Num new The N hierarchical causal structure models B obtained by analyzing the mth causal structure learning algorithm m,1 ~B m,N Integrated into a data scale-aware causal structure model DCSM m ;

[0013] Among them, DCSM m There are K configuration parameters; K≤X m,n ;

[0014] Traversing M causal structure learning algorithms to obtain M data scale-aware causal structure models DCSM1~DCSM M ;

[0015] Step 4: Based on DCSM m Given the mth recommended configuration parameter set C m ;

[0016] Traverse DCSM1~DCSM M Get M recommended configuration parameter sets C1~C M ;

[0017] For C1~C M Take the intersection to get the running number new Recommended configuration parameters.

[0018] This method for configuring and tuning a big data processing framework based on causal reasoning implements a method or process according to an embodiment of the present disclosure.

[0019] In a second aspect, the present invention discloses a big data processing configuration tuning system based on causal reasoning, which uses the big data processing framework configuration tuning method based on causal reasoning disclosed in the first aspect.

[0020] The big data processing configuration tuning system based on causal reasoning includes: sample collection module, causal model construction module, causal model integration module, and configuration tuning module.

[0021] The sample collection module is used to process N data of different sizes Num1~Num N Parameter samples are collected during operation to obtain N sample data sets A1~A N .

[0022] The causal model building module is used to use M causal structure learning algorithms to learn A n Analyze and get A n M hierarchical causal structure models B 1,n ~B M,n .

[0023] The causal model integration module is used to combine Num new The N hierarchical causal structure models B obtained by analyzing the mth causal structure learning algorithm m,1 ~B m,N Integrated into a data scale-aware causal structure model DCSM m , and is also used to traverse M causal structure learning algorithms to obtain data scale-aware causal structure models DCSM1~DCSM M .

[0024] Configuration tuning module for DCSM m Give the recommended configuration parameter set C m , and is also used to traverse DCSM1~DCSM M Get the recommended configuration parameter set C1~C M , also used for C1~C M Take the intersection to get the running number new Recommended configuration parameters.

[0025] This big data processing configuration tuning system based on causal reasoning implements the method or process according to the embodiment of the present disclosure.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] 1. The present invention collects multiple sample data sets and uses multiple causal structure learning algorithms to analyze the sample data sets to construct a hierarchical causal structure model related to the data scale. Then, the target scale data is taken into consideration and a data scale-aware causal structure model is constructed to provide a configuration parameter set adapted to the target new data, thereby achieving adaptive configuration tuning for the target scale.

[0028] 2. The present invention overcomes the subjectivity and uncertainty of the traditional method of manually selecting and setting parameters, and improves the scientificity and rationality of the configuration parameter set.

[0029] 3. The present invention can dynamically adjust the key performance parameter set and optimal configuration according to the change of data volume, thereby improving resource utilization and shortening job execution time.

[0030] 4. The present invention improves the overall processing performance of big data processing operations by deeply analyzing cause-effect relationships, adaptively and specifically performing parameter selection and performance optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0032] Figure 1 A flowchart of a method for configuring and optimizing a big data processing framework based on causal reasoning provided in Example 1 of the present invention;

[0033] Figure 2 A structural diagram of a big data processing framework configuration and tuning system based on causal reasoning provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] It should be noted that when a component is referred to as being "mounted on" another component, it may be directly on the other component or there may be a central component. When a component is considered to be "set on" another component, it may be directly set on the other component or there may be a central component at the same time. When a component is considered to be "fixed to" another component, it may be directly fixed on the other component or there may be a central component at the same time.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "or / and" used herein includes any and all combinations of one or more of the related listed items.

[0037] Example 1

[0038] See also Figure 1 , Figure 1 A flowchart of a method for configuring and tuning a big data processing framework based on causal reasoning is presented, which includes:

[0039] Step 1: The big data processing framework is used for N data of different sizes Num1~Num N Parameter samples are collected during operation to obtain N sample data sets A1~A N .

[0040] Generally speaking, the big data processing framework can be deployed on a Spark cluster to run Num1~Num N The same operation of the same processor is performed by collecting parameter samples in real time or quasi real time.

[0041] It should be noted that in order to balance the time cost of sample collection and the need for uniqueness, the size difference between any two datasets should be greater than 20%, and the value of N should not be too small or too large. Generally, 5 is recommended.

[0042] In order to build a more accurate and interpretable hierarchical causal structure model in the future, the parameters in the sample data set can be selected based on empirical values ​​or specified based on the user's specific needs and understanding. However, no matter which method is used, the sample data set should comply with the following requirements:

[0043] The nth sample data set A n Includes: Big data processing framework for n-th scale data Num n X configuration parameters, Z intermediate performance indicators, and Y performance goals at runtime; n∈[1,N].

[0044] It should be noted that the configuration parameters are: the number of processor cores, memory allocation size, data partitioning strategy and other configurable parameters of the big data processing framework.

[0045] The intermediate performance indicators are: task execution time, data throughput, memory usage, and other indicators that reflect the changes in resources of the big data processing framework.

[0046] The performance goals are: a series of indicators such as running time that reflect the performance changes of the big data processing framework.

[0047] In this embodiment 1, X, Y, and Z satisfy the following conditions: X≥K (see subsequent step 3 for K, which is generally 15); Z≥1; and Y≥1.

[0048] Then with Num n The following operation is used as an example to illustrate the specific collection process of step 1:

[0049] ① Generate configuration vector Config for performance evaluation n ; Among them, Config n It consists of X configuration parameters;

[0050] ②Actual execution of Config n The running time under this configuration is evaluated as Y performance goals; it should be noted that in order to reduce the impact of performance noise, Config n Repeat the run multiple times (5 times is recommended) and take the average of the run times as the final run time ExecTime n ;

[0051] ③ Parse the Spark cluster system execution log Spark_events_log and extract the values ​​of Z intermediate performance indicators; it should be noted that due to Config n If the operation is repeated multiple times, Spark_events_log will also have multiple records. The average value of the intermediate performance indicators is taken as the final intermediate performance indicator EventFeat n .

[0052] Then, Num n You will get an A for your homework. n ={Config n ,EventFeat n ,ExecTime n}.

[0053] Step 2: Use M causal structure learning algorithms to learn A n Analyze and get A n M hierarchical causal structure models B1,n ~B M,n ;

[0054] Among them, the mth causal structure learning algorithm is n Analyze and get A n The mth hierarchical causal structure model B m,n ; m∈[1,M].

[0055] The causal structure learning algorithm uses conditional independence, scoring functions and other methods to evaluate the statistical relationship between variables, determine causal connections, and construct a directed acyclic graph (DAG) to describe causal relationships. In other words, the causal structure learning algorithm aims to analyze the causal relationship between X configuration parameters and Y performance targets - that is, the causal path with configuration parameters as the starting point, performance targets as the end point, and intermediate performance indicators in the middle. These causal paths constitute a hierarchical causal structure model.

[0056] Therefore, B m,n Includes: A n Several causal paths obtained under the analysis of the mth causal structure learning algorithm.

[0057] It should be emphasized that the causal path in the hierarchical causal structure model does not necessarily cover all X configuration parameters, Z intermediate performance indicators, and Y performance targets. This is because it is very likely that some configuration parameters, intermediate performance indicators, and performance targets do not have a causal relationship.

[0058] Therefore, B m,n There is X m,n Configuration parameters, Z m,n Intermediate performance indicators, Y m,n performance goal.

[0059] Among them, X m,n ≤X,Y m,n ≤Y,Z m,n ≤Z.

[0060] In addition, different causal structure learning algorithms processing the same sample data set will reveal the relationship between parameters and performance objectives from different perspectives, thus obtaining different hierarchical causal structure models; the same causal structure learning algorithm processing data sets of different sizes can construct a hierarchical causal structure model related to the size, which contains different configuration parameters, intermediate performance indicators, and performance objectives.

[0061] Similar to the value of N, the value of M should not be too small or too large. Generally, 3 is recommended for M.

[0062] In this embodiment 1, three causal structure learning algorithms are recommended: PC algorithm, FCI algorithm, and MMHC algorithm. Of course, other causal structure learning algorithms can also be used.

[0063] Among them, the PC algorithm uses conditional independence as a constraint, eliminates the causal connection between conditionally independent variables from the complete graph, and then constructs the causal structure through the directed propagation rule.

[0064] The FCI algorithm is a causal structure learning algorithm that adds additional steps to the PC algorithm to identify latent variables and confounding variables. It constructs a causal graph starting with a fully connected undirected graph, removes edges that are independent of conditional independent variables to restore the initial skeleton, then locates edges by identifying "V" and "Y" structures, and locates the remaining edges according to the entropy orientation rule.

[0065] The MMHC algorithm first builds the Bayesian network skeleton and then uses greedy search and K2 scoring criteria to adjust the edges.

[0066] Then, PC algorithm is used as the first causal structure learning algorithm, and five causal structure learning models B are obtained through analysis. 1,1 ~B 1,5 ; FCI algorithm is used as the second causal structure learning algorithm, and five causal structure learning models B are obtained by analysis 2,1 ~B 2,5 MMHC algorithm is used as the third causal structure learning algorithm, and five causal structure learning models are obtained by analysis. 3,1 ~B 3,5 The 15 causal structure learning models mentioned above contain different causal paths, configuration parameters, intermediate performance indicators, and performance targets.

[0067] Step 3: When the big data processing framework needs to run the target size data Num new , then combined with Num new The N hierarchical causal structure models B obtained by analyzing the mth causal structure learning algorithm m,1 ~B m,N Integrated into a data scale-aware causal structure model DCSM m ;

[0068] Traversing M causal structure learning algorithms to obtain M data scale-aware causal structure models DCSM1~DCSM M .

[0069] Step 3 is the core of the present invention, which aims to convert Num new Taking into account to integrate the hierarchical causal structure model B m,1 ~B m,N Data Scale Aware Causal Structure Model DCSMm , which is equivalent to according to Num new Dynamically adjust the causal structure to better adapt to the changes in Num new Big data processing tasks.

[0070] Specifically, DCSM m Take the process of step three as an example:

[0071] S301, calculate Num new With Num n The Manhattan distance L n To determine the nth weight w n .

[0072] Among them, in S301, Num is first calculated new With Num n The Manhattan distance L n , and then L n Converted into weight w n ——L n The smaller, w n The bigger.

[0073] Specifically, w n The calculation formula is:

[0074] S302, calculate the xth configuration parameter among the X configuration parameters in B m,n The average causal effect ACE of the causal path in m,n,x . Where x∈[1,X].

[0075] It should be noted that, due to B m,n At this time, only X m,n configuration parameters, not all X configuration parameters. Therefore, the xth configuration parameter among X configuration parameters may be in B m,n In, or maybe not B m,n middle.

[0076] Then, if the xth configuration parameter among X configuration parameters is in B m,n In B, set it as configuration parameter r, then r in B m,n The causal path in includes: starting from r, Y m,n performance goal as the end point, through Z m,n Several causal relationships composed of intermediate performance indicators.

[0077] Specific:

[0078] ① If the xth configuration parameter among X configuration parameters is in B m,n , then ACE m,n,x Can be seen as ACEm,n,r , whose expression is:

[0079]

[0080] In the formula, |Num n | means Num n The size of the scale; Dom(r) represents the value range of r; a represents the value a; b represents the value b; E[Ydo(r=b),Num n ] represents the expectation corresponding to the performance target when r is b in Numn; E[Ydo(r=a), Num n ] represents the expectation corresponding to the performance target when r is a in Numn.

[0081] ② If the xth configuration parameter among the X configuration parameters is not in B m,n Then in B m,n There is no corresponding causal path in ACE m,n,x =0.

[0082] S303, based on w n ACE m,n,x Weighted, calculate the xth configuration parameter among X configuration parameters in B m,1 ~B m,N CuACE is the cumulative average causal effect of the causal path in m,x .

[0083] Specifically, CuACE m,x The expression is:

[0084]

[0085] S304, traverse X configuration parameters to obtain X cumulative average causal effects CuACE m,1 ~CuACE m,X .

[0086] See above, B m,1 ~B m,N The configuration parameters included are different; that is, X m,1 ~X m,N Not exactly the same, but meets: X m,1 ~X m,N ≤X.

[0087] Then, using X as the traversal standard can ensure that each configuration parameter is calculated.

[0088] S305, CuACE m,1 ~CuACE m,XThe K cumulative average causal effects with the top K values ​​in the middle are used to form a DCSM with the causal paths corresponding to these K cumulative average causal effects. m .

[0089] Therefore, DCSM m There are K configuration parameters, Z m Intermediate performance indicators, Y m performance goals;

[0090] Where K≤X m,n , Z m ≤Y m,n , Z m ≤Z m,n .

[0091] Step 4: Based on DCSM m Give the recommended configuration parameter set C m ;

[0092] Traverse DCSM1~DCSM M Get M recommended configuration parameter sets C1~C M ;

[0093] For C1~C M Take the intersection to get the running number new Recommended configuration parameters.

[0094] Among them, due to DCSM m It is a path diagram composed of causal paths, which includes configuration parameters - it not only relies on A1~A N , also relying on Num new Therefore, from DCSM m Extract K configuration parameters, which constitute C m .

[0095] Then, for DCSM1~DCSM M Do the same process to get C1~C M ; Then C1~C M The intersection is taken, and the result obtained takes into account M kinds of causal structure learning algorithms to improve the configuration tuning effect.

[0096] It should be noted that C1~C M The intersection of the two does not guarantee that there will be X configuration parameters. The remaining number will constitute the running number. new Recommended configuration parameters.

[0097] It should be noted that the value of K is generally set to 15, which is obtained through experiments and past experience. M The running number obtained after taking the intersection newThe number of recommended configuration parameters is close to the threshold α=6.

[0098] For the threshold α, its value is determined based on the specific application scenario, system characteristics, performance goals, and resource constraints. As the α value increases, the contribution of each additional configuration parameter to performance improvement may gradually decrease. Therefore, it is necessary to select appropriate K and α values ​​to avoid C1~C M The intersection of is neither an empty set nor a complete set, and achieves a good balance between performance improvement and tuning cost - if all X configuration parameters are adjusted, the tuning cost will be too high, and the corresponding performance improvement will not be obvious after exceeding the threshold α.

[0099] In addition, considering steps 3 and 4 as a whole, it can handle the situation where the size of the running data changes, and incorporate the size of the data into the construction process of the data scale-aware causal structure model:

[0100] 1. The calculation of the cumulative average causal effect takes into account different data sizes and can more accurately identify key parameters under different data sizes;

[0101] 2. Use multiple causal structure learning algorithms to build multiple data scale-aware causal structure models, which can make full use of the differences between different algorithms in mining causal relationships and adaptively select appropriate models or model combinations based on the characteristics of observed data, so as to more accurately identify key performance parameters and improve configuration tuning effects;

[0102] 3. The combined use of the hierarchical causal structure model and the data scale-aware causal structure model has better interpretability, can clearly show the causal hierarchical relationship between parameters and performance, and helps to understand the system behavior; in terms of generalization ability, through causal analysis and reasoning, it has stronger adaptability to new and unseen data scale system environments, and can be more effectively extended to different application scenarios and data set sizes.

[0103] In summary, the configuration tuning method of a big data processing framework based on causal reasoning provided in this embodiment 1 can run Num new Adaptive configuration tuning is performed when the server is running.

[0104] Example 2

[0105] This embodiment 2 provides a big data processing configuration tuning system based on causal reasoning, which uses the big data processing framework configuration tuning method based on causal reasoning disclosed in embodiment 1.

[0106] See also Figure 2 ,The big data processing configuration tuning system based on causal reasoning includes: sample collection module, causal model building module, causal model integration module, and ,configuration tuning module.

[0107] The sample collection module is configured to process the big data processing framework in N scales of data Num1 to Num N Parameter samples are collected during operation to obtain N sample data sets A1~A N .

[0108] The causal model building module is configured to use M causal structure learning algorithms to learn A n Analyze and get A n M hierarchical causal structure models B 1,n ~B M,n .

[0109] The causal model integration module is configured to combine Num new The N hierarchical causal structure models B obtained by analyzing the mth causal structure learning algorithm m,1 ~B m,N Integrated into a data scale-aware causal structure model DCSM m , and is also used to traverse M causal structure learning algorithms to obtain data scale-aware causal structure models DCSM1~DCSM M .

[0110] Configuration tuning module is configured according to DCSM m Give the recommended configuration parameter set C m , and is also used to traverse DCSM1~DCSM M Get the recommended configuration parameter set C1~C M , also used for C1~C M Take the intersection to get the running number new Recommended configuration parameters.

[0111] Since this system uses the big data processing framework configuration tuning method based on causal reasoning in Example 1, it also has the same effect and will not be repeated here.

[0112] Of course, the big data processing configuration tuning system based on causal reasoning can also add a user interaction module for users or professionals to interact with the tuning system - transmitting user input to the tuning system for processing. User input includes but is not limited to: parameter type and value range of sample data sets and other related settings, Num1~Num N 、Num new , to ensure the normal operation of the tuning system.

[0113] Example 3

[0114] This embodiment 3 discloses a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the big data processing framework configuration tuning method based on causal reasoning disclosed in embodiment 1 are implemented.

[0115] This embodiment 3 also discloses a readable storage medium, in which computer program instructions are stored. When the computer program instructions are read and executed by a processor, the steps of the big data processing framework configuration tuning method based on causal reasoning disclosed in embodiment 1 are executed.

[0116] This embodiment 3 also discloses a computer program product, including a computer program. When the computer program is executed by a processor, the steps of the method for optimizing the configuration of a big data processing framework based on causal reasoning disclosed in embodiment 1 are implemented.

[0117] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A configuration tuning method for a big data processing framework based on causal reasoning, characterized in that: include: Step 1: The big data processing framework is used for N data of different sizes Num1~Num N Parameter samples are collected during operation to obtain N sample data sets A1~A N ; Among them, the nth sample data set A n Includes: Big data processing framework for n-th scale data Num n X configuration parameters, Z intermediate performance indicators, and Y performance goals at runtime; n∈[1,N]; Step 2: Use M causal structure learning algorithms to learn A n Analyze and get A n M hierarchical causal structure models B 1,n ~B M,n ; Among them, the mth causal structure learning algorithm is n Analyze and get A n The mth hierarchical causal structure model B m,n ; m∈[1,M]; Among them, B m,n Includes: A n Several causal paths obtained by analyzing the mth causal structure learning algorithm; B m,n There is X m,n Configuration parameters; X m,n ≤X; Step 3: When the big data processing framework needs to run the target size data Num new , then combined with Num new The N hierarchical causal structure models B obtained by analyzing the mth causal structure learning algorithm m,1 ~B m,N Integrated into a data scale-aware causal structure model DCSM m ; Among them, DCSM m There are K configuration parameters; K≤X m,n ; Traversing M causal structure learning algorithms to obtain M data scale-aware causal structure models DCSM1~DCSM M ; Step 4: Based on DCSM m Given the mth recommended configuration parameter set C m ; Traverse DCSM1~DCSM M Get M recommended configuration parameter sets C1~C M ; For C1~C M Take the intersection to get the running number new Recommended configuration parameters.

2. The method for optimizing the configuration of a big data processing framework based on causal reasoning according to claim 1 is characterized in that: In step 1, N = 5; The size difference between any two datasets is greater than 20%.

3. The method for optimizing the configuration of a big data processing framework based on causal reasoning according to claim 1 is characterized in that: X≥15; Z≥1; Y≥1.

4. The method for optimizing the configuration of a big data processing framework based on causal reasoning according to claim 1 or 3, characterized in that: In step 1, the configuration parameters are a series of parameters that can be configured in the big data processing framework. Intermediate performance indicators are a series of indicators that reflect the changes in resources of the big data processing framework; Performance goals are a series of indicators that reflect the performance changes of the big data processing framework.

5. The method for optimizing the configuration of a big data processing framework based on causal reasoning according to claim 1, characterized in that: In step 2, M=3.

6. The method for optimizing the configuration of a big data processing framework based on causal reasoning according to claim 1, characterized in that: In step 3, DCSM m The integration methods include: Calculate Num new With Num n Manhattan distance L n To determine the nth weight w n ; Calculate the xth configuration parameter among X configuration parameters in B m,n The average causal effect ACE of the causal path in m,n,x ; where x∈[1,X]; Based on w n ACE m,n,x Weighted, calculate the xth configuration parameter among X configuration parameters in B m,1 ~B m,N CuACE is the cumulative average causal effect of the causal path in m,x ; Traverse X configuration parameters and obtain X cumulative average causal effects CuACE m,1 ~CuACE m,X ; CuACE m,1 ~CuACE m,X The K cumulative average causal effects with the top K values ​​in the middle are used to form a DCSM with the causal paths corresponding to these K cumulative average causal effects. m .

7. The method for optimizing the configuration of a big data processing framework based on causal reasoning according to claim 1 or 6, characterized in that: K＝15。 8. The method for optimizing the configuration of a big data processing framework based on causal reasoning according to claim 1, characterized in that: In step 4, C m The methods for obtaining include: From DCSM m Extract K configuration parameters, which constitute C m .

9. A big data processing configuration tuning system based on causal reasoning, characterized in that: It uses the big data processing framework configuration tuning method based on causal reasoning as described in any one of claims 1 to 8; The big data processing configuration tuning system based on causal reasoning includes: The sample collection module is used to collect N-scale data Num1~Num N Parameter samples are collected during operation to obtain N sample data sets A1~A N ; The causal model building module is used to use M causal structure learning algorithms to learn A n Analyze and get A n M hierarchical causal structure models B 1,n ~B M,n ; The causal model integration module is used to combine Num new The N hierarchical causal structure models B obtained by analyzing the mth causal structure learning algorithm m,1 ~B m,N Integrated into a data scale-aware causal structure model DCSM m , and is also used to traverse M causal structure learning algorithms to obtain M data scale-aware causal structure models DCSM1~DCSM M ; as well as Configuration tuning module, which is used to configure the DCSM m Give the recommended configuration parameter set C m , and is also used to traverse DCSM1~DCSM M Get the recommended configuration parameter set C1~C M , also used for C1~C M Take the intersection to get the running number new Recommended configuration parameters.

10. A computer program product, characterized in that It includes a computer program; when the computer program is executed by a processor, it implements the steps of the big data processing framework configuration tuning method based on causal reasoning as described in any one of claims 1-8.

Images