Multi-dimensional root cause analysis method, system and equipment based on clustering and regression tree algorithm and medium
By using clustering and regression tree algorithms in multi-dimensional root cause analysis, the shortcomings of traditional algorithms in branch combination and dimension processing are solved, efficient and accurate root cause analysis and intuitive display tools are achieved, and the clarity and efficiency of business decisions are improved.
Patent Information
- Application Number
- CN202411890015.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-20
- Publication Date
- 2025-04-18
AI Technical Summary
Traditional algorithms in multi-dimensional roots have problems such as difficult to remove impurities caused by branch combinations and difficult to deal with in the upper and lower dimensions, resulting in low analysis efficiency and accuracy.
The multi-dimensional root cause analysis method based on clustering and regression tree algorithms was adopted, and the significant branches were extracted through the N+1 branching method, and the upper and lower dimensions were processed by two modeling, and the expansion coefficient and penalty term log (N+1) were introduced during the pruning process to improve the analysis efficiency and accuracy.
It realizes automated multi-dimensional traceability analysis of business analysis indicators, has the functions of automated decomposition and precise positioning of abnormal data, improves the efficiency and accuracy of root cause analysis, and provides intuitive display tools for business decision-making.
Smart Images
Figure CN119849623A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of big data analysis, and in particular to a multidimensional root cause analysis method, system, device and medium based on clustering and regression tree algorithms. Background Art
[0002] The CART (Classification and Regression Trees) algorithm is a popular decision tree learning technique that can be used for both classification and regression problems. The CART algorithm recursively divides the data set into smaller subsets and constructs a tree, where each node represents a judgment condition and each leaf node represents a prediction result.
[0003] The CHAID (Chi-squared Automatic Interaction Detection) algorithm is a decision tree statistical modeling method based on the chi-square test. It is mainly used in the fields of market segmentation, customer classification, and predictive analysis. It constructs a decision tree by detecting the interaction between features, providing decision makers with an intuitive basis for decision making.
[0004] AHC (Agglomerative Hierarchical Clustering): is a bottom-up clustering method. Starting from the data point, each point is initially regarded as a separate cluster, and then the closest clusters are gradually merged until a certain termination condition is reached or a single cluster is formed. When used in root cause analysis, it has higher interpretability than other clustering ensemble algorithms.
[0005] ClickHouse (ClickStream, DataWareHouse, ClickHouse for short) is a completely column-based distributed database management system (DBMS) with high data compression ratio, support for real-time query processing, distributed query, support for linear expansion, high reliability, high fault tolerance, etc. It is particularly suitable for business scenarios that require high performance and real-time analysis.
[0006] The CART (Classification and Regression Trees) algorithm is a popular decision tree learning technique that can be used for both classification and regression problems. The CART algorithm recursively divides the data set into smaller subsets and constructs a tree, where each node represents a judgment condition and each leaf node represents a prediction result.
[0007] The CHAID (Chi-squared Automatic Interaction Detection) algorithm is a decision tree statistical modeling method based on the chi-square test. It is mainly used in the fields of market segmentation, customer classification, and forecasting analysis. It constructs a decision tree by detecting the interaction between features, providing decision makers with an intuitive basis for decision making.
[0008] AHC (Agglomerative Hierarchical Clustering): is a bottom-up clustering method. Starting from the data point, each point is initially regarded as a separate cluster, and then the closest clusters are gradually merged until a certain termination condition is reached or a single cluster is formed. When used in root cause analysis, it has higher interpretability than other clustering ensemble algorithms.
[0009] ClickHouse (ClickStream, DataWareHouse, ClickHouse for short) is a completely column-based distributed database management system (DBMS) with high data compression ratio, support for real-time query processing, distributed query, support for linear expansion, high reliability, high fault tolerance, etc. It is particularly suitable for business scenarios that require high performance and real-time analysis. Summary of the invention
[0010] In view of the deficiencies in the prior art, the present invention provides a multi-dimensional root cause analysis method, system, device and medium based on clustering and regression tree algorithms. To solve the problem that impurities generated by the branch combination of traditional algorithms are difficult to remove: an N+1 branching method is adopted, that is, the multi-branching method formed by the CART dichotomy and the CHAID multi-branching method is not used, but the significant branches are extracted according to the change value (that is, the weighted change rate) through a clustering algorithm, and the dimension values with significant changes are used as independent branches, that is, "N", and all dimension values with non-significant changes are merged into one branch, that is, "1". To solve the problem that multiple upper and lower dimensions are difficult to handle: two modelings are adopted. The first modeling generally adopts the upper / coarse-grained dimension, and the leaf node based on the result of the first modeling is used for secondary modeling, and the lower / fine-grained dimension field is generally used for modeling. Improve the efficiency and accuracy of root cause analysis: introduce the expansion coefficient, consider the scale change and the change rate at the same time, and assist in tree pruning. Due to the multi-branch model, the number of branches of different dimensional fields may vary greatly. By using the penalty term log(N+1), the algorithm tends to select dimensional fields with small branches. Provide a visual display of the root cause analysis results. With the help of graphical means, managers and front-line operators can quickly and intuitively identify the root causes of business development and improve the efficiency of operational analysis.
[0011] To achieve the above object, the present invention adopts the following technical solutions:
[0012] A multidimensional root cause analysis method based on clustering and regression tree algorithm comprises the following steps:
[0013] Step 1: Collect and pre-process data from different business scenarios to obtain a wide data table; the wide data table includes different fields and data values under the fields;
[0014] Step 2: According to the dimensions analyzed by the business needs, select fields related to the problem from the wide data table, wherein the fields include upper-level fields and lower-level fields subordinate to the upper-level fields, and construct a first root cause tree based on the upper-level fields and the data in the wide data table;
[0015] Step 3, pruning the first root cause tree to obtain a pruned first root cause tree;
[0016] Step 4: Based on the leaf nodes of the pruned first root cause tree, the first root cause tree is further branched based on the subordinate fields to obtain a second root cause tree; the second root cause tree is pruned to obtain a final root cause tree;
[0017] Step 5: Extract the attributes from the root node to the leaf node in the final root cause tree, and visualize the abnormal data of different business scenarios to find the root cause of the problem.
[0018] To optimize the above technical solutions, the specific measures taken also include:
[0019] Furthermore, step 2 specifically includes:
[0020] Step 2.1, initialize the data wide table and load it into memory;
[0021] Step 2.2: According to the dimensions analyzed by the business needs, select the fields in the data wide table that may be related to the problem, and use the fields as candidate fields for root cause tree modeling;
[0022] Step 2.3, initialize the list of nodes to be processed as an empty list, and put all the data in the data wide table into the list of nodes to be processed;
[0023] Step 2.4: extract a node from the list of nodes to be processed. If the list of nodes to be processed is empty, stop expanding the tree and jump to step 3. If the list of nodes to be processed is not empty, select one of the nodes to be processed and go to step 2.5.
[0024] Step 2.5: Determine whether the current tree meets the stop expansion condition according to the selected node to be processed, wherein the stop expansion condition is: the depth of the node to be processed in the current tree exceeds a specified threshold, or the weight scale of the branch reaches a specified threshold; if the stop expansion condition is met, the node terminates further branch expansion operations, and removes the node from the list of nodes to be processed, and returns to step 2.4; if the stop expansion condition is not met, proceed to step 2.6;
[0025] Step 2.6: Select fields from the candidate fields, and calculate the branches of the nodes to be processed based on agglomerative hierarchical clustering. Fields that have already participated in modeling will no longer be selected for execution. N+1 branches are obtained, where N represents the number of clusters that meet the coverage threshold.
[0026] Step 2.7, calculate the branch measure value;
[0027] Step 2.8: Select the branch with the smallest branch measure value as the optimal branch, remove the processed nodes from the list of nodes to be processed; put the expanded new node into the list of nodes to be processed, and jump to step 2.4.
[0028] Furthermore, step 2.6 is specifically as follows:
[0029] Select fields from the candidate fields, group the data values corresponding to the fields into two groups: positive change values and negative change values. Perform agglomerative hierarchical clustering on each group according to the degree of change of each data value to obtain several clusters. Select N clusters that meet the coverage threshold as N branches, and merge the remaining clusters that do not meet the coverage threshold as one branch.
[0030] Furthermore, step 2.7 is specifically as follows:
[0031]
[0032] Where BMV represents the branching measure value, W j represents the weighted standard deviation of the j-th branch, ω i represents the weight of the i-th data point in the cluster, x i is the ratio indicator, is the average value of the ratio index.
[0033] Furthermore, step 3 is specifically as follows:
[0034] Step 3.1, traverse each branch node from the leaf node upwards. If the evaluation values of all nodes below a branch node are less than the evaluation value of the branch node, then delete the branch node, otherwise keep the branch node; if the change scale of the branch node is lower than the specified threshold, prune it;
[0035] Step 3.2: For the tree pruned in step 3.1, calculate the value attribute V of each node. The calculation formula is as follows:
[0036] V=C*R*(E) D
[0037] Among them, V represents the value attribute, C represents the change amount, R represents the change rate, C*R represents the interest, E represents the expansion coefficient, and D represents the node depth, which indicates the level of the node in the tree structure;
[0038] The value attributes of all nodes are sorted from large to small and put into the leaf list in sequence. If the parent node or child node of the node is already in the leaf node list, the node will not be put into the leaf node list.
[0039] The nodes retained in the final leaf node list are all the nodes of the first root tree after pruning.
[0040] The present invention also proposes a multidimensional root cause analysis system based on clustering and regression tree algorithm, comprising:
[0041] A data collection module is used to collect and pre-process data from different business scenarios to obtain a wide data table; the wide data table includes different fields and data values under the fields;
[0042] A first modeling module is used to select fields related to the problem from the wide data table according to the dimensions analyzed by the business needs, wherein the fields include upper-level fields and lower-level fields subordinate to the upper-level fields, and to construct a first root cause tree based on the upper-level fields and the data in the wide data table;
[0043] A pruning module is used to prune the first root cause tree to obtain a pruned first root cause tree;
[0044] The second modeling module is used to, based on the leaf nodes of the pruned first root cause tree and the lower-level fields, continue to branch the first root cause tree to obtain a second root cause tree; prune the second root cause tree to obtain a final root cause tree;
[0045] The display module is used to extract the attributes from the root node to the leaf node in the final root cause tree, and visualize the abnormal data of different business scenarios to find the root cause of the problem.
[0046] The present invention also proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the multidimensional root cause analysis method based on clustering and regression tree algorithm as described above is implemented.
[0047] The present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the multidimensional root cause analysis method based on clustering and regression tree algorithms as described above.
[0048] The beneficial effects of the present invention are:
[0049] 1. The present invention adopts machine learning technologies such as agglomerative hierarchical clustering algorithm and regression tree, which are more interpretable and independent of initial conditions. At the same time, relying on the powerful data processing capabilities of Clickhouse database, it provides a multi-dimensional root cause analysis method and system, realizes the automated multi-dimensional traceability analysis of business analysis indicators, and has the functions of automated decomposition and precise positioning of abnormal data.
[0050] 2. The present invention optimizes the regression tree algorithm, mainly including: (1) adopting the N+1 branching method to solve the problem that impurities generated by the branch combination of the traditional method are difficult to remove. (2) adopting two-time modeling to solve the problem that the traditional method is difficult to handle multiple upper and lower dimensions. (3) When pruning, the scale change and change rate are comprehensively considered, and the expansion coefficient and penalty term log(N+1) are introduced to improve the efficiency and accuracy of root cause analysis.
[0051] 3. The present invention provides intuitive display tools for root cause analysis, including root cause decomposition trees and exception lists. These tools not only clearly display the root causes of the problem, but also help the business team quickly identify areas that need priority attention through prompts of importance (high, medium). This intuitive and layered display method makes business decisions clearer and more focused.
[0052] 4. It realizes multi-dimensional root cause analysis, improves the execution efficiency of the root cause analysis algorithm, and solves the problems in traditional methods that impurities generated by branch combinations are difficult to remove, the root cause tree has poor interpretability and is difficult to understand, and the analysis dimensions involve many upper and lower dimensions that are difficult to handle.
[0053] 5. It improves the efficiency and accuracy of business data analysis, solves the problem that business development changes cannot be automatically analyzed and the root causes cannot be deeply explored, and solves the problem that it is time-consuming and labor-intensive to locate the causes of business development changes.
[0054] 6. It realizes the ecological, automated and personalized data services. With the help of graphical means, it directly displays the root causes of abnormal business development, allowing managers and front-line operators to quickly find the abnormal causes of business development and provide strong data support for the subsequent formulation of business marketing plans. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] Figure 1 The overall flow chart of the multidimensional root cause analysis method based on clustering and regression tree algorithm proposed by the present invention;
[0056] Figure 2 Flowchart for constructing the first root cause tree. DETAILED DESCRIPTION
[0057] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0058] Embodiment 1
[0059] The present invention proposes a multidimensional root cause analysis method based on clustering and regression tree algorithm, the flow chart is as follows: Figure 1 As shown, the following steps are included:
[0060] Step 1: Collect and pre-process data from different business scenarios, such as product ordering channel sources, ordering methods, sales product categories, sales product IDs, sales product ordering time / cancellation time / city / cooperative sales number / billing fees, etc. Pre-processing includes processing missing values, outliers, and data conversion to obtain a wide data table; the wide data table includes different fields and data values under the fields; sort out the fields in the wide table that have hierarchical relationships, such as: region-related fields from cities-districts and counties-branch bureaus to finer grid levels, the affiliation of sales product categories and specific sales product specifications, etc. These relationships will be considered as one of the factors for subsequent modeling.
[0061] Step 2: According to the dimensions analyzed by the business needs, select fields related to the problem from the data wide table, including upper-level fields and lower-level fields subordinate to the upper-level fields, and build the first root cause tree based on the upper-level fields and the data in the data wide table; the process of building the first root cause tree is as follows: Figure 2 As shown, specifically including:
[0062] Step 2.1, initialize the data wide table and load it into memory;
[0063] Step 2.2: According to the dimensions analyzed by the business needs, select the fields in the data wide table that may be related to the problem, and use the fields as candidate fields for root cause tree modeling;
[0064] Step 2.3, initialize the list of nodes to be processed as an empty list, and put all the data in the data wide table into the list of nodes to be processed;
[0065] Step 2.4: extract a node from the list of nodes to be processed. If the list of nodes to be processed is empty, stop expanding the tree and jump to step 3. If the list of nodes to be processed is not empty, select one of the nodes to be processed and go to step 2.5.
[0066] Step 2.5: Determine whether the current tree meets the stop expansion condition according to the selected node to be processed, wherein the stop expansion condition is: the depth of the node to be processed in the current tree exceeds a specified threshold, or the weight scale of the branch reaches a specified threshold; if the stop expansion condition is met, the node terminates further branch expansion operations, and removes the node from the list of nodes to be processed, and returns to step 2.4; if the stop expansion condition is not met, proceed to step 2.6;
[0067] Step 2.6: Select fields from the candidate fields, and calculate the branches of the nodes to be processed based on agglomerative hierarchical clustering. Fields that have already participated in modeling will no longer be selected for execution. N+1 branches are obtained, where N represents the number of clusters that meet the coverage threshold. Specifically:
[0068] Select fields from the candidate fields, group the data values corresponding to the fields into two groups: positive change values and negative change values. Perform agglomerative hierarchical clustering on each group according to the degree of change of each data value to obtain several clusters. Select N clusters that meet the coverage threshold as N branches, and merge the remaining clusters that do not meet the coverage threshold as one branch.
[0069] Step 2.7: Calculate the branch measure value; specifically:
[0070]
[0071] Where BMV represents the branching measure value, W j represents the weighted standard deviation of the j-th branch, ω i represents the weight of the i-th data point in the cluster, x i is the ratio index, and x is the average value of the ratio index.
[0072] Step 2.8: Select the branch with the smallest branch measure value as the optimal branch, remove the processed nodes from the list of nodes to be processed; put the expanded new node into the list of nodes to be processed, and jump to step 2.4.
[0073] Step 3: Prune the first root cause tree to obtain the pruned first root cause tree; specifically:
[0074] Step 3.1, traverse each branch node from the leaf node upwards. If the evaluation values of all nodes below a branch node are less than the evaluation value of the branch node, then delete the branch node, otherwise keep the branch node; if the change scale of the branch node is lower than the specified threshold, prune it;
[0075] Step 3.2: For the tree pruned in step 3.1, calculate the value attribute V of each node. The calculation formula is as follows:
[0076] V=C*R*(E) D
[0077] Among them, V represents the value attribute, C represents the change amount, R represents the change rate, C*R represents the interest, E represents the expansion coefficient, and D represents the node depth, which indicates the level of the node in the tree structure;
[0078] The value attributes of all nodes are sorted from large to small and put into the leaf list in sequence. If the parent node or child node of the node is already in the leaf node list, the node will not be put into the leaf node list.
[0079] The nodes retained in the final leaf node list are all the nodes of the first root tree after pruning.
[0080] Step 4: Based on the leaf nodes of the pruned first root cause tree, the first root cause tree is further branched based on the subordinate fields to obtain a second root cause tree; the second root cause tree is pruned to obtain a final root cause tree;
[0081] Step 5: Extract the attributes from the root node to the leaf node in the final root cause tree, and visualize the abnormal data of different business scenarios to find the root cause of the problem.
[0082] Embodiment 2
[0083] The present invention proposes a multidimensional root cause analysis system based on clustering and regression tree algorithm corresponding to the method of embodiment 1, comprising:
[0084] A data collection module is used to collect and pre-process data from different business scenarios to obtain a wide data table; the wide data table includes different fields and data values under the fields;
[0085] A first modeling module is used to select fields related to the problem from the wide data table according to the dimensions analyzed by the business needs, wherein the fields include upper-level fields and lower-level fields subordinate to the upper-level fields, and to construct a first root cause tree based on the upper-level fields and the data in the wide data table;
[0086] A pruning module is used to prune the first root cause tree to obtain a pruned first root cause tree;
[0087] The second modeling module is used to, based on the leaf nodes of the pruned first root cause tree and the lower-level fields, continue to branch the first root cause tree to obtain a second root cause tree; prune the second root cause tree to obtain a final root cause tree;
[0088] The display module is used to extract the attributes from the root node to the leaf node in the final root cause tree, and visualize the abnormal data of different business scenarios to find the root cause of the problem.
[0089] The implementation methods of each module and module function in the system are completely consistent with the steps of the method in Example 1, so they will not be repeated here.
[0090] Embodiment 3
[0091] The present invention proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the multidimensional root cause analysis method based on clustering and regression tree algorithm as described in Example 1 is implemented.
[0092] Embodiment 4
[0093] The present invention provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the multidimensional root cause analysis method based on clustering and regression tree algorithm as described in the first embodiment.
[0094] Below we will analyze the root causes of the revenue changes in June 2024 in conjunction with the video ringback tone business.
[0095] Step A: Data aggregation and processing
[0096] Collect and obtain full data based on business sales products, main products and user income dimensions. The fields involved include city / district / county / branch / grid, sales product / main product name and ID, main product / sales product acceptance time / effective time / cancellation time, acceptance channel (primary, secondary and tertiary channels), cancellation channel (primary, secondary and tertiary channels), primary and secondary cards, user points, billing expenses, billing items, user payment type, three-zero user identification and other data. Through the Clickhouse cluster database, the multi-dimensional data is processed based on the product instance ID, mainly including processing missing values, outliers and data conversion, and finally a wide data table is formed.
[0097] Step B: First modeling and construction of root cause tree
[0098] Based on the wide data table generated in step A, and according to the business revenue change analysis scenario, select analysis dimensions, such as city, major sales product category, user points classification, user payment type, customer type, cancellation channel (primary), and ordering channel (primary), etc.
[0099] The parameters for the first tree construction are as follows: the tree depth is set to 20, the branch weight scale is set to 1 / 1000, the branch scale coverage threshold is set to 95%, the fitting parameter is the month-on-month change rate of video ringback tone revenue, and the weight in the branch measurement value calculation is the scale of video ringback tone revenue last month.
[0100] Step C: Prune the root tree
[0101] Based on the first analysis root cause tree generated in step B, the first pruning threshold is set to 1 / 5000. In the second pruning, the expansion factor E used to remove insignificant branches is set to 1.05, so that the purity (or impurity) of each child node can be as high (or low) as possible to achieve the best classification of the data.
[0102] Step D: Secondary modeling based on each leaf node
[0103] The second modeling is performed for fine-grained or multi-attribute dimension such as county, specific sales product specifications, user points, user industry, subscription channel (three-level channel) and cancellation channel (three-level channel). The difference from the first modeling parameters is that the tree depth is set to 5 layers, the scale threshold is set to 1 / 100, and the expansion factor E used in the second pruning is set to 1.5. Other parameters remain unchanged (for example, the fitting parameter is still the month-on-month change rate of video ringback tone revenue, and the weight in the calculation of branch measurement value is still the scale of video ringback tone revenue last month), as follows:
[0104] Each leaf node is analyzed based on user industry, subscription channel (third-level channel), and unsubscription channel (third-level channel), and the leaf node is placed in the "to-be-processed list" to determine whether the node in the to-be-processed list needs further branch expansion. The determination method is: compare whether the depth of the node in the current tree exceeds the specified threshold of 5 layers, and whether the weight scale of the branch reaches the specified threshold of 1 / 100. (1) If the stop expansion condition is met, the node terminates further branch expansion operations, and removes the node from the "to-be-processed list", and determines whether the "to-be-processed list" is empty. If the "to-be-processed list" is empty, prune the node. If it is not empty, continue to obtain new nodes and determine whether the node needs to be expanded. (2) If the stop expansion condition is not met, expand the node: perform "N+1" hierarchical clustering, treat the significantly changed dimension values as independent branches, that is, N in "N+1", and pack all non-significant dimension values into one branch, that is, "1" in "N+1". Then, based on the N+1 branches output by clustering, the measurement value is calculated, where the fitting parameter is the month-on-month change rate of video ringback tone revenue, and the weight in the branch measurement value (BMV) calculation is the scale of video ringback tone revenue last month. According to the BMV of each dimension field, the dimension field with the smallest BMV is selected as the corresponding branch as the optimal solution. And do the following processing: (a) remove the processed nodes from the list of nodes to be processed; (b) put the expanded new nodes into the "list of nodes to be processed", and then obtain the nodes from the "nodes in the list of nodes to be processed", and determine whether further branch expansion is needed until all the nodes in the "list of nodes to be processed" are processed.
[0105] Pruning: If the "Node List to be Processed" is empty, pruning is performed. First, from the leaf nodes upward, the branch nodes that do not meet the change scale threshold (1 / 100) are removed. Then the value attribute V of each node is calculated, and the expansion coefficient E is set to 1.5. After the calculation is completed, the V values of all nodes are sorted from large to small, and cyclically put into the leaf list. If the parent node or child node of the node is already in the leaf node list, the node will not be put in.
[0106] Step E: Extract the final leaf node information and display it visually
[0107] For the complete root cause tree formed from step B to step D, extract the branch attributes from the root node to the leaf node, and use the sequence outlier method of analyzing the index value to display the abnormalities or significant changes in various combinations using trend analysis, anomaly list, and TOP change list to display relevant information.
[0108] After analysis, it was finally found that the change in sales revenue of the video ringback tone (enterprise version) business function package of 25 yuan / month / household in June 2024 had a significant impact on the video ringback tone business revenue in June 24. The reduction in revenue was particularly affected by the changes in the Nanjing local network, mainly due to the serious decline in revenue from physical acceptance channels, and the customer industries involved included agriculture, forestry, animal husbandry, sideline production, fishery, real estate and construction industries.
[0109] In the embodiments disclosed in the present application, the computer storage medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. The computer storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of computer storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.
[0110] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0111] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.
Claims
1. A multidimensional root cause analysis method based on clustering and regression tree algorithm, characterized in that: The following steps are involved: Step 1: Collect and pre-process data from different business scenarios to obtain a wide data table; the wide data table includes different fields and data values under the fields; Step 2: According to the dimensions analyzed by the business needs, select fields related to the problem from the wide data table, wherein the fields include upper-level fields and lower-level fields subordinate to the upper-level fields, and construct a first root cause tree based on the upper-level fields and the data in the wide data table; Step 3, pruning the first root cause tree to obtain a pruned first root cause tree; Step 4: Based on the leaf nodes of the pruned first root cause tree, the first root cause tree is further branched based on the subordinate fields to obtain a second root cause tree; Prune the second root cause tree to obtain the final root cause tree; Step 5: Extract the attributes from the root node to the leaf node in the final root cause tree, and visualize the abnormal data of different business scenarios to find the root cause of the problem.
2. The multidimensional root cause analysis method based on clustering and regression tree algorithm as claimed in claim 1, characterized in that: Step 2 specifically includes: Step 2.1, initialize the data wide table and load it into memory; Step 2.2: According to the dimensions analyzed by the business needs, select the fields in the data wide table that may be related to the problem, and use the fields as candidate fields for root cause tree modeling; Step 2.3, initialize the list of nodes to be processed as an empty list, and put all the data in the data wide table into the list of nodes to be processed; Step 2.4: extract a node from the list of nodes to be processed. If the list of nodes to be processed is empty, stop expanding the tree and jump to step 3. If the list of nodes to be processed is not empty, select one of the nodes to be processed and go to step 2.
5. Step 2.5: Determine whether the current tree meets the stop expansion condition according to the selected node to be processed, wherein the stop expansion condition is: the depth of the node to be processed in the current tree exceeds a specified threshold, or the weight scale of the branch reaches a specified threshold; if the stop expansion condition is met, the node terminates further branch expansion operations, and removes the node from the list of nodes to be processed, and returns to step 2.4; if the stop expansion condition is not met, proceed to step 2.6; Step 2.6: Select fields from the candidate fields, and calculate the branches of the nodes to be processed based on agglomerative hierarchical clustering. Fields that have already participated in modeling will no longer be selected for execution. N+1 branches are obtained, where N represents the number of clusters that meet the coverage threshold. Step 2.7, calculate the branch measure value; Step 2.8: Select the branch with the smallest branch measure value as the optimal branch, remove the processed nodes from the list of nodes to be processed; put the expanded new node into the list of nodes to be processed, and jump to step 2.
4.
3. The multidimensional root cause analysis method based on clustering and regression tree algorithm as claimed in claim 2, characterized in that: Step 2.6 is as follows: Select fields from the candidate fields, group the data values corresponding to the fields into two groups: positive change values and negative change values. Perform agglomerative hierarchical clustering on each group according to the degree of change of each data value to obtain several clusters. Select N clusters that meet the coverage threshold as N branches, and merge the remaining clusters that do not meet the coverage threshold as one branch.
4. The multidimensional root cause analysis method based on clustering and regression tree algorithm as claimed in claim 3, characterized in that: Step 2.7 is as follows: Where BMV represents the branching measure value, W j represents the weighted standard deviation of the j-th branch, ω i represents the weight of the i-th data point in the cluster, x i is the ratio indicator, is the average value of the ratio index.
5. The multidimensional root cause analysis method based on clustering and regression tree algorithm as claimed in claim 1, characterized in that: Step 3 is as follows: Step 3.1, traverse each branch node from the leaf node upwards in turn. If the evaluation values of all nodes below a branch node are less than the evaluation value of the branch node, then delete the branch node, otherwise keep the branch node; if the change scale of the branch node is lower than the specified threshold, prune it; Step 3.2: For the tree pruned in step 3.1, calculate the value attribute V of each node. The calculation formula is as follows: V=C*R*(E) D Among them, V represents the value attribute, C represents the change amount, R represents the change rate, C*R represents the interest, E represents the expansion coefficient, and D represents the node depth, which indicates the level of the node in the tree structure; The value attributes of all nodes are sorted from large to small and put into the leaf list in sequence. If the parent node or child node of the node is already in the leaf node list, the node will not be put into the leaf node list. The nodes retained in the final leaf node list are all the nodes of the first root tree after pruning.
6. A multidimensional root cause analysis system based on clustering and regression tree algorithm, characterized in that: include: The data collection module is used to collect and preprocess data from different business scenarios to obtain a wide data table; The data wide table includes different fields and data values under the fields; A first modeling module is used to select fields related to the problem from the wide data table according to the dimensions analyzed by the business needs, wherein the fields include upper-level fields and lower-level fields subordinate to the upper-level fields, and to construct a first root cause tree based on the upper-level fields and the data in the wide data table; A pruning module is used to prune the first root cause tree to obtain a pruned first root cause tree; The second modeling module is used to branch the first root cause tree based on the leaf nodes of the pruned first root cause tree and the lower-level fields to obtain a second root cause tree; Prune the second root cause tree to obtain the final root cause tree; The display module is used to extract the attributes from the root node to the leaf node in the final root cause tree, and visualize the abnormal data of different business scenarios to find the root cause of the problem.
7. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the multidimensional root cause analysis method based on clustering and regression tree algorithm as described in any one of claims 1 to 5 is implemented.
8. A computer-readable storage medium storing a computer program, characterized in that: The computer program enables a computer to execute the multidimensional root cause analysis method based on clustering and regression tree algorithm as described in any one of claims 1 to 5.