Data skew causal identification processing method, device and equipment and storage medium

Abstract

The invention provides a causal identification processing method and device for data skew, equipment and a storage medium. The method comprises the steps of obtaining multiple pieces of multi-dimensional original data; preprocessing each piece of multi-dimensional original data to generate a plurality of pieces of standardized feature data so as to construct a standardized feature data set; performing feature extraction on the data set to obtain a task feature vector; performing causal effect processing and inclination index prediction on the vector by adopting a double-machine learning model to generate a causal effect value and a prediction inclination index; according to the causal effect values, sorting each intervention strategy, screening out a target intervention strategy with the maximum causal effect value, and adding the target intervention strategy to a task plan of the to-be-executed task; therefore, the processing efficiency and stability of data skew are improved.

Description

Technical Field

[0001] This application relates to the field of causal inference technology, and in particular to a method, apparatus, device and storage medium for causal identification processing of data skew. Background Technology

[0002] As a core driver of current digital transformation, the Internet of Things (IoT) industry is experiencing exponential growth in data volume, characterized by massive scale, complexity, heterogeneity, and strong real-time requirements. In IoT security monitoring scenarios, data warehouses are responsible for storing, analyzing, and responding in real-time to massive amounts of device data. For example, in IoT SIM card security monitoring, real-time analysis of communication behavior, geographical location, and traffic patterns of billions of devices is required to identify abnormal behavior. Such scenarios place extremely high demands on the real-time performance, stability, and resource utilization of data computing tasks. However, data skew can significantly lead to task execution delays, resource waste, and even task failures. Furthermore, the dynamic nature of IoT data further exacerbates the unpredictability of data skew, necessitating a causal identification and processing method for data skew.

[0003] In existing technologies, the main methods for identifying and handling data skew include: early warning mechanisms that set static thresholds based on monitoring data distribution indicators; using machine learning algorithms such as regression models and decision trees to model historical task data to predict the probability of skew; and, after identifying risks, relying on the experience of R&D personnel to subjectively propose intervention measures such as key salting and adjusting shuffle strategies.

[0004] However, existing technologies suffer from several drawbacks: static thresholds have poor dynamic adaptability; prediction models rely solely on correlations without removing causal interference from confounding variables; intervention strategies lack quantitative effect evaluation and are highly subjective; and model construction lacks generality and adaptive optimization. These shortcomings collectively lead to delayed responses to dynamic scenarios and uncertain intervention effects, resulting in low processing efficiency and poor stability in existing technologies for handling data skew. Summary of the Invention

[0005] The data tilt causal identification processing method, apparatus, equipment and storage medium provided in this application are used to solve the technical problems of low processing efficiency and poor stability in the prior art.

[0006] In a first aspect, this application provides a method for identifying the causal relationship of data skew, comprising:

[0007] Acquire multiple multi-dimensional raw data, including metadata, task log information, cluster status information and sampled data. The task log information includes the task plan of the task to be executed and the historical task execution log.

[0008] Each multi-dimensional raw data is preprocessed to obtain a standardized feature dataset containing multiple standardized feature data;

[0009] Feature extraction is performed on the standardized feature dataset to obtain the corresponding task feature vector; the task feature vector includes data distribution features, task operator features, and cluster state features.

[0010] A dual machine learning model is used to process the causal effect of the task feature vector and predict the skew index, so as to generate the causal effect value and the predicted skew index of each intervention strategy. The intervention strategies include adding random labels to the key skew fields, adjusting the association operator type to broadcast association, adjusting the data redistribution parameters, and adjusting the parallel execution degree of the task.

[0011] Based on the causal effect value, each intervention strategy is ranked to select the target intervention strategy with the largest causal effect value, and then the target intervention strategy is added to the task plan of the task to be executed.

[0012] In one possible implementation, feature extraction is performed on the standardized feature dataset to obtain the corresponding task feature vector, including:

[0013] Feature extraction is performed on sampled data from the standardized feature dataset to obtain the corresponding data distribution features;

[0014] Based on the task plan of the task to be executed, feature extraction is performed on the task log information and metadata in the standardized feature dataset to obtain the corresponding task operator features; among them, the task operator features include the association type and the amount of data redistributed.

[0015] Feature extraction is performed on the cluster state information in the standardized feature dataset to obtain the corresponding cluster state features, which include node load, network bandwidth, number of concurrent tasks, and cluster resource utilization.

[0016] In one possible implementation, a dual machine learning model is used to process the causal effect of the task feature vector and predict the tilt index, in order to generate the causal effect value and the predicted tilt index of each intervention strategy, including:

[0017] By using a dual machine learning model, the task feature vector is split or cross-fitted according to the confounding variables to determine the first conditional expectation of the tilt index after the implementation of the intervention strategy and the second conditional expectation of the confounding variables on the intervention strategy.

[0018] Based on the first and second conditional expectations, the corresponding causal effect values ​​are determined using a linear regression formula.

[0019] Based on the task feature vector, the tilt index is calculated to obtain the predicted tilt index corresponding to the intervention strategy.

[0020] In one possible implementation, adding the target intervention strategy to the task schedule of the task to be performed includes:

[0021] Based on the target intervention strategy, the configuration information of the task plan is modified to obtain the modified task plan for the task to be executed.

[0022] Add the target intervention strategy to the task plan of the modified pending task.

[0023] In one possible implementation, after adding the target intervention strategy to the task schedule for the task to be performed, the method further includes:

[0024] After the task plan for the tasks to be executed is completed, the corresponding actual data tilt indicators are determined.

[0025] Determine the corresponding calculation deviation based on the predicted tilt index and the actual data tilt index;

[0026] The parameters of the dual machine learning model are updated based on the calculated bias.

[0027] In one possible implementation, the corresponding calculation deviation is determined based on the predicted tilt index and the actual data tilt index, including:

[0028] Based on the predicted tilt index and the actual tilt index, determine the corresponding absolute difference, and use the absolute difference as the calculation bias; or,

[0029] Based on the predicted tilt index and the actual tilt index, the corresponding absolute difference is determined. The ratio of the absolute difference to the predicted tilt index is taken as the relative difference, and the relative difference is used as the calculation deviation.

[0030] In one possible implementation, the parameters of the dual machine learning model are updated based on the computational bias, including:

[0031] Based on the calculation bias, the weight parameters, regularization coefficients of the loss function, and sample splitting ratio of the dual machine learning model are updated.

[0032] In one possible implementation, preprocessing includes standardization, normalization, and semantic unification.

[0033] Secondly, this application provides a data skew causality identification processing apparatus, comprising:

[0034] The acquisition module is used to acquire multiple multi-dimensional raw data, including metadata, task log information, cluster status information and sampled data. The task log information includes the task plan of the task to be executed and the historical task execution log.

[0035] The first processing module is used to preprocess each multi-dimensional raw data to obtain a standardized feature dataset containing multiple standardized feature data.

[0036] The extraction module is used to extract features from the standardized feature dataset to obtain the corresponding task feature vector; wherein, the task feature vector includes data distribution features, task operator features and cluster state features;

[0037] The second processing module is used to perform causal effect processing and skew index prediction on the task feature vector using a dual machine learning model, so as to generate the causal effect value and the predicted skew index of each intervention strategy. The intervention strategies include adding random labels to the skew key fields, adjusting the association operator type to broadcast association, adjusting the data redistribution parameters, and adjusting the parallel execution degree of the task.

[0038] The filtering module is used to sort each intervention strategy according to the causal effect value, so as to filter out the target intervention strategy with the largest causal effect value and add the target intervention strategy to the task plan of the task to be executed.

[0039] In one possible implementation, the extraction module is further configured to:

[0040] Feature extraction is performed on sampled data from the standardized feature dataset to obtain the corresponding data distribution features;

[0041] Based on the task plan of the task to be executed, feature extraction is performed on the task log information and metadata in the standardized feature dataset to obtain the corresponding task operator features; among them, the task operator features include the association type and the amount of data redistributed.

[0042] Feature extraction is performed on the cluster state information in the standardized feature dataset to obtain the corresponding cluster state features, which include node load, network bandwidth, number of concurrent tasks, and cluster resource utilization.

[0043] In one possible implementation, the second processing module is further configured to:

[0044] By using a dual machine learning model, the task feature vector is split or cross-fitted according to the confounding variables to determine the first conditional expectation of the tilt index after the implementation of the intervention strategy and the second conditional expectation of the confounding variables on the intervention strategy.

[0045] Based on the first and second conditional expectations, the corresponding causal effect values ​​are determined using a linear regression formula.

[0046] Based on the task feature vector, the tilt index is calculated to obtain the predicted tilt index corresponding to the intervention strategy.

[0047] In one possible implementation, the filtering module is also used for:

[0048] Based on the target intervention strategy, the configuration information of the task plan is modified to obtain the modified task plan for the task to be executed.

[0049] Add the target intervention strategy to the task plan of the modified pending task.

[0050] In one possible implementation, the filtering module is also used for:

[0051] After the task plan for the tasks to be executed is completed, the corresponding actual data tilt indicators are determined.

[0052] Determine the corresponding calculation deviation based on the predicted tilt index and the actual data tilt index;

[0053] The parameters of the dual machine learning model are updated based on the calculated bias.

[0054] In one possible implementation, the filtering module is also used for:

[0055] Based on the predicted tilt index and the actual tilt index, determine the corresponding absolute difference, and use the absolute difference as the calculation bias; or,

[0056] Based on the predicted tilt index and the actual tilt index, the corresponding absolute difference is determined. The ratio of the absolute difference to the predicted tilt index is taken as the relative difference, and the relative difference is used as the calculation deviation.

[0057] In one possible implementation, the filtering module is also used for:

[0058] Based on the calculation bias, the weight parameters, regularization coefficients of the loss function, and sample splitting ratio of the dual machine learning model are updated.

[0059] In one possible implementation, the first processing module is further used for preprocessing, which includes standardization, normalization and semantic unification.

[0060] Thirdly, this application provides a data skew causal identification processing device, including: a memory and a processor;

[0061] The memory stores instructions that the computer executes;

[0062] The processor executes computer execution instructions stored in memory, causing the processor to perform the first aspect or various possible implementations of the first aspect as described above.

[0063] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect or various possible implementations of the first aspect.

[0064] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the first aspect or various possible implementations of the first aspect.

[0065] This application provides a method, apparatus, device, and storage medium for identifying and processing causal data skew. By acquiring multi-dimensional raw data, it comprehensively covers information from various aspects of data, tasks, and clusters, laying the foundation for stable data skew analysis. Preprocessing generates a standardized feature dataset, unifying data specifications, reducing interference from data differences, and enhancing the stability of data processing. Feature extraction yields task feature vectors, accurately focusing on key features, avoiding the influence of irrelevant factors, and improving analytical stability. Dual machine learning models handle predictions, combining the advantages of multiple algorithms to reduce the error of a single model, making causal effect values ​​and predicted skew indicators more reliable. Targeted intervention strategies are selected and added to the task plan to address data skew specifically, avoiding blind processing that could lead to new problems and ensuring stable system operation. Thus, it achieves the technical effect of improving the processing efficiency and stability of data skew. Attached Figure Description

[0066] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0067] Figure 1 This application provides a schematic diagram of an application data processing system architecture.

[0068] Figure 2 A flowchart illustrating the causal identification processing method for data skew provided in the embodiments of this application. Figure 1 ;

[0069] Figure 3 A flowchart illustrating the causal identification processing method for data skew provided in the embodiments of this application. Figure 2 ;

[0070] Figure 4 A flowchart illustrating the causal identification processing method for data skew provided in the embodiments of this application. Figure 3 ;

[0071] Figure 5A flowchart illustrating the causal identification processing method for data skew provided in the embodiments of this application. Figure 4 ;

[0072] Figure 6 A flowchart illustrating the causal identification processing method for data skew provided in the embodiments of this application. Figure 5 ;

[0073] Figure 7 A schematic diagram of the data skew causality identification processing device provided in an embodiment of this application;

[0074] Figure 8 A schematic diagram of the structure of the data tilt causal identification processing device provided in the embodiments of this application.

[0075] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0076] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0077] Existing technologies suffer from several shortcomings: static thresholds exhibit poor dynamic adaptability; prediction models rely solely on correlation without removing causal interference from confounding variables; intervention strategies lack quantitative effect evaluation and are highly subjective; and model construction lacks generality and adaptive optimization. These deficiencies collectively lead to delayed responses and uncertain intervention effects in dynamic scenarios, resulting in low processing efficiency and poor stability when dealing with data skew.

[0078] To address the aforementioned issues, this application provides a method, apparatus, device, and storage medium for identifying the causal relationship of data skew. By acquiring multi-dimensional raw data, comprehensively covering information from various aspects of data, tasks, and clusters, it lays the foundation for stable data skew analysis. Preprocessing generates a standardized feature dataset, unifying data specifications, reducing interference from data differences, and enhancing the stability of data processing. Feature extraction yields task feature vectors, accurately focusing on key features, avoiding the influence of irrelevant factors, and improving analytical stability. Dual machine learning models handle predictions, combining the advantages of multiple algorithms to reduce the error of a single model, making causal effect values ​​and predicted skew indicators more reliable. Targeted intervention strategies are selected and added to the task plan to specifically address data skew, avoiding blind processing that could lead to new problems and ensuring stable system operation. Thus, it achieves the technical effect of improving processing efficiency and stability.

[0079] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0080] Figure 1 This is a schematic diagram of an application data processing system architecture provided in an embodiment of this application. The application data processing system is a computer device. Figure 1 As shown, the above architecture includes at least one of a data acquisition device 101, a processing device 102, and a display device 103.

[0081] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the architecture of the application data processing system. In other feasible embodiments of this application, the above architecture may include more or fewer components than illustrated, or combine some components, or split some components, or arrange different components, which can be determined according to the actual application scenario and is not limited here. Figure 1 The components shown can be implemented in hardware, software, or a combination of both.

[0082] In the specific implementation process, the data acquisition device 101 may include an input / output interface or a communication interface, and the data acquisition device 101 can be connected to the processing device through the input / output interface or the communication interface.

[0083] The processing device 102 can first acquire multi-dimensional raw data, construct a standardized feature dataset through standardization and other preprocessing, then extract task feature vectors, use dual machine learning models to process the vectors for causal effects and predict skew indicators, obtain the causal effect values ​​and predicted skew indicators of each intervention strategy, select the target intervention strategy and add it to the task plan, thereby achieving efficient and stable data skew causal identification and processing.

[0084] The display device 103 can also be a touch screen or the screen of a terminal device, used to receive user commands while displaying the above-mentioned content, so as to realize interaction with the user.

[0085] It should be understood that the aforementioned processing device can be implemented by a processor reading instructions from memory and executing those instructions, or it can be implemented by a chip circuit.

[0086] Furthermore, the network architecture and business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0087] Figure 2 A flowchart illustrating the causal identification processing method for data skew provided in the embodiments of this application. Figure 1 ,like Figure 2 As shown, the data skew causal identification processing method provided in this embodiment includes:

[0088] S201. Obtain multiple multi-dimensional raw data.

[0089] In this embodiment, the multi-dimensional raw data includes metadata, task log information, cluster status information, and sampled data. The task log information includes the task plan for the task to be executed and the historical task execution log. The metadata includes the field names, field types, field business meanings, creation time, modification records, and static statistical information of the data warehouse table. The execution plan includes association, grouping, and sorting operator information.

[0090] Multiple dimensions of raw data are acquired, including metadata, task log information, cluster status information, and sampled data. The task log information contains the task plans of the tasks to be executed, providing a comprehensive data foundation for subsequent analysis.

[0091] For example, the process of obtaining metadata from multi-dimensional raw data can be as follows: obtain it from a relational database using a data change capture tool.

[0092] For example, the process of obtaining sampled data from multi-dimensional raw data can be as follows: randomly sample the data in the data warehouse table, generate time-series key field distribution statistics based on the sampling results, and record the key field distribution statistics as sampled data.

[0093] For example, the process of obtaining task log information from multi-dimensional raw data can be as follows: For historical task execution logs: obtain the task execution duration, subtask execution duration, central processing unit (CPU) consumption, memory consumption, amount of data read, amount of data written, and number of data redistribution times from the historical task execution logs; the execution plan of the task to be executed is obtained by setting up data collection points in the scheduling engine and execution engine.

[0094] S202. Preprocess each multi-dimensional original data to obtain a standardized feature dataset containing multiple standardized feature data.

[0095] In this embodiment, preprocessing includes standardization, normalization, and semantic unification.

[0096] Each acquired multi-dimensional raw data is standardized, normalized, and semantically unified. Standardization unifies the original data of different dimensions in terms of units, which facilitates comparison and analysis. Normalization scales the data proportionally to make it fall within a specific range. Semantic unification eliminates semantic ambiguity in the data. After these processes, multiple standardized feature data are generated, which are then aggregated to construct a standardized feature dataset, unifying the data format for subsequent processing.

[0097] S203. Extract features from the standardized feature dataset to obtain the corresponding task feature vector.

[0098] In this embodiment, the task feature vector includes data distribution features, task operator features, and cluster state features.

[0099] Feature extraction is performed on the standardized feature dataset to obtain a task feature vector containing data distribution features, task operator features, and cluster state features, thus clarifying the direction of analysis.

[0100] S204. A dual machine learning model is used to process the causal effect of the task feature vector and predict the tilt index, so as to generate the causal effect value and the predicted tilt index of each intervention strategy.

[0101] In this embodiment, the intervention strategies include adding random identifiers to the skewed key fields, adjusting the association operator type to broadcast association, adjusting the data redistribution parameters, and adjusting the parallel execution degree of the tasks.

[0102] A dual machine learning model is used to process the causal effect of the task feature vector and predict the skew index, generating the causal effect value and predicted skew index for each intervention strategy, providing a basis for screening intervention strategies.

[0103] S205. Based on the causal effect value, sort each intervention strategy to select the target intervention strategy with the largest causal effect value, and add the target intervention strategy to the task plan of the task to be executed.

[0104] In one possible implementation, adding the target intervention strategy to the task plan of the task to be executed includes: modifying the configuration information of the task plan according to the target intervention strategy to obtain the modified task plan of the task to be executed; and adding the target intervention strategy to the modified task plan of the task to be executed.

[0105] Intervention strategies are ranked according to their causal effect values. The target intervention strategy with the largest causal effect value is selected. The task plan configuration information of the task to be executed is modified accordingly, and then the target intervention strategy is added to the modified task plan.

[0106] For example, the specific process of adding a targeted intervention strategy to the task plan of a task to be executed can be as follows: After selecting the targeted intervention strategy based on the causal effect value, firstly, analyze the specific requirements of the targeted intervention strategy in depth. For example, if adding random identifiers to skewed key fields, clarify the rules and scope for adding the identifiers. Next, obtain the task plan of the task to be executed and review its configuration information in detail to find the parts related to the targeted intervention strategy, such as the configuration involving skewed key field processing and operator type settings. Then, according to the requirements of the targeted intervention strategy, precisely modify these related configuration information. For example, when adjusting the association operator type to broadcast association, modify the type parameters of the corresponding operator in the task plan. After completing the modification, obtain the modified task plan of the task to be executed. Finally, add the targeted intervention strategy to the task plan in a standardized form to ensure that the targeted intervention strategy can be accurately implemented according to this plan during subsequent task execution, effectively solving the data skew problem.

[0107] This application provides a causal identification and processing method for data skew. By acquiring multi-dimensional raw data, comprehensively covering information from various aspects of the data, task, and cluster, it lays the foundation for stable data skew analysis. Preprocessing generates a standardized feature dataset, unifying data specifications, reducing interference from data differences, and enhancing the stability of data processing. Feature extraction yields task feature vectors, accurately focusing on key features, avoiding the influence of irrelevant factors, and improving analytical stability. Dual machine learning models are used for prediction, combining the advantages of multiple algorithms to reduce the error of a single model, making causal effect values ​​and predicted skew indicators more reliable. Targeted intervention strategies are selected and added to the task plan to address data skew specifically, avoiding blind processing that could lead to new problems and ensuring stable system operation. Thus, it achieves the technical effect of improving processing efficiency and stability.

[0108] Figure 3 A flowchart illustrating the causal identification processing method for data skew provided in the embodiments of this application. Figure 2 ,like Figure 3 As shown, this embodiment, based on the above embodiments, provides a detailed explanation of the specific process for obtaining the task feature vector, including:

[0109] S301. Extract features from the sampled data in the standardized feature dataset to obtain the corresponding data distribution features.

[0110] For example, extracting features from sampled data in a standardized feature dataset to obtain data distribution characteristics can be done as follows: First, classify the sampled data according to specific rules, such as by data field or business type. Then, calculate statistical indicators for each type of data, such as the Gini coefficient to measure the degree of data imbalance and entropy to reflect the degree of data disorder. The distribution quantity and proportion of data in different intervals can also be statistically analyzed. For instance, for sampled data containing skewed key fields, the frequency of occurrence of each key field value can be counted, and its distribution pattern analyzed to obtain data distribution characteristics that accurately reflect the data distribution, providing a data foundation for subsequent analysis of the causes of data skew and the development of intervention strategies.

[0111] S302. Based on the task plan of the task to be executed, extract features from the task log information and metadata in the standardized feature dataset to obtain the corresponding task operator features.

[0112] In this embodiment, the task operator features include association type and data redistribution data volume.

[0113] For example, based on the task plan of the task to be executed, the task execution duration, subtask execution duration, etc., are extracted from the task log information of the standardized feature dataset, and table structure information is obtained from the metadata. For the association types in the task operator features, operators such as join and group by are examined in the task plan to determine their types, such as inner join and outer join. The data redistribution volume is then calculated based on the data redistribution operations in the task plan, and the amount of data involved is statistically analyzed. Through these operations, task operator features that accurately reflect the characteristics of operators and changes in data distribution during task execution can be obtained.

[0114] S303. Extract features from the cluster state information in the standardized feature dataset to obtain the corresponding cluster state features.

[0115] In this embodiment, the cluster status characteristics include node load, network bandwidth, number of concurrent tasks, and cluster resource utilization.

[0116] For example, features are extracted from the cluster status information in the standardized feature dataset, and the load status of each node in the cluster, such as CPU and memory usage, is collected in real time. Network bandwidth data is obtained through network monitoring tools to understand data transmission capacity. The number of tasks executing concurrently in the cluster is counted to assess the cluster's workload. Simultaneously, the overall utilization rate of cluster resources, including storage and computing resources, is calculated. Combining this information yields a comprehensive cluster status feature that reflects the current operating status of the cluster, providing a basis for analyzing the relationship between data skew and cluster status.

[0117] The data skew causal identification method provided in this application extracts data distribution features that accurately present data distribution patterns, providing key information for identifying data skew points. The obtained task operator features clearly demonstrate the operation of operators and data changes during task execution, aiding in the analysis of the impact of operators on data skew. The acquired cluster state features reflect the cluster's operational status, clarifying the correlation between cluster state and data skew. These three elements collectively lay the foundation for generating accurate task feature vectors, facilitating subsequent causal effect processing and skew index prediction.

[0118] Figure 4 A flowchart illustrating the causal identification processing method for data skew provided in the embodiments of this application. Figure 3 ,like Figure 4 As shown, this embodiment, based on the above embodiments, provides a detailed explanation of the generation process of the causal effect value and the prediction tilt index, including:

[0119] S401. Using a dual machine learning model, based on the confounding variables, the task feature vector is split into samples or cross-fitted to determine the first conditional expectation of the bias index after the implementation of the intervention strategy and the second conditional expectation of the confounding variables on the intervention strategy.

[0120] In this embodiment, the confounding variable refers to the variable in the task feature vector that has an impact on both the tilt index and the intervention strategy.

[0121] The task feature vector is split or cross-fitted based on the confounding variables by using a dual machine learning model. The sample splitting involves randomly dividing the dataset corresponding to the task feature vector into two non-overlapping sub-sample sets. One set is used for model training to estimate the conditional expectation, and the other set is used for subsequent calculation of causal effect values.

[0122] Cross-fitting is achieved through multiple rounds of sample partitioning and alternating training and computation. This process can remove bias and ensure orthogonality, avoiding model overfitting and ensuring the asymptotic unbiasedness and normality of subsequent estimation results. Finally, the first conditional expectation of the skewness index after the intervention strategy is determined by the confounding variable, namely E[Y|X] (Y is the skewness index, X is the confounding variable), denoted as... And the second conditional expectation of the intervention strategy for the confounding variables, namely E[T|X] (T is the intervention strategy, X is the confounding variable), denoted as .

[0123] S402. Based on the first conditional expectation and the second conditional expectation, determine the corresponding causal effect value using the linear regression formula.

[0124] In this embodiment, the causal effect value is used to quantify the direct impact of the intervention strategy on data skew, where a negative value indicates that the corresponding intervention strategy can reduce the degree of data skew.

[0125] Based on the first condition expectation Second conditional expectation To eliminate the interference of confounding variables on skewed indicators and intervention strategies, a linear regression model without intercept is constructed. The core relationship of this model is: For dependent variable, Let be the independent variable. The causal effect value is obtained by solving the model by minimizing the loss function of the sample version. , The parsing expression is:

[0126]

[0127] Where Y is the skewness indicator, T is the intervention strategy, and X is the confounding variable. For the second conditional expectation, This is the first conditional expectation.

[0128] S403. Calculate the tilt index based on the task feature vector to obtain the predicted tilt index corresponding to the intervention strategy.

[0129] The task feature vector comprises data distribution features, task operator features, and cluster state features. Data distribution features include the Gini coefficient and entropy value of key fields; task operator features include association type and the amount of data redistributed; and cluster state features include node load, network bandwidth, number of concurrent tasks, and cluster resource utilization. Based on this task feature vector and the determined intervention strategy, the tilt index is calculated using the prediction branch in the dual machine learning model. The result is the predicted tilt index corresponding to the intervention strategy, which is subsequently compared with the actual tilt index to determine the calculation deviation.

[0130] The data skew causal identification and processing method provided in this application accurately determines two conditional expectations through sample splitting or cross-fitting, providing a foundation for causal effect analysis; it uses linear regression formulas to determine causal effect values, clearly quantifying the impact of intervention strategies on skew indicators; and it calculates and predicts skew indicators, enabling advance estimation of skew indicator conditions after the implementation of intervention strategies. These three aspects collectively provide crucial data support for solving the data skew problem, improving overall processing efficiency and accuracy.

[0131] Figure 5 A flowchart illustrating the causal identification processing method for data skew provided in the embodiments of this application. Figure 4 ,like Figure 5 As shown, this embodiment, based on the above embodiment, provides supplementary explanations of the subsequent process after adding the target intervention strategy to the task plan of the task to be executed, including:

[0132] S501. After the task plan for the task to be executed is completed, determine the corresponding actual data tilt index, and determine the corresponding calculation deviation based on the predicted tilt index and the actual data tilt index.

[0133] After the task plan for the task to be executed is completed, the actual data tilt index is obtained and compared with the predicted tilt index to determine the calculation deviation, which is used to evaluate the accuracy of the model prediction.

[0134] S502. Update the parameters of the dual machine learning model based on the calculation deviation.

[0135] In one possible implementation, the parameters of the dual machine learning model are updated based on the computational bias, including updating the weight parameters, regularization coefficients of the loss function, and sample splitting ratio of the dual machine learning model based on the computational bias.

[0136] Based on the calculation bias, the weight parameters, regularization coefficients of the loss function, and sample splitting ratio of the dual machine learning model are updated to make the model more accurate.

[0137] For example, based on the calculated deviation between the predicted skew index and the actual data skew index, a dual machine learning model parameter update process is initiated: First, based on the magnitude and direction of the calculated deviation, the model's bias tendency in prediction is determined. If the deviation is large and exhibits a specific direction, the weight parameters are adjusted accordingly, increasing the weight of key features or weakening the influence of irrelevant features. For the regularization coefficient of the loss function, if the calculated deviation indicates model overfitting, the coefficient is appropriately increased to constrain model complexity; if underfitting, the coefficient is decreased. Simultaneously, the samples are re-divided according to the calculated deviation, the sample splitting ratio is adjusted, and the new ratio is used to re-segment or cross-fit the samples to avoid overfitting and ensure the asymptotic unbiasedness and normality of the estimation. Through these operations, the parameters of the dual machine learning model are gradually optimized, enabling it to more accurately handle causal effects and predict skew indexes.

[0138] The data skew causal identification processing method provided in this application obtains the actual data skew index and calculates the deviation from the predicted value after the task plan is completed. This can intuitively reflect the accuracy of the model prediction and provide a key basis for model optimization. Updating the parameters of the dual machine learning model based on the calculated deviation, adjusting the weight parameters can optimize the model's attention to different features, updating the regularization coefficient of the loss function can prevent overfitting, and modifying the sample split ratio can improve the unbiasedness and normality of the model estimation, thereby improving the accuracy of subsequent predictions.

[0139] Figure 6 A flowchart illustrating the causal identification processing method for data skew provided in the embodiments of this application. Figure 5 ,like Figure 6 As shown, this embodiment, based on the above embodiments, provides a detailed explanation of the process for determining the calculation deviation, including:

[0140] S601. Based on the predicted tilt index and the actual data tilt index, determine the corresponding absolute difference and use the absolute difference as the calculation deviation.

[0141] The absolute difference between the predicted tilt index and the actual tilt index is obtained by calculating the absolute value of the numerical difference between the predicted tilt index and the actual tilt index, and directly reflects the difference in magnitude between the two.

[0142] S602. Based on the predicted tilt index and the actual tilt index, determine the corresponding absolute difference, take the ratio of the absolute difference to the predicted tilt index as the relative difference, and take the relative difference as the calculation deviation.

[0143] S602 and S601 are parallel schemes. The absolute difference between the predicted tilt index and the actual data tilt index is calculated using the same steps as in S601. Then, the absolute difference is divided by the predicted tilt index to obtain the relative difference. The relative difference is used to eliminate the influence of the magnitude difference of the predicted tilt index.

[0144] For example, by using both absolute and relative differences, the deviation assessment scenarios for data warehouse computing tasks of different sizes can be adapted. When the magnitude of the predicted skew index is fixed, the absolute difference can intuitively reflect the deviation magnitude. When the magnitude of the predicted skew index fluctuates greatly, the relative difference can accurately reflect the relative degree of deviation, thereby improving the applicability and accuracy of the deviation assessment and providing reliable data support for subsequent updates of dual machine learning model parameters.

[0145] The data skew causal identification processing method provided in this application embodiment directly uses the absolute difference as the calculation deviation, which can intuitively reflect the size of the gap between the prediction and the actual situation, and simply and quickly measure the prediction accuracy; or it uses the relative difference as the calculation deviation, which takes into account the base of the prediction skew index, and can more reasonably compare the deviation under different scales of prediction. Both provide key basis for evaluating the prediction effect and adjusting the intervention strategy.

[0146] Figure 7 This is a schematic diagram of the data skew causality identification processing device provided in an embodiment of this application. The device in this embodiment can be in software or hardware form. Figure 7 As shown, the data tilt causal identification processing device 700 provided in this application embodiment includes: an acquisition module 701, a first processing module 702, an extraction module 703, a second processing module 704, and a filtering module 705.

[0147] The acquisition module 701 is used to acquire multiple multi-dimensional raw data, including metadata, task log information, cluster status information and sampled data. The task log information includes the task plan of the task to be executed and the historical task execution log.

[0148] The first processing module 702 is used to preprocess each multi-dimensional raw data to obtain a standardized feature dataset containing multiple standardized feature data.

[0149] The extraction module 703 is used to extract features from the standardized feature dataset to obtain the corresponding task feature vector; wherein, the task feature vector includes data distribution features, task operator features and cluster state features;

[0150] The second processing module 704 is used to perform causal effect processing and skew index prediction on the task feature vector using a dual machine learning model, so as to generate the causal effect value and the predicted skew index of each intervention strategy. The intervention strategies include adding random labels to the skew key fields, adjusting the association operator type to broadcast association, adjusting the data redistribution parameters, and adjusting the parallel execution degree of the task.

[0151] The filtering module 705 is used to sort each intervention strategy according to the causal effect value, so as to filter out the target intervention strategy with the largest causal effect value and add the target intervention strategy to the task plan of the task to be executed.

[0152] In one possible implementation, the extraction module 703 is further configured to:

[0153] Feature extraction is performed on sampled data from the standardized feature dataset to obtain the corresponding data distribution features;

[0154] Based on the task plan of the task to be executed, feature extraction is performed on the task log information and metadata in the standardized feature dataset to obtain the corresponding task operator features; among them, the task operator features include the association type and the amount of data redistributed.

[0155] Feature extraction is performed on the cluster state information in the standardized feature dataset to obtain the corresponding cluster state features, which include node load, network bandwidth, number of concurrent tasks, and cluster resource utilization.

[0156] In one possible implementation, the second processing module 704 is further configured to:

[0157] By using a dual machine learning model, the task feature vector is split or cross-fitted according to the confounding variables to determine the first conditional expectation of the tilt index after the implementation of the intervention strategy and the second conditional expectation of the confounding variables on the intervention strategy.

[0158] Based on the first and second conditional expectations, the corresponding causal effect values ​​are determined using a linear regression formula.

[0159] Based on the task feature vector, the tilt index is calculated to obtain the predicted tilt index corresponding to the intervention strategy.

[0160] In one possible implementation, the filtering module 705 is further configured to:

[0161] Based on the target intervention strategy, the configuration information of the task plan is modified to obtain the modified task plan for the task to be executed.

[0162] Add the target intervention strategy to the task plan of the modified pending task.

[0163] In one possible implementation, the filtering module 705 is further configured to:

[0164] After the task plan for the tasks to be executed is completed, the corresponding actual data tilt indicators are determined.

[0165] Determine the corresponding calculation deviation based on the predicted tilt index and the actual data tilt index;

[0166] The parameters of the dual machine learning model are updated based on the calculated bias.

[0167] In one possible implementation, the filtering module 705 is further configured to:

[0168] Based on the predicted tilt index and the actual tilt index, determine the corresponding absolute difference, and use the absolute difference as the calculation bias; or,

[0169] Based on the predicted tilt index and the actual tilt index, the corresponding absolute difference is determined. The ratio of the absolute difference to the predicted tilt index is taken as the relative difference, and the relative difference is used as the calculation deviation.

[0170] In one possible implementation, the filtering module 705 is further configured to:

[0171] Based on the calculation bias, the weight parameters, regularization coefficients of the loss function, and sample splitting ratio of the dual machine learning model are updated.

[0172] In one possible implementation, the first processing module 702 is further used for preprocessing, which includes standardization processing, normalization processing, and semantic unification processing.

[0173] The data tilt causal identification processing device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0174] Figure 8 A schematic diagram of the structure of the data skew causal identification processing device provided in an embodiment of this application. Figure 8 As shown, the data skew causal identification processing device 800 provided in this embodiment includes at least one processor 801 and a memory 802. Optionally, the device 800 further includes a communication component 803. The processor 801, memory 802, and communication component 803 are connected via a bus.

[0175] In a specific implementation, at least one processor 801 executes computer execution instructions stored in memory 802, causing at least one processor 801 to perform the above-described method.

[0176] The specific implementation process of processor 801 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0177] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0178] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0179] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0180] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0181] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0182] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0183] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0184] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0185] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0186] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0187] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0188] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0189] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for identifying the cause of data skew, characterized in that, include: Acquire multiple multi-dimensional raw data, including metadata, task log information, cluster status information and sampled data, wherein the task log information includes the task plan of the task to be executed and the historical task execution log; Each of the multi-dimensional raw data is preprocessed to obtain a standardized feature dataset containing multiple standardized feature data. Feature extraction is performed on the standardized feature dataset to obtain the corresponding task feature vector; wherein, the task feature vector includes data distribution features, task operator features and cluster state features; A dual machine learning model is used to process the causal effect and predict the skew index of the task feature vector to generate the causal effect value and the predicted skew index of each intervention strategy. The intervention strategies include adding random labels to the skew key fields, adjusting the association operator type to broadcast association, adjusting the data redistribution parameters, and adjusting the task parallel execution degree. Based on the causal effect value, each intervention strategy is sorted to select the target intervention strategy with the largest causal effect value, and the target intervention strategy is added to the task plan of the task to be executed.

2. The method according to claim 1, characterized in that, The step of extracting features from the standardized feature dataset to obtain the corresponding task feature vector includes: Feature extraction is performed on the sampled data in the standardized feature dataset to obtain the corresponding data distribution features; Based on the task plan of the task to be executed, feature extraction is performed on the task log information and metadata in the standardized feature dataset to obtain the corresponding task operator features; wherein, the task operator features include association type and data redistribution data volume; Feature extraction is performed on the cluster state information in the standardized feature dataset to obtain the corresponding cluster state features, wherein the cluster state features include node load, network bandwidth, number of concurrent tasks, and cluster resource utilization.

3. The method according to claim 2, characterized in that, The step of employing a dual machine learning model to process the causal effect of the task feature vector and predict the tilt index, in order to generate the causal effect value corresponding to each intervention strategy and the predicted tilt index of the intervention strategy, includes: Using the dual machine learning model, the task feature vector is split or cross-fitted based on the confounding variable to determine the first conditional expectation of the tilt index after the intervention strategy is implemented and the second conditional expectation of the confounding variable on the intervention strategy. Based on the first conditional expectation and the second conditional expectation, the corresponding causal effect value is determined using a linear regression formula; Based on the task feature vector, the tilt index is calculated to obtain the predicted tilt index corresponding to the intervention strategy.

4. The method according to claim 3, characterized in that, Adding the target intervention strategy to the task plan for the task to be executed includes: Based on the target intervention strategy, the configuration information of the task plan for the task to be executed is modified to obtain the modified task plan for the task to be executed. Add the target intervention strategy to the task plan of the modified task to be executed.

5. The method according to any one of claims 1 to 4, characterized in that, After adding the target intervention strategy to the task plan of the task to be performed, the method further includes: After the task plan for the task to be executed is completed, the corresponding actual data skew index is determined. Based on the predicted tilt index and the actual data tilt index, determine the corresponding calculation deviation; The parameters of the dual machine learning model are updated based on the calculated deviation.

6. The method according to claim 5, characterized in that, The step of determining the corresponding calculation deviation based on the predicted tilt index and the actual data tilt index includes: Based on the predicted tilt index and the actual data tilt index, determine the corresponding absolute difference, and use the absolute difference as the calculation deviation; or, Based on the predicted tilt index and the actual data tilt index, the corresponding absolute difference is determined, the ratio of the absolute difference to the predicted tilt index is taken as the relative difference, and the relative difference is taken as the calculation deviation.

7. The method according to claim 6, characterized in that, The step of updating the parameters of the dual machine learning model based on the calculated deviation includes: Based on the calculated deviation, the weight parameters, regularization coefficients of the loss function, and sample splitting ratio of the dual machine learning model are updated.

8. The method according to any one of claims 1 to 4, characterized in that, The preprocessing includes standardization, normalization, and semantic unification.

9. A causal identification processing device for data skew, characterized in that, include: The acquisition module is used to acquire multiple multi-dimensional raw data, wherein the multi-dimensional raw data includes metadata, task log information, cluster status information and sampled data, and the task log information includes the task plan of the task to be executed and the historical task execution log; The first processing module is used to preprocess each of the multi-dimensional raw data to obtain a standardized feature dataset containing multiple standardized feature data. The extraction module is used to extract features from the standardized feature dataset to obtain the corresponding task feature vector; wherein, the task feature vector includes data distribution features, task operator features and cluster state features; The second processing module is used to perform causal effect processing and skew index prediction on the task feature vector using a dual machine learning model, so as to generate the causal effect value corresponding to each intervention strategy and the predicted skew index of the intervention strategy. The intervention strategy includes adding random identifiers to the skew key fields, adjusting the association operator type to broadcast association, adjusting the data redistribution parameters, and adjusting the task parallel execution degree. The filtering module is used to sort each intervention strategy according to the causal effect value, so as to filter out the target intervention strategy with the largest causal effect value, and add the target intervention strategy to the task plan of the task to be executed.

10. A data skew causal identification processing device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-8.

12. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-8.

Images