Data skew adjustment method and device, electronic equipment and storage medium

Abstract

The invention provides a data skew adjustment method and device, electronic equipment and a storage medium, and the method comprises the steps: generating an abnormal score corresponding to each operator through a first model, and generating a distance measure corresponding to each operator through a second model under the condition that the data size processed by the operators is detected to be abnormal, the first model and the second model are constructed based on the key features, and the key features are determined through real-time monitoring data and an analysis result of a target execution plan; determining whether the operator distribution state is in a data skew state or not based on the abnormal scores corresponding to all operators and the distance measurement; and if yes, performing adaptive adjustment on the operator distribution state based on a preset adjustment rule. According to the method, the data distribution is continuously monitored by using the anomaly detection algorithm, so that the problem of data skew can be automatically found without manual intervention.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of big data processing technology, and in particular to a data tilt adjustment method, device, electronic device and storage medium. Background Art

[0002] Data skew is a common problem in big data processing and distributed computing. Data skew can seriously affect the execution efficiency of tasks and the utilization of computing power, thereby reducing the performance of the entire system.

[0003] Existing data skew processing solutions mainly include manual monitoring, rule-based automatic detection and adjustment, etc., but these methods often require a lot of manual intervention and experience accumulation, and cannot adapt to complex and changeable big data scenarios. Summary of the invention

[0004] Embodiments of the present application provide a data skew adjustment method, device, system, electronic device, and computer-readable storage medium.

[0005] In a first aspect, an embodiment of the present application provides a data tilt adjustment method, the method comprising:

[0006] In the case of detecting that there is an abnormal amount of data processed by an operator, generating an abnormal score corresponding to each operator through a first model, and generating a distance metric corresponding to each operator through a second model, wherein the first model and the second model are constructed based on key features, and the key features are determined by parsing results of real-time monitoring data corresponding to the operator and a target execution plan corresponding to the operator;

[0007] Determining whether an operator distribution state is in a data skew state based on the abnormal scores corresponding to all the operators and the distance metric;

[0008] If so, the operator distribution state is adaptively adjusted based on a preset adjustment rule.

[0009] Optionally, determining whether the current operator is in a data skew state based on the anomaly score and the distance metric includes:

[0010] When it is detected that the abnormal score reaches a first threshold and the distance metric reaches a second threshold, it is determined whether the operator is in a data skew state.

[0011] Optionally, the adaptively adjusting the operator distribution state based on a preset adjustment rule includes:

[0012] The parallelism of the operator is adjusted based on the amount of data processed by the operator and the resource status of the system corresponding to the operator.

[0013] Optionally, the adaptively adjusting the operator distribution state based on a preset adjustment rule includes:

[0014] The resource allocation state of the target execution plan corresponding to the operator is adjusted through the resource manager.

[0015] Optionally, the adaptively adjusting the operator distribution state based on a preset adjustment rule includes:

[0016] Determine whether the data volume corresponding to the target partition is abnormal;

[0017] If yes, re-partition the data.

[0018] Optionally, the adaptively adjusting the operator distribution state based on a preset adjustment rule includes:

[0019] Merge several operators based on preset operator merging rules.

[0020] Optionally, after the step of adaptively adjusting the operator distribution state based on a preset adjustment rule, the method includes:

[0021] Analyze the real-time monitoring data corresponding to the operator to determine the target time period corresponding to the data tilt state when the operator distribution state is in the data tilt state;

[0022] Obtain the distribution state of the adaptively adjusted target operator;

[0023] Performing label processing on each adaptive adjustment result based on the target time period and the target operator distribution state to obtain a label corresponding to each adaptive adjustment result;

[0024] Feedback optimization is performed on the first model and the second model based on the label corresponding to each of the adaptive adjustment results.

[0025] In a second aspect, an embodiment of the present application provides a data tilt adjustment device, the device comprising:

[0026] A generation module, configured to generate an abnormality score corresponding to each operator through a first model and generate a distance metric corresponding to each operator through a second model when detecting that an abnormal amount of data processed by an operator exists, wherein the first model and the second model are constructed based on key features, and the key features are determined by parsing results of real-time monitoring data corresponding to the operator and a target execution plan corresponding to the operator;

[0027] A determination module, configured to determine whether an operator distribution state is in a data skew state based on the abnormal scores and the distance metrics corresponding to all the operators;

[0028] The adjustment module is used to, if yes, adaptively adjust the operator distribution state based on a preset adjustment rule.

[0029] Optionally, the determining module includes:

[0030] The first determination submodule is used to determine whether the operator is in a data skew state when it is detected that the abnormal score reaches a first threshold and the distance metric reaches a second threshold.

[0031] Optionally, the adjustment module includes:

[0032] The second adjustment submodule is used to adjust the parallelism of the operator based on the amount of data processed by the operator and the resource status of the system corresponding to the operator.

[0033] Optionally, the adjustment module includes:

[0034] The third adjustment submodule is used to adjust the resource allocation state of the target execution plan corresponding to the operator through the resource manager.

[0035] Optionally, the adjustment module includes:

[0036] The fourth adjustment submodule is used to determine whether the data volume corresponding to the target partition is abnormal;

[0037] The fifth adjusting submodule is used for re-partitioning the data if yes.

[0038] Optionally, the adjustment module includes:

[0039] The sixth adjustment submodule is used to merge several operators based on a preset operator merging rule.

[0040] Optionally, the device further comprises:

[0041] The first module is used to analyze the real-time monitoring data corresponding to the operator to determine the target time period corresponding to the data tilt state when the operator distribution state is in the data tilt state;

[0042] The second module is used to obtain the distribution state of the target operator after adaptive adjustment;

[0043] A third module is used to perform label processing on each of the adaptive adjustment results based on the target time period and the target operator distribution state to obtain a label corresponding to each of the adaptive adjustment results;

[0044] The fourth module is used to perform feedback optimization on the first model and the second model based on the label corresponding to each adaptive adjustment result.

[0045] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor, wherein the processor is configured to execute the instructions to implement any of the data tilt adjustment methods described above.

[0046] In a fourth aspect, an embodiment of the present application further provides a storage medium, which, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute any of the data tilt adjustment methods described above.

[0047] In a fifth aspect, an embodiment of the present application further provides a vehicle, comprising the data tilt adjustment device as described above.

[0048] In an embodiment of the present application, when detecting that there is an abnormal amount of data processed by an operator, an abnormal score corresponding to each operator is generated through a first model, and a distance metric corresponding to each operator is generated through a second model, wherein the first model and the second model are constructed based on key features, and the key features are determined by the analysis results of the real-time monitoring data corresponding to the operator and the target execution plan corresponding to the operator; based on the abnormal scores and the distance metric corresponding to all the operators, it is determined whether the operator distribution state is in a data tilt state; if so, the operator distribution state is adaptively adjusted based on a preset adjustment rule. That is, in an embodiment of the present application, by continuously monitoring the data distribution using an anomaly detection algorithm, the data tilt problem can be automatically discovered without manual intervention. Compared with the traditional static threshold monitoring method, this method is more intelligent and flexible, and can adapt to the dynamic changes of data distribution. This solution can not only detect the data tilt itself, but also locate the specific reasons for the tilt, such as data distribution deviation, uneven computing load, etc., which helps to formulate more targeted optimization strategies and improve the optimization effect. By timely discovering and repairing data tilt problems, it can ensure the stable and efficient operation of the big data application system, effectively alleviate the performance bottleneck caused by data tilt, and improve the throughput and response speed of the overall system.

[0049] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0051] Figure 1 It is a flowchart of the steps of a data tilt adjustment method provided in an embodiment of the present application;

[0052] Figure 2 is a flowchart of another method for adjusting data skew provided in an embodiment of the present application;

[0053] Figure 3 is a device block diagram of a data tilt adjustment device provided in an embodiment of the present application;

[0054] Figure 4 is a structural diagram of an electronic device provided in an embodiment of the present application;

[0055] Figure 5 This is an exemplary data tilt adjustment flow chart provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0057] Figure 1 is a flowchart of a method for adjusting data skew provided in an embodiment of the present application. Figure 1 As shown, the method may include:

[0058] Step 101, when detecting that there is an abnormal amount of data processed by an operator, generating an abnormality score corresponding to each operator through the first model, and generating a distance metric corresponding to each operator through the second model, wherein the first model and the second model are constructed based on the key features, and the key features are determined by the real-time monitoring data corresponding to the operator and the analysis results of the target execution plan corresponding to the operator;

[0059] It should be noted that in the embodiments of the present application, first, by real-time monitoring of the amount of data processed by each operator in Flink and Spark, and calculating the ratio of these data amounts to the overall average data amount, operators with abnormally high data processing volumes can be identified, which may be the source of data skew. Then, by obtaining and analyzing the Flink job graph or the physical execution plan of Spark, especially the wide dependency operations and data flows therein, the structure of the job and the potential risk of data skew can be further understood, thereby providing a qualitative analysis basis for subsequent optimization and adjustment.

[0060] Key features may be selected based on the real-time monitoring data corresponding to the operator and the analysis results of the target execution plan corresponding to the operator. In an embodiment of the present application, the key features may include at least one of the operator data volume ratio, operator type, and parallelism.

[0061] After determining the key features, an abnormal operator recognition model can be constructed based on the key features, that is, the first model and the second model in the embodiment of the present application, wherein the first model can be an IF model and the second model can be an OC-SVM model.

[0062] Specifically, IF model training, that is, training the Isolation Forest model, learns how to quickly isolate abnormal data points; OC-SVM model training, that is, training the One-Class Support Vector Machine model, determines the normal boundaries of the data.

[0063] Isolation Forest (IF) is an anomaly detection algorithm based on random forests. It identifies abnormal data points by randomly dividing the feature space. Abnormal data points are more easily isolated than normal data points, so their isolation depth will be shallower. In the embodiment of the present application, the IF algorithm is used to isolate the data set and calculate the abnormal score of each data point. Data points with high abnormal scores are considered to be abnormal data, which helps to identify data skew. By randomly dividing the feature space, IF can quickly identify abnormal data points because they are easier to isolate. The IF algorithm outputs the abnormal score of each data point. The higher the score, the more likely it is abnormal data.

[0064] One-Class Support Vector Machine (OC-SVM), OC-SVM is an unsupervised single-class support vector machine algorithm that identifies abnormal data by learning the boundaries of normal data. OC-SVM finds a hyperplane so that normal data is as far away from the hyperplane as possible and abnormal data is located outside the hyperplane; OC-SVM identifies abnormal data by learning the boundaries of normal data and calculating the distance from each data point to the hyperplane. The smaller the distance, the more likely the data point is abnormal data. OC-SVM helps determine the normal boundaries of a data set, thereby identifying data points that deviate from the normal range.

[0065] It should be noted that in the embodiments of the present application, in distributed computing frameworks such as Apache Flink and ApacheSpark, data points are abnormal data, which means that some data points exhibit abnormal behavior during processing, for example, they may cause a certain operator to process much more data than other operators. This uneven data distribution can lead to load imbalance and affect the performance of the entire system. By monitoring the amount of data processed by each operator, it is possible to identify which operators process too many data points, which may be the source of data skew. Then, machine learning algorithms (such as IsolationForest and One-Class Support Vector Machine) can be used to further analyze these data points to determine whether they are indeed abnormal data, and take corresponding adjustment measures accordingly, such as repartitioning data, adjusting the parallelism of operators, etc., to optimize data distribution and improve system performance.

[0066] Step 102, determining whether the operator distribution state is in a data skew state based on the abnormal scores corresponding to all operators and the distance metric;

[0067] The degree of anomaly of an operator can be quantified by using the Isolation Forest (IF) model to calculate the anomaly score of each operator and the One-Class Support Vector Machine (OC-SVM) model to calculate the distance from the data point to the hyperplane. Combining the output results of IF and OC-SVM, especially when a data point has a high anomaly score in IF and a small distance metric in OC-SVM, it is possible to more accurately judge the data point as abnormal data, thereby identifying data skew. This method improves the accuracy and reliability of data skew detection through algorithm combination.

[0068] Further, the determining whether the current operator is in a data skew state based on the anomaly score and the distance metric includes:

[0069] When it is detected that the abnormal score reaches a first threshold and the distance metric reaches a second threshold, it is determined whether the operator is in a data skew state.

[0070] Specifically, the thresholds of the anomaly score and the distance metric can be set, and then a comprehensive score can be performed, that is, a comprehensive score is calculated for each operator by combining the operator data volume ratio, real-time data monitoring results, execution plan analysis, and the weights of the prediction results of the IF and OC-SVM models.

[0071] Based on the scores of all operators and whether each operator is an abnormal operator, it can be determined whether the operator distribution state is in a data skew state.

[0072] Step 103: If yes, adaptively adjust the operator distribution state based on a preset adjustment rule.

[0073] For operators whose comprehensive scores exceed a threshold and whose distribution status is in a data skew state, the operator distribution status and each operator can be automatically adjusted for parallelism, repartitioned, or the execution plan optimized based on preset adjustment rules.

[0074] Furthermore, the adaptively adjusting the distribution state of the operators based on preset adjustment rules includes: adjusting the parallelism of the operators based on the amount of data processed by the operators and the resource state of the system corresponding to the operators.

[0075] The parallelism of operators is dynamically adjusted according to the amount of data processed by the operators and the system resources. If the amount of data of an operator is much larger than that of other operators, the system can increase the parallelism of this operator to distribute the load.

[0076] Furthermore, the adaptively adjusting the operator distribution state based on a preset adjustment rule includes: adjusting the resource allocation state of the target execution plan corresponding to the operator through a resource manager.

[0077] Use a resource manager, such as YARN, Mesos, or Kubernetes, to adjust resource allocation for your job.

[0078] Furthermore, the adaptively adjusting the operator distribution state based on a preset adjustment rule includes: determining whether there is an abnormality in the amount of data corresponding to the target partition; and if so, re-partitioning the data.

[0079] If data skew is detected due to excessive data in some partitions, the system can repartition the data. In Spark, you can use repartition() or coalesce(), and in Flink, you can adjust the parallelism of the keyBy operation or use a custom partition function.

[0080] Furthermore, the partitioning strategy is optimized according to the distribution characteristics of the data, such as using partitioning based on data volume or data range.

[0081] Furthermore, analyze the wide dependency operations in the execution plan, such as Shuffle, and optimize these operations, for example, by adjusting the number of Shuffle partitions or using a more efficient join algorithm.

[0082] Furthermore, the adaptively adjusting the operator distribution state based on a preset adjustment rule includes: merging a plurality of the operators based on a preset operator merging rule.

[0083] When possible, merge multiple operators to reduce the flow of data between operators and reduce the impact of data skew.

[0084] In addition, the query plan can be automatically adjusted and the optimal execution path can be selected based on real-time monitoring data and execution plan analysis results.

[0085] In an embodiment of the present application, when detecting that there is an abnormal amount of data processed by an operator, an abnormal score corresponding to each operator is generated through the first model, and a distance metric corresponding to each operator is generated through the second model, wherein the first model and the second model are constructed based on the key features, and the key features are determined by the real-time monitoring data corresponding to the operator and the analysis results of the target execution plan corresponding to the operator; based on the abnormal scores and the distance metrics corresponding to all operators, it is determined whether the operator distribution state is in a data tilt state; if so, the operator distribution state is adaptively adjusted based on the preset adjustment rules. That is, in an embodiment of the present application, by continuously monitoring the data distribution using an anomaly detection algorithm, the data tilt problem can be automatically discovered without manual intervention. Compared with the traditional static threshold monitoring method, this method is more intelligent and flexible, and can adapt to the dynamic changes of data distribution. This solution can not only detect the data tilt itself, but also locate the specific causes of the tilt, such as data distribution deviation, uneven computing load, etc., which helps to formulate more targeted optimization strategies and improve the optimization effect.

[0086] In addition, the embodiments of the present application are independent of the specific data processing framework, and can monitor and optimize data skew across multiple frameworks such as Spark and Flink. They can be extended to other types of big data application scenarios and have good versatility and portability.

[0087] Figure 2 is a flowchart of another method for adjusting data skew provided in an embodiment of the present application. Figure 2 As shown, the method may include:

[0088] Step 201: when detecting that the amount of data processed by an operator is abnormal, generating an abnormal score corresponding to each operator through the first model, and generating a distance metric corresponding to each operator through the second model, wherein the first model and the second model are constructed based on the key features, and the key features are determined by the real-time monitoring data corresponding to the operator and the analysis results of the target execution plan corresponding to the operator;

[0089] Step 202, determining whether the operator distribution state is in a data skew state based on the abnormal scores corresponding to all operators and the distance metric;

[0090] Step 203: If yes, adaptively adjust the operator distribution state based on a preset adjustment rule;

[0091] It should be noted that in the embodiment of the present application, the above steps 201-203 refer to the previous discussion and will not be repeated here.

[0092] Step 204, analyzing the real-time monitoring data corresponding to the operator to determine a target time period corresponding to the data tilt state when the operator distribution state is in the data tilt state;

[0093] Step 205, obtaining the target operator distribution state after adaptive adjustment;

[0094] Step 206, performing label processing on each of the adaptive adjustment results based on the target time period and the target operator distribution state to obtain a label corresponding to each of the adaptive adjustment results;

[0095] After the adaptive processing measures are implemented, key performance indicators such as operator data volume ratio, throughput, processing delay, resource utilization, etc. are continuously monitored to ensure that the monitoring system can provide real-time feedback to quickly respond to performance changes.

[0096] Analyze performance monitoring data to determine if adaptive processing actions are effective and, if performance metrics are not improving, identify new issues or ongoing causes of data skew.

[0097] Analyze data collected by performance monitoring systems, including processing time, throughput, resource utilization, etc.

[0098] Determine which operators or jobs exhibit characteristics of data skew during a specific time period.

[0099] Evaluate the effectiveness of adaptive processing measures, such as whether the data volume ratios of all operators are close to the average after data redistribution.

[0100] For the identified data skew cases, classify and label them according to the effects of the adaptive processing measures. The labels can be "solved", "partially solved", "unsolved", etc. The specific label categories are set according to actual needs. The detailed information of each case is recorded, including the operator name, job ID, timestamp, performance indicator data, adaptive measures taken, etc.

[0101] Step 207: Feedback optimization is performed on the first model and the second model based on the label corresponding to each of the adaptive adjustment results.

[0102] By designing a feedback loop, the labeled data and results are fed back to the model training process for model learning and optimization. For model feedback and retraining, prepare the data set for model retraining, including historical data and newly collected labeled cases; perform feature selection and engineering on the retraining data set to ensure that the model can learn from key features; use the updated data set to retrain the IF and OC-SVM models to improve the model's prediction accuracy for data skew; verify the performance of the model through methods such as cross-validation to ensure that the model performs well on new data, deploy the trained and verified model to the production environment, and replace or update the existing model. Through these detailed steps, build a more accurate and efficient system to detect and handle data skew.

[0103] Implement an automated system that can monitor, detect, and process data skew in real time, and continuously optimize itself based on performance indicator feedback, thereby improving data processing efficiency and system stability through continuous performance monitoring and model optimization.

[0104] It should be noted that in the embodiments of the present application, reference is made to Figure 5 , it can be seen that an exemplary overall process of the entire data skew adjustment is to determine whether there is a possibility of data skew based on the distribution of monitoring data. If so, the abnormal data is located based on the IF algorithm and the OCSVM algorithm. After the location, adaptive optimization can be performed. The adaptive optimization process is based on the adaptive optimization strategy. The details can be referred to the previous description. Furthermore, the monitoring performance indicators can be used as feedback signals input into the algorithm for further continuous self-optimization to achieve adaptive adjustment.

[0105] In an embodiment of the present application, when detecting that there is an abnormal amount of data processed by an operator, an abnormal score corresponding to each operator is generated through the first model, and a distance metric corresponding to each operator is generated through the second model, wherein the first model and the second model are constructed based on the key features, and the key features are determined by the real-time monitoring data corresponding to the operator and the analysis results of the target execution plan corresponding to the operator; based on the abnormal scores and the distance metrics corresponding to all operators, it is determined whether the operator distribution state is in a data tilt state; if so, the operator distribution state is adaptively adjusted based on the preset adjustment rules. That is, in an embodiment of the present application, by continuously monitoring the data distribution using an anomaly detection algorithm, the data tilt problem can be automatically discovered without manual intervention. Compared with the traditional static threshold monitoring method, this method is more intelligent and flexible, and can adapt to the dynamic changes of data distribution. This solution can not only detect the data tilt itself, but also locate the specific causes of the tilt, such as data distribution deviation, uneven computing load, etc., which helps to formulate more targeted optimization strategies and improve the optimization effect.

[0106] In addition, in the embodiment of the present application, through a closed-loop adaptive optimization mechanism, it is possible to continuously learn and improve the optimization strategy based on performance feedback. This automated optimization process greatly reduces the cost of manual operation and maintenance and improves the autonomy of the system.

[0107] Corresponding to the method provided in the above-mentioned data tilt adjustment method embodiment of the present application, see Figure 3 The present application also provides a device block diagram of a data tilt adjustment device. In this embodiment, the device includes:

[0108] A generation module 301 is used to generate an abnormal score corresponding to each operator through the first model and generate a distance metric corresponding to each operator through the second model when detecting that there is an abnormal amount of data processed by the operator, wherein the first model and the second model are constructed based on the key features, and the key features are determined by the real-time monitoring data corresponding to the operator and the analysis results of the target execution plan corresponding to the operator;

[0109] A determination module 302, configured to determine whether the operator distribution state is in a data skew state based on the abnormal scores and the distance metrics corresponding to all operators;

[0110] The adjustment module 303 is configured to, if yes, adaptively adjust the operator distribution state based on a preset adjustment rule.

[0111] Optionally, the determining module includes:

[0112] The first determination submodule is used to determine whether the operator is in a data skew state when it is detected that the abnormal score reaches a first threshold and the distance metric reaches a second threshold.

[0113] Optionally, the adjustment module includes:

[0114] The second adjustment submodule is used to adjust the parallelism of the operator based on the amount of data processed by the operator and the resource status of the system corresponding to the operator.

[0115] Optionally, the adjustment module includes:

[0116] The third adjustment submodule is used to adjust the resource allocation state of the target execution plan corresponding to the operator through the resource manager.

[0117] Optionally, the adjustment module includes:

[0118] The fourth adjustment submodule is used to determine whether the data volume corresponding to the target partition is abnormal;

[0119] The fifth adjusting submodule is used for re-partitioning the data if yes.

[0120] Optionally, the adjustment module includes:

[0121] The sixth adjustment submodule is used to merge several operators based on a preset operator merging rule.

[0122] Optionally, the device further comprises:

[0123] The first module is used to analyze the real-time monitoring data corresponding to the operator to determine the target time period corresponding to the data tilt state when the operator distribution state is in the data tilt state;

[0124] The second module is used to obtain the distribution state of the target operator after adaptive adjustment;

[0125] A third module is used to perform label processing on each of the adaptive adjustment results based on the target time period and the target operator distribution state to obtain a label corresponding to each of the adaptive adjustment results;

[0126] The fourth module is used to perform feedback optimization on the first model and the second model based on the label corresponding to each of the adaptive adjustment results.

[0127] In summary, a data tilt adjustment device provided in an embodiment of the present application, by detecting that there is an abnormal amount of data processed by an operator, generating an abnormal score corresponding to each operator through the first model, and generating a distance metric corresponding to each operator through the second model, wherein the first model and the second model are constructed based on the key features, and the key features are determined by the real-time monitoring data corresponding to the operator and the analysis results of the target execution plan corresponding to the operator; based on the abnormal scores and the distance metrics corresponding to all operators, determine whether the operator distribution state is in a data tilt state; if so, the operator distribution state is adaptively adjusted based on the preset adjustment rules. That is, in the embodiment of the present application, by continuously monitoring the data distribution using an anomaly detection algorithm, the data tilt problem can be automatically discovered without manual intervention. Compared with the traditional static threshold monitoring method, this method is more intelligent and flexible, and can adapt to the dynamic changes of data distribution. This solution can not only detect the data tilt itself, but also locate the specific causes of the tilt, such as data distribution deviation, uneven computing load, etc., which helps to formulate more targeted optimization strategies and improve the optimization effect.

[0128] Figure 4This is a structural diagram of an electronic device M00 provided in an embodiment of the present application. In the diagram, the electronic device M00 includes a processor M01 and a memory M02. The electronic device includes: a processor; a memory for storing instructions executable by the processor, wherein the processor is configured to execute the instructions to implement any of the data tilt adjustment methods described above, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0129] In an embodiment of the present application, the memory M02 can be used to store software programs and various data. The memory M02 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory M02 may include a volatile memory or a non-volatile memory, or the memory x09 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory M02 in the embodiment of the present application includes but is not limited to these and any other suitable types of memories.

[0130] The processor M01 may include one or more processing units; optionally, the processor M01 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor M01.

[0131] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned data tilt adjustment method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0132] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0133] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned data tilt adjustment method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0134] It should be understood that the chip involved in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0135] An embodiment of the present application further provides a storage medium. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute any of the data tilt adjustment methods described above.

[0136] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned data tilt adjustment method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0137] An embodiment of the present application also provides a vehicle, which includes the data tilt adjustment device as described above.

[0138] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0139] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0140] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A data tilt adjustment method, characterized in that: The method comprises: In the case of detecting that there is an abnormal amount of data processed by an operator, generating an abnormal score corresponding to each operator through a first model, and generating a distance metric corresponding to each operator through a second model, wherein the first model and the second model are constructed based on key features, and the key features are determined by parsing results of real-time monitoring data corresponding to the operator and a target execution plan corresponding to the operator; Determining whether an operator distribution state is in a data skew state based on the abnormal scores corresponding to all the operators and the distance metric; If so, the operator distribution state is adaptively adjusted based on a preset adjustment rule.

2. The method according to claim 1, characterized in that: The determining whether the current operator is in a data tilt state based on the anomaly score and the distance metric includes: When it is detected that the abnormal score reaches a first threshold and the distance metric reaches a second threshold, it is determined whether the operator is in a data skew state.

3. The method according to claim 1, characterized in that The adaptively adjusting the operator distribution state based on a preset adjustment rule includes: The parallelism of the operator is adjusted based on the amount of data processed by the operator and the resource status of the system corresponding to the operator.

4. The method according to claim 1, characterized in that: The adaptively adjusting the operator distribution state based on a preset adjustment rule includes: The resource allocation state of the target execution plan corresponding to the operator is adjusted through the resource manager.

5. The method according to claim 1, characterized in that The adaptively adjusting the operator distribution state based on a preset adjustment rule includes: Determine whether the data volume corresponding to the target partition is abnormal; If yes, re-partition the data.

6. The method according to claim 1, characterized in that The adaptively adjusting the operator distribution state based on a preset adjustment rule includes: Merge several operators based on preset operator merging rules.

7. The method according to claim 1, characterized in that After the step of adaptively adjusting the operator distribution state based on a preset adjustment rule, the method includes: Analyze the real-time monitoring data corresponding to the operator to determine the target time period corresponding to the data tilt state when the operator distribution state is in the data tilt state; Obtain the distribution state of the adaptively adjusted target operator; Performing label processing on each adaptive adjustment result based on the target time period and the target operator distribution state to obtain a label corresponding to each adaptive adjustment result; Feedback optimization is performed on the first model and the second model based on the label corresponding to each of the adaptive adjustment results.

8. A data tilt adjustment device, characterized in that: The device comprises: A generation module, configured to generate an abnormality score corresponding to each operator through a first model and generate a distance metric corresponding to each operator through a second model when detecting that an abnormal amount of data processed by an operator exists, wherein the first model and the second model are constructed based on key features, and the key features are determined by parsing results of real-time monitoring data corresponding to the operator and a target execution plan corresponding to the operator; A determination module, configured to determine whether an operator distribution state is in a data skew state based on the abnormal scores and the distance metrics corresponding to all the operators; The adjustment module is used to, if yes, adaptively adjust the operator distribution state based on a preset adjustment rule.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instruction to implement the data tilt adjustment method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the data tilt adjustment method as claimed in any one of claims 1 to 7.

Images