Data skew evaluation method, device and system, medium and program product

Abstract

The invention discloses a data skew evaluation method, device and system, a medium and a program product. The method relates to the field of artificial intelligence and big data and comprises the steps that performance data of a target database cluster is collected, a target data set is obtained, and the target database cluster comprises a cluster constructed by a distributed database; the target data set is input into a target evaluation model, an evaluation result is output, the target evaluation model comprises a long-short-term memory network model used for evaluating the data skew risk, and the evaluation result is used for indicating the time period when the target database cluster has the data skew risk. Through the data processing method and device, the problem that the efficiency of processing the data skew condition is low due to the fact that the data skew condition is manually identified after a business system reports an error in the related technology is solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence and big data, and specifically, to a data tilt assessment method, device, system, medium and program product. Background Art

[0002] At present, in distributed database systems, especially in the MPP (Massively Parallel Processing) architecture, query performance and system throughput are generally improved by processing data in parallel. However, for business tables with extremely large capacity, the data skew problem is a key challenge. Data skew refers to the uneven distribution of data among multiple nodes. Uneven data distribution can lead to performance bottlenecks. For example, some nodes may be under too high a load, resulting in slow processing speed of data-skewed nodes, while other nodes may be under low load or idle, resulting in resource waste. For example, uneven load distribution may result in a large number of computing resources not being fully utilized, and may also limit system expansion. For example, in a data-skewed environment, since newly added nodes may not be able to balance the already skewed load, simply increasing the number of nodes cannot effectively solve the performance problem.

[0003] In related technologies, data skew risk assessment is mostly performed manually after a business error occurs. There is a lack of effective mechanisms to dynamically predict and mark potential data skew events, resulting in the inability to take preventive or corrective measures in a timely manner. Moreover, after the system reports an error and identifies data skew, analysis is performed on a case-by-case basis based on the situation of each table, and expert experience is used to highly access and handle data skew situations. There is a lack of fast and automated response solutions, resulting in low efficiency in handling data skew situations.

[0004] Currently, no effective solution has been proposed to the problem that after an error occurs in a business system in related technologies, data skew is manually identified, resulting in low efficiency in processing data skew. Summary of the invention

[0005] The main purpose of this application is to provide a data skew assessment method, device, system, medium and program product to solve the problem of low efficiency in processing data skew due to manual identification of data skew after an error occurs in the business system in related technologies.

[0006] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a data skew assessment method is provided. The method comprises: collecting performance data of a target database cluster to obtain a target data set, wherein the target database cluster comprises: a cluster constructed by a distributed database; inputting the target data set into a target assessment model, and outputting an assessment result, wherein the target assessment model comprises: a long short-term memory network model for assessing data skew risk, and the assessment result is used to indicate a time period in which the target database cluster has a data skew risk.

[0007] Furthermore, after inputting the target data set into the target evaluation model and outputting the evaluation result, it also includes: when the evaluation result indicates that the target database cluster has a risk of data skew in the target time period, generating a tuning strategy, wherein the tuning strategy includes: a strategy for distributing and adjusting the load of the target database cluster; before the target time period, adjusting the distribution key of the target data set based on the tuning strategy, wherein the distribution key is used to adjust the data distribution in the target database cluster.

[0008] Furthermore, the target data set is input into a target evaluation model and an evaluation result is output, including: determining a data format of the data in the target data set, and based on the data format, converting the data in the target data set into a time series format to obtain a converted target data set; inputting the converted target data set into the target evaluation model and outputting the evaluation result.

[0009] Furthermore, the target evaluation model is obtained in the following manner: obtaining an initial training sample, wherein the initial training sample includes: performance data generated by the target database cluster in a historical time period; preprocessing the initial training sample to obtain a target training sample; performing model training on the initial evaluation model based on the target training sample, and when the initial evaluation model training converges, determining the converged initial evaluation model as the target evaluation model, wherein, during the model training process of the initial evaluation model, a cross-validation strategy is used to verify the initial evaluation model, and the initial evaluation model includes: a long short-term memory network model to be trained.

[0010] Furthermore, the preprocessing method includes at least one of the following: data cleaning, feature engineering, time series construction and data labeling, wherein the time series construction is used to convert the data into a time series format, and the data labeling is used to label data with data skew.

[0011] Furthermore, performance data of the target database cluster is collected to obtain a target data set, including: collecting resource data of the target database cluster to obtain a first data set; collecting time data of the target database cluster to obtain a second data set, wherein the time data includes at least one of the following: average response time, average waiting time; collecting task processing data of the target database cluster to obtain a third data set, wherein the task processing data includes: data on concurrent execution of tasks; and determining the target data set based on the first data set, the second data set and the third data set.

[0012] Furthermore, after inputting the target data set into the target evaluation model and outputting the evaluation result, it also includes: detecting whether there are data fragments in the target database cluster to obtain a detection result, wherein the data fragments include: data generated by multiple update operations or multiple deletion operations on the data in the target database cluster; when the detection result indicates that there are data fragments in the target database cluster, the data fragments are cleaned up.

[0013] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a data skew assessment system is provided, which is used to execute the data skew assessment method, including: a target database cluster, wherein the target database cluster includes: a cluster constructed by a distributed database; a big data service platform, which is used to collect performance data of the target database cluster to obtain a target data set; a data skew prediction management module, which is used to input the target data set into a target assessment model and output an assessment result, wherein the target assessment model includes: a long short-term memory network model for assessing data skew risks, and the assessment result is used to indicate a time period when the target database cluster has data skew risks.

[0014] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a data skew assessment device is provided. The device comprises: a collection unit, used to collect performance data of a target database cluster to obtain a target data set, wherein the target database cluster comprises: a cluster constructed by a distributed database; a processing unit, used to input the target data set into a target assessment model and output an assessment result, wherein the target assessment model comprises: a long short-term memory network model for assessing data skew risk, and the assessment result is used to indicate a time period in which the target database cluster has a data skew risk.

[0015] Furthermore, the data skew assessment device also includes: a generation unit, which is used to generate a tuning strategy after inputting the target data set into a target assessment model and outputting an assessment result, when the assessment result indicates that the target database cluster has a data skew risk in a target time period, wherein the tuning strategy includes: a strategy for distributing and adjusting the load of the target database cluster; an adjustment unit, which is used to adjust the distribution key of the target data set based on the tuning strategy before the target time period, wherein the distribution key is used to adjust the data distribution in the target database cluster.

[0016] Furthermore, the processing unit includes: a first processing sub-unit, used to determine the data format of the data in the target data set, and based on the data format, convert the data in the target data set into a time series format to obtain a converted target data set; a second processing sub-unit, used to input the converted target data set into the target evaluation model and output the evaluation result.

[0017] Furthermore, the target evaluation model is obtained through the following units: an acquisition unit, used to acquire an initial training sample, wherein the initial training sample includes: performance data generated by the target database cluster in a historical time period; a preprocessing unit, used to preprocess the initial training sample to obtain a target training sample; a training unit, used to perform model training on the initial evaluation model based on the target training sample, and when the initial evaluation model training converges, determine the converged initial evaluation model as the target evaluation model, wherein, in the process of model training the initial evaluation model, a cross-validation strategy is used to verify the initial evaluation model, and the initial evaluation model includes: a long short-term memory network model to be trained.

[0018] Furthermore, the preprocessing method includes at least one of the following: data cleaning, feature engineering, time series construction and data labeling, wherein the time series construction is used to convert the data into a time series format, and the data labeling is used to label data with data skew.

[0019] Furthermore, the collection unit includes: a first collection subunit, used to collect resource data of the target database cluster to obtain a first data set; a second collection subunit, used to collect time data of the target database cluster to obtain a second data set, wherein the time data includes at least one of the following: average response time, average waiting time; a third collection subunit, used to collect task processing data of the target database cluster to obtain a third data set, wherein the task processing data includes: data on concurrent execution of tasks; a determination subunit, used to determine the target data set based on the first data set, the second data set and the third data set.

[0020] Furthermore, the data skew assessment device also includes: a detection unit, which is used to detect whether there are data fragments in the target database cluster after inputting the target data set into the target assessment model and outputting the assessment result, and obtain a detection result, wherein the data fragments include: data generated by multiple update operations or multiple deletion operations on the data in the target database cluster; a cleaning unit, which is used to clean up the data fragments when the detection result indicates that there are data fragments in the target database cluster.

[0021] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored executable program, wherein when the executable program is run, the device where the computer-readable storage medium is located is controlled to execute the data skew assessment method.

[0022] According to another aspect of the present application, an electronic device is provided, comprising: a memory storing an executable program; and a processor for running the program, wherein the data skew assessment method is executed when the program is running.

[0023] According to another aspect of the present application, a computer program product is provided, comprising computer instructions, which implement the steps of the data skew assessment method when executed by a processor.

[0024] In an embodiment of the present application, performance data of a target database cluster is collected to obtain a target data set, wherein the target database cluster includes: a cluster constructed by a distributed database; the target data set is input into a target evaluation model, and an evaluation result is output, wherein the target evaluation model includes: a long short-term memory network model for evaluating data skew risk, and the evaluation result is used to indicate a time period when the target database cluster has a data skew risk, thereby solving the technical problem of manually identifying data skew after an error occurs in a business system in related technologies, resulting in low efficiency in processing data skew.

[0025] In the present invention, the collected performance data of the distributed database cluster is input into a pre-trained long short-term memory network model (target evaluation model) for data skew risk assessment, thereby avoiding the situation in related technologies where data skew is manually identified after an error occurs in the business system, resulting in low efficiency in assessing data skew, thereby achieving the technical effect of improving the evaluation efficiency of data skew assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0027] Figure 1 A hardware structure block diagram of a computer terminal for implementing a data tilt assessment method is shown;

[0028] Figure 2 is a flow chart of a data tilt assessment method provided according to an embodiment of the present application;

[0029] Figure 3 is a schematic diagram of a data skew management tool provided according to an embodiment of the present application;

[0030] Figure 4 is a schematic diagram of a data tilt assessment system provided according to an embodiment of the present application;

[0031] Figure 5 is a flow chart of data tilt assessment provided according to an embodiment of the present application;

[0032] Figure 6 is a schematic diagram of a data tilt assessment device provided according to an embodiment of the present application;

[0033] Figure 7 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0036] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0037] Long Short-Term Memory (LSTM): A special type of recurrent neural network (RNN) suitable for processing and predicting long-term dependencies in time series data. LSTM overcomes the difficulties of traditional RNN in processing long sequence data through special structural design (such as forget gate, input gate, output gate).

[0038] MPP distributed database cluster: Massively Parallel Processing (MPP) database, a distributed database system for storing and processing large amounts of data. Performance and scalability are improved by executing complex query operations in parallel on multiple server nodes. A distributed database refers to a database system in which data is physically distributed but logically centrally managed. PB (Petabyte)-level distributed databases based on the Shared-nothing architecture are increasingly being used in the field of financial technology, among which analytical distributed databases are supporting key business scenarios such as massive data warehouses, data marts, real-time analysis, real-time decision-making, and mixed loads in the financial industry.

[0039] Data skew: refers to the phenomenon that data is unevenly distributed on different computing or storage nodes in a distributed database system. This phenomenon usually causes some nodes to bear more data processing or storage load than other nodes, thus becoming a bottleneck for system performance. Data skew can be caused by a variety of factors, such as uneven data distribution strategies, uneven query loads, or natural concentration of data.

[0040] Database lock mechanism: It is used to manage the access of multiple concurrent operations to the same data to ensure the consistency and integrity of the data. There are multiple levels of lock mechanisms, including row-level lock, page-level lock, table-level lock, and higher-level database lock or global lock.

[0041] Intelligent Operations (AIOps): Intelligent Operations, or AIOps (Artificial Intelligence for IT Operations), refers to the use of artificial intelligence, machine learning, and big data technologies to optimize and automate IT operations. AIOps aims to improve operation efficiency, reduce manual intervention, and solve complex operation challenges in an intelligent way. AIOps is applied to a variety of IT operations scenarios, including performance monitoring, anomaly detection, fault diagnosis, and predictive maintenance, helping organizations quickly respond to and solve IT problems.

[0042] It should be noted that the collected information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data are in compliance with relevant laws, regulations and standards, necessary confidentiality measures are taken, and public order and good customs are not violated, and corresponding operation entrances are provided for users to choose to authorize or refuse. For example, an interface is set up between this system and relevant users or institutions to provide users with corresponding operation entrances for users to choose to agree or refuse the results of automated decision-making; if the user chooses to refuse, the expert decision-making process will be entered.

[0043] Embodiment 1

[0044] According to an embodiment of the present application, a method embodiment of a data skew assessment method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0045] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a data tilt assessment method. Figure 1As shown, the computer terminal 10 (or mobile device) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0046] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0047] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the data tilt assessment method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the above-mentioned data tilt assessment method is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0048] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0049] The display may be, for example, a touch screen liquid crystal display (LCD), which may enable a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0050] Under the above operating environment, this application provides Figure 2 The data skew assessment method shown. Figure 2 It is a flowchart of a data tilt assessment method according to the first embodiment of the present application.

[0051] Step S201 : collecting performance data of a target database cluster to obtain a target data set, wherein the target database cluster includes: a cluster constructed by a distributed database.

[0052] The above-mentioned target database cluster may include: a cluster constructed by a distributed database, for example, an MPP distributed database cluster. The above-mentioned performance data may include but is not limited to: (1) System resource usage data: including CPU (Central Processing Unit) utilization, memory usage, disk I / O (Input / Output) operations, network traffic, etc. (2) Historical SQL (Structured Query Language) statement resource consumption details: resource consumption of each SQL statement, such as execution time, CPU and memory resources consumed. (3) Average response time and waiting time: the speed and efficiency of the database responding to user requests. Waiting time may include lock waiting, I / O waiting, etc. (4) Task concurrency of the scheduling system: concurrent execution of system tasks at different time points.

[0053] Step S202: input the target data set into the target evaluation model and output the evaluation result, wherein the target evaluation model includes: a long short-term memory network model for evaluating data skew risk, and the evaluation result is used to indicate the time period when the target database cluster has data skew risk.

[0054] The above-mentioned target evaluation model may include a long short-term memory network model for evaluating whether the target database cluster has a risk of data skew. The above-mentioned target evaluation model may be a model obtained by training an LSTM model using historical monitoring data (performance data generated by the target database cluster in a historical time period), wherein the training goal of the model training is to identify patterns and trends that may cause data skew.

[0055] In this embodiment, the target data set may be input into a target evaluation model, and the target evaluation model may be used to evaluate the time period during which data skew will occur in the target database cluster to obtain an evaluation result.

[0056] In this embodiment, through the above steps, the performance data of the collected distributed database cluster is input into the pre-trained long short-term memory network model (target evaluation model) for data tilt risk assessment, which avoids the situation in the related art where the business system reports an error and the data tilt situation is manually identified, resulting in low efficiency in evaluating the data tilt situation, thereby achieving the technical effect of improving the evaluation efficiency of data tilt evaluation. This solves the technical problem in the related art where the business system reports an error and the data tilt situation is manually identified, resulting in low efficiency in processing the data tilt situation.

[0057] Optionally, in the data skew assessment method provided in the embodiment of the present application, after inputting the target data set into the target assessment model and outputting the assessment result, it also includes: when the assessment result indicates that the target database cluster has a data skew risk in the target time period, generating a tuning strategy, wherein the tuning strategy includes: a strategy for distributing and adjusting the load of the target database cluster; before the target time period, adjusting the distribution key of the target data set based on the tuning strategy, wherein the distribution key is used to adjust the data distribution in the target database cluster.

[0058] After identifying the risk of data skew, a load distribution adjustment strategy (corresponding to a tuning strategy) can be automatically generated or recommended. This includes rebalancing the data distribution on each data node (DN). For example, a tuning strategy may include an adjustment strategy for the data distribution key, or optimizing SQL statements to reduce reliance on specific nodes. When generating a tuning strategy, multiple factors may be considered, including the overall load of the database cluster, the performance characteristics of the node, historical SQL query patterns, and data growth forecasts. For example, if the potential data skew is caused by improper distribution key settings, the distribution key of the target data set may be adjusted based on the tuning strategy before the target time period to adjust the distribution of data within the database cluster.

[0059] In this embodiment, the adjustment of the distribution key involves analyzing the data access mode, data volume distribution and business requirements, and selecting one or more fields as new distribution keys to ensure that the data is more evenly distributed on each node. After obtaining the tuning strategy, the distribution key can be adjusted, data can be redistributed, SQL statements can be optimized, and the capacity of nodes can be expanded or reduced when necessary.

[0060] In an optional example, the distribution key may be selected by executing a preset script. For example, the preset script may be executed to determine whether data distribution is uniform after a distribution key is selected, thereby determining whether to select the distribution key.

[0061] In this embodiment, intelligent data skew prediction based on the LSTM model is used to automatically generate and execute tuning strategies to ensure that the MPP distributed database cluster can provide stable and efficient performance within the target time period, thereby ensuring the continuity of the business system and the smooth completion of data processing tasks.

[0062] For example, after the target assessment model assesses a specific risk of skew, such as a node's load being consistently higher than other nodes. Tuning solutions can be automatically proposed based on the prediction results. These solutions may include adjusting distribution keys, redistributing data, increasing node processing power, etc. Specifically, analyze whether the existing distribution keys are effective and whether there are better distribution key options. According to the data skew pattern and prediction results, adjust the distribution key in advance to prevent potential skew in the future. After confirming the new distribution key, perform data redistribution operations to ensure that the data is more evenly distributed among the nodes. Since this process may involve a large amount of data migration, the execution time and resource usage can be carefully planned to minimize the impact on normal business.

[0063] Optionally, in the data skew assessment method provided in an embodiment of the present application, the target data set is input into a target assessment model and an assessment result is output, including: determining the data format of the data in the target data set, and based on the data format, converting the data in the target data set into a time series format to obtain a converted target data set; inputting the converted target data set into the target assessment model and outputting the assessment result.

[0064] In this embodiment, before the target data set is input into the target evaluation model, the data format in the target data set can be determined, for example, the type of data (such as numbers, text, date and time, etc.), structure (such as JSON (lightweight data exchange format), XML (Extensible Markup Language), etc.) and the range and precision of the data can be identified. In order to ensure that the data can be correctly understood and processed by the model, because different data types and formats may require different preprocessing methods. Since the LSTM model is good at processing time series data, it can learn and predict time-based patterns. Therefore, the target data set can be converted into a time series format to obtain a converted target data set, which can specifically include: Timestamp addition: If the data set itself does not contain time information, a timestamp can be added according to the generation time or acquisition time of the data, and then the original data can be resampled according to the needs of model training to ensure that the data is presented at fixed time intervals (such as every minute, every hour or every day).

[0065] In this embodiment, the converted target data set can be input into the target assessment model for data skew risk assessment, and the assessment results can be output, thereby achieving the technical effect of improving the accuracy of the target assessment model in performing data skew risk assessment.

[0066] Optionally, in the data skew assessment method provided in an embodiment of the present application, a target evaluation model is obtained in the following manner: obtaining an initial training sample, wherein the initial training sample includes: performance data generated by a target database cluster in a historical time period; preprocessing the initial training sample to obtain a target training sample; performing model training on the initial evaluation model based on the target training sample, and when the initial evaluation model training converges, determining the converged initial evaluation model as the target evaluation model, wherein, during the model training process of the initial evaluation model, a cross-validation strategy is used to verify the initial evaluation model, and the initial evaluation model includes: a long short-term memory network model to be trained.

[0067] In this embodiment, the performance data generated by the target database cluster in the historical time period can be collected to obtain the initial training sample, and then the initial training sample can be preprocessed to obtain the preprocessed data (target training sample). The LSTM model can be trained based on the preprocessed data to learn to identify the timing of data tilt and tilt tuning. Since the LSTM model is suitable for processing long-term relationships in long-sequence data learning data, learning the patterns and characteristics of data tilt, it can be used to predict the time point when data tilt may occur in some table objects in a complex data platform system. In this embodiment, cross-validation and other technologies can be used to avoid overfitting during model training to ensure that the model has good generalization ability.

[0068] It should be noted that during the model training process, historical operation data, data tuning strategies and tuning results can be used to train the LSTM model, so that it can batch predict tables that may experience data skew in the future and time windows available for tuning based on database usage patterns and skew trends.

[0069] After model training, the prediction performance of the model can also be evaluated, and model parameters can be adjusted based on feedback. The specific process includes: evaluating the difference between the prediction results of the LSTM model and the actual event point where data tilt occurs and the actual optimal tilt tuning time point; tuning the model based on the evaluation results, such as the learning rate, the number and size of hidden layers, etc., and conducting continuous iterative training and evaluation to ensure the accuracy of the prediction and the generalization ability of the model.

[0070] In an optional example, the process of predicting data skew of a distributed database cluster using an LSTM model is as follows: (1) Model overview: LSTM is a special recurrent neural network (RNN) that is suitable for processing and predicting important events with variable intervals and lengths in time series data. In the context of data skew prediction, LSTM can effectively learn and memorize the historical load and resource usage patterns of an MPP distributed database cluster. (2) Data input: Model input includes time series data such as system resource usage, resource consumption history of SQL queries, task concurrency, and response time. (3) Training process: The LSTM model is trained using historical monitoring data, and the training goal is to identify patterns and trends that may lead to data skew. (4) Prediction execution: After the model training is completed, the current and recently collected preprocessed data is input into the model in real time to predict data skew that may occur in the future, thereby achieving the technical effect of automatically predicting data skew risks.

[0071] Optionally, in the data skew assessment method provided in an embodiment of the present application, the preprocessing method includes at least one of the following: data cleaning, feature engineering, time series construction, and data labeling, wherein the time series construction is used to convert the data into a time series format, and the data labeling is used to label the data with data skew.

[0072] In this embodiment, after a series of preprocessing steps are performed on the collected data, it is used for training and prediction of the LSTM model to improve the data quality of the training samples. The specific preprocessing methods may include but are not limited to: (1) Data cleaning: removing invalid or abnormal data, such as erroneous data entries, duplicate records, etc. (2) Feature engineering: extracting useful features from the original data, which may include transforming certain original data (such as normalized resource usage data) or calculating new features (for example, the average execution time growth rate of SQL queries). (3) Time series construction: converting the data into a format suitable for time series analysis, which is particularly important for using the LSTM model for data tilt risk assessment because LSTM is superior to processing sequence data, such as dividing data sets by time windows. (4) Data annotation: For supervised learning, data can be annotated, for example, indicating which data points belong to the normal range and which indicate tilt. This may require manually or automatically annotating the data based on historical tilt events, thereby achieving the technical effect of improving the data quality of the training samples.

[0073] Optionally, in the data skew assessment method provided in an embodiment of the present application, performance data of the target database cluster is collected to obtain a target data set, including: collecting resource data of the target database cluster to obtain a first data set; collecting time data of the target database cluster to obtain a second data set, wherein the time data includes at least one of the following: average response time, average waiting time; collecting task processing data of the target database cluster to obtain a third data set, wherein the task processing data includes: data on concurrent execution of tasks; and determining the target data set based on the first data set, the second data set, and the third data set.

[0074] In this embodiment, the fragmentation information of each data table (corresponding to the target data set) can be collected regularly from the distributed database cluster, including but not limited to: (1) System resource usage data: including CPU utilization, memory usage, disk I / O operations, network traffic, etc. The system resource usage data can be used to reflect the overall load and performance status of the database system. (2) Historical SQL statement resource consumption details: record the resource consumption of each SQL statement in detail, such as execution time, CPU and memory resources consumed, so as to identify complex situations that lead to resource tilt, wherein the above resource data may include: system resource usage data and historical SQL statement resource consumption details (3) Average response time and waiting time (corresponding to time data): can be used to measure the speed and efficiency of the distributed database in responding to user requests. The waiting time may include lock waiting, I / O waiting, etc. (4) Task concurrency of the scheduling system (corresponding to task processing data): used to obtain the concurrent execution of system tasks at different time points, which can reveal the load and pressure of the system during peak periods. In addition to monitoring data, you can also consider incremental changes in business data sources. For example, incremental changes in data: Track incremental updates to data, such as the amount of newly inserted and updated data, to improve the prediction of data growth trends and possible skew risks.

[0075] Optionally, in the data skew assessment method provided in the embodiment of the present application, after inputting the target data set into the target assessment model and outputting the assessment result, it also includes: detecting whether there is data fragmentation in the target database cluster to obtain a detection result, wherein the data fragmentation includes: data generated by multiple update operations or multiple deletion operations on the data in the target database cluster; when the detection result indicates that there is data fragmentation in the target database cluster, the data fragmentation is cleaned up.

[0076] The above-mentioned data fragments may refer to invalid data or discontinuous data storage fragments caused by frequent update (UPDATE) or delete (DELETE) operations of the distributed data cluster (target database cluster). Since frequent update (UPDATE) or delete (DELETE) operations may cause the originally continuous data layout to become scattered, increasing the complexity of data access and management, thereby affecting query performance, therefore, in this embodiment, the data fragments are cleaned up regularly.

[0077] In this embodiment, after the target data set is input into the target evaluation model and the evaluation results are output, it is possible to further detect whether data fragmentation exists in the target database cluster. Specifically, this can be achieved by analyzing database logs, query execution plans, or directly querying system tables. By checking whether there are a large number of update or deletion operations, and whether these operations result in discontinuity in data storage or excessive blank space, it is determined whether data fragmentation exists.

[0078] After the data fragmentation detection is completed, a detection result will be output to indicate whether the target database cluster has data fragmentation. If so, the detection result can specifically indicate which tables or partitions are affected and the severity of the data fragmentation. If the detection result indicates that the target database cluster has data fragmentation, a data fragmentation cleanup strategy is generated, which may include reorganizing the data, or compressing or deleting the data fragments, etc., to reduce data fragmentation, optimize the use of storage space, and improve the data query efficiency of the target database cluster.

[0079] For example, in this embodiment, data fragmentation can be checked and cleaned up regularly to optimize the use of storage space and query performance. Data fragmentation is usually a byproduct of frequent update and delete operations. Cleaning up fragments can reduce data access time and improve system response speed, and reduce the amount of data with data skew to a certain extent. In this embodiment, cluster data nodes that may have data skew can be automatically expanded based on the load of the target database cluster and data growth forecast. The expansion operation includes adding physical nodes or optimizing the configuration of existing nodes to provide more processing power and storage space.

[0080] In this embodiment, the results of the tilt governance activities can also be analyzed, and the prediction models and tuning strategies can be continuously optimized and adjusted based on these results. This ensures that it is possible to learn from practice and adapt to changes, thereby continuously improving the overall efficiency and effectiveness of the system. Specifically, it includes: automatic tilt tuning. For example, once the tuning plan is generated, it can be semi-automatically implemented by the execution according to the operation and maintenance management mechanism, while monitoring the tuning effect and recording the execution results in real time. The main means include: adjusting and optimizing the LSTM model and tuning strategy based on the actual tuning effect feedback. Generate a detailed analysis report and improvement suggestion document for reference by other modules and management. The analysis report can include key performance indicators, problem identification, improvement measures and their expected effects.

[0081] An optional data skew management tool can be used to execute the data skew assessment method provided in this embodiment. Figure 3 is a schematic diagram of a data skew management tool provided according to an embodiment of the present application, such as Figure 3 As shown, three major components are integrated in series through bus 31: configuration management unit 32, intelligent prediction unit 35 and operation and maintenance management unit 39. The specific functions are as follows:

[0082] Bus 31: As the communication and data transmission hub within the data tilt management tool, Bus 31 connects the key parts of the device to ensure coordination and data consistency between modules.

[0083] Configuration management unit 32: used to manage the content related to the deployed servers and variables. It consists of parameter parsing module 33 and environment checking module 34.

[0084] Parameter parsing module 33: used to parse and process configuration parameters, including working directory path, key file path, configuration of distributed database connection, etc.

[0085] Environment check module 34: Check the environment connectivity and key configuration of the MPP distributed database cluster (corresponding to the target database cluster), verify whether all parts of the data skew management tool are working properly, and ensure the stable operation of the data skew management tool.

[0086] Intelligent prediction unit 35: Based on the prediction component of the LSTM model, it analyzes the operation, monitoring, and alarm data of the big data platform and its upstream and downstream systems, analyzes and predicts the storage status of the data table in the MPP distributed database cluster, predictively identifies potential data tilt risks, and generates specific tilt governance solutions and optimal tilt governance time windows. Data processing module 36, model training module 37, and model parameter adjustment module 38.

[0087] Data processing module 36: In the data processing stage, it is responsible for collecting necessary operation, monitoring, and alarm data from the MPP distributed database, job scheduling platform, and data asset platform, and pre-processing the collected data, including cleaning, normalization, and feature extraction, and converting it into a format suitable for LSTM model training to ensure that the model can receive accurate and meaningful inputs and provide prepared input data for high-quality predictions. The required data includes but is not limited to: data lineage of business data tables, upstream data field-level data volume, batch job frequency settings for processing business tables, business data table call frequency (combing hotspot tables, backup tables, temporary tables, and prioritizing), business data table downstream call information, database cluster downtime maintenance time (higher tolerance for unavailability caused by data tilt management during upgrades and updates), business system downtime maintenance time (available for tilt management), etc.

[0088] Model training module 37: Based on the preprocessed data, the LSTM model is trained at this stage to learn to identify data skew and the timing of skew tuning. Since the LSTM model is suitable for processing long-sequence data to learn long-term relationships in the data, learn the patterns and characteristics of data skew, and can be used to predict the time points when data skew may occur in some table objects in complex data platform systems. Cross-validation and other techniques are used to avoid overfitting and ensure that the model has good generalization ability. Model training involves: using historical operation data, data tuning strategies and results to train the LSTM model, so that it can batch predict tables that may experience data skew in the future and the time windows available for tuning based on database usage patterns and skew trends.

[0089] Model parameter adjustment module 38: After model training, the parameter adjustment module is responsible for evaluating the prediction performance of the model and adjusting the model parameters based on feedback. The specific process includes: evaluating the difference between the prediction results of the LSTM model and the actual event point where data tilt occurs and the actual optimal tilt tuning time point; tuning the model according to the evaluation results, such as learning rate, number and size of hidden layers, etc., and conducting continuous iterative training and evaluation to ensure the accuracy of the prediction and the generalization ability of the model.

[0090] Through the close collaboration of the data processing module, model training module, and model parameter adjustment module, the intelligent prediction unit 35 can provide accurate time window predictions, providing strong support for data tilt predictions in the MPPDB distributed database. This not only optimizes the performance of the database, but also ensures the continuity and stability of the business, significantly improving the efficiency and intelligence level of database management.

[0091] The operation and maintenance management unit 39 is the core component of the intelligent data tilt prediction tool, which is responsible for converting the output of the fragmentation intelligent prediction unit component into actual maintenance operations. This module mainly includes two sub-modules: the tilt tuning module 310 and the maintenance prompt module 311, which aims to ensure the efficient execution of the defragmentation process while minimizing the impact on the business system.

[0092] The tilt tuning module 310 performs operations such as distribution key reselection, table reconstruction, and MPP distributed database cluster data node expansion according to the tilt management solution provided by the intelligent prediction unit.

[0093] Maintenance reminder module 311: sends maintenance reminders to other business systems during the tilt management process to ensure that the tilt management operation does not affect the normal business operation and minimize the interference with the business system.

[0094] Through efficient management and automated governance of data skew, the stability and high-performance operation of the MPPDB distributed database cluster are guaranteed, and the impact of maintenance activities on the business is minimized. This intelligent and automated data skew prediction strategy improves the maintenance efficiency of distributed databases and ensures the stable operation of business systems.

[0095] In this embodiment, the target evaluation model uses an LSTM model, which can accurately process and analyze time series data, so that the target evaluation model can predict the best time for defragmentation based on historical SQL statement performance records, database load, business activities and other data. There are complex dependencies between the state of the database and business activities. With its long-term memory ability, LSTM can learn these dependencies and improve the accuracy of prediction.

[0096] Since the database and business environment are constantly changing, and the LSTM model can continuously learn from new data and adapt to environmental changes, the target evaluation model provided in this embodiment can provide accurate data tilt management suggestions.

[0097] In summary, the target evaluation model uses the LSTM model, which has become an indispensable tool in AIOps through its powerful time series data processing capabilities and ability to learn long-term dependencies. In the MPPDB intelligent data tilt prediction, the use of LSTM to predict batch data tilt can not only effectively reduce the impact of table data tilt on database performance, but also ensure that the tilt governance process has minimal interference with the running business, thereby achieving continuous optimization of database performance and stable operation of the business.

[0098] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0099] Embodiment 2

[0100] Embodiment 2 of the present invention optionally provides a data skew assessment system, where the data skew assessment system is used to execute a data skew assessment method.

[0101] The data skew assessment system includes: a target database cluster, wherein the target database cluster includes: a cluster constructed by a distributed database; a big data service platform, which is used to collect performance data of the target database cluster and obtain a target data set; a data skew prediction management module, which is used to input the target data set into a target assessment model and output an assessment result, wherein the target assessment model includes: a long short-term memory network model for assessing data skew risks, and the assessment result is used to indicate the time period when the target database cluster has data skew risks.

[0102] Figure 4 is a schematic diagram of a data tilt assessment system provided according to an embodiment of the present application, such as Figure 4As shown, it includes: a big data service platform 412 (BigData Service Platform), an MPP distributed database cluster 413 (MPP-DBMS1, MPP-DBMS2...MPP-DBMSN, including the cluster's coordination CN node 414 (Coordinator Node) and the cluster's data DN node 415 (Data Node)), several big data scheduling systems 416 (ETL System 1, ETL System 2...), several big data analysis systems 417 (Analyze System 3), and an intelligent data skew prediction management module 418 (MPPDB AIOpsData Skew Manager). The collaboration and functions of each module are as follows:

[0103] Big data service platform 412: As the core platform for data processing and analysis, it provides data storage, computing and analysis services. It interacts directly with the MPP distributed database cluster 413, receives data from data collection and preprocessing, and provides it for use by the intelligent data tilt prediction management module 418.

[0104] MPP distributed database cluster 413 (corresponding to the target database cluster): This module represents the cluster environment of the MPP distributed database, including multiple database instances and nodes. CN node in the distributed database cluster: This node is the coordination node in the MPP distributed database cluster, responsible for query parsing and optimization, task scheduling and other functions. DN node in the MPP distributed database cluster: This node is the data node in the MPP distributed database cluster, responsible for storing and processing actual data.

[0105] Big data scheduling system 416: It is composed of multiple task scheduling nodes deployed in a distributed manner, and each node can be statelessly and infinitely expanded horizontally. Task scheduling is based on preset trigger conditions (such as time points, time intervals, judgment mechanisms, etc.) to execute a given job. Among them, the control node (Scheduler) is responsible for job scheduling, involving job status management, sending and receiving job scheduling requests, and specific job allocation. The execution node (Worker) is responsible for specific operations on business data stored in the MPP distributed database cluster 413 and implementing the execution of data processing jobs. The processed result set is used by users of the downstream big data analysis system 417.

[0106] Big data analysis system 417: is an integrated software tool and process designed to efficiently process and analyze large-scale data sets so that data analyst users can extract useful information, insights and knowledge to support policy making and strategic planning.

[0107] Intelligent data tilt prediction management module 418 (corresponding to data tilt prediction management module): responsible for implementing the strategy formulated by the resource allocation decision module and monitoring its execution, ensuring that the system operates according to the predetermined strategy and monitoring the strategy execution effect in real time.

[0108] The multiple modules in the data skew assessment system together form a complete system that can effectively predict data skew events in MPP distributed database clusters, and dynamically allocate resources and optimize skew based on these predictions to ensure the stable operation and efficient performance of the big data platform.

[0109] In an optional example, the MPP distributed database cluster of the big data analysis platform can be integrated into the analysis business system of the financial data processing department to process a large amount of transaction data and customer information. During the peak trading period, especially the trading sprint at the end of the quarter, a key business table is imported with a large amount of data in an instant, causing the originally set distribution key to no longer apply, resulting in data skew. This skew causes the response time of a certain node to be much longer than other nodes, affecting the performance of the entire system. Performance tuning can be performed through the following steps:

[0110] Data preparation and prediction phase: Specifically, data collection: The system collects key indicators including the CPU and memory usage of nodes, I / O performance data of each node, and the data volume changes of key business tables. In addition, the pre-configured jobs in the big data platform scheduling system that may cause a significant increase in data volume are monitored. Key data analysis: Analyze the growth trend of business table data at the end of the quarter, especially those tables with large amounts of instantaneous data imported, and how these increases affect the current distribution keys and node loads.

[0111] Data skew prediction model training and application: Specifically, LSTM model application: The LSTM model is trained to identify data skew patterns derived from key business tables and node monitoring data. The model uses historical skew events, data growth patterns of business tables, and performance indicators of nodes to predict possible data skew in the future. Prediction implementation: The model can predict which business tables may have severe data skew at a specific time point, such as the end of a quarter, due to the instantaneous import of large amounts of data. The prediction results can be seen in the visualization interface.

[0112] Real-time monitoring and early warning: Specifically, early warning mechanism: Before the end of the quarter, the LSTM model successfully predicted that several business tables might be severely tilted due to the instantaneous import of large amounts of data. Tuning strategy: Based on the model's predictions and the historical SQL performance data of the MPP distributed database, as well as the call cycle of business hotspot tables, the system automatically proposed specific solutions. This includes temporarily adjusting the distribution key, pre-scheduling data redistribution, and increasing computing resources during predicted high load periods.

[0113] Solution and effect: Specifically, adjustment execution: During the high-risk period predicted by the model, the system automatically executed the tilt tuning strategy, including the redistribution of distribution keys and the dynamic redistribution of data. Effect evaluation: After the adjustment was implemented, monitoring data showed that the response time of related nodes was significantly improved and the overall performance of the system was improved. Financial business often has very obvious peak and valley characteristics. For example, during the peak trading period at the end of the quarter, the database processing capacity remains stable and the response time remains at a normal level. Through this LSTM model-driven intelligent tilt management system, the financial big data analysis platform successfully avoids the problem of data tilt and ensures the continuity and high-performance operation of key business applications.

[0114] Figure 5 is a flow chart of data tilt assessment provided according to an embodiment of the present application, such as Figure 5 As shown, including:

[0115] 1. Data collection and preprocessing. Data collection includes but is not limited to the following: (1) System resource usage data: including CPU utilization, memory usage, disk I / O operations, network traffic, etc. These data reflect the overall load and performance of the database system. (2) Historical SQL statement resource consumption details: Detailed records of resource consumption of each SQL statement, such as execution time, CPU and memory resources consumed, which helps to identify complex queries that cause resource skew. (3) Average response time and waiting time: measures the speed and efficiency of the database in responding to user requests. Waiting time includes lock waiting, I / O waiting, etc. (4) Task concurrency of the scheduling system. Preprocessing includes but is not limited to the following: (1) Data cleaning: removing invalid or abnormal data, such as incorrect data entries, duplicate records, etc. (2) Feature engineering: extracting useful features from raw data, which may include transforming some raw data (such as normalizing resource usage data) or calculating new features (for example, the average execution time growth rate of SQL queries). (3) Time series construction: converting data into a format suitable for time series analysis.

[0116] 2. The tilt prediction module uses the preprocessed data target evaluation model to predict the data tilt risk.

[0117] 3. Operation and maintenance management module, used to generate tilt optimization strategies based on tilt prediction results.

[0118] 4. The execution and monitoring module executes the tilt tuning strategy and monitors the performance data in real time, such as distribution key reselection, table reconstruction, and MPP distributed database cluster data node expansion, and transmits the performance data to the feedback and optimization module.

[0119] 5. Feedback and optimization module, which is mainly responsible for analyzing the results of tilt governance activities, and continuously optimizing and adjusting the prediction model and tuning strategy based on these results.

[0120] In this embodiment, the application of the long short-term memory network (LSTM) model in the field of financial technology big data is adopted, especially the innovation in predicting the data skew of the MPP distributed database cluster to ensure the operation stability.

[0121] The following technical effects can be achieved:

[0122] 1. Optimize performance and resource efficiency: Through real-time data skew prediction and automatic skew adjustment of important business tables in the MPP distributed database cluster, performance bottlenecks can be effectively prevented, ensuring rapid response and high efficiency of distributed database operations. At the same time, intelligent resource allocation strategies optimize the use of hardware resources and reduce costs caused by resource waste.

[0123] 2. Improve system reliability and scalability: Automatic tilt tuning solution generation and execution enable the distributed database to dynamically adjust resources and storage capacity according to actual needs, significantly enhancing the scalability and flexibility of the system and avoiding the passive situation of waiting until hidden dangers cause business interruptions before solving them. In addition, continuous monitoring and real-time adjustment ensure the stability and reliability of the system in the face of emergencies.

[0124] 4. Enhanced automation and operational efficiency: Highly automated tilt management reduces the need for manual intervention, improves the accuracy of database management and the response efficiency of the operation and maintenance team. The use of automated tools also greatly improves the accuracy and timeliness of data analysis and reporting.

[0125] Embodiment 3

[0126] The present application also provides a data tilt assessment device. It should be noted that the data tilt assessment device of the present application can be used to execute the data tilt assessment method provided in the present application. The data tilt assessment device provided in the present application is introduced below.

[0127] According to an embodiment of the present application, a device for implementing the above-mentioned data tilt assessment method is also provided, such as Figure 6 As shown, the device includes: a collection unit 61 and a processing unit 62.

[0128] The collection unit 61 is used to collect performance data of a target database cluster to obtain a target data set, wherein the target database cluster includes: a cluster constructed by a distributed database;

[0129] The processing unit 62 is used to input the target data set into the target evaluation model and output the evaluation result, wherein the target evaluation model includes: a long short-term memory network model for evaluating data skew risk, and the evaluation result is used to indicate the time period when the target database cluster has data skew risk.

[0130] In the data tilt evaluation device provided in the embodiment of the present application, the performance data of the target database cluster can be collected by the collection unit 61 to obtain the target data set, wherein the target database cluster includes: a cluster constructed by a distributed database; the processing unit 62 is used to input the target data set into the target evaluation model and output the evaluation result, wherein the target evaluation model includes: a long short-term memory network model for evaluating the risk of data tilt, and the evaluation result is used to indicate the time period when the target database cluster has the risk of data tilt. Thus, the technical problem of the low efficiency of processing the data tilt situation caused by manual identification of the data tilt situation after the business system has an error in the related technology is solved. In this embodiment, the performance data of the distributed database cluster collected is input into the pre-trained long short-term memory network model (target evaluation model) for data tilt risk evaluation, which avoids the situation in the related technology that the business system has an error, and the data tilt situation is manually identified, resulting in the low efficiency of evaluating the data tilt situation, thereby achieving the technical effect of improving the evaluation efficiency of data tilt evaluation.

[0131] Optionally, in the data skew assessment device provided in the embodiment of the present application, the data skew assessment device also includes: a generation unit, which is used to, after inputting the target data set into the target assessment model and outputting the assessment result, generate a tuning strategy when the assessment result indicates that the target database cluster has a data skew risk in the target time period, wherein the tuning strategy includes: a strategy for distributing and adjusting the load of the target database cluster; an adjustment unit, which is used to adjust the distribution key of the target data set based on the tuning strategy before the target time period, wherein the distribution key is used to adjust the data distribution in the target database cluster.

[0132] Optionally, in the data tilt assessment device provided in an embodiment of the present application, the processing unit includes: a first processing sub-unit, used to determine the data format of the data in the target data set, and based on the data format, convert the data in the target data set into a time series format to obtain a converted target data set; a second processing sub-unit, used to input the converted target data set into a target assessment model and output an assessment result.

[0133] Optionally, in the data tilt evaluation device provided in the embodiment of the present application, the target evaluation model is obtained through the following units: an acquisition unit, used to acquire an initial training sample, wherein the initial training sample includes: performance data generated by the target database cluster in a historical time period; a preprocessing unit, used to preprocess the initial training sample to obtain a target training sample; a training unit, used to perform model training on the initial evaluation model based on the target training sample, and when the initial evaluation model training converges, determine the converged initial evaluation model as the target evaluation model, wherein, in the process of model training the initial evaluation model, a cross-validation strategy is used to verify the initial evaluation model, and the initial evaluation model includes: a long short-term memory network model to be trained.

[0134] Optionally, in the data skew assessment device provided in an embodiment of the present application, the preprocessing method includes at least one of the following: data cleaning, feature engineering, time series construction, and data labeling, wherein the time series construction is used to convert the data into a time series format, and the data labeling is used to label the data with data skew.

[0135] Optionally, in the data tilt assessment device provided in the embodiment of the present application, the collection unit includes: a first collection subunit, used to collect resource data of the target database cluster to obtain a first data set; a second collection subunit, used to collect time data of the target database cluster to obtain a second data set, wherein the time data includes at least one of the following: average response time, average waiting time; a third collection subunit, used to collect task processing data of the target database cluster to obtain a third data set, wherein the task processing data includes: data on concurrent execution of tasks; a determination subunit, used to determine the target data set based on the first data set, the second data set and the third data set.

[0136] Optionally, in the data skew assessment device provided in the embodiment of the present application, the data skew assessment device also includes: a detection unit, which is used to detect whether there are data fragments in the target database cluster after inputting the target data set into the target assessment model and outputting the assessment result, and obtain a detection result, wherein the data fragments include: data generated by multiple update operations or multiple deletion operations on the data in the target database cluster; a cleaning unit, which is used to clean up the data fragments when the detection result indicates that there are data fragments in the target database cluster.

[0137] It should be noted that the above-mentioned acquisition unit 61 and processing unit 62 correspond to steps S201 to S202 in the first embodiment, and the two modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the first embodiment. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n), and the above-mentioned modules may also be part of the device and may be run in the computer terminal 10 provided in the first embodiment.

[0138] Embodiment 4

[0139] An embodiment of the present application may provide an electronic device, Figure 7 is a structural block diagram of an electronic device according to an embodiment of the present application. Figure 7 As shown, the electronic device may include: one or more ( Figure 7 (only one is shown) processor 1002, memory 1004, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0140] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0141] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: collect performance data of the target database cluster to obtain a target data set, wherein the target database cluster includes: a cluster constructed by a distributed database; input the target data set into a target evaluation model, and output an evaluation result, wherein the target evaluation model includes: a long short-term memory network model for evaluating data skew risks, and the evaluation result is used to indicate a time period when the target database cluster has data skew risks.

[0142] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: after inputting the target data set into the target evaluation model and outputting the evaluation results, it also includes: when the evaluation results indicate that the target database cluster has a risk of data skew in the target time period, generating a tuning strategy, wherein the tuning strategy includes: a strategy for distributing and adjusting the load of the target database cluster; before the target time period, adjusting the distribution key of the target data set based on the tuning strategy, wherein the distribution key is used to adjust the data distribution in the target database cluster.

[0143] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: input the target data set into the target evaluation model and output the evaluation results, including: determining the data format of the data in the target data set, and based on the data format, converting the data in the target data set into a time series format to obtain a converted target data set; inputting the converted target data set into the target evaluation model and outputting the evaluation results.

[0144] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: the target evaluation model is obtained in the following manner: an initial training sample is obtained, wherein the initial training sample includes: performance data generated by the target database cluster in a historical time period; the initial training sample is preprocessed to obtain a target training sample; the initial evaluation model is trained based on the target training sample, and when the initial evaluation model training converges, the converged initial evaluation model is determined as the target evaluation model, wherein, during the model training process of the initial evaluation model, a cross-validation strategy is used to verify the initial evaluation model, and the initial evaluation model includes: a long short-term memory network model to be trained.

[0145] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: The preprocessing method includes at least one of the following: data cleaning, feature engineering, time series construction and data labeling, wherein the time series construction is used to convert the data into a time series format, and the data labeling is used to label the data with data skew.

[0146] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: collect performance data of the target database cluster to obtain a target data set, including: collect resource data of the target database cluster to obtain a first data set; collect time data of the target database cluster to obtain a second data set, wherein the time data includes at least one of the following: average response time, average waiting time; collect task processing data of the target database cluster to obtain a third data set, wherein the task processing data includes: data on concurrent execution of tasks; determine the target data set based on the first data set, the second data set and the third data set.

[0147] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: after inputting the target data set into the target evaluation model and outputting the evaluation results, it also includes: detecting whether there are data fragments in the target database cluster to obtain the detection results, wherein the data fragments include: data generated by multiple update operations or multiple deletion operations on the data in the target database cluster; when the detection result indicates that there are data fragments in the target database cluster, the data fragments are cleaned up.

[0148] By adopting the embodiment of the present application, the performance data of the distributed database cluster collected is input into the pre-trained long short-term memory network model (target evaluation model) for data tilt risk assessment, thereby avoiding the situation in the related art where the business system reports an error and the data tilt situation is manually identified, resulting in low efficiency in evaluating the data tilt situation, thereby achieving the technical effect of improving the evaluation efficiency of data tilt evaluation. This further solves the technical problem in the related art where the business system reports an error and the data tilt situation is manually identified, resulting in low efficiency in processing the data tilt situation.

[0149] It can be understood by those skilled in the art that Figure 7 The structure shown is for illustration only, and the electronic device may also be a smart phone, a tablet computer, a PDA, a mobile Internet device (MID), a PAD or other terminal device. Figure 7 The structure of the electronic device is not limited. Figure 7 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 7 Different configurations shown.

[0150] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0151] Embodiment 5

[0152] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the data tilt assessment method provided in the first embodiment.

[0153] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0154] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing the steps of the data tilt assessment method.

[0155] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0156] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0157] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0158] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0159] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0160] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0161] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for evaluating data tilt, characterized in that: include: Collecting performance data of a target database cluster to obtain a target data set, wherein the target database cluster includes: a cluster constructed by a distributed database; The target data set is input into a target evaluation model, and an evaluation result is output, wherein the target evaluation model includes: a long short-term memory network model for evaluating data skew risk, and the evaluation result is used to indicate a time period when the target database cluster has a data skew risk.

2. The data tilt assessment method according to claim 1, characterized in that: After inputting the target data set into the target evaluation model and outputting the evaluation result, the method further includes: If the evaluation result indicates that the target database cluster has a data skew risk in the target time period, generating a tuning strategy, wherein the tuning strategy includes: a strategy for distributing and adjusting the load of the target database cluster; Before the target time period, the distribution key of the target data set is adjusted based on the tuning strategy, wherein the distribution key is used to adjust the data distribution in the target database cluster.

3. The data tilt assessment method according to claim 1, characterized in that: The target data set is input into the target evaluation model, and the evaluation results are output, including: Determining a data format of the data in the target data set, and based on the data format, converting the data in the target data set into a time series format to obtain a converted target data set; The converted target data set is input into the target evaluation model, and the evaluation result is output.

4. The data tilt assessment method according to claim 1, characterized in that: The target evaluation model is obtained by: Acquire an initial training sample, wherein the initial training sample includes: performance data generated by the target database cluster in a historical time period; Preprocessing the initial training samples to obtain target training samples; The initial evaluation model is trained based on the target training sample, and when the initial evaluation model training converges, the converged initial evaluation model is determined as the target evaluation model, wherein, during the model training of the initial evaluation model, a cross-validation strategy is used to verify the initial evaluation model, and the initial evaluation model includes: a long short-term memory network model to be trained.

5. The data tilt assessment method according to claim 4, characterized in that: The preprocessing method includes at least one of the following: data cleaning, feature engineering, time series construction and data labeling, wherein the time series construction is used to convert the data into a time series format, and the data labeling is used to label data with data skew.

6. The data tilt assessment method according to claim 1, characterized in that: Collect performance data of the target database cluster to obtain the target data set, including: Collecting resource data of the target database cluster to obtain a first data set; Collecting time data of the target database cluster to obtain a second data set, wherein the time data includes at least one of the following: average response time and average waiting time; Collecting the task processing data of the target database cluster to obtain a third data set, wherein the task processing data includes: data on concurrent execution of tasks; The target data set is determined based on the first data set, the second data set, and the third data set.

7. The data tilt assessment method according to claim 1, characterized in that: After inputting the target data set into the target evaluation model and outputting the evaluation result, the method further includes: Detecting whether data fragments exist in the target database cluster and obtaining a detection result, wherein the data fragments include: data generated by performing multiple update operations or multiple deletion operations on the data in the target database cluster; When the detection result indicates that data fragments exist in the target database cluster, the data fragments are cleaned up.

8. A data tilt assessment system, characterized in that: The data tilt assessment system is used to execute the data tilt assessment method according to any one of claims 1 to 7, comprising: A target database cluster, wherein the target database cluster includes: a cluster constructed by a distributed database; A big data service platform, used to collect performance data of the target database cluster to obtain a target data set; A data skew prediction management module is used to input the target data set into a target evaluation model and output an evaluation result, wherein the target evaluation model includes: a long short-term memory network model for evaluating data skew risk, and the evaluation result is used to indicate the time period when the target database cluster has a data skew risk.

9. A data tilt assessment device, characterized in that: include: A collection unit, used to collect performance data of a target database cluster to obtain a target data set, wherein the target database cluster includes: a cluster constructed by a distributed database; A processing unit is used to input the target data set into a target evaluation model and output an evaluation result, wherein the target evaluation model includes: a long short-term memory network model for evaluating data skew risk, and the evaluation result is used to indicate a time period when the target database cluster has a data skew risk.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is executed, the device where the computer-readable storage medium is located is controlled to execute the data skew assessment method according to any one of claims 1 to 7.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the data tilt assessment method described in any one of claims 1 to 7 are implemented.

Images