Optimization strategy determination method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to an optimization strategy determination method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: reading a performance index set of a target database; the target database is a distributed database; calculating a data skew condition of the target database according to the performance index set; under the condition that data skew exists in the target database, abnormal factors influencing the data skew are determined; and generating a target optimization strategy according to the abnormal factors. The method can effectively solve the problem of data skew.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to an optimization strategy determination method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] Distributed databases achieve high availability and scalability by storing data on multiple nodes, and are widely used in big data scenarios. However, due to the uneven distribution of data among nodes, data skew is prone to occur, that is, some nodes are overloaded and the overall performance is affected. Data skew often causes excessive resource consumption on some nodes, which slows down query efficiency and reduces system stability.

[0003] At present, the existing technology usually adopts methods such as distribution key optimization and data redistribution to solve the data skew problem. For example, by adjusting the distribution key, the data is distributed more evenly on each node to reduce the load of some nodes. However, these methods often rely on predefined distribution keys or static partitioning schemes, which are difficult to cope with the dynamic changes in data access patterns. In practical applications, as the amount of data and access requests continue to change, a single distribution key strategy is difficult to maintain load balancing, resulting in the possibility that the data skew problem will reappear, and the data skew problem cannot be effectively solved. Summary of the invention

[0004] Based on this, it is necessary to provide an optimization strategy determination method, device, computer equipment, computer-readable storage medium and computer program product that can effectively solve the data skew problem in response to the above technical problems.

[0005] In a first aspect, the present application provides a method for determining an optimization strategy, the method comprising:

[0006] Reading a performance indicator set of a target database; the target database is a distributed database;

[0007] Calculating the data skew of the target database according to the performance indicator set;

[0008] In the case where data skew exists in the target database, determining abnormal factors affecting the data skew;

[0009] Based on the abnormal factors, a target optimization strategy is generated.

[0010] In one embodiment, calculating the data skew of the target database according to the performance indicator set includes:

[0011] Constructing a random tree corresponding to the performance indicator set; the random tree is obtained by randomly selecting any performance indicator from the performance indicator set as a root node, and recursively dividing other performance indicators based on the root node;

[0012] Calculating the path length from the other performance indicators to the root node;

[0013] Calculating an anomaly score corresponding to the performance indicator set according to the path length;

[0014] When the abnormal score is greater than the abnormal condition threshold, it is determined that data skew exists in the performance indicator.

[0015] In one embodiment, determining the abnormal factors affecting data skew includes:

[0016] Calculate the correlation coefficient corresponding to each performance indicator in the performance indicator set;

[0017] The abnormal factor is determined according to the correlation coefficient and a preset abnormal threshold.

[0018] In one embodiment, generating a target optimization strategy according to the abnormal factors includes:

[0019] Get historical optimization records;

[0020] Perform prediction based on the abnormal factors and the historical optimization to obtain an initial optimization strategy;

[0021] The operating effect of the initial optimization strategy is predicted, and the initial optimization strategy is optimized based on the operating effect until the initial optimization strategy reaches the optimization target, thereby obtaining the target optimization strategy.

[0022] According to the above method, the method further comprises:

[0023] monitoring the set of performance indicators;

[0024] If the degree of change of the performance indicator set reaches the optimization standard, an optimization model is trained according to the performance indicator set; the optimization model is used to optimize the initial optimization strategy according to the operation effect;

[0025] The training process of the optimization model includes:

[0026] Get sample indicators;

[0027] Inputting the sample indicators into the initial model for training to obtain a prediction optimization strategy;

[0028] A feedback result is generated based on the optimization strategy, and the parameters of the initial model are optimized according to the feedback result until the iteration of the initial model is completed to obtain the optimized model.

[0029] In one embodiment, after generating the target optimization strategy, the method further includes:

[0030] The target optimization strategy is tested in a simulated test environment, and the target optimization strategy is applied to the target database after the test passes.

[0031] In a second aspect, the present application provides an optimization strategy determination device, the device comprising:

[0032] An acquisition module, used to read a set of performance indicators of a target database; the target database is a distributed database;

[0033] A tilt calculation module, used to calculate the data tilt of the target database according to the performance indicator set;

[0034] A factor determination module, used to determine abnormal factors affecting data skew when data skew exists in the target database;

[0035] The strategy generation module is used to generate a target optimization strategy based on the abnormal factors.

[0036] In a third aspect, the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0037] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0038] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the steps of the above method when executed by a processor.

[0039] The above optimization strategy determination method, device, computer device, computer readable storage medium and computer program product, after determining that the target database has data skew according to the performance indicator set, further determine the abnormal factors causing the data skew, and finally generate the corresponding target optimization strategy based on the abnormal factors. Since the target optimization strategy is generated based on the abnormal factors, the target optimization strategy can effectively cope with dynamic application environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0041] FIG1 is a diagram of an application environment of an optimization strategy determination method in one embodiment;

[0042] FIG2 is a schematic diagram of a flow chart of a method for determining an optimization strategy in one embodiment;

[0043] FIG3 is a structural block diagram of an optimization strategy determination device in one embodiment;

[0044] FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0046] The optimization strategy determination method provided in the embodiment of the present application can be applied to the application system shown in Figure 1. The application system shown in Figure 1 includes a distributed database, a performance monitoring module, and a performance optimization module. Specifically, the distributed database is responsible for storing and managing massive data to ensure efficient access and consistency of the data; the client interacts with the distributed database, initiates query requests, and receives query results.

[0047] The performance monitoring module deploys a lightweight performance monitoring agent on each distributed database node. The agent collects system-level performance indicators such as CPU utilization, memory usage, disk I / O statistics, network traffic, etc. in real time as a performance indicator set. At the same time, the monitoring agent also collects the SQL query statements being executed, records the query text, execution plan, query time, number of returned rows, and other indicators as a performance indicator set, and transmits the collected performance indicator set to a centralized time series database through a message queue, such as InfluxDB (an open source time series database) or Prometheus (a monitoring and alarm tool). The performance monitoring module also collects database metadata information, such as table structure definition, index information, partition configuration, etc., to provide a basis for subsequent performance analysis.

[0048] Among them, the performance optimization module is mainly used to read the collected performance indicator set from the time series database, and analyze it according to the performance indicator set to obtain the target optimization strategy. Specifically, the performance optimization module first reads the performance indicator set of the target database, and then calculates the data skew of the target database according to the performance indicator set. When there is data skew in the target database, the abnormal factors affecting the data skew are determined, and finally the target optimization strategy is generated according to the abnormal factors.

[0049] In an exemplary embodiment, as shown in FIG2 , a method for determining an optimization strategy is provided, and the method is applied to the performance optimization module in FIG1 as an example for explanation, including the following steps 202 to 206. Among them:

[0050] Step 202, read the performance indicator set of the target database; the target database is a distributed database.

[0051] Optionally, the performance optimization module reads the collected performance indicator set from the time series database according to a preset time interval. The time series database is used to store the performance indicator set collected by the performance monitoring module from the target database.

[0052] Furthermore, the preset time interval can be dynamically adjusted according to actual business needs. For example, during periods of high database load, a shorter time interval (such as collecting data once every minute) can be selected to capture potential performance bottlenecks in a timely manner; during periods of low load, the collection interval can be appropriately extended (such as once every hour) to reduce system burden.

[0053] Since database performance indicators are usually time-series, volatile, and periodic, using a time series database to store performance indicator sets can provide the following advantages to ensure efficient, real-time, and flexible performance data processing, thereby supporting performance optimization based on historical data and real-time data.

[0054] Step 204: Calculate the data skew of the target database based on the performance indicator set.

[0055] Since the performance indicator set reflects the key data of various aspects of the database operation status, such as query response time, load, I / O throughput, CPU usage, etc., these data can help identify data skew problems.

[0056] For example, if the load on a certain node or partition is significantly higher than that on other nodes or partitions, this usually means that the data distribution on this part of the database is uneven, and some nodes need to process more requests or a larger amount of data, causing performance bottlenecks. Therefore, by analyzing performance indicators such as CPU usage and disk I / O, it is possible to determine which nodes or partitions are bearing too many requests or data processing loads. This unbalanced load distribution is often directly related to uneven data distribution (i.e., data skew).

[0057] For example, if the response time of some queries on a certain database node is significantly longer than that on other nodes, it is likely that the data distribution of that node is skewed. Especially in distributed databases, "hot spots" of data can cause certain queries to be particularly concentrated on a certain node or partition, resulting in increased latency. Therefore, by analyzing the query response time and latency of different nodes, it is possible to identify which nodes have performance bottlenecks. If the response time of some queries is consistently high, it may be because these queries access skewed parts of the data.

[0058] Optionally, the performance indicator set can be analyzed by a data distribution analysis method to determine whether the target database has data skew. The data distribution method is to check the distribution of data between nodes, partitions or tables to find out whether a large amount of data is stored on some partitions or nodes, causing load imbalance.

[0059] Optionally, the data skew can be calculated by a pre-trained machine learning model, which can quickly identify the data skew of the database. The machine learning model can be isolation forest, K-means clustering, and principal component analysis (PCA).

[0060] Step 206: When data skew exists in the target database, determine abnormal factors that affect the data skew.

[0061] The phenomenon of data skew is only a superficial manifestation of system performance problems, and the factors that cause data skew may be very diverse, such as uneven data distribution, query mode deviation, abnormal load on certain nodes, etc. If the specific cause of data skew is not clear, direct optimization may not fundamentally solve the problem and may even cause other problems. Therefore, further identifying the abnormal factors will help locate the root cause of the problem, so that more targeted optimization measures can be taken.

[0062] In this embodiment, after determining that the target database has a data skew problem, the abnormal factors causing the data skew are further determined.

[0063] Optionally, abnormal factors can be identified through abnormal correlation analysis. For example, different performance indicators are correlated and analyzed to see if there is any correlation between the indicators. If indicators such as CPU usage, disk I / O, and response time have synchronous peaks on certain nodes, it may be a systemic problem caused by data skew. Abnormal correlation analysis can help to more fully understand the impact of data skew.

[0064] Optionally, abnormal factors can be identified through correlation analysis. Correlation analysis is very useful in detecting and analyzing anomalies, especially in multi-indicator systems, where it can reveal the correlations between different performance indicators, thereby helping to locate abnormal factors that may cause data skew.

[0065] Step 208: Generate a target optimization strategy based on abnormal factors.

[0066] After determining the abnormal factors causing data skew, a corresponding target optimization strategy is generated according to the abnormal factors. The target optimization strategy refers to an optimization strategy generated according to the abnormal factors to solve the data skew caused by the abnormal factors.

[0067] Optionally, a target optimization strategy can be generated by combining abnormal factors and historical optimization records. By analyzing historical optimization records, an optimization solution that has been effective in alleviating similar abnormalities can be selected. For example, if a previous query optimization strategy was successful in dealing with hotspot data skew, the system can give priority to applying similar query optimization solutions.

[0068] Optionally, the specific parameters of the optimization strategy can be adjusted based on the feedback of the optimization effect in the historical records. For example, if the effect of a resource expansion strategy is not ideal, the historical records may show that the amount of resources added at that time is insufficient or unevenly distributed. In this case, when generating the target optimization strategy, the amount of expanded resources can be increased or a more appropriate distribution can be selected.

[0069] In the above embodiment, after determining that the target database has data skew according to the performance indicator set, the abnormal factors causing the data skew are further determined, and finally, the corresponding target optimization strategy is generated according to the abnormal factors. Since the target optimization strategy is generated according to the abnormal factors, the target optimization strategy can effectively face the dynamic application environment.

[0070] In one embodiment, the above-mentioned calculation of the data skew of the target database based on the performance indicator set includes: constructing a random tree corresponding to the performance indicator set; the random tree is obtained by randomly selecting any performance indicator from the performance indicator set as a root node, and recursively dividing other performance indicators based on the root node; calculating the path length from other performance indicators to the root node; calculating the abnormal score corresponding to the performance indicator set based on the path length; when the abnormal score is greater than the abnormal condition threshold, it is determined that the target database has data skew.

[0071] In this embodiment, firstly, a random number corresponding to the performance indicator set firmware is used to determine whether the target database has a data skew problem.

[0072] Optionally, constructing a random number corresponding to the performance indicator set includes randomly selecting any performance indicator from the performance indicator set as a root node, and then recursively dividing the performance indicators based on the root node to obtain the random number.

[0073] For example, randomly select "CPU usage" as a feature. Randomly select a split point between the minimum value (such as 50) and the maximum value (such as 100) of CPU usage, such as 70. At this time, the first node generated is the root node, and the data is divided into two parts: a subset with CPU usage less than or equal to 70 and a subset greater than 70. For each subset, randomly select features (such as disk I / O) and split points again, and continue recursive splitting until each data point is isolated to a leaf node.

[0074] Optionally, for each performance indicator in the performance indicator set, the path length from the root node to the leaf node is calculated. Each time a node is traversed, the path length is increased by 1, and the recursion continues until each performance indicator is isolated to a leaf node. If a data point is isolated to a leaf node during the traversal process, the path calculation stops and the current path length is returned.

[0075] Optionally, in order to make data sets of different sizes comparable, the path length is usually normalized. The normalized path length is used to calculate the anomaly score. The shorter the path, the easier it is to isolate the data point and the higher the anomaly score.

[0076] It should be noted that one performance indicator set corresponds to one anomaly score, because a performance indicator set usually represents the overall change status at a point in time or a location.

[0077] Optionally, the abnormality score is compared with an abnormality condition threshold, wherein the abnormality condition threshold is a pre-set criterion for determining whether there is data skew in the target database. The abnormality condition threshold may be obtained based on historical experience data.

[0078] Exemplarily, when the abnormality score is greater than the abnormality condition threshold, it is determined that the target database has data skew. Otherwise, the target database does not have data skew.

[0079] In this embodiment, the abnormality score corresponding to the performance indicator set can be calculated quickly and accurately by establishing a random tree.

[0080] Furthermore, in one embodiment, the above-mentioned determination of abnormal factors affecting data skew includes: calculating a correlation coefficient corresponding to each performance indicator in the performance indicator set; and determining the abnormal factor according to the correlation coefficient and a preset abnormal threshold.

[0081] Optionally, a target variable is first selected, and then the correlation coefficient between each performance indicator and the target variable is calculated. The correlation coefficient is used to measure the correlation between the performance indicator and the target variable. Exemplarily, the higher the correlation coefficient, the stronger the relationship between the two variables. After obtaining the correlation coefficient, the correlation coefficient is compared with a preset abnormal threshold to determine the abnormal factor. Exemplarily, if the correlation coefficient is greater than the preset abnormal threshold, the performance indicator is judged as an abnormal factor.

[0082] Exemplarily, the target variable may be set as query performance, and then the correlation coefficient between each performance indicator and the target variable is calculated.

[0083] Optionally, the Pearson correlation coefficient can be used to quantify the relationship between each performance metric and the target variable.

[0084] Exemplarily, the performance indicator set is as follows:

[0085] Query ID Query response time Index usage CPU usage Disk operations 1 200 miss 70 300 2 50 Hit 40 100 3 250 miss 80 400 4 60 Hit 45 110 5 300 miss 85 500

[0086] In this embodiment, the target variable is the query response time. The Pearson correlation coefficient is used to quantify the relationship between each performance indicator and the query response time.

[0087] Assume that the calculated correlation coefficient is as follows:

[0088] index Correlation coefficient with query response time Index usage 0.85 CPU usage 0.65 Disk operations -0.75

[0089] Since index usage and CPU usage are strongly correlated with query response time, it means that index usage and CPU usage have a significant impact on query performance. Further, assuming that the abnormal condition threshold is 0.7, index usage is set as an abnormal factor.

[0090] Furthermore, in one embodiment, the above-mentioned generation of a target optimization strategy based on abnormal factors includes: obtaining historical optimization records; making predictions based on abnormal factors and historical optimizations to obtain an initial optimization strategy; predicting the operating effect of the initial optimization strategy, and optimizing the initial optimization strategy based on the operating effect until the initial optimization strategy reaches the optimization goal, thereby obtaining a target optimization strategy.

[0091] First, the system obtains historical optimization records and retrieves historical cases similar to the current abnormal factors and corresponding optimization strategies. These records contain successful experience in handling similar problems, which can provide important reference for the optimization strategy of the current abnormal factors. Based on the characteristics of the current abnormal factors and historical optimization records, the system predicts an initial optimization strategy. For example, if historical records show that the "data skew" problem is mostly solved by "adjusting the distribution key", "adjusting the distribution key" will be used as the initial optimization strategy.

[0092] Optionally, the initial optimization strategy may be predicted in combination with historical optimization records of the previous time period.

[0093] First, the operating effect of the initial optimization strategy is predicted, and the operating effect of the initial optimization strategy is estimated through simulation or based on historical data. If the operating effect is close to the expected optimization goal, the initial optimization strategy is used as the target optimization strategy and is prepared for further verification in the test environment; if the operating effect is not ideal, the strategy will be readjusted or other optimization combinations and parameters will be explored in order to find a more effective solution. In subsequent iterations, the feedback effect of each strategy is continuously evaluated, and the performance of different strategies under different conditions is recorded and accumulated to learn which strategies are more effective in the current situation. In this way, in the subsequent optimization process, it will be more inclined to choose a strategy combination with better results and reduce attempts at inefficient or ineffective strategies. Finally, after multiple rounds of feedback and improvement, it gradually converges to a candidate strategy with the best effect, ensuring that the performance is maximized on the basis of achieving the optimization goal.

[0094] Furthermore, the above method also includes: monitoring a set of performance indicators; if the degree of change of the performance indicator set reaches the optimization standard, training an optimization model according to the performance indicator set; the optimization model is used to optimize the initial optimization strategy according to the operating results; the training process of the optimization model includes: obtaining sample indicators; inputting the sample indicators into the initial model for training to obtain a predictive optimization strategy; generating feedback results based on the optimization strategy, and optimizing the parameters of the initial model according to the feedback results, until the iteration of the initial model is completed to obtain an optimized model.

[0095] Optionally, first monitor the performance indicator set, and continuously track various performance indicators, such as CPU usage, memory usage, disk I / O, etc., in order to understand the operating status in real time. If the performance is stable and the indicator changes are within the normal range, continue to monitor and maintain the current operating status. However, if the degree of change in the performance indicator set is detected to reach the preset optimization standard, it means that the performance may have a bottleneck or load imbalance. At this time, the optimization process needs to be triggered to ensure that the performance is restored to the optimal state. Therefore, the optimization process is triggered based on the current performance indicator set to start the training process of the optimization model.

[0096] In the process of training the optimization model, we first obtain sample indicators and collect data samples related to the current performance status. These sample indicators are used as model inputs to provide data support for the model. Then, the sample indicators are input into the initial model for training. The initial model generates a prediction optimization strategy based on these input data, that is, a preliminary optimization plan, and then generates corresponding feedback results based on the optimization strategy.

[0097] In order to enable the optimization model to continuously learn and improve, the parameters of the initial model will be optimized and adjusted based on the feedback results. The reason for this is that the actual operation is complicated, and the initial model cannot fully adapt to all scenarios in the early stages of training. It needs to be continuously adjusted through feedback to find a more suitable combination of parameters and strategies. Through repeated feedback and adjustments, the prediction ability of the initial model gradually improves, and the effect of the optimization strategy will continue to improve. The entire process goes through multiple rounds of feedback iterations, that is, feedback is recorded after each round of applying the optimization strategy, and model parameters are adjusted based on feedback to continuously optimize the performance prediction ability of the initial model.

[0098] The iterative process continues until the training process of the initial model reaches the preset convergence condition or the iterative effect is stable, at which time the model training is terminated to obtain the final optimized model.

[0099] In one embodiment, after the target optimization strategy is generated, the method further includes: testing the target optimization strategy in a simulated test environment, and applying the target optimization strategy to the target database after the test passes.

[0100] Optionally, the target optimization strategy is tested in a simulated test environment to verify its performance and stability in the simulated test environment to ensure that there is no negative impact on the actual target database.

[0101] Optionally, the test verification process includes evaluating the improvement effect of the target optimization strategy on performance indicators, such as query response time, resource utilization, etc. If the test verification passes, proving that the target optimization strategy can run stably in the actual environment and achieve the expected effect, the target optimization strategy is applied to the target database to achieve system performance optimization and load balancing. By first verifying in the test environment and then applying it to the target database, the potential risk of the optimization strategy to the production environment can be effectively reduced.

[0102] One thing that needs to be explained is that the simulation test environment should be consistent with the target database.

[0103] Optionally, the process of building a simulated database may include: creating a similar data set in a test environment based on the actual data size and distribution of the production environment. You can extract a portion of the data from the production environment or use data generation tools to simulate the characteristics of the data, including data volume, distribution, index, and table structure. If the target database has a data skew problem, you can reproduce the situation in the test environment and test whether the optimization strategy can effectively alleviate the load imbalance problem caused by data skew through uneven data distribution.

[0104] In the above embodiment, after the test verification is passed, the target optimization strategy is applied to the target database to reduce the risk to the actual system.

[0105] In other embodiments, the application system also includes a visual management module, which provides a visual management console and displays a real-time performance data dashboard for each node. Administrators can view the root cause analysis report of performance issues to understand the key factors that lead to performance degradation. The system will provide optimization suggestions for performance issues, and administrators can view the predicted effects of the optimization plan and perform manual intervention and adjustments. Administrators can input knowledge about business characteristics, SLA requirements, etc. through the console to help the system provide optimization plans that are closer to actual needs. This module will record previous optimization behaviors and effects, and provide administrators with visual optimization history tracking.

[0106] It should be noted that the optimization strategy determination method proposed in this application can be applied to large-scale e-commerce platforms, financial industry applications, Internet of Things data platforms, medical and health information systems, and online education platforms.

[0107] Among them, large-scale e-commerce platforms usually deploy large-scale distributed databases to carry massive amounts of user, order, product and other data. The optimization strategy determination method can automatically discover performance issues such as data skew and hot query, and provide targeted optimization solutions to ensure that the e-commerce system can run stably and efficiently.

[0108] In one embodiment, a method for determining an optimization strategy for an online education platform is provided, comprising the following steps:

[0109] 1. Data collection and monitoring

[0110] Step 1: Deploy the intelligent monitoring component on the database server of the education platform. The component automatically collects various performance indicators, including course access delay, user learning record query time, CPU / memory usage, disk I / O, etc., and stores the collected data as a performance indicator data set in the time series database.

[0111] Step 2: Develop a real-time performance monitoring dashboard to display key performance indicators and trends. The dashboard displays the latency of various queries, system resource usage, etc., and supports administrators to view the operating status of the database in real time.

[0112] Step 3: Record metadata. The monitoring component records metadata such as executed SQL statements, table structure, index information, etc., providing basic data for subsequent analysis.

[0113] 2. Data skew analysis

[0114] Step 1: Design and deploy an analysis engine. Develop a machine learning-based analysis engine and deploy it in the data center.

[0115] The analysis engine periodically obtains performance indicator data from the time series database.

[0116] Step 2: Automatically identify data skew issues, use unsupervised learning algorithms to analyze the duration of the query's mapping and simplification phases, and perform correlation analysis with other performance indicators. Automatically identify data skew issues during course access and exams, and the system determines and issues alerts to administrators.

[0117] 3. Automatic distribution key optimization

[0118] Step 1: Collect and analyze the distributed database SQL execution status. The system regularly collects and analyzes the executed SQL statements and records the redistribution cost and non-redistribution cost.

[0119] Step 2: Machine learning model prediction. Use the machine learning model to predict the performance of each candidate distribution key based on SQL execution information, table structure and other features. The model outputs the distribution key with the best predicted performance in each table.

[0120] Step 3: Dynamically apply distribution keys. The system automatically selects and applies the optimal distribution key to the distributed database to optimize the storage of course data and user learning records.

[0121] 4. Adaptive performance optimization

[0122] Step 1: Continuously monitor overall performance indicators. The system continuously monitors the overall performance indicators of the database and automatically triggers the optimization process when performance deviations are detected.

[0123] Step 2: Execute the optimization process, dynamically adjust the distribution key, add indexes, adjust data partitions, etc. Use reinforcement learning technology to learn and execute the best optimization strategy based on historical optimization experience, and gradually improve optimization efficiency.

[0124] 5. Visualization and interactive optimization

[0125] Step 1: Develop a management console. Develop a visual management console to display real-time performance monitoring data and optimization suggestions.

[0126] Step 2: Provide a manual intervention interface. Administrators can view specific performance problem descriptions and optimization plans through the console and perform manual intervention and adjustments.

[0127] Step 3: Integrate business knowledge. The console supports administrators to input business knowledge, such as data access patterns, key business indicators, etc. The system combines this knowledge to provide more practical optimization suggestions.

[0128] Furthermore, container orchestration tools such as Kubernetes or Docker Swarm can be deployed to achieve elastic expansion of applications, automatically expanding or reducing application instances based on real-time traffic and load.

[0129] Furthermore, you can use automated operation and maintenance tools such as Ansible, Chef, and Puppet to simplify the operation and maintenance process. Automated deployment, configuration management, and fault repair can reduce the workload of the operation and maintenance team.

[0130] Furthermore, tools such as Google Analytics and Mixpanel can be integrated to collect user behavior data including course browsing, learning progress, test scores, etc., for building a personalized recommendation system. Finally, big data analysis and machine learning technology are used to analyze user behavior data, build a personalized recommendation algorithm, and recommend courses and learning resources based on users' learning habits and interests.

[0131] In the above embodiments, performance problems such as data tilt and hotspot query can be automatically discovered, and targeted optimization solutions can be provided. By automatically adjusting the distribution key, adding indexes and other measures, the performance bottleneck problem is effectively solved, and the query processing speed and transaction response time are improved. It should be understood that although the various steps in the flowcharts involved in the embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless clearly stated in this article, the execution of these steps is not strictly restricted in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the embodiments described above may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequentially, but can be executed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.

[0132] Based on the same inventive concept, the embodiment of the present application also provides an optimization strategy determination device for implementing the optimization strategy determination method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more optimization strategy determination device embodiments provided below can refer to the limitations of the optimization strategy determination method above, and will not be repeated here.

[0133] In an exemplary embodiment, as shown in FIG3 , an optimization strategy determination device is provided, comprising: a reading module 310, a tilt calculation module 320, a factor determination module 330 and a strategy generation module 340, wherein:

[0134] The reading module 310 is used to read the performance indicator set of the target database; the target database is a distributed database.

[0135] The tilt calculation module 320 is used to calculate the data tilt of the target database according to the performance indicator set.

[0136] The factor determination module 330 is used to determine abnormal factors that affect data skew when there is data skew in the target database.

[0137] The strategy generation module 340 is used to generate a target optimization strategy based on abnormal factors.

[0138] In one embodiment, the tilt calculation module 320 includes:

[0139] The random number construction unit is used to construct a random tree corresponding to the performance indicator set; the random tree is obtained by randomly selecting any performance indicator from the performance indicator set as a root node and recursively dividing other performance indicators based on the root node.

[0140] The path calculation unit is used to calculate the path length from other performance indicators to the root node.

[0141] The anomaly score calculation unit is used to calculate the anomaly score corresponding to the performance indicator set according to the path length.

[0142] The anomaly judgment unit is used to determine whether data skew exists in the target database when the anomaly score is greater than the anomaly condition threshold.

[0143] In one embodiment, the factor determination module 330 includes:

[0144] The correlation calculation unit is used to calculate the correlation coefficient corresponding to each performance indicator in the performance indicator set.

[0145] The selection unit is used to determine the abnormal factors according to the correlation coefficient and the preset abnormal threshold.

[0146] In one embodiment, the policy generation module 340 includes:

[0147] The record acquisition unit is used to acquire historical optimization records.

[0148] The prediction unit is used to make predictions based on abnormal factors and historical optimization to obtain an initial optimization strategy.

[0149] The optimization unit is used to predict the operating effect of the initial optimization strategy and optimize the initial optimization strategy based on the operating effect until the initial optimization strategy reaches the optimization target, thereby obtaining the target optimization strategy.

[0150] In one embodiment, the strategy generation module 340 further includes:

[0151] A detection unit, used to monitor a set of performance indicators;

[0152] The training unit is used to train the optimization model according to the performance indicator set if the degree of change of the performance indicator set reaches the optimization standard; the optimization model is used to optimize the initial optimization strategy according to the operation effect.

[0153] In one embodiment, the training unit includes:

[0154] The sample acquisition subunit is used to obtain sample indicators.

[0155] The model training subunit is used to input sample indicators into the initial model for training to obtain a prediction optimization strategy.

[0156] The model optimization subunit is used to generate feedback results based on the prediction optimization strategy, and optimize the parameters of the initial model according to the feedback results until the initial model iteration is completed to obtain the optimized model.

[0157] In one embodiment, the above device further comprises:

[0158] The test module is used to test the target optimization strategy in a simulated test environment, and apply the target optimization strategy to the target database after the test passes.

[0159] Each module in the above-mentioned optimization strategy determination device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0160] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG4. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store performance indicator data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, an optimization strategy determination method is implemented.

[0161] Those skilled in the art will understand that the structure shown in FIG. 4 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0162] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the method in any one of the above embodiments when executing the computer program.

[0163] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in any one of the above embodiments are implemented.

[0164] In one embodiment, a computer program product is provided, including a computer program, which implements the steps of the method in any one of the above embodiments when executed by a processor.

[0165] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0166] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0167] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for determining an optimization strategy, characterized in that: The method comprises: Reading a performance indicator set of a target database; the target database is a distributed database; Calculating the data skew of the target database according to the performance indicator set; In the case where data skew exists in the target database, determining abnormal factors affecting the data skew; Based on the abnormal factors, a target optimization strategy is generated.

2. The method according to claim 1, characterized in that: The calculating the data skew of the target database according to the performance indicator set includes: Constructing a random tree corresponding to the performance indicator set; the random tree is obtained by randomly selecting any performance indicator from the performance indicator set as a root node, and recursively dividing other performance indicators based on the root node; Calculating the path length from the other performance indicators to the root node; Calculating an anomaly score corresponding to the performance indicator set according to the path length; When the abnormal score is greater than the abnormal condition threshold, it is determined that data skew exists in the target database.

3. The method according to claim 1, characterized in that: The determining of abnormal factors affecting data skew includes: Calculate the correlation coefficient corresponding to each performance indicator in the performance indicator set; The abnormal factor is determined according to the correlation coefficient and a preset abnormal threshold.

4. The method according to claim 1, characterized in that The step of generating a target optimization strategy according to the abnormal factors includes: Get historical optimization records; Perform prediction based on the abnormal factors and the historical optimization to obtain an initial optimization strategy; The operating effect of the initial optimization strategy is predicted, and the initial optimization strategy is optimized based on the operating effect until the initial optimization strategy reaches the optimization target, thereby obtaining the target optimization strategy.

5. The method according to claim 4, characterized in that The method further comprises: monitoring the set of performance indicators; If the degree of change of the performance indicator set reaches the optimization standard, an optimization model is trained according to the performance indicator set; the optimization model is used to optimize the initial optimization strategy according to the operation effect; The training process of the optimization model includes: Get sample indicators; Inputting the sample indicators into the initial model for training to obtain a prediction optimization strategy; A feedback result is generated based on the prediction optimization strategy, and the parameters of the initial model are optimized according to the feedback result until the iteration of the initial model is completed to obtain the optimized model.

6. The method according to claim 1, characterized in that After generating the target optimization strategy, the method further includes: The target optimization strategy is tested in a simulated test environment, and the target optimization strategy is applied to the target database after the test passes.

7. An optimization strategy determination device, characterized in that: The device comprises: A reading module, used to read a set of performance indicators of a target database; the target database is a distributed database; A tilt calculation module, used to calculate the data tilt of the target database according to the performance indicator set; A factor determination module, used to determine abnormal factors affecting data skew when data skew exists in the target database; The strategy generation module is used to generate a target optimization strategy based on the abnormal factors.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images