Database load automatic analysis and auxiliary diagnosis method and system

Abstract

The invention discloses a database load automatic analysis and auxiliary diagnosis method and system. Comprising a data acquisition module, a time window division module, a feature extraction module, a mode recognition module, an anomaly detection and diagnosis module, a report generation module, a data storage and management module, a system monitoring and early warning module, a user management module and a system optimization and upgrade module. According to the method and the system, multi-dimensional load data can be comprehensively acquired in real time through distributed acquisition agents, high efficiency and low resource occupation of data acquisition are ensured by adopting a lightweight communication protocol, and meanwhile, the acquired load data are divided according to different time granularities by adopting a self-adaptive time window division algorithm, so that the data acquisition efficiency is improved. According to the method, the instantaneous change and the long-term trend of the load can be more accurately captured, feature extraction is performed on the load data in each time window, the accuracy of feature extraction is improved, the running state of the system is monitored in real time, and the usability and the stability of the system are improved.

Description

Technical Field

[0001] This invention relates to an automatic analysis and auxiliary diagnosis technology, and more particularly to a method and system for automatic analysis and auxiliary diagnosis of database load. Background Technology

[0002] A database is a "warehouse" for storing data. Its storage space is enormous, capable of holding millions, tens of millions, or even hundreds of millions of data entries. However, databases follow certain rules; data is not stored arbitrarily, otherwise query efficiency would be very low. The internet today is filled with massive amounts of data from various sources, such as travel records, consumption records, browsed web pages, and sent messages. This data includes text, images, music, and sound, among other types. A database is a computer software system that stores and manages data according to a data structure. The concept of a database encompasses two aspects: firstly, a database is an entity, a "warehouse" that can reasonably store data, where users store the transactional data they need to manage; secondly, databases represent new methods and technologies for data management, enabling more appropriate organization, easier maintenance, more rigorous control, and more efficient utilization of data. As the most important foundational software, databases are the cornerstone of ensuring the stable operation of computer systems. They include relational databases and non-relational databases, differing in storage methods, structures, standardization, expansion methods, query methods, normalization, transactional nature, read / write performance, and authorization methods.

[0003] Existing technologies largely rely on single-node or centralized data collection methods, making it difficult to acquire multi-dimensional load data during database operation in real time, such as key indicators like CPU utilization and memory usage. This collection method is not only inefficient but also prone to missing important information, leading to inaccurate assessments of database load status. Furthermore, existing technologies often use fixed-size time windows when processing database load data, failing to dynamically adjust the window size based on historical load patterns and real-time load trends. This static approach struggles to accurately capture instantaneous load changes and long-term trends, resulting in decreased accuracy in feature extraction and pattern recognition. Additionally, the lack of a load change sensitivity coefficient often leads to imprecise time window divisions, further impacting the accurate assessment of database load status.

[0004] In feature extraction, existing technologies mostly employ static weight allocation mechanisms, which fail to assign different weights to data within a sliding window based on the impact of data at different time points on the overall features. This fixed weight allocation method makes it difficult for extracted features to reflect the actual database load, reducing the accuracy of feature extraction. Furthermore, the lack of real-time applications of sliding window technology severely impacts the real-time performance and accuracy of feature computation. Simultaneously, when using machine learning algorithms for pattern recognition of extracted load features, existing technologies often lack effective feature selection and cross-validation mechanisms, resulting in insufficient accuracy and robustness in pattern recognition, making it difficult to accurately classify database load into different pattern categories. Summary of the Invention

[0005] Purpose of the invention: The purpose of this invention is to provide a method for automatically analyzing and assisting in the diagnosis of database load. Another purpose of this invention is to provide a system for implementing the method.

[0006] Technical solution: The database load automatic analysis and auxiliary diagnosis method of the present invention includes the following steps:

[0007] Multi-dimensional data acquisition: Through a distributed acquisition agent deployed on the server, multi-dimensional load data of the database operation is collected in real time, and data interaction is carried out using a lightweight communication protocol;

[0008] Dynamic time window partitioning: An adaptive time window partitioning algorithm is adopted to divide the collected multi-dimensional load data according to different time granularities; the adaptive time window partitioning algorithm dynamically adjusts the size of the time window based on the historical load pattern and real-time load change trend of the database.

[0009] Load feature extraction: Feature extraction is performed on the load data within each time window;

[0010] Load pattern recognition: An ensemble learning algorithm is used to perform pattern recognition on the extracted load features, classifying the database load into different categories;

[0011] Anomaly detection and diagnosis: Based on load categories, rule-based and statistical anomaly detection methods are used to detect anomalies in the database load and further analyze the causes of the anomalies; the anomaly detection method adopts dynamic threshold adjustment technology, which automatically adjusts the anomaly detection threshold according to the real-time load of the database;

[0012] Assisted Diagnostic Report Generation: Generates visual assisted diagnostic reports based on anomaly detection and diagnostic results;

[0013] Prediction and Early Warning: Based on historical load data and current load trends, time series analysis methods and prediction algorithms are used to predict the future load of the database; the prediction algorithm uses online learning technology to update model parameters in real time.

[0014] The multi-dimensional data collection process includes data preprocessing, which includes data cleaning, data normalization, and missing value imputation. Data cleaning employs statistical and rule-based methods to remove noise and outliers from the collected load data. Data normalization uses normalization methods to unify load data of different dimensions into a consistent dimension range. Missing value imputation uses interpolation or prediction methods to handle missing data that occurs during the collection process.

[0015] The load pattern recognition also includes load pattern mining, which discovers the correlation and evolution of load patterns, and uses association rule mining algorithm and sequence pattern mining algorithm to mine the frequency of occurrence, order of occurrence and mutual influence of different load patterns in different time periods.

[0016] The method also supports load analysis for various types of databases, including relational databases, non-relational databases, and distributed databases. For different types of databases, corresponding data acquisition interfaces and analysis strategies are adopted to analyze and diagnose the load of various databases.

[0017] After the auxiliary diagnostic report is generated, it also includes real-time feedback and adjustment; the real-time feedback and adjustment adjusts the load analysis and diagnostic strategies in real time based on user feedback and actual processing results, and optimizes the load analysis and diagnostic strategies using online learning algorithms.

[0018] The database load automatic analysis and auxiliary diagnosis system of the present invention includes the following modules:

[0019] Data acquisition module: Collects multi-dimensional load data in real time during database operation through a distributed acquisition agent; the distributed acquisition agent is deployed on the database server and related application servers, and uses a lightweight communication protocol to interact with the analysis server. It also has an intelligent downsampling function. When high network bandwidth usage is detected, it can downsample some non-critical load data according to a preset priority strategy.

[0020] Time window segmentation module: Based on the periodic characteristics of database load, an adaptive time window segmentation algorithm is adopted to divide the collected load data according to different time granularities. The adaptive time window segmentation algorithm dynamically adjusts the size of the time window according to the historical load pattern and real-time load change trend of the database. At the same time, a load change sensitivity coefficient is introduced to automatically adjust the segmentation accuracy of the time window according to the rate of change of load data. When the load changes drastically, the time window is reduced, and when the load changes steadily, the time window is increased.

[0021] Feature extraction module: Extracts features from the load data within each time window, including mean, variance, maximum, minimum, median, quantile, slope, volatility, and periodicity. It also employs a sliding window technique and introduces a dynamic weight allocation mechanism, assigning different weights to the data within the sliding window based on the degree of influence of data at different time points on the overall features.

[0022] Pattern recognition module: Utilizes ensemble learning algorithms to perform pattern recognition on extracted load features, classifying database load into different pattern categories; these pattern categories include normal load mode, burst high load mode, periodic load mode, abnormal low load mode, and mixed load mode; the ensemble learning introduces a feature cross-validation mechanism to perform cross-validation on different feature combinations, selecting the optimal feature combination for pattern recognition.

[0023] Anomaly detection and diagnosis module: Based on the identified load patterns, it uses a combination of rule-based and statistical anomaly detection methods to detect anomalies in the database load. When an anomaly is detected, the cause of the anomaly is further analyzed. The anomaly detection method uses dynamic threshold adjustment technology and introduces an adaptive learning mechanism to automatically adjust the anomaly detection threshold according to historical anomaly data and real-time load conditions.

[0024] Report generation module: Based on the anomaly detection and diagnosis results, it generates a visual auxiliary diagnostic report. The report includes an anomaly description, anomaly occurrence time, anomaly impact range, cause analysis, and suggested solutions. It also introduces interactive analysis functions into the visualization, allowing users to conduct in-depth analysis of the data in the report through interactive operations.

[0025] Data storage and management module: Stores the collected multi-dimensional load data, extracted feature data, identified load patterns, and generated auxiliary diagnostic reports. It adopts a storage method that combines a distributed file system and a relational database, classifies and stores different types of data, and introduces data compression and encryption technologies to compress and encrypt the stored data.

[0026] System monitoring and early warning module: Real-time monitoring of the system's operating status, including data acquisition, processing, and analysis of each stage of operation. When a system malfunctions or abnormalities, it promptly issues early warning information to notify relevant personnel for handling. The early warning information includes the fault type, fault occurrence time, and fault impact range.

[0027] User Management Module: Manages system users, including user registration, login, and permission allocation. Based on user roles and permissions, it provides different operation interfaces and functional permissions for users. It also introduces user behavior analysis function to analyze user operation behavior and promptly issue warnings and handle abnormal operations.

[0028] System Optimization and Upgrade Module: Regularly optimizes and upgrades the system, including algorithm optimization, function expansion, and performance improvement. It also optimizes and upgrades the system based on its operation and user feedback, and introduces an automatic update mechanism so that the system can automatically detect and prompt users to update when a new version is released.

[0029] The distributed acquisition agent in the data acquisition module also has a data caching function. When communication with the analysis server fails, the load data is temporarily cached locally, and the cached data is transmitted to the analysis server after communication is restored. The adaptive time window division algorithm in the time window division module also introduces a seasonal factor adjustment mechanism to adjust the time window according to the seasonal variation of the database load.

[0030] The sliding window of the feature extraction module has a data smoothing function, which smooths the data within the sliding window and removes noise and outliers; the ensemble learning method of the pattern recognition module integrates decision tree, support vector machine and neural network algorithms.

[0031] The rule-based and statistical anomaly detection method of the anomaly detection and diagnosis module also has a dynamic update function, which dynamically updates the anomaly detection rules and statistical models according to historical anomaly data and real-time load conditions; the distributed file system of the data storage and management module adopts a redundant storage strategy to back up important data, and introduces a data lifecycle management function to hierarchically manage storage according to data access frequency and importance.

[0032] The system monitoring and early warning module also has a fault self-diagnosis function, which can automatically diagnose faults and analyze their root causes; the automatic update mechanism of the system optimization and upgrade module also has an update risk assessment function, which assesses the impact of the update before updating.

[0033] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0034] (1) Through the distributed acquisition agent, it is possible to collect multi-dimensional load data in the database operation process in real time and comprehensively. The distributed acquisition agent is deployed on the server and a lightweight communication protocol is adopted to ensure the high efficiency and low resource consumption of data acquisition. At the same time, the intelligent downsampling function can downsample some non-critical load data according to the preset priority strategy when the network bandwidth consumption is too high, further improving the efficiency and flexibility of data acquisition.

[0035] (2) Based on the periodic characteristics of database load, an adaptive time window partitioning algorithm is adopted, which can dynamically adjust the size of the time window according to the historical load pattern and the real-time load change trend. The load change sensitivity coefficient is introduced to make the time window partitioning more accurate and can more accurately capture the instantaneous changes and long-term trends of the load, providing a solid foundation for subsequent feature extraction and pattern recognition.

[0036] (3) Extract features from the load data in each time window. Use sliding window technology to ensure the real-time performance and accuracy of feature calculation. At the same time, introduce a dynamic weight allocation mechanism to allocate different weights to the data in the sliding window according to the degree of influence of data at different time points on the overall features. This makes the extracted features more reflective of the actual situation of the database load and improves the accuracy of feature extraction.

[0037] (4) The extracted load features are pattern recognized by machine learning algorithms, and the database load is divided into different pattern categories. The feature cross-validation mechanism is introduced to select the optimal feature combination for pattern recognition, which improves the accuracy and robustness of pattern recognition and provides a reliable basis for subsequent anomaly detection and diagnosis. At the same time, the rule-based and statistical anomaly detection methods are combined, dynamic threshold adjustment technology is adopted, and an adaptive learning mechanism is introduced to automatically adjust the threshold of anomaly detection according to historical anomaly data and real-time load conditions, which improves the sensitivity and accuracy of anomaly detection and can further analyze the causes of anomalies, providing targeted solutions for database administrators.

[0038] (5) Generate a visual auxiliary diagnostic report based on the abnormal detection and diagnosis results. It includes interactive analysis functions such as zooming in, zooming out, and filtering, which makes it convenient for users to quickly locate problems and improve processing efficiency. It adopts a storage method that combines a distributed file system and a relational database to improve the efficiency and reliability of data storage. At the same time, it introduces data compression and encryption technology to reduce storage space occupation, ensure data security, and support the long-term stable operation of the system.

[0039] (6) Monitor the system's operating status in real time. When the system malfunctions or becomes abnormal, it can issue early warning information in a timely manner, which will help relevant personnel to quickly locate and solve the problem, thereby improving the system's availability and stability. Attached Figure Description

[0040] Figure 1 This is a flowchart of the method of the present invention;

[0041] Figure 2 This is a schematic diagram of the system modules of the present invention. Detailed Implementation

[0042] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0043] Example 1

[0044] This embodiment provides a basic implementation process for automatic database load analysis and assisted diagnosis, such as... Figure 1 As shown, the specific steps are as follows:

[0045] (1) Multi-dimensional data collection: Through distributed deployment of lightweight collection agent programs, multi-dimensional load data during database operation is collected in real time. Multi-dimensional load data includes, but is not limited to, CPU utilization, memory usage, disk I / O read / write speed, network bandwidth utilization, database connection count, query response time, transaction processing speed, lock wait time, cache hit rate, and SQL statement execution frequency. The distributed collection agent is deployed on the database server and related application servers, and uses a lightweight communication protocol to interact with the central analysis server to ensure the high efficiency and low resource consumption of data collection.

[0046] (2) Dynamic time window division: Based on the periodic characteristics of database load, an adaptive time window division algorithm is adopted to divide the collected load data according to different time granularities, including second level, minute level, hour level and day level; the adaptive time window division algorithm dynamically adjusts the size of the time window according to the historical load pattern and real-time load change trend of the database in order to capture the instantaneous changes and long-term trends of the load.

[0047] (3) Load feature extraction: Feature extraction is performed on the load data within each time window. The extracted features include, but are not limited to, mean, variance, maximum, minimum, median, quantile, slope, volatility and periodicity. The feature extraction process uses sliding window technology to ensure the real-time performance and accuracy of feature calculation.

[0048] (4) Load pattern recognition: Using machine learning algorithms, the extracted load features are used to perform pattern recognition, and the database load is divided into different pattern categories. The pattern categories include, but are not limited to, normal load pattern, sudden high load pattern, periodic load pattern, abnormal low load pattern and mixed load pattern. The machine learning algorithm adopts the ensemble learning method, which combines the advantages of multiple classification algorithms to improve the accuracy and robustness of pattern recognition.

[0049] (5) Anomaly detection and diagnosis: Based on the identified load patterns, a combination of rule-based anomaly detection methods and statistical anomaly detection methods is used to detect anomalies in the database load. When an anomaly is detected, the cause of the anomaly is further analyzed. The cause includes, but is not limited to, hardware failure, software configuration errors, SQL statement performance issues, concurrent access conflicts, and excessively rapid data growth. The anomaly detection method uses dynamic threshold adjustment technology to automatically adjust the anomaly detection threshold according to the real-time load of the database, thereby improving the sensitivity of anomaly detection.

[0050] (6) Auxiliary diagnostic report generation: Based on the results of anomaly detection and diagnosis, a detailed auxiliary diagnostic report is generated. The report includes, but is not limited to, anomaly description, anomaly occurrence time, anomaly impact range, cause analysis, and suggested solutions. The auxiliary diagnostic report is displayed using visualization technology to facilitate users' quick understanding and handling of database load issues.

[0051] (7) Prediction and early warning: Based on historical load data and current load trends, time series analysis and machine learning prediction algorithms are used to predict the future load of the database; when the load is predicted to exceed the preset safety threshold, an early warning message is issued in a timely manner to remind users to take corresponding measures; the prediction algorithm adopts online learning technology, which can update the model parameters in real time and improve the accuracy of prediction.

[0052] In practical applications, multi-dimensional data collection is first carried out using a distributed data acquisition agent. This agent is flexibly deployed on the database server and its associated application servers, interacting with the analysis server via a lightweight communication protocol. During database operation, the acquisition agent collects multi-dimensional load data in real time and comprehensively, covering key indicators such as CPU utilization, memory usage, disk I / O read / write speed, network bandwidth utilization, database connection count, query response time, transaction processing speed, lock wait time, cache hit rate, and SQL statement execution frequency. This distributed and real-time data acquisition method ensures both the comprehensiveness and timeliness of the data, providing rich and accurate foundational data for subsequent analysis, while effectively avoiding analytical errors caused by missing or biased data, thus solving the core issues of data source completeness and timeliness.

[0053] Next, dynamic time window segmentation technology is used to process the collected load data. Based on the periodic characteristics of database load, an adaptive time window segmentation algorithm is adopted to segment the load data according to different time granularities such as seconds, minutes, hours, and days. This algorithm can dynamically adjust the size of the time window according to the historical load patterns and real-time load change trends of the database, enabling the system to capture both instantaneous load changes (such as sudden query peaks and short-term hardware performance fluctuations) and long-term load trends (such as regular changes between peak and off-peak periods). This reasonable time window segmentation method provides a more accurate and targeted data foundation for subsequent load feature extraction and pattern recognition, solving the problem of inaccurate capture of load features due to unreasonable time scale settings.

[0054] Subsequently, the load feature extraction stage begins. For the load data within each time window, a sliding window technique is used to extract various features, including mean, variance, maximum, minimum, median, quantile, slope, volatility, and periodicity. The use of the sliding window technique ensures the real-time performance and accuracy of feature calculation, enabling continuous and stable calculation of feature values ​​within each time window as the data is constantly updated. These feature values ​​characterize the state and changes of the database load from different perspectives, providing rich and detailed evidence for load pattern recognition and solving the technical problem of how to extract effective features from massive load data to accurately describe the load state.

[0055] Subsequently, machine learning algorithms were used for load pattern recognition. An ensemble learning approach was employed, combining the advantages of multiple classification algorithms to perform pattern recognition on the extracted load features. Database load was categorized into different types: normal load patterns, sudden high load patterns, periodic load patterns, abnormally low load patterns, and mixed load patterns. The ensemble learning method, by integrating the results of multiple classification algorithms, improves the accuracy and robustness of pattern recognition, effectively addressing complex and variable database load conditions. Accurate pattern recognition provides a clear classification basis for subsequent anomaly detection and diagnosis, solving the technical challenge of accurately determining database load patterns for targeted analysis.

[0056] In the anomaly detection and diagnosis phase, based on the identified load patterns, a combination of rule-based and statistical anomaly detection methods is used to comprehensively monitor database load. Simultaneously, dynamic threshold adjustment technology is employed to automatically adjust anomaly detection thresholds according to the real-time database load. When anomalies are detected, the causes are further analyzed, covering multiple aspects such as hardware failures, software configuration errors, SQL statement performance issues, concurrent access conflicts, and excessively rapid data growth. This comprehensive anomaly detection and diagnosis approach, combining the explicitness of rules with the flexibility of statistics, can promptly identify various types of anomalies and accurately trace their root causes, solving the technical challenge of efficiently and accurately detecting database load anomalies and determining their causes.

[0057] Following the anomaly detection and diagnosis results, a detailed auxiliary diagnostic report is generated. The report is presented using visualization technology and includes anomaly description, occurrence time, impact scope, cause analysis, and suggested solutions. This visualization approach allows users to quickly and intuitively understand database load issues, facilitating appropriate action based on the report's information. This addresses the technical challenge of clearly and effectively communicating analysis results to users to assist in problem-solving.

[0058] Finally, forecasting and early warning systems are implemented. Based on historical load data and current load trends, time series analysis and machine learning algorithms are used to predict future database load. The forecasting algorithm employs online learning technology, enabling real-time updates of model parameters and improving prediction accuracy. When the predicted load exceeds a preset safety threshold, a timely warning is issued, reminding users to take appropriate measures, such as optimizing database configuration, increasing hardware resources, or adjusting business strategies. This forecasting and early warning mechanism helps users prepare in advance, avoiding system performance degradation or even crashes caused by excessive load, thus solving the technical problem of how to predict database load risks and take timely preventative measures.

[0059] Example 2

[0060] This embodiment provides a database load analysis workflow that includes data preprocessing and load pattern mining. After multi-dimensional data acquisition, a data preprocessing step is included, comprising data cleaning, data normalization, and missing value imputation. Data cleaning removes noise and outliers from the acquired load data, employing a combination of statistical and rule-based methods. Data normalization unifies load data of different dimensions to the same dimension, using min-max normalization or Z-score normalization. Missing value imputation handles missing data encountered during acquisition, using interpolation based on historical data or prediction methods based on machine learning. After load pattern identification, a load pattern mining step is included, discovering the correlations and evolutionary patterns between load patterns. Association rule mining and sequence pattern mining algorithms are used to mine the frequency, sequence, and mutual influence of different load patterns across different time periods.

[0061] In practical applications, after the system starts up, it first enters the multi-dimensional data collection stage. In this stage, through the data collection modules deployed at each key node of the database system, the load information of the database under different operating states is comprehensively collected, covering multiple dimensions of indicators such as CPU utilization, memory usage, disk I / O rate, network traffic, query response time, and transaction processing volume, to ensure the comprehensiveness and accuracy of the collected data and provide rich basic data for subsequent analysis.

[0062] The collected raw data then enters the data preprocessing stage, which includes three key steps:

[0063] (1) Data cleaning: The collected load data is deeply cleaned by combining statistical and rule-based methods. The statistical method identifies and removes noisy data and outliers that deviate from the normal range by analyzing the distribution characteristics of the data; the rule-based method further filters data that does not conform to the actual scenario according to the pre-set business rules and thresholds to ensure the accuracy and reliability of the data.

[0064] (2) Data normalization: For loading data with different dimensions, the min-max normalization method or Z-score normalization method is used to map the data to the same dimension range. Min-max normalization scales the data to the [0,1] interval through linear transformation, which is suitable for cases where the data distribution is relatively uniform; Z-score normalization calculates the mean and standard deviation of the data and converts the data into a standard normal distribution with a mean of 0 and a standard deviation of 1, which is suitable for cases where the data distribution has large differences. Through normalization, the influence of dimension differences on subsequent analysis is eliminated, and the accuracy of the analysis results is improved.

[0065] (3) Data missing value imputation: For missing data that appears during the collection process, interpolation methods based on historical data or prediction methods based on machine learning are used to imput the missing values. Interpolation methods based on historical data estimate the missing values ​​by using linear interpolation or spline interpolation based on the data characteristics of adjacent time points; prediction methods based on machine learning use historical data to train prediction models, such as regression models or time series models, to predict and imput the missing values. By imputing the missing values, the integrity and continuity of the data are ensured, providing a complete dataset for subsequent analysis.

[0066] After data preprocessing, the system enters the load pattern identification stage. This stage applies advanced pattern recognition algorithms, such as cluster analysis and time series analysis, to deeply mine the preprocessed load data and identify load patterns in the database under different operating conditions, such as peak load patterns, off-peak load patterns, and abnormal load patterns, providing a foundation for subsequent load pattern mining.

[0067] Subsequently, the system enters the load pattern mining phase, employing association rule mining and sequence pattern mining algorithms to conduct in-depth analysis of the identified load patterns. This analysis uncovers the frequency of occurrence, sequence, and interrelationships of different load patterns across different time periods. The association rule mining algorithm analyzes the co-occurrence relationships between load patterns to discover potential association rules, such as "when CPU utilization exceeds 80%, disk I / O rate usually increases significantly." The sequence pattern mining algorithm analyzes the time-series characteristics of load patterns to discover their evolutionary patterns, such as "during daily peak business periods, database load typically shows a trend of first rising and then falling." Through load pattern mining, the system gains a deep understanding of the dynamic characteristics of database load changes, providing strong support for database performance optimization and fault prediction.

[0068] Finally, the system integrates and analyzes the results of load pattern mining to generate a detailed diagnostic report and recommendations. The diagnostic report includes the current database load status, potential performance bottlenecks, causes of failures, and corresponding optimization suggestions. This helps database administrators fully understand the database's operational status, take timely measures for performance optimization and troubleshooting, and ensure the efficient and stable operation of the database system.

[0069] Example 3

[0070] This embodiment provides a method for load analysis of multiple database types, combined with real-time feedback for continuous optimization of diagnostic capabilities. The method supports load analysis of various database types, including but not limited to relational databases (such as MySQL, Oracle, SQL Server), non-relational databases (such as MongoDB, Redis, Cassandra), and distributed databases (such as HBase, Greenplum). For different types of databases, corresponding data acquisition interfaces and analysis strategies are adopted to ensure accurate analysis and diagnosis of various database loads. After the auxiliary diagnostic report is generated, a real-time feedback and adjustment step is included. Real-time feedback and adjustment are used to adjust the load analysis model and diagnostic strategy in real time based on user feedback and actual processing results. A user feedback mechanism is established to collect user satisfaction with the auxiliary diagnostic report and processing results information. Based on user feedback, an online learning algorithm is used to optimize the load analysis model and diagnostic strategy, improving the system's adaptability and accuracy.

[0071] In practical applications, after system startup, the first step is multi-source data acquisition. The system automatically selects appropriate data acquisition interfaces based on the characteristics of different database types: for relational databases, key performance indicators (such as CPU utilization, memory usage, disk I / O, and query response time) are obtained through SQL query interfaces; for non-relational databases, operation logs, cache hit rates, and cluster node status data are collected using their provided APIs or monitoring tools; for distributed databases, global load distribution, data skew, and network latency information are obtained through management nodes. This multi-source data acquisition strategy ensures the comprehensiveness and accuracy of data collection, laying a solid foundation for subsequent analysis.

[0072] The collected raw data then enters the preprocessing module for data cleaning, format standardization, and outlier detection to eliminate noise interference and improve data quality. Subsequently, the system invokes appropriate analysis strategies based on the database type. For example, for relational databases, statistical analysis methods are used to identify performance bottlenecks such as slow queries and missing indexes; for non-relational databases, machine learning algorithms, such as cluster analysis, are used to discover imbalances in data access patterns; and for distributed databases, graph theory algorithms are used to analyze load balancing among nodes and identify potential data hotspots.

[0073] After completing the load analysis, the system enters the auxiliary diagnostic report generation phase. The report includes not only a detailed description of the current load status but also precise identification of the root causes of problems and improvement suggestions. For example, for identified slow queries, the report provides specific solutions for optimizing SQL statements; for data skew issues, it suggests adjusting the data distribution strategy or adding nodes. After the report is generated, the system immediately initiates a user feedback collection mechanism, proactively obtaining user satisfaction ratings and feedback on actual processing effectiveness through interface interaction, email notifications, or API callbacks. The system's built-in user feedback mechanism can record detailed user satisfaction scores for the diagnostic report, problem resolution status, and additional suggestions. Feedback data is transmitted in real-time to the online learning module, which uses advanced online learning algorithms, such as stochastic gradient descent and reinforcement learning, to dynamically adjust the load analysis model and diagnostic strategy. For example, if users generally report inaccurate load predictions for a certain type of database, the system will automatically adjust the analysis parameters for that database type or introduce new feature variables to improve prediction accuracy.

[0074] Furthermore, the system possesses self-verification and iterative capabilities. After each adjustment, the system verifies the optimization effect in a simulated environment or a small-scale actual deployment to ensure the correctness of the adjustment direction. Through this closed-loop cycle of "analysis-diagnosis-feedback-optimization," the system can continuously adapt to changes in the database environment, improve the accuracy and timeliness of diagnosis, and ultimately form a self-evolving and continuously optimizing ecosystem for automatic database load analysis and assisted diagnosis.

[0075] Example 4

[0076] This embodiment provides a system architecture and specific application scheme for automatic database load analysis and auxiliary diagnosis, such as... Figure 2 As shown, the system for automatic database load analysis and assisted diagnosis includes:

[0077] (1) Data acquisition module: It is used to collect multi-dimensional load data in real time during the operation of the database through a distributed acquisition agent. The multi-dimensional load data includes, but is not limited to, CPU utilization, memory usage, disk I / O read / write speed, network bandwidth utilization, database connection number, query response time, transaction processing speed, lock wait time, cache hit rate and SQL statement execution frequency. The distributed acquisition agent is deployed on the database server and related application server and uses a lightweight communication protocol to interact with the analysis server. The distributed acquisition agent has an intelligent downsampling function. When it detects that the network bandwidth usage is too high, it can downsample some non-critical load data according to the preset priority strategy, so as to reduce the amount of data transmission and improve the data acquisition efficiency while ensuring the integrity of critical data.

[0078] (2) Time window partitioning module: Based on the periodic characteristics of database load, the adaptive time window partitioning algorithm is used to divide the collected load data according to different time granularities, including second, minute, hour and day. The adaptive time window partitioning algorithm dynamically adjusts the size of the time window according to the historical load pattern and real-time load change trend of the database. The time window partitioning algorithm introduces a load change sensitivity coefficient and automatically adjusts the partitioning accuracy of the time window according to the rate of change of load data. When the load changes drastically, the time window is reduced to capture the details of load change more accurately. When the load changes steadily, the time window is increased to reduce the amount of calculation.

[0079] (3) Feature extraction module: used to extract features from the load data in each time window. The extracted features include, but are not limited to, mean, variance, maximum, minimum, median, quantile, slope, volatility and periodicity. The feature extraction process adopts the sliding window technique. In the sliding window technique, a dynamic weight allocation mechanism is introduced. According to the influence of data at different time points on the overall features, different weights are assigned to the data in the sliding window, so that the extracted features can better reflect the actual situation of the database load.

[0080] (4) Pattern recognition module: It is used to perform pattern recognition on the extracted load features using machine learning algorithms, and divide the database load into different pattern categories. The pattern categories include, but are not limited to, normal load mode, burst high load mode, periodic load mode, abnormal low load mode and mixed load mode. The machine learning algorithm adopts the ensemble learning method. In the ensemble learning process, a feature cross-validation mechanism is introduced to cross-validate different feature combinations, select the optimal feature combination for pattern recognition, and improve the accuracy of pattern recognition.

[0081] (5) Anomaly Detection and Diagnosis Module: Based on the identified load patterns, a combination of rule-based anomaly detection and statistical anomaly detection methods is used to detect anomalies in the database load. When an anomaly is detected, the cause of the anomaly is further analyzed. The cause includes, but is not limited to, hardware failure, software configuration error, SQL statement performance problem, concurrent access conflict, and excessively rapid data growth. The anomaly detection method adopts dynamic threshold adjustment technology. In the dynamic threshold adjustment technology, an adaptive learning mechanism is introduced to automatically adjust the anomaly detection threshold according to historical anomaly data and real-time load conditions, thereby improving the sensitivity and accuracy of anomaly detection.

[0082] (6) Report generation module: It is used to generate detailed auxiliary diagnostic reports based on the results of anomaly detection and diagnosis. The report includes, but is not limited to, anomaly description, anomaly occurrence time, anomaly impact range cause analysis and suggested solutions. The auxiliary diagnostic report is displayed using visualization technology. Interactive analysis functions are introduced in the visualization display. Users can perform in-depth analysis of the data in the report through interactive operations, such as zooming in, zooming out and filtering, so that users can quickly locate the problem.

[0083] (7) Data storage and management module: used to store the collected multi-dimensional load data, extracted feature data, identified load patterns and generated auxiliary diagnostic reports; adopts a storage method that combines distributed file system and relational database to classify and store different types of data, thereby improving the efficiency and reliability of data storage; at the same time, data compression and encryption technology is introduced to compress and encrypt the stored data, reduce storage space occupation and ensure data security.

[0084] (8) System monitoring and early warning module: used to monitor the system's operating status in real time, including the operation of each link of data acquisition, processing and analysis; when the system has a fault or abnormality, it will issue an early warning information in a timely manner to notify relevant personnel to handle it; the early warning information includes the fault type, fault occurrence time, fault impact range and other detailed information to facilitate relevant personnel to quickly locate and solve the problem.

[0085] (9) User Management Module: Used to manage users of the system, including user registration, login, and permission allocation functions; based on the user's role and permissions, it provides different operation interfaces and function permissions to ensure the security of the system and the confidentiality of data; at the same time, it introduces user behavior analysis function to analyze user operation behavior, and promptly warn and handle abnormal operations.

[0086] (10) System optimization and upgrade module: used to perform regular optimization and upgrade of the system, including algorithm optimization, function expansion and performance improvement; based on the system operation and user feedback, the system is optimized and upgraded in a targeted manner to improve the system performance and user experience; at the same time, an automatic update mechanism is introduced, and when a new version is released, the system can automatically detect and prompt the user to update.

[0087] In practical applications, during the data acquisition phase, a distributed acquisition agent is deployed on the database server and its associated application servers. The agent uses a lightweight communication protocol to interact with the analysis server and collects multi-dimensional load data in real time during the database operation process, covering key indicators such as CPU utilization, memory usage, disk I / O read / write speed, and network bandwidth utilization. The intelligent downsampling function plays a crucial role when network bandwidth usage is too high. Based on a preset priority strategy, it downsamples some non-critical load data, ensuring the integrity of critical data while reducing data transmission volume and improving data acquisition efficiency, thus providing a high-quality and efficient data foundation for subsequent analysis.

[0088] The collected load data then enters the time window segmentation module. Based on the periodic characteristics of the database load, an adaptive time window segmentation algorithm is adopted. This algorithm dynamically adjusts the size of the time window by combining the historical load patterns and real-time load change trends of the database. By introducing a load change sensitivity coefficient, the segmentation precision of the time window is automatically adjusted according to the rate of change of the load data. When the load changes drastically, the time window is narrowed to more accurately capture the details of the load change; when the load changes steadily, the time window is widened to reduce the computational load. This flexible time window segmentation method allows subsequent feature extraction to obtain more representative feature data for different load conditions. The feature extraction module performs in-depth analysis of the load data within each time window. A sliding window technique is adopted, and a dynamic weight allocation mechanism is introduced. Different weights are assigned to the data within the sliding window according to the degree of influence of data at different time points on the overall features, so that the extracted features, such as mean, variance, maximum, minimum, median, quantiles, slope, volatility, and periodicity, can better reflect the actual situation of the database load. In this way, the data within the time window is transformed into a feature vector with clear meaning, providing effective input for pattern recognition.

[0089] The pattern recognition module utilizes ensemble learning to perform pattern recognition on the extracted load features. During the ensemble learning process, a feature cross-validation mechanism is introduced to cross-validate different feature combinations and select the optimal feature combination for pattern recognition. In this way, the database load is categorized into different modes: normal load, sudden high load, periodic load, abnormal low load, and mixed load. Accurate pattern recognition provides crucial information for subsequent anomaly detection and diagnosis, enabling the system to adopt appropriate detection and diagnostic strategies for different load modes.

[0090] The anomaly detection and diagnosis module employs a combination of rule-based and statistical anomaly detection methods based on identified load patterns. The dynamic threshold adjustment technology incorporates an adaptive learning mechanism, automatically adjusting the anomaly detection threshold based on historical anomaly data and real-time load conditions. When an anomaly is detected, the system further analyzes its causes, including hardware failures, software configuration errors, SQL statement performance issues, concurrent access conflicts, and excessively rapid data growth. The report generation module generates detailed auxiliary diagnostic reports based on the anomaly detection and diagnosis results. These reports are presented using visualization technology and incorporate interactive analysis functions. Users can interact with the report by zooming in, zooming out, and filtering to conduct in-depth analysis of the data, facilitating rapid problem localization. The visualized report presents anomaly descriptions, occurrence times, impact ranges, cause analysis, and suggested solutions in an intuitive manner, enabling users to quickly understand the database load anomaly and take appropriate measures.

[0091] The data storage and management module employs a combination of distributed file systems and relational databases to categorize and store collected multi-dimensional load data, extracted feature data, identified load patterns, and generated auxiliary diagnostic reports. Simultaneously, data compression and encryption technologies are introduced to compress and encrypt stored data. This storage method improves data storage efficiency and reliability, reduces storage space usage, and ensures data security, providing data support for the long-term stable operation of the system. The system monitoring and early warning module monitors the system's operational status in real time, covering all aspects of data acquisition, processing, and analysis. When a system fault or anomaly occurs, it promptly issues early warning information, including detailed information such as the fault type, time of occurrence, and scope of impact. This enables relevant personnel to quickly locate and resolve problems, ensuring the normal operation of the system and reducing business interruptions caused by system failures.

[0092] The user management module provides comprehensive management of system users, including user registration, login, and permission allocation. Based on user roles and permissions, it provides different user interfaces and functional permissions, ensuring system security and data confidentiality. Simultaneously, it introduces user behavior analysis to analyze user actions, promptly issuing warnings and handling abnormal operations to further enhance system security. The system optimization and upgrade module regularly optimizes and upgrades the system based on its operational status and user feedback. This includes algorithm optimization, feature expansion, and performance improvements. An automatic update mechanism is introduced; when a new version is released, the system automatically detects and prompts users to update, enabling the system to continuously adapt to new demands in database load analysis, maintaining advanced technology and a good user experience.

[0093] Example 5

[0094] This embodiment upgrades the functionality of four key modules: data acquisition, time window segmentation, feature extraction, and pattern recognition. This further enhances the system's data integrity, analytical adaptability, feature accuracy, and pattern recognition robustness. Specific details and applications are as follows:

[0095] The distributed acquisition agent in the data acquisition module also has a data caching function. When communication with the analysis server fails, the acquired load data can be temporarily stored in the local cache. After the communication is restored, the cached data is transmitted to the analysis server to ensure the integrity and continuity of the data. The adaptive time window partitioning algorithm in the time window partitioning module also introduces a seasonal factor adjustment mechanism. Based on the changing pattern of database load in different seasons, the partitioning of the time window is seasonally adjusted so that the partitioning of the time window is more in line with the actual changes in database load.

[0096] The sliding window technique in the feature extraction module also has a data smoothing function during the sliding process. It smooths the data within the sliding window, removes noise and outliers, and improves the accuracy of feature extraction. The ensemble learning method in the pattern recognition module integrates a variety of different machine learning algorithms, including decision trees, support vector machines, and neural networks. By combining the recognition results of different algorithms, it improves the accuracy and robustness of pattern recognition.

[0097] In practical applications, under normal communication conditions, the collected load data is transmitted to the analysis server in real time and stably. However, when communication with the analysis server fails, the distributed acquisition agent immediately activates a local caching mechanism to temporarily store the collected load data in a local cache, effectively preventing data loss due to communication interruption and ensuring data integrity and continuity. Once communication is restored, the distributed acquisition agent automatically transmits the cached data to the analysis server in chronological order, guaranteeing data integrity and continuity and providing a reliable data foundation for subsequent analysis and diagnosis.

[0098] After receiving complete load data from the data acquisition module, the time window segmentation module initiates an adaptive time window segmentation algorithm. This algorithm does not use a fixed time window segmentation method but incorporates a seasonal adjustment mechanism. Database load often exhibits different variation patterns in different seasons. For example, during peak business periods (such as e-commerce promotional seasons), the database load increases significantly; while during off-peak seasons, the load is relatively low. The adaptive time window segmentation algorithm dynamically adjusts the time window segmentation based on these seasonal variations. In seasons with drastic load changes, a smaller time window is used to capture instantaneous load changes more precisely; while in seasons with relatively stable loads, the time window is appropriately increased to reduce unnecessary computation and improve analysis efficiency. Through this seasonal adjustment, the time window segmentation more closely reflects the actual changes in database load, providing a more reasonable time granularity for subsequent feature extraction and pattern recognition.

[0099] The feature extraction module uses a sliding window technique to extract features from the load data, based on the time window segmentation module's generated time window. During the sliding process, the sliding window technique not only analyzes data within different time windows one by one but also provides data smoothing. Because database load data is subject to various noises and outliers during acquisition, such as instantaneous network fluctuations and brief hardware failures, noise and outliers can severely affect the accuracy of feature extraction. The data smoothing function effectively removes noise and outliers by performing weighted averaging and filtering operations on the data within the sliding window, making the extracted features more accurate and stable. For example, for query response time data over a period of time, data smoothing can eliminate individual abnormally high response time values, more realistically reflecting the average response performance of the database during that time period, providing high-quality feature input for subsequent pattern recognition.

[0100] Ensemble learning methods do not rely on a single machine learning algorithm, but rather integrate multiple different machine learning algorithms such as decision trees, support vector machines, and neural networks. Each algorithm has its unique advantages and applicable scenarios. Decision trees can intuitively display classification rules in a tree structure, making them easy to understand and interpret; support vector machines have powerful classification capabilities in high-dimensional spaces and can effectively handle complex nonlinear problems; neural networks, through the connection and training of multiple layers of neurons, can automatically learn complex patterns in data. During pattern recognition, the system inputs the same set of feature data into these different algorithms for recognition, and then combines the recognition results from different algorithms. Through weighted voting and averaging ensemble strategies, the recognition results of each algorithm are merged, fully leveraging the advantages of each algorithm and compensating for the shortcomings of a single algorithm, thereby significantly improving the accuracy and robustness of pattern recognition. For example, when a certain algorithm deviates in recognizing a certain type of load pattern, the recognition results of other algorithms can play a corrective role, ensuring the reliability of the final recognition result.

[0101] Example 6

[0102] This embodiment upgrades the functions of four core modules: anomaly detection and diagnosis, data storage and management, system monitoring and early warning, and system optimization and upgrade. Combined with inter-module collaboration mechanisms, it further enhances the system's anomaly detection adaptability, data storage reliability, fault handling efficiency, and update security. It also clarifies the definition and scope of protection of technical terms. Specific details are as follows:

[0103] The rule-based and statistics-based anomaly detection methods in the anomaly detection and diagnosis module also have dynamic update capabilities for rules and statistical models. Based on historical anomaly data and real-time load conditions, the rules and statistical models for anomaly detection are dynamically updated to improve the adaptability and accuracy of anomaly detection. The distributed file system in the data storage and management module employs a redundant storage strategy, performing multiple backups of important data to ensure data security and reliability. Simultaneously, a data lifecycle management function is introduced, hierarchically storing and managing data according to its access frequency and importance, thereby improving data storage efficiency.

[0104] The system monitoring and early warning module also has a fault self-diagnosis function. When a system fault occurs, it can automatically diagnose the fault, analyze the cause of the fault, and provide corresponding solutions to facilitate relevant personnel to quickly resolve the fault. The automatic update mechanism in the system optimization and upgrade module also has an update risk assessment function. Before updating the system, it conducts a risk assessment on the update content and analyzes the impact of the update, such as data compatibility and system stability, to ensure the safety and reliability of the update process.

[0105] In practical applications, the system employs a dual-engine collaborative detection mechanism in the anomaly detection and diagnosis phase. A rule-based anomaly detection engine performs pattern matching on load metrics using a pre-defined rule base, while a statistical anomaly detection engine uses a dynamic threshold algorithm to determine anomalies in time-series data. Both engines share a historical anomaly database as training samples and continuously optimize the detection model through machine learning algorithms. When real-time load data flows in, the system first performs rule matching. If no rule threshold is triggered, statistical detection is initiated to calculate the deviation of the current metric from the dynamic baseline. During the detection process, the system continuously collects newly emerging anomaly patterns. After manual confirmation, it automatically updates the rule base and statistical model parameters, forming a closed-loop mechanism of "detection-feedback-optimization." This dynamic update mechanism enables the system to adapt to constantly changing load characteristics, such as automatically adjusting detection sensitivity during peak business periods and recalibrating baseline parameters after system upgrades.

[0106] The data storage and management layer employs a three-tier redundancy architecture. Raw data is first written to the primary storage node of the distributed file system, and simultaneously replicated on backup nodes in two different racks via an asynchronous replication mechanism. For critical business data, the system implements cross-datacenter synchronous replication to ensure that no data loss occurs due to the failure of any single datacenter. The data lifecycle management module categorizes data into three tiers—hot, warm, and cold—based on access frequency: frequently accessed hot data is stored on SSDs with multiple replicas; moderately accessed warm data is stored on large-capacity hard drives with a single replica; and infrequently accessed cold data is automatically migrated to object storage and compressed for archiving. The system runs a storage efficiency analysis program every 24 hours, dynamically adjusting storage strategies based on data access patterns, such as downgrading hot data that has not been accessed for 30 consecutive days to warm data storage.

[0107] The system monitoring and early warning module constructs a three-dimensional monitoring network. By deploying monitoring agents at key nodes such as database servers, storage devices, and network switches, it collects over 200 metrics in real time, including CPU utilization, memory usage, disk I / O, and network latency. The monitoring data is analyzed in real time by a streaming engine. When an anomaly is detected, the fault self-diagnosis subsystem immediately initiates multi-dimensional source tracing analysis: first, it uses time series correlation analysis to locate the time window of the anomaly; then, it uses root cause analysis algorithms to construct a fault propagation graph, identifying the initial fault point and its impact path. For example, when a sudden increase in query response time is detected, the system simultaneously checks the CPU load, memory usage, and lock status of relevant nodes, ultimately determining that it is caused by an anomaly in the execution plan of a specific SQL statement, and generating a diagnostic report containing optimization suggestions.

[0108] The system optimization and upgrade module implements a risk-controlled update strategy. Before initiating an update, the update risk assessment subsystem performs triple verification: first, it checks the compatibility of the new version with existing components using dependency analysis tools; second, it runs a regression test suite to verify core functionality; and finally, it performs stress tests in a simulated environment to assess system stability. The assessment report includes a detailed risk matrix, clearly marking the affected functional modules and key indicators such as expected recovery time. If the risk level exceeds a preset threshold, the system automatically triggers a rollback plan, reverting the component version to its stable state before the update. For critical patches, the system adopts a canary release strategy, deploying them first on a small number of nodes and monitoring their operation for 24 hours before full rollout.

[0109] All modules communicate and collaborate via a unified event bus. When the anomaly detection module detects a severe load anomaly, it simultaneously triggers the alarm mechanism of the monitoring and early warning module and the data protection process of the storage management module: the former sends an alarm notification containing fault location information to the operations and maintenance personnel, while the latter immediately initiates additional backups of critical data. The system optimization module adjusts resource configuration strategies based on the anomaly type, such as automatically increasing memory allocation for database instances that frequently experience query timeouts.

Claims

1. A method for automatic analysis and aided diagnosis of database load, characterized in that, Comprise: (1) Multi-dimensional data collection: through the deployment of distributed collection agent on the server, real-time collection of multi-dimensional load data in the database running, and data interaction using lightweight communication protocol; (2) Dynamic time window division: using adaptive time window division algorithm, the collected multi-dimensional load data is divided according to different time granularity; the adaptive time window division algorithm dynamically adjusts the size of the time window according to the historical load mode and real-time load trend of the database; (3) Load feature extraction: load data in each time window is extracted; (4) Load pattern recognition: using integrated learning algorithm to identify the load features extracted, and dividing the database load into different categories; (5) Abnormal detection and diagnosis: based on the load category, combining rule and statistical anomaly detection method to detect the abnormality of the database load, and further analyzing the cause of the abnormality; the abnormal detection method uses dynamic threshold adjustment technology to automatically adjust the threshold of abnormal detection according to the real-time load of the database; (6) Auxiliary diagnosis report generation: based on the results of abnormal detection and diagnosis, generating visual auxiliary diagnosis report; (7) Prediction and early warning: based on the historical load data and current load trend, using time series analysis method and prediction algorithm to predict the future load of the database; the prediction algorithm uses online learning technology to update model parameters in real time.

2. The database load automatic analysis and aided diagnosis method according to claim 1, characterized in that, In step (1), the multi-dimensional data collection further comprises data preprocessing; the data preprocessing comprises data cleaning, data normalization and data missing value filling; the data cleaning uses statistical and rule-based method to clean up the noise data and abnormal values in the collected load data; the data normalization uses normalization method to unify the load data of different dimensions to the same dimension range; the data missing value filling uses interpolation method or prediction method to handle the missing data in the collection process.

3. The database load automatic analysis and aided diagnosis method according to claim 1, characterized in that, In step (4), the load pattern recognition further comprises load pattern mining, which finds the association relationship and evolution law between load patterns, and uses association rule mining algorithm and sequence pattern mining algorithm to mine the appearance frequency, sequence and mutual influence relationship of different load patterns in different time periods.

4. The database load automatic analysis and aided diagnosis method according to claim 1, characterized in that, The method also supports load analysis of multiple types of databases, including relational databases, non-relational databases and distributed databases, and uses corresponding data collection interface and analysis strategy for different types of databases to analyze and diagnose various database loads.

5. The method of automatic analysis and aided diagnosis of database load according to claim 1, characterized in that, In step (6), the auxiliary diagnosis report generation further comprises real-time feedback and adjustment; the real-time feedback and adjustment adjusts the load analysis and diagnosis strategy in real time according to the feedback opinions of users and actual processing effect, and optimizes the load analysis and diagnosis strategy using online learning algorithm.

6. A database load automatic analysis and aided diagnosis system, characterized in that, Comprise: Data collection module: through the distributed collection agent, real-time collection of multi-dimensional load data in the database running process; The distributed collection agent is deployed on a database server and an associated application server, adopts a lightweight communication protocol to interact with an analysis server, and has an intelligent downsampling function, which can perform downsampling processing on part of non-key load data according to a preset priority strategy when detecting that network bandwidth occupation is too high. The time window division module divides the collected load data according to different time granularities based on the periodic characteristics of the database load, by using an adaptive time window division algorithm. The adaptive time window division algorithm dynamically adjusts the size of the time window according to the historical load mode and real-time load change trend of the database, and introduces a load change sensitivity coefficient to automatically adjust the division precision of the time window according to the change rate of the load data. When the load changes dramatically, the time window is reduced. When the load changes smoothly, the time window is increased. The feature extraction module extracts features of the load data in each time window, including mean, variance, maximum, minimum, median, quantile, slope, volatility, and periodicity intensity. The sliding window technology is used to introduce a dynamic weight distribution mechanism, and different weights are assigned to the data in the sliding window according to the influence degree of the data at different time points on the overall features. The pattern recognition module uses an ensemble learning algorithm to recognize the load features, and divides the database load into different pattern categories. The pattern categories include normal load pattern, sudden high load pattern, periodic load pattern, abnormal low load pattern, and mixed load pattern. The ensemble learning introduces a feature cross-validation mechanism to cross-validate different feature combinations, and selects the optimal feature combination for pattern recognition. The anomaly detection and diagnosis module uses a combination of rule-based and statistical anomaly detection methods to detect anomalies in the database load based on the recognized load patterns. When an anomaly is detected, the cause of the anomaly is further analyzed. The anomaly detection method uses a dynamic threshold adjustment technique and introduces an adaptive learning mechanism to automatically adjust the threshold of anomaly detection according to historical anomaly data and real-time load conditions. The report generation module generates a visual auxiliary diagnosis report based on the anomaly detection and diagnosis results. The report includes anomaly description, anomaly occurrence time, anomaly impact range cause analysis, and recommended solutions. An interactive analysis function is introduced in the visual display, and users can perform in-depth analysis on the data in the report through interactive operations. The data storage and management module stores the collected multi-dimensional load data, extracted feature data, recognized load patterns, and generated auxiliary diagnosis report. A combination of a distributed file system and a relational database is used to store different types of data in a classified manner. Data compression and encryption techniques are introduced to compress and encrypt the stored data. The system monitoring and early warning module monitors the running state of the system in real time, including data collection, processing and analysis of the operation of each link, and when the system fails or abnormally, timely early warning information is sent to inform relevant personnel to handle; the early warning information includes fault type, fault occurrence time and fault influence range information; The user management module manages the users of the system, including user registration, login and permission allocation functions, and provides different operation interfaces and function permissions for users according to their roles and permissions, while introducing user behavior analysis function to analyze user operation behavior and find abnormal operation for timely early warning and handling; The system optimization and upgrading module periodically optimizes and upgrades the system, including algorithm optimization, function expansion and performance improvement, and optimizes and upgrades the system according to the running situation and user feedback of the system, while introducing an automatic update mechanism, which can automatically detect and prompt users to update when a new version is released.

7. The database load automatic analysis and diagnostic assistance system of claim 6, wherein, The distributed collection agent in the data collection module also has data caching function, which temporarily stores the load data in local cache when the communication with the analysis server fails, and transmits the cached data to the analysis server after the communication is restored; the adaptive time window division algorithm in the time window division module also introduces a seasonal factor adjustment mechanism to adjust the time window according to the seasonal variation law of database load.

8. The database load automatic analysis and diagnostic assistance system of claim 6, wherein, The sliding window of the feature extraction module has data smoothing function, which smoothes the data in the sliding window to remove noise and outliers in the data; the integrated learning method of the pattern recognition module integrates decision tree, support vector machine and neural network algorithm.

9. The database load automatic analysis and diagnostic assistance system of claim 6, wherein, The rule and statistics based anomaly detection method of the anomaly detection and diagnosis module also has dynamic updating function, which dynamically updates the rules and statistical models of anomaly detection according to historical anomaly data and real-time load; the distributed file system of the data storage and management module adopts redundant storage strategy to backup important data, and introduces data life cycle management function to store and manage data according to access frequency and importance.

10. The database load automatic analysis and diagnostic assistance system of claim 6, wherein, The system monitoring and early warning module also has fault self-diagnosis function, which can automatically diagnose faults and analyze the root cause; the automatic update mechanism of the system optimization and upgrading module also has update risk assessment function, which evaluates the impact of update before update.

Images