Unified real-time monitoring and operation optimization method and system for service-oriented multi-type databases

Abstract

The application provides a service-oriented multi-type database unified real-time monitoring and operation optimization method and system, which establishes a distributed data collection node network for relational, document and time series databases, receives running state data streams of each database instance, performs standardized conversion processing across database types, generates standardized state data sequences with unified data formats and timestamp alignment, constructs a database performance correlation graph based on the standardized state data sequences, calculates performance influence factors of each database instance through the dependency relationship weight between graph nodes, performs database abnormal pattern recognition according to the performance influence factors and historical baseline data of the standardized state data sequences, generates an abnormal diagnosis result, and generates an operation optimization scheme by calling a preset operation strategy generation rule library. The application improves the intelligent level and operation efficiency of database operation.

Description

Technical Field

[0001] This invention relates to the field of data processing, and more specifically, to a method and system for unified real-time monitoring and operation and maintenance optimization of service-oriented multi-type databases. Background Technology

[0002] As enterprises deepen their digital transformation, the industry currently relies heavily on independent monitoring tools that connect to different types of databases. These tools acquire operational status data through common data collection protocols, depend on human experience for anomaly detection, and implement maintenance strategies primarily based on pre-set scripts or post-event manual intervention. However, these independent tools suffer from data silos, making cross-database integration difficult. Common data collection protocols struggle to adapt to the native characteristics of various databases, easily resulting in the loss of critical performance data. Data standardization often remains at the format conversion level, failing to effectively preserve performance correlation characteristics. Furthermore, maintenance strategies lack precise matching to dynamic anomaly scenarios, limiting response efficiency and optimization effectiveness. Summary of the Invention

[0003] This invention provides a method and system for unified real-time monitoring and operation and maintenance optimization of service-oriented multi-type databases.

[0004] In a first aspect, embodiments of the present invention provide a unified real-time monitoring and operation and maintenance optimization method for service-oriented multi-type databases. The method includes: establishing a distributed data acquisition node network for relational, document-oriented, and time-series databases; receiving runtime status data streams from each database instance through a protocol adaptation interface; the runtime status data streams include database connection session records, structured query statement execution trajectories, and storage resource usage snapshots; performing cross-database type standardization conversion processing on the runtime status data streams to generate a standardized status data sequence with a unified data format and timestamp alignment; the standardized status data sequence includes connection feature vectors, query behavior matrices, and resource consumption tensors; and based on the standardized status data... A database performance correlation graph is constructed based on the sequence. The performance impact factor of each database instance is calculated by the dependency weights between graph nodes. The performance impact factor is used to characterize the propagation path of performance disturbances between different database instances. Based on the performance impact factor and historical baseline data of the standardized state data sequence, database anomaly pattern recognition is performed to generate anomaly diagnosis results that include anomaly occurrence probability distribution and impact range assessment. Based on the anomaly occurrence probability distribution and impact range assessment in the anomaly diagnosis results, a pre-set operation and maintenance strategy is invoked to generate a rule base, generating an operation and maintenance optimization plan that includes resource scheduling instructions and query optimization suggestions. The operation and maintenance optimization plan is then pushed to the database management platform for automated operation and maintenance.

[0005] Secondly, embodiments of the present invention provide a computer system, including: a memory storing a computer program; and a processor for loading the computer program to implement the service-oriented multi-type database unified real-time monitoring and operation and maintenance optimization method as described above.

[0006] This invention establishes a distributed data acquisition node network for relational, document-oriented, and time-series databases, achieving native compatibility in acquiring operational status data streams from different database types. This effectively solves the problems of low data acquisition efficiency or data loss caused by differences in protocols among multiple database types. By standardizing the operational status data streams across database types, heterogeneous data is converted into multimodal features such as connection feature vectors, query behavior matrices, and resource consumption tensors, preserving performance correlation features across different dimensions. A database performance correlation graph is constructed based on the standardized status data sequence. The performance impact factor of each database instance is calculated through the dependency weights between graph nodes, achieving quantitative analysis of the propagation path of performance disturbances between database instances. Database anomaly pattern identification is performed based on the performance impact factor and historical baseline data of the standardized status data sequence. Combining multi-dimensional deviation calculation and impact factor weighting improves the accuracy of anomaly identification and enables a comprehensive assessment of the anomaly's impact range. Based on the anomaly diagnosis results, a pre-set operation and maintenance strategy rule base is invoked to generate an operation and maintenance optimization plan containing resource scheduling instructions and query optimization suggestions. This ensures precise matching of operation and maintenance strategies with anomaly scenarios, improving the intelligence level and operational efficiency of database operation and maintenance. Attached Figure Description

[0007] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0008] Figure 1 This is a flowchart of a unified real-time monitoring and operation and maintenance optimization method for service-oriented multi-type databases provided in an embodiment of the present invention.

[0009] Figure 2 This is a schematic diagram of the composition of a computer system provided in an embodiment of the present invention. Detailed Implementation

[0010] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0011] Please see Figure 1 This is a flowchart of a unified real-time monitoring and operation and maintenance optimization method for service-oriented multi-type databases provided in an embodiment of the present invention. The method can be executed by a computer system and includes the following steps: Step S100: Establish a distributed data acquisition node network for relational, document-oriented, and time-series databases. Receive the running status data streams of each database instance through a protocol adaptation interface. The running status data streams include database connection session records, structured query statement execution traces, and storage resource usage snapshots.

[0012] A distributed data acquisition node network consists of multiple data acquisition nodes located in different places. These nodes can collect data in parallel, improving the efficiency and reliability of data acquisition. Relational databases are databases based on the relational model, storing data through tables, such as MySQL and Oracle. Document-oriented databases store data in units of documents, such as using JSON or XML formats, like MongoDB. Time-series databases are specifically designed for processing time-series data, such as InfluxDB, suitable for storing and querying data that changes over time.

[0013] When establishing a distributed data acquisition node network, the data acquisition nodes need to be deployed reasonably according to the distribution and scale of the database. For example, for a large-scale distributed database system, one or more data acquisition nodes can be deployed in each data center or cluster. The implementation of the protocol adaptation interface can adopt a plug-in approach, developing corresponding plugins for different database protocols. When receiving runtime status data streams, the data acquisition nodes establish connections with each database instance through the protocol adaptation interface to obtain data such as database connection session records, structured query statement execution traces, and storage resource usage snapshots in real time.

[0014] Step S200: Perform cross-database type standardization transformation on the running status data stream to generate a standardized status data sequence with a unified data format and timestamp alignment. The standardized status data sequence includes connection feature vectors, query behavior matrices, and resource consumption tensors.

[0015] In one implementation, step S200 may specifically include the following steps S210 to S260: Step S210: Parse the protocol identification information of the running status data stream, identify the data stream source of relational, document, and time-series databases, and call the corresponding data parsing program to convert the original data records of different databases into intermediate format data units containing field names, data types, and value ranges. The field configuration of the intermediate format data units is adapted to the characteristics of the database type.

[0016] When parsing the protocol identification information of the runtime status data stream, the data source can be determined by analyzing the specified fields or identifiers in the data stream. After identifying the data stream source, the corresponding data parsing program is invoked to parse the raw data records. The data parsing program will convert the raw data records into intermediate format data units according to the database type and structure. For example, for a relational database, the data parsing program may convert each row of data in the table into an intermediate format data unit, where each field corresponds to a column in the table.

[0017] Step S220: Extract the time attribute information from the intermediate format data unit, perform time base unification processing on the transaction commit time of the relational database, the update time of the document database, and the acquisition time of the time-series database, and calibrate the time deviation of each database through the system clock synchronization mechanism to generate a time calibration data unit with a unified time mark.

[0018] Time attribute information refers to time-related information in intermediate format data units, such as transaction commit time, update time, and data acquisition time. Time base unification processing standardizes time information from different types of databases to a single time base, eliminating time discrepancies. The system clock synchronization mechanism is used to calibrate the clocks of different database systems, ensuring time consistency across databases. Time calibration data units are data units that have undergone time base unification processing and clock synchronization, possessing a unified time stamp.

[0019] When extracting time attribute information from intermediate format data units, time information can be obtained by searching the fields within the data unit. For example, for relational databases, the transaction commit time might be stored in a dedicated field. When performing time base unification processing on the transaction commit times of relational databases, the update times of document databases, and the acquisition times of time-series databases, a unified time base, such as Coordinated Universal Time (UTC), can be selected. Through system clock synchronization mechanisms, the Network Time Protocol (NTP) can be used to calibrate the system clocks of various databases, reducing time deviations. Finally, the calibrated time information is added to the intermediate format data unit to generate a time calibration data unit.

[0020] Step S230: Perform feature structure integration on the connection session records in the time calibration data unit, convert non-numerical data such as connection status identifier, authentication method and protocol version into classification coding vectors, and perform interval mapping processing on numerical data such as connection duration and data transmission volume to generate a connection feature vector containing a comprehensive description of the connection status. Each dimension of the connection feature vector corresponds to a standardized representation of a connection attribute.

[0021] When performing feature-structured integration of connection session records in the time calibration data unit, non-numerical data such as connection status identifiers, authentication methods, and protocol versions are first classified and encoded. For example, the connection status identifier "connected" can be encoded as 1, and "disconnected" as 0. Authentication methods can be numbered according to different authentication methods. Then, numerical data such as connection duration and data transmission volume are processed using interval mapping. For example, connection duration can be mapped to the [0,1] interval, using a linear mapping based on the maximum and minimum values ​​of the connection duration. Finally, the classified and encoded non-numerical data and the interval-mapped numerical data are merged to generate a connection feature vector.

[0022] Step S240: Parse the execution trajectory of the structured query statement, identify the syntax of the query operation, the data tables involved and the conditional filtering logic, and convert the query statements of different databases into a unified query operation model through abstract syntax tree transformation. Statistically calculate the execution frequency and average time consumption of various types of queries per unit time and the size of the result set to construct a query behavior matrix containing multi-dimensional features of query behavior. The row dimension of the query behavior matrix corresponds to the query operation category and the column dimension corresponds to the time slice segment.

[0023] When parsing the execution trajectory of a structured query statement, a parser can be used to analyze the query statement, identifying its syntactic structure, involved data tables, and conditional filtering logic. Through abstract syntax tree (AST) transformation, the query statement is converted into an AST, which is then transformed into a unified query operation model. When calculating the execution frequency, average execution time, and result set size of various query types within a unit of time, the query execution trajectory can be grouped according to time slices, and then relevant information for each type of query within each time slice can be collected. Finally, the statistical results are used to construct a query behavior matrix.

[0024] To facilitate execution, for each query operation category and each time slice segment, observations of execution frequency, average time consumption, and result set size are obtained. Each observation is then regularized to eliminate differences in units. Based on the regularized execution frequency, average time consumption, and result set size, a comprehensive query behavior feature value is calculated using a weighted fusion algorithm. The comprehensive query behavior feature value is then filled in with each query operation category as rows and each time slice segment as columns to construct a query behavior matrix. Each element of the query behavior matrix represents the overall behavioral intensity of the corresponding query type within the corresponding time period.

[0025] Step S250: Collect hardware resource usage information from the storage resource usage snapshot, including storage access characteristics, memory usage characteristics, and computing resource usage characteristics. Map heterogeneous resource data to a unified resource measurement framework through resource dimension alignment processing. Integrate the data of time, resource type, and database partition into a resource consumption tensor. Each element of the resource consumption tensor corresponds to the resource usage of the corresponding time slice, resource type, and partition.

[0026] When collecting hardware resource usage information from storage resource usage snapshots, relevant data can be obtained through system monitoring tools or the database's built-in monitoring functions. For example, operating system performance monitoring tools can be used to obtain information such as CPU utilization and memory usage, while database monitoring views can be used to obtain information such as storage access characteristics. Through resource dimension alignment processing, different types of resource data are mapped to a unified resource measurement framework. For example, storage access characteristics, memory usage characteristics, and computing resource usage characteristics can be uniformly converted into percentage forms for comparison. Finally, the data from the three dimensions of time, resource type, and database partition are integrated into a resource consumption tensor. Resource data can be grouped according to time slices, with each time slice corresponding to a two-dimensional matrix. The rows of the matrix represent resource types, the columns represent database partitions, and the elements in the matrix represent the resource usage for that time slice, resource type, and partition.

[0027] Step S260: Integrate the connection feature vector, query behavior matrix, and resource consumption tensor; perform cross-field verification according to preset data verification rules; supplement missing data using a prediction and filling method based on historical data from similar databases; correct outliers using a mean replacement method within the same period; and generate a standardized state data sequence containing data integrity verification identifiers. The data integrity verification identifiers are generated through the verification and calculation of all fields.

[0028] When fusing join feature vectors, query behavior matrices, and resource consumption tensors, they can be combined according to certain rules. For example, join feature vectors can be used as the first part, query behavior matrices as the second part, and resource consumption tensors as the third part. When performing cross-field validation based on preset data validation rules, the logical relationships between different fields in the dataset are checked. For example, the relationship between query execution frequency in the query behavior matrix and computational resource usage in the resource consumption tensor is checked to see if they match expectations. When supplementing missing data using a predictive imputation method based on historical data from similar databases, historical data from similar databases is first collected, then a predictive model is built to predict the values ​​of missing data based on the historical data. When correcting outliers using the same-period mean replacement method, the average value of other data within the same period is calculated, and this average value is used to replace the outliers. Finally, the checksum of all fields is calculated, a data integrity check identifier is generated, and it is added to the standardized state data sequence.

[0029] Step S300: Construct a database performance correlation graph based on the standardized state data sequence. Calculate the performance impact factor of each database instance through the dependency weights between graph nodes. The performance impact factor is used to characterize the propagation path of performance disturbances between different database instances.

[0030] In one implementation, step S300 may specifically include the following steps S310 to S360: Step S310: Extract database instance metadata from the standardized state data sequence, including the instance's unique identifier, deployment node address, and service port information. Create an initial association graph with the database instance as the core node and data interaction relationships as edges. The node attributes of the initial association graph include database type labels and deployment physical location information.

[0031] When extracting database instance metadata from a normalized state data sequence, relevant information can be obtained by parsing the fields in the normalized state data sequence. For example, the instance's unique identifier might be stored in a dedicated field. When creating the initial association graph, each database instance is treated as a node, and edges are established based on data interaction relationships. Data interaction relationships can be determined by analyzing query behavior, connection session records, and other information in the normalized state data sequence. For example, if a query operation of one database instance involves data from another database instance, an edge is established between the two nodes. Adding database type labels and deployment physical location information to nodes can be obtained from the normalized state data sequence or other configuration files.

[0032] Step S320: Analyze the cross-database query records in the query behavior matrix, identify the interaction relationship between the query initiating instance and the target instance, count the number of cross-database query requests, average response latency, and data transmission volume per unit time, and generate query dependency weights that characterize the coupling strength between instances. The query dependency weights are positively correlated with the cross-database query frequency and negatively correlated with the average response latency.

[0033] In one implementation, step S320 may specifically include the following steps S321 to S326: Step S321: Traverse each row of the query behavior matrix, identify query operation records containing cross-database identifiers, extract the target database instance name and access data table information from the query statement to determine the bidirectional interaction relationship between the query initiating instance and the target instance, and generate a query interaction triplet containing the initiator, receiver and interaction type.

[0034] When traversing each row of the query behavior matrix, check if each record contains a cross-database identifier. If it does, extract the target database instance name and accessed table information from the query statement. Based on this information, determine the bidirectional interaction relationship between the query initiating instance and the target instance, i.e., identify the initiator and receiver. Determine the interaction type based on the query statement type and generate a query interaction triplet. For example, if the query statement is a SELECT statement, the initiator is database instance A, and the receiver is database instance B, then the generated query interaction triplet is (A, B, SELECT).

[0035] Step S322: Using the system sampling period as the time unit, divide the query interaction triplet into time windows, count the number of query requests between the same initiator and receiver within each time window to generate a query request frequency sequence, and the length of the time window is the same as the time granularity of the standardized state data sequence.

[0036] When dividing query interaction triples into time windows based on the system sampling period, the triples are assigned to different time windows according to the time information within them. To count the number of query requests between the same initiator and receiver within each time window, the query interaction triples within each time window are traversed, and the number of query requests between the same initiator and receiver is counted. The number of query requests in each time window is recorded to generate a query request frequency sequence.

[0037] Step S323: Calculate the response latency of a single query in each query interaction triplet. The response latency is the time interval from the time the query is initiated to the time the result is returned. Calculate the mean and variance of the response latency of all queries within the time window to generate an average response latency sequence. The average response latency sequence reflects the stability characteristics of the query interaction.

[0038] When calculating the response latency of a single query in each query interaction triplet, the query initiation time and result return time are obtained from the query interaction triplet, and the time interval between them is calculated. To calculate the mean and variance of the response latency for all queries within a time window, the query interaction triplets within each time window are traversed, the response latency of each query is summed, and then divided by the number of queries to obtain the average response latency. The square of the difference between the response latency of each query and the average response latency is calculated, these squares are summed, and then divided by the number of queries to obtain the variance. The average response latency and variance for each time window are recorded to generate an average response latency sequence.

[0039] Step S324: Parse the data volume field in the query response results, count the number of records returned and the data size for each query, calculate the total data transmission volume and the average single transmission volume within the time window to generate a data transmission volume sequence, and the ratio of the data transmission volume sequence to the query request frequency sequence reflects the data load of a single query.

[0040] When parsing the data volume field in the query response results, the number of records and data size information are extracted. The number of records and data size returned for each query are counted, and then the number of records and data size within each time window are added together to obtain the total data transfer volume. The total data transfer volume is divided by the number of queries to obtain the average transfer volume per query. The total data transfer volume and average transfer volume per query for each time window are recorded to generate a data transfer volume sequence.

[0041] Step S325: Associate the query request frequency sequence, average response delay sequence, and data transmission volume sequence. Calculate the correlation coefficient of the three sequences using a sliding window to identify the positive correlation between query request frequency and response delay, and the positive correlation between data transmission volume and response delay, generating a cross-database query interaction feature set.

[0042] When linking the query request frequency sequence, average response latency sequence, and data transfer volume sequence, these three sequences are aligned in chronological order. When calculating the correlation coefficient of the three sequences using a sliding window, within each sliding window, the correlation coefficient between the query request frequency sequence and the average response latency sequence, as well as the correlation coefficient between the data transfer volume sequence and the average response latency sequence, are calculated. If the correlation coefficient is positive, it indicates a positive correlation. The correlation coefficient for each sliding window is recorded to generate a cross-database query interaction feature set. For example, using the Pearson correlation coefficient to calculate the correlation, within one sliding window, the Pearson correlation coefficient between the query request frequency sequence and the average response latency sequence is 0.8, indicating a strong positive correlation between them.

[0043] Step S326: Based on the cross-database query interaction feature set, construct a query dependency evaluation model. The query request frequency, average response latency, and data transmission volume are used as input parameters to generate query dependency weights through linear combination. In the calculation of query dependency weights, the weight coefficients that adapt to the performance impact are used to highlight the response latency factor.

[0044] When constructing a query dependency evaluation model based on cross-database query interaction feature sets, linear regression models or other machine learning models can be used. Query request frequency, average response latency, and data transfer volume are used as input parameters, with weighting coefficients adjusted to reflect their performance impact. For example, a weighting coefficient of 0.3 for query request frequency, 0.5 for average response latency, and 0.2 for data transfer volume can be set. The input parameters are multiplied by their respective weighting coefficients and then summed to obtain the query dependency weight. This approach highlights the influence of average response latency on the query dependency weight.

[0045] Step S330: Analyze the resource contention relationship in the resource consumption tensor, identify the set of database instances sharing the storage resources, calculate the resource contention coefficient and the resource waiting queue length to generate resource dependency weights based on the resource contention relationship. The resource dependency weights are positively correlated with the resource contention coefficient.

[0046] When analyzing resource contention relationships in the resource consumption tensor, the resource usage of different database instances within the tensor at the same time slice, resource type, and partition is analyzed. If multiple database instances simultaneously exceed the resource's capacity in their usage of the same resource, resource contention exists. When identifying a set of database instances sharing storage resources, the database instances using the same storage resource are located. The resource contention coefficient can be calculated based on resource usage and capacity. For example, if the resource capacity is 100 and multiple database instances simultaneously use 80 units, the resource contention coefficient is 80 / 100 = 0.8. When calculating the resource waiting queue length, the number of database instances waiting to use the resource is counted. Resource dependency weights are generated based on the resource contention coefficient and the resource waiting queue length. A linear combination method can be used, taking the resource contention coefficient and the resource waiting queue length as input parameters, and calculating the resource dependency weights through linear combination.

[0047] Step S340: Integrate query dependency weights and resource dependency weights, and calculate the comprehensive dependency weights of each edge connection in the initial association graph using a linear combination algorithm. The range of comprehensive dependency weights is mapped to the interval of 0 to 1 through normalization.

[0048] When merging query dependency weights and resource dependency weights, a linear combination algorithm is used. The weight coefficient for the query dependency weight is set to 'a', and the weight coefficient for the resource dependency weight is set to 'b', with a + b = 1. The query dependency weight is multiplied by 'a', and the resource dependency weight is multiplied by 'b', and then the results are added together to obtain the overall dependency weight. For example, if a = 0.6, b = 0.4, the query dependency weight is 0.8, and the resource dependency weight is 0.7, then the overall dependency weight is 0.6 × 0.8 + 0.4 × 0.7 = 0.76. When normalizing the overall dependency weight to the interval between 0 and 1, a minimum-maximum normalization method can be used. The minimum and maximum values ​​of the overall dependency weight are found, each weight is subtracted from the minimum value, and then divided by the difference between the maximum and minimum values ​​to obtain the normalized overall dependency weight.

[0049] Step S350: Perform edge filtering on the initial association graph based on the comprehensive dependency weights, remove weak dependency connections with weight values ​​below a preset threshold to generate a simplified database performance association graph, and retain key dependencies in the simplified database performance association graph.

[0050] When performing edge filtering on the initial association graph based on comprehensive dependency weights, each edge in the initial association graph is traversed, and its comprehensive dependency weight is checked. If the weight value is lower than a preset threshold, the edge is removed from the graph. For example, if the preset threshold is 0.3, and an edge has a comprehensive dependency weight of 0.2, it is removed. After edge filtering, a simplified database performance association graph is obtained, which retains edges with weight values ​​higher than the preset threshold, i.e., key dependencies.

[0051] Step S360: Calculate the node influence of the database performance correlation graph using the node influence propagation algorithm. Starting from the initial node, propagate the influence value according to the edge weight. After multiple iterations, generate the performance influence factor of each database instance. The performance influence factor represents the degree of influence of the instance on the overall system performance.

[0052] In one implementation, step S360 may specifically include the following steps S361 to S366: Step S361: Initialize the influence value of all nodes in the database performance correlation graph, set the initial influence value of each node to a uniform baseline value, and match the sum of the initial influence values ​​with the total number of nodes in the graph.

[0053] Initializing node influence values ​​provides each node with an initial influence metric for influence propagation calculations. A unified baseline value is a pre-defined influence value set to this value for all nodes. The sum of the initial influence values ​​matches the total number of nodes in the graph, ensuring that the total influence remains constant during influence propagation. When initializing the influence values ​​of all nodes in the database performance association graph, the initial influence value of each node is set to the unified baseline value.

[0054] Step S362: Select the node with the largest sum of edge weights in the graph as the initial propagation source node. Distribute the influence value of the initial propagation source node to the adjacent nodes according to the weight ratio of the connecting edges. The influence distribution value obtained by each adjacent node is equal to the current influence value of the propagation source node multiplied by the ratio of the corresponding edge weight to the sum of all edge weights of the propagation source node.

[0055] When selecting the node with the largest sum of edge weights in the graph as the initial propagation source node, each node in the graph is traversed, and its sum of edge weights is calculated. The node with the largest sum of edge weights is then selected as the initial propagation source node. When distributing the influence value of the initial propagation source node to adjacent nodes according to the weight ratio of the connecting edges, for each adjacent node, the ratio of its corresponding edge weight to the sum of all edge weights of the propagation source node is calculated. Then, the current influence value of the propagation source node is multiplied by this ratio to obtain the influence allocation value obtained by that adjacent node.

[0056] Step S363: In each propagation iteration, all nodes simultaneously receive the influence allocation value from their neighboring nodes, and add the received value to their current influence value to obtain a new influence value. The number of propagation iterations is a preset multiple of the graph diameter to ensure that the influence covers all nodes.

[0057] Propagation iteration refers to the process of influence spreading multiple times in the graph. In each iteration, all nodes simultaneously receive influence allocation values ​​from their neighboring nodes. The graph diameter is the maximum length of the shortest path between any two nodes in the graph. The number of propagation iterations is a preset multiple of the graph diameter to ensure that influence can cover all nodes in the graph.

[0058] In each propagation iteration, all nodes simultaneously receive influence allocation values ​​from their neighboring nodes. For each node, the received value is added to its current influence value to obtain a new influence value. The number of propagation iterations is determined by a preset multiple of the graph diameter. For example, if the graph diameter is 2 and the preset multiple is 3, then the number of propagation iterations is 2 × 3 = 6, ensuring that influence covers all nodes.

[0059] Step S364: During the propagation iteration process, the influence value of each node is decayed. After each iteration, the current influence value of the node is multiplied by the decay coefficient. The decay coefficient is set according to the characteristics of the database type of the node. The decay coefficient of relational database is lower than that of document-type and time-series database.

[0060] Attenuation processing simulates the loss of influence during propagation, preventing its unlimited spread. The attenuation coefficient is a value less than 1 used to reduce the influence value of a node. The attenuation coefficient is set according to the characteristics of the node's database type; the attenuation coefficient for relational databases is lower than that for document-oriented and time-series databases because relational databases have relatively stable performance and experience less loss during influence propagation.

[0061] During the propagation iteration process, when attenuating the influence value of each node, the node's current influence value is multiplied by a attenuation coefficient after each iteration. The attenuation coefficient is set according to the characteristics of the node's database type. For relational databases, the attenuation coefficient can be set to 0.9; for document-based databases, the attenuation coefficient can be set to 0.8; and for time-series databases, the attenuation coefficient can be set to 0.85.

[0062] Step S365: Stop the propagation iteration when the change in node influence value between two consecutive iterations is less than the preset convergence threshold, and record the final influence value of each node. The convergence threshold is related to the number of nodes in the graph; the more nodes there are, the smaller the convergence threshold should be.

[0063] During the propagation iteration process, the change in influence value of each node is checked after each iteration. The absolute value of the difference between the influence value of each node in the current iteration and the previous iteration is calculated. If the change in influence value of all nodes is less than a preset convergence threshold, the propagation iteration stops. For example, if the preset convergence threshold is 0.01, and a node has an influence value of 5.01 in the current iteration and an influence value of 5 in the previous iteration, then the change in influence value is |5.01 - 5| = 0.01, which satisfies the convergence condition.

[0064] Step S366: Normalize the final influence value by dividing it by the sum of the influence values ​​of all nodes and converting it into a relative value in the range of 0 to 1. Use the relative value as the performance influence factor of each database instance. The change in the value of the performance influence factor reflects the change in the degree of influence of the instance on the overall system performance.

[0065] When normalizing the final influence value, the sum of the influence values ​​of all nodes is first calculated. Then, the final influence value of each node is divided by the sum of the influence values ​​of all nodes to obtain the normalized relative value. For example, if the sum of the influence values ​​of all nodes is 20, and the final influence value of a node is 5, then the normalized relative value is 5 / 20 = 0.25. The normalized relative value is used as the performance impact factor for each database instance. The larger the performance impact factor value, the greater the impact of that database instance on the overall system performance.

[0066] Step S400: Based on the historical baseline data of the performance impact factor and standardized state data sequence, perform database anomaly pattern recognition to generate anomaly diagnosis results that include anomaly occurrence probability distribution and impact range assessment.

[0067] In one implementation, step S400 may specifically include the following steps S410 to S460: Step S410: Retrieve the standardized status data sequence history from the database historical operation data storage system, filter the historical data within the historical operation period where the system has no abnormal alarms and the performance indicators are within the normal range, divide it into multiple baseline periods according to the time period, and generate a historical baseline data set by each baseline period containing a complete standardized status data sequence. Each baseline period of the historical baseline data set is associated with the corresponding system configuration version information.

[0068] In one implementation, step S410 may specifically include the following steps S411 to S416: Step S411: Connect to the database historical data archiving service, send a query request containing historical running cycles and data types, and receive the returned historical data file set. The historical data file set is compressed and stored according to time units, and each file contains a standardized status data sequence of all database instances within the corresponding time unit.

[0069] When connecting to the database historical data archiving service, appropriate network protocols and interfaces can be used. For example, a connection can be established using a remote connection interface provided by the database management system, employing standard network communication protocols such as TCP / IP. When sending a query request, the historical execution cycle and data type information are encapsulated and sent according to the service's specified format. For instance, if the service uses the HTTP protocol to receive requests, the query information can be added as a request parameter to the URL, or sent in JSON format as the request body. When receiving the returned collection of historical data files, the response is parsed according to the service's response format. If the returned file is a compressed file, a suitable decompression tool can be used for decompression. For example, for files compressed in ZIP format, a dedicated ZIP decompression library can be used.

[0070] Step S412: Parse the metadata information of the historical data file set, extract the system operation status label corresponding to each file, and filter the historical data files with the label indicating normal operation. The normal operation status is determined by the system having no alarm records, performance indicators within the preset range, and stable business transaction volume.

[0071] Metadata information describes relevant information about historical data files, including file creation time, data type, and the database instance to which it belongs. System operation status tags identify the database system's operating status within the time frame corresponding to that file, such as normal operation or abnormal operation. When parsing the metadata information of a collection of historical data files, appropriate parsing methods can be used based on the file format. For example, if the metadata is stored in JSON format, a JSON parsing library can be used to parse it into an operable data structure. When extracting the system operation status tag for each file, the tag field is searched in the parsed metadata. When filtering historical data files with the tag indicating normal operation, the system operation status tag for each file is checked. Only files whose tags indicate normal operation and meet three conditions—no system alarm records, performance indicators within a preset range, and stable business transaction volume—are included in the filtering results. For the determination of no system alarm records, the alarm information field in the metadata can be viewed; for the determination of performance indicators within a preset range, the performance indicators recorded in the metadata are compared with the preset range; for the determination of stable business transaction volume, the trend of business transaction volume changes can be analyzed, for example, calculating the fluctuation range of transaction volume. If the fluctuation range is within a certain range, the business transaction volume is considered stable.

[0072] Step S413: Extract performance index data from the filtered historical data files, including database response characteristics, query success characteristics, and resource utilization characteristics. Verify whether these indicators are all within the preset normal range. The normal range is determined by statistically analyzing the distribution characteristics of historical performance indicators.

[0073] When extracting performance metrics data from filtered historical data files, the extraction is performed based on the file structure and data storage method. For example, if the data is stored in tabular form, the column names can be used to locate the columns containing the performance metrics data for extraction. To verify whether these metrics are within a preset normal range, each extracted performance metric is compared to its corresponding normal range. For database response characteristics, if the response time exceeds the upper limit of the normal range, the metric is considered abnormal; for query success characteristics, if the query success rate is lower than the lower limit of the normal range, it is considered abnormal; for resource utilization characteristics, if CPU utilization is higher than the upper limit of the normal range, it indicates abnormal resource utilization. When statistically analyzing the distribution characteristics of historical performance metrics, statistical analysis methods can be used, such as calculating the mean, variance, and median, combined with visualization tools such as histograms, to visually observe the distribution of performance metrics and thus determine the normal range.

[0074] Step S414: Segment the verified historical data according to the time period to generate multiple baseline periods. Each baseline period contains a standardized state data sequence of all normal operating states within the corresponding period. Establish a baseline period index table to record the time range, data integrity check value and associated system configuration version of each period.

[0075] When segmenting verified historical data according to a time period, the historical data is divided chronologically based on a pre-defined time period. For example, if the time period is set to one day, the historical data is segmented into daily periods. After generating multiple baseline periods, the standardized state data sequences within each period are organized and stored. When establishing the baseline period index table, for each baseline period, its start and end times are recorded as the time range; the integrity check value of the data within that period is calculated using hash algorithms such as MD5 and SHA-1; the associated system configuration version can be obtained from metadata or by querying system configuration records.

[0076] Step S415: Perform data checks on the standardized state data sequence for each baseline period, including field completeness, timestamp continuity and numerical rationality verification. For data with a missing proportion lower than the preset ratio, use a prediction and filling method based on historical data from similar databases to supplement it. For outliers, use the mean replacement method of the same period to correct them, and generate a complete historical baseline data set.

[0077] When performing data checks on the standardized state data sequence for each baseline period, for field integrity verification, each field in the data sequence can be traversed to check for null or missing values. For timestamp continuity verification, timestamps can be sorted, and the difference between adjacent timestamps can be checked to see if it conforms to the normal time interval. For numerical reasonableness verification, reasonable value ranges are set according to the characteristics of different indicators. When data below the preset missing proportion is found, historical data from similar databases are collected, and appropriate prediction models, such as linear regression models or time series models, are selected to predict and impute the missing data. For outliers, the average value of the indicator within the period is calculated, and the outliers are replaced with the average value. After these processes, a complete historical baseline data set is generated.

[0078] Step S416: According to the preset cycle, incrementally add new normal operation status data periodically, redivide the baseline cycle and update the baseline cycle index table.

[0079] When operating according to a preset cycle, a scheduled task tool can be used. For example, a scheduled task scheduler can be used in an operating system, or the built-in scheduled job function can be used in a database management system. When judging newly added normal operating status data, the same normal operating status judgment method as when filtering historical data is used, i.e., checking that the system has no alarm records, performance indicators are within the preset range, and business transaction volume is stable. When performing incremental addition operations, the new data is merged with the original historical baseline data. When re-dividing the baseline cycle, the boundaries of each baseline cycle are redefined according to the new data distribution and time range. When updating the baseline cycle index table, for new baseline cycles, their time range is updated, new data integrity check values ​​are calculated, and the corresponding system configuration version is associated; for the original baseline cycles affected by the new data, corresponding updates and adjustments are also made.

[0080] Step S420: Extract baseline data from the historical baseline data set that is identical to the current database instance and has the same deployment environment. Divide the current standardized state data sequence and the historical baseline data into segments according to the time granularity alignment principle. Each segment contains the same number of time units to generate a comparison data pair between the current data segment and the baseline data segment. The time length of the comparison data pair is a preset multiple of the system sampling period.

[0081] When extracting baseline data from the historical baseline dataset that is identical to the current database instance and deployed in the same environment, data meeting the criteria can be filtered by querying the metadata information in the historical baseline dataset. During segmentation, the current standardized state data sequence and the historical baseline data are divided according to a predetermined time granularity and segment length. For example, if the time granularity is set to one hour and the segment length is set to three time units, each segment contains three hours of data. When generating comparison data pairs, the current data segment is paired with the corresponding baseline data segment. For example, the first segment of the current data and the first segment of the historical baseline data form a comparison data pair.

[0082] Step S430: Calculate the vector space difference of the connection feature vectors in the comparison data pair. Calculate the vector space angle between the current vector and the baseline vector, convert the cosine value of the angle into a similarity metric, and then convert (1 - similarity metric) into vector deviation. Changes in the vector deviation value reflect changes in the degree of difference in the connection features.

[0083] When calculating the vector space difference of the connection feature vectors in the comparison data pairs, the first step is to calculate the vector space angle between the current vector and the baseline vector. The cosine value of the angle can be calculated using the dot product formula and the vector magnitude formula. The cosine value is then used as a similarity measure, and the vector deviation is calculated as (1 - similarity measure). By continuously calculating the vector deviation of different comparison data pairs, the changes in the degree of difference in connection features can be observed.

[0084] Step S440: Calculate the distribution difference of the query behavior matrix in the comparison data pair, normalize the matrix by row to form a probability distribution matrix, calculate the cross-entropy value between the current distribution matrix and the baseline distribution matrix, convert the cross-entropy value into the distribution deviation, and the change in the distribution deviation value reflects the change in the degree of distribution difference of the query behavior.

[0085] When calculating the distribution difference of the query behavior matrix in the comparison data pairs, the query behavior matrix is ​​first normalized row by row. For each row of the matrix, each element of that row is divided by the sum of the elements in that row to obtain the probability distribution matrix. Then, the cross-entropy value between the current distribution matrix and the baseline distribution matrix is ​​calculated using the cross-entropy formula. Using the cross-entropy value as the basis for the distribution deviation, appropriate transformations are performed according to the actual situation, such as linear transformations, to obtain the final distribution deviation. By comparing the distribution deviations of different comparison data pairs, changes in the degree of difference in query behavior distribution can be observed.

[0086] Step S450: Perform tensor space difference calculation on the resource consumption tensor in the comparison data pair. Convert the three-dimensional tensor into a matrix sequence through tensor decomposition. Calculate the tensor norm difference between the current matrix sequence and the baseline matrix sequence. Convert the norm difference into tensor deviation. The change in the tensor deviation value reflects the change in the degree of difference in resource consumption patterns.

[0087] When calculating the tensor space difference of resource consumption tensors in comparative data pairs, tensor decomposition methods are used to decompose the three-dimensional tensor into a sequence of matrices. For example, Tucker decomposition and CP decomposition can be employed. After decomposition, the current matrix sequence and the baseline matrix sequence are obtained. The tensor norm of the matrix sequences is calculated, using common tensor norm definitions such as the Frobenius norm. The difference in tensor norm between the current matrix sequence and the baseline matrix sequence is calculated, and this difference is used as the basis for the tensor deviation, undergoing appropriate transformations to obtain the final tensor deviation. By analyzing the tensor deviation of different comparative data pairs, changes in the degree of difference in resource consumption patterns can be observed.

[0088] Step S460: Integrate vector deviation, distribution deviation, and tensor deviation, and perform multi-dimensional combination calculations based on the performance impact factors of each database instance to generate a comprehensive deviation value that reflects the overall performance deviation. When the comprehensive deviation value exceeds the preset anomaly threshold, trigger the anomaly pattern recognition process. The anomaly threshold is dynamically adjusted based on the deviation distribution characteristics of historical baseline data.

[0089] In one implementation, step S460 may specifically include the following steps S461 to S466: Step S461: Normalize the three deviation values ​​respectively, and convert them into relative deviation values ​​in the range of 0 to 1 by dividing them by their respective historical maximum values. The historical maximum values ​​are obtained statistically from the historical baseline data set.

[0090] When normalizing the three deviation values, for vector deviation, the historical maximum value is calculated from the historical baseline dataset, and then the current vector deviation value is divided by this historical maximum value to obtain the normalized vector relative deviation value. The same method is used to process distribution deviation and tensor deviation.

[0091] Step S462: Extract the performance impact factor of each database instance as the weight coefficient for the deviation calculation. The database instance with the higher the performance impact factor value, the higher the proportion of the deviation value in the combined calculation.

[0092] When extracting the performance impact factor of each database instance as the weighting coefficient for the deviation combination calculation, the performance impact factor value of each database instance is obtained from the previously calculated performance impact factor results. During the deviation combination calculation, the vector relative deviation value, distribution relative deviation value, and tensor relative deviation value of each database instance are multiplied by the performance impact factor of that database instance, so that the deviation value of database instances with higher performance impact factors plays a greater role in the combination calculation.

[0093] Step S463: Combine the normalized vector deviation, distribution deviation, and tensor deviation in multiple dimensions to generate a single-instance comprehensive deviation value. The weight coefficient of the vector deviation is determined based on the contribution analysis of each deviation in historical anomaly cases.

[0094] When combining the normalized vector deviation, distribution deviation, and tensor deviation in multiple dimensions, the weighting coefficients for each parameter are first determined. For example, analysis of historical anomaly cases reveals that vector deviation contributes 0.4 to the anomaly, distribution deviation contributes 0.3, and tensor deviation contributes 0.3. Then, the normalized vector deviation is multiplied by 0.4, the distribution deviation by 0.3, and the tensor deviation by 0.3, and these three results are summed to obtain the single-instance comprehensive deviation value.

[0095] Step S464: Summarize the single-instance comprehensive deviation values ​​of all database instances, and calculate the overall comprehensive deviation value of the system through multi-dimensional combination. The weight of each instance is its performance impact factor. The overall comprehensive deviation value of the system reflects the degree of performance deviation under the synergistic effect of multiple instances.

[0096] The aggregated single-instance comprehensive deviation value of all database instances is obtained by combining the individual comprehensive deviation values ​​of each database instance to obtain the overall comprehensive evaluation result of the system. Multi-dimensional combination calculation takes into account the possible synergistic effects between different database instances, calculating the overall comprehensive deviation value of the system through a reasonable combination method. The weight of each instance is its performance impact factor, meaning that database instances with higher performance impact factors have a greater influence on the calculation of the overall comprehensive deviation value of the system. The overall comprehensive deviation value of the system reflects the degree of performance deviation of multiple database instances under synergistic effects, enabling a more comprehensive evaluation of the overall performance of the database system. When aggregating the individual-instance comprehensive deviation values ​​of all database instances, the individual-instance comprehensive deviation value of each database instance is multiplied by its performance impact factor, and then all results are summed.

[0097] Step S465: Perform time-series filtering on the overall system deviation value, set the window size to multiple time units, and calculate the average deviation within the window as the filtered overall deviation value at the current moment.

[0098] When performing time-series filtering on the overall system deviation value, the overall system deviation value within a certain number of time units preceding the current time is selected according to a pre-set window size. For example, if the window size is set to 5 time units, the overall system deviation values ​​of the current time and the previous 4 time units are selected. These values ​​are added together and then divided by 5 to obtain the filtered overall deviation value at the current time.

[0099] Step S466: Compare the filtered overall deviation value with the dynamic anomaly threshold. The dynamic anomaly threshold is dynamically adjusted according to the distribution characteristics of the overall deviation of historical baseline data. When the filtered overall deviation value exceeds the dynamic anomaly threshold, it is determined that the system currently has a performance anomaly, triggering the anomaly pattern recognition process.

[0100] When comparing the filtered overall deviation value with the dynamic anomaly threshold, the appropriate dynamic anomaly threshold is first determined based on the distribution characteristics of the overall deviation of historical baseline data, such as calculating statistics like the mean and standard deviation. Then, the current filtered overall deviation value is compared with the dynamic anomaly threshold. If the filtered overall deviation value is greater than the dynamic anomaly threshold, the system is determined to have a performance anomaly, triggering the anomaly pattern recognition process to conduct an in-depth analysis of the anomaly and identify its cause and scope of impact.

[0101] As one implementation method, in step S460, when the overall deviation value exceeds a preset anomaly threshold, an anomaly pattern recognition process is triggered, which may specifically include the following steps S467~S4614: Step S467: Extract the standardized state data sequence fragments when the anomaly is triggered, including complete data from multiple time windows before and after the anomaly occurs. The length of the time window is consistent with the fragment length when calculating the comprehensive deviation. Generate an anomaly analysis data window, which contains performance change process data before and after the anomaly occurs.

[0102] Standardized state data sequence fragments are portions of data extracted from standardized state data sequences, containing key information before and after an anomaly. Complete data from multiple time windows before and after the anomaly comprehensively reflects the process and impact of the anomaly. The length of the time windows matches the fragment length used in the comprehensive deviation calculation, ensuring data consistency and comparability. The anomaly analysis data window consists of extracted standardized state data sequence fragments, containing performance change data before and after the anomaly.

[0103] When extracting standardized state data sequence fragments triggered by an anomaly, a certain number of time windows are selected before and after the anomaly occurs. For example, if the fragment length for calculating the overall deviation is 3 time units, complete data from the 3 time windows before and after the anomaly can be selected. These data are then organized and integrated to generate anomaly analysis data windows.

[0104] Step S468: Analyze the trend of connection feature vector changes in the anomaly analysis data window, calculate the first difference of the connection feature vector within a continuous time unit, identify the time point when the absolute value of the difference exceeds the historical baseline fluctuation range as the connection status change point, calculate the deviation rate of the fluctuation amplitude of the number of connections within a unit time window from the historical fluctuation range, calculate the deviation of the connection type proportion value from the historical distribution interval, and integrate the change point, fluctuation deviation rate and proportion deviation to generate a connection anomaly feature set.

[0105] When analyzing the trend of connection feature vector changes in the anomaly analysis data window, the connection feature vectors are arranged in chronological order, and the first-order difference of the connection feature vectors within adjacent time units is calculated. For example, for a certain element in the connection feature vector, the difference between the current time unit and the previous time unit is calculated. The absolute value of the difference is compared with the historical baseline fluctuation range, and the time point where the absolute value of the difference exceeds the historical baseline fluctuation range is identified as the connection status change point. The fluctuation amplitude of the number of connections within a unit time window is statistically analyzed, and its deviation rate from the historical fluctuation range is calculated. For example, the difference between the maximum and minimum values ​​of the number of connections in the current time window is calculated as the fluctuation amplitude, and then compared with the fluctuation amplitude of the same period in history to calculate the deviation rate. The deviation of the proportion of connection types from the historical distribution range is calculated, which can be achieved by calculating the absolute value of the difference between the current proportion of connection types and the center value of the historical distribution range. Finally, this information is integrated into the connection anomaly feature set.

[0106] Step S469: Parse the query behavior matrix in the anomaly analysis data window, calculate the probability density function of the query type distribution and the cumulative distribution difference of the historical distribution model, identify the query types whose distribution difference exceeds the set threshold as distribution change types, count the deviation of the time consumption ratio of each query type from the historical ratio interval, calculate the ratio of the sliding window variance of the execution frequency sequence to the historical variance benchmark, and integrate the distribution change type, time consumption ratio deviation and frequency variance ratio to generate a query anomaly feature set.

[0107] When parsing the query behavior matrix in the anomaly analysis data window, the probability density function of the query type distribution is first calculated. This probability density function can be estimated using statistical methods based on the frequency of occurrence of each query type in the query behavior matrix. Then, this probability density function is compared with the cumulative distribution of the historical distribution model to calculate the cumulative distribution difference. Query types whose distribution differences exceed a set threshold are identified as distribution change types. The time consumption percentage of each query type is statistically analyzed, and its deviation from the historical percentage interval is calculated. For example, the absolute value of the difference between the current query type's time consumption percentage and the median value of the historical percentage interval is calculated. The sliding window variance of the execution frequency sequence is calculated and compared with the historical variance benchmark to obtain the ratio. Finally, the distribution change types, time consumption percentage deviations, and frequency variance ratios are integrated into the query anomaly feature set.

[0108] Step S4610: Analyze the resource consumption tensor in the anomaly analysis data window, calculate the growth rate of the resource usage sequence and the residual of the historical growth trend model, identify resource types with residuals exceeding the confidence interval as resources with changing usage, calculate the ratio of the frequency of resource competition events to the historical average frequency, calculate the coefficient of variation of the resource utilization rate sequence and the deviation from the historical stable interval, and integrate the resources with changing usage, the competition frequency ratio, and the utilization deviation to generate a resource anomaly feature set.

[0109] When parsing the resource consumption tensor in the anomaly analysis data window, the resource usage sequence is extracted, and its growth rate is calculated. Historical growth trend models, such as linear regression models, are built using historical data, and the residuals between the resource usage sequence's growth rate and the historical growth trend model are calculated. Resource types with residuals exceeding the confidence interval are identified as resources with changing usage. The frequency of resource competition events is statistically analyzed, and its ratio to the historical average frequency is calculated. The coefficient of variation of the resource utilization rate sequence is calculated and compared with the historical stable interval to obtain the deviation. Finally, the resources with changing usage, the competition frequency ratio, and the utilization deviation are integrated into a resource anomaly feature set.

[0110] Step S4611: Input the connection anomaly feature set, query anomaly feature set, and resource anomaly feature set into the preset anomaly pattern matching library. The anomaly pattern matching library contains feature templates of historical anomaly cases. By comparing feature similarity, the matching degree between the current anomaly and historical cases is identified, and a preliminary anomaly type candidate list is generated. The candidate list contains multiple anomaly types sorted by matching degree.

[0111] When inputting connection anomaly feature sets, query anomaly feature sets, and resource anomaly feature sets into a pre-defined anomaly pattern matching library, similarity calculation methods, such as Euclidean distance and cosine similarity, are used to compare these feature sets with feature templates in the anomaly pattern matching library. For each feature template, its similarity score with the input feature set is calculated. Based on the similarity scores, historical anomaly cases are sorted to generate a preliminary anomaly type candidate list. For example, if cosine similarity is used for calculation, a cosine similarity value closer to 1 indicates a higher similarity, and anomaly types with higher similarity are ranked higher.

[0112] Step S4612: Based on the preliminary candidate list of anomaly types, construct an anomaly probability calculation model, take the feature items in the connection, query and resource anomaly feature set as input variables, calculate the occurrence probability of each candidate anomaly type through the correlation between the features of historical anomaly cases and the anomaly type, and generate an anomaly occurrence probability distribution.

[0113] When constructing an anomaly probability calculation model based on a preliminary list of anomaly types, machine learning algorithms such as Bayesian classifiers and decision trees can be used. The model uses features from the connection and query sets and the resource anomaly feature set as input variables, extracting the association data between these features and anomaly types from historical anomaly cases. For example, it statistically analyzes the frequency of different anomaly types when a certain feature appears in historical anomaly cases. This association data is used to train the anomaly probability calculation model, and then the current anomaly features are input into the model to calculate the probability of occurrence for each candidate anomaly type. These probabilities are then organized according to anomaly type to generate an anomaly occurrence probability distribution.

[0114] Step S4613: Based on the database performance correlation graph and the probability distribution of anomalies, assess the scope of the anomaly's impact. Starting from the instance where the anomaly occurred, propagate the impact to adjacent instances according to the edge weights in the graph. Calculate the probability that each instance is affected by the anomaly. The probability of being affected is equal to the probability of the anomaly occurring multiplied by the weight of the comprehensive dependency relationship between instances.

[0115] When assessing the impact range of an anomaly based on the database performance correlation graph and the anomaly occurrence probability distribution, the impact is propagated from the anomaly-occurring instance to neighboring instances according to the edge weights in the graph. For each neighboring instance, the anomaly occurrence probability is multiplied by the overall dependency weight between that instance and the anomaly-occurring instance to obtain the spillover probability for that instance. Then, for the neighboring instances of these neighboring instances, the spillover probability is calculated in the same way until it propagates to all potentially affected instances. By continuously updating the spillover probability, the probability of each instance in the entire database system being affected by the anomaly can be obtained, thus determining the scope of the anomaly's impact.

[0116] Step S4614: Summarize the results of the probability distribution and impact range assessment of anomalies, and generate anomaly diagnosis results that include anomaly types, probability of occurrence, a list of impact instances, and propagation paths.

[0117] When summarizing the anomaly occurrence probability distribution and impact scope assessment results, each anomaly type and its probability of occurrence in the anomaly occurrence probability distribution are compiled. Based on the impact scope assessment results, a list of affected instances is determined, including instances with a ripple probability greater than a certain threshold. By analyzing the database performance correlation graph and the propagation process of the ripple probability, the propagation path of the anomaly is determined. All this information is integrated to generate an anomaly diagnostic result that includes the anomaly type, occurrence probability, a list of affected instances, and the propagation path.

[0118] Step S500: Based on the probability distribution and impact range assessment of anomalies in the anomaly diagnosis results, call the preset operation and maintenance strategy to generate a rule base, generate an operation and maintenance optimization plan containing resource scheduling instructions and query optimization suggestions, and push the operation and maintenance optimization plan to the database management platform for automated operation and maintenance.

[0119] In one implementation, step S500 may specifically include the following steps S510-S560: Step S510: Analyze the probability distribution of abnormal occurrence in the abnormal diagnosis results, extract the abnormal type with the highest probability value as the dominant abnormal type probability value, and the next highest probability value as the associated abnormal type, sort them from high to low probability value, and generate an abnormal type priority sequence.

[0120] The anomaly probability distribution includes information on the probability of occurrence of different anomaly types. The dominant anomaly type is the one with the highest probability of occurrence, having the most significant impact on the database system. Related anomaly types are those with the next highest probability of occurrence; these types may be related to or influence the dominant anomaly type. The anomaly type priority sequence is a sequence ordered from highest to lowest anomaly probability, clarifying the priority of anomaly handling and helping to rationally allocate operational resources.

[0121] When analyzing the probability distribution of anomalies in the diagnostic results, each anomaly type and its probability are compiled. The anomaly type with the highest probability value is identified as the dominant anomaly type, and its probability value is recorded. Then, the anomaly type with the second highest probability value is identified as the associated anomaly type. All anomaly types are sorted according to their probability values ​​from high to low to generate an anomaly type priority sequence.

[0122] Step S520: Based on the priority sequence of anomaly types, query the preset operation and maintenance strategy to generate a rule base, match the core processing rules for the dominant anomaly type and the auxiliary processing rules for related anomaly types. The core processing rules contain key strategies to directly mitigate anomalies, and the auxiliary processing rules contain supplementary strategies to prevent the development of related anomalies.

[0123] Core processing rules are for handling dominant anomaly types. These rules include key strategies for directly mitigating anomalies, such as resource reallocation and query optimization. Auxiliary processing rules are for handling related anomaly types, primarily used to prevent the development of related anomalies, playing a supplementary and auxiliary role.

[0124] In one implementation, step S520 may specifically include the following steps S521 to S526: Step S521: Parse the dominant exception type name and feature description in the exception type priority sequence, query the core rule table of the operation and maintenance strategy generation rule base, and match the rule entries that are completely consistent with the dominant exception type features. The core rule table contains fields such as exception type, triggering conditions, processing actions, and verification indicators.

[0125] When parsing the dominant anomaly type name and feature description in the anomaly type priority sequence, relevant information about the dominant anomaly type is extracted from the sequence. Then, a query is performed in the core rule table of the operation and maintenance strategy generation rule base to find rule entries that are completely consistent with the dominant anomaly type's features. For example, if the dominant anomaly type is "resource contention anomaly," and the feature description is "the utilization rate of a specific resource exceeds a threshold and contention events occur frequently," then the core rule table is searched for rule entries with the anomaly type "resource contention anomaly" and triggering conditions that match this feature description. By matching these rule entries, the core processing rules for the dominant anomaly type can be found.

[0126] Step S522: Extract the sequence of processing actions from the matched core processing rules, including resource adjustment actions, service configuration actions, and data operation actions. Each processing action includes action type, target parameters, and execution constraints.

[0127] When extracting the sequence of processing actions from the matched core processing rules, information related to the processing actions is found from the matched rule entries. The processing actions are categorized into resource adjustment actions, service configuration actions, and data operation actions. For each processing action, its action type, target parameters, and execution constraints are extracted. For example, if the processing action is "increase the memory allocation of a database instance," then the action type is "resource adjustment action," the target parameter is "the database instance and the increased memory size," and the execution constraint might be "the system has sufficient available memory."

[0128] Step S523: Parse the associated anomaly types in the anomaly type priority sequence, query the auxiliary rule table of the rule base, match the preventive processing rules associated with the associated anomaly types, the auxiliary rule table records the mapping relationship between associated anomaly types and preventive strategies, and the preventive strategies block the anomaly propagation path.

[0129] Related anomaly types are anomaly types in the anomaly type priority sequence that are other than the dominant anomaly type and may be related to or influence the dominant anomaly type. The auxiliary rule table in the rule base is specifically used to store auxiliary processing rules, recording the mapping relationship between related anomaly types and prevention strategies. Preventive processing rules are rules formulated for related anomaly types, aiming to block the propagation path of anomalies and prevent the further development of related anomalies.

[0130] When parsing associated anomaly types in the anomaly type priority sequence, the names and related characteristics of the associated anomaly types are obtained from the sequence. Then, a query is performed in the auxiliary rule table of the rule base to find preventative handling rules associated with the associated anomaly types. For example, if the associated anomaly type is "query performance anomaly," the auxiliary rule table is searched for preventative strategies related to "query performance anomaly," such as suggesting optimization of query statements or creating appropriate indexes. By matching preventative handling rules, auxiliary handling strategies for the associated anomaly types can be found.

[0131] Step S524: Merge the core processing rules and auxiliary processing rules, check the conflict relationship between the rules, and when the execution target instance of two rules is the same and the resource requirements conflict, retain the resource requirements of the core processing rule and adjust the resource allocation ratio of the auxiliary processing rule.

[0132] When merging core processing rules and auxiliary processing rules, they are combined. When checking for conflicts between rules, all rules are traversed, and their target instances and resource requirements are compared. If two rules are found to have the same target instance and conflicting resource requirements (e.g., one rule requires increasing memory allocation for a database instance while another requires decreasing it), the resource requirements of the core processing rule are retained, and the resource allocation ratio of the auxiliary processing rule is adjusted.

[0133] Step S525: Adjust the execution order of processing rules according to the priority of exception type. The core processing rules for the dominant exception type are executed first, and the auxiliary processing rules for related exception types are executed in descending order of impact probability, generating an ordered sequence of processing rules. The sequence of processing rules includes rule identifiers, execution order, and dependencies.

[0134] The priority of exception types determines the execution order of processing rules. Core processing rules for dominant exception types should be executed first, as they have the most significant impact on the database system. Auxiliary processing rules for related exception types are executed in descending order of their impact probability; rules for related exception types with higher impact probabilities are executed first, thus prioritizing related exceptions with potentially larger impact ranges. An ordered sequence of processing rules is a set of rules arranged in execution order, containing information such as rule identifiers, execution order, and dependencies. Rule identifiers uniquely identify each rule; the execution order specifies the order in which rules are executed; and dependencies describe the mutual dependencies between rules.

[0135] When adjusting the execution order of processing rules based on exception type priority, the core processing rules for the dominant exception type are first placed at the top. Then, auxiliary processing rules for related exception types are sorted from highest to lowest impact probability. All rules are arranged in the adjusted order to generate an ordered sequence of processing rules. Each rule is assigned a rule identifier, and its execution order is recorded. Simultaneously, dependencies between rules are analyzed; for example, the execution of some rules may depend on the execution results of other rules, and these dependencies are recorded in the processing rule sequence.

[0136] Step S526: Add policy metadata to the processing rule sequence, including policy generation time, applicable scenario tags, and rollback mechanism description. The rollback mechanism description includes recovery steps and data backup requirements in case of abnormal processing failure to generate a complete set of processing rules.

[0137] When adding policy metadata to the processing rule sequence, record the current time as the policy generation time. Add applicable scenario tags to the processing rule sequence based on information such as the exception type and database status. For example, if the processing rule sequence addresses the simultaneous occurrence of "resource contention exception and query performance exception," the applicable scenario tag could be marked as "combined resource contention and query performance exception." Write a rollback mechanism description, clearly defining the recovery steps in case of exception handling failure, such as restoring resources to their pre-processing state and rolling back data operations. Simultaneously, specify data backup requirements, such as the backup interval and backup storage location. Add the policy metadata to the processing rule sequence to generate a complete set of processing rules.

[0138] Step S530: Combining the instance spillover probability in the impact range assessment, apply the first processing strategy to instances with spillover probability exceeding the preset value, and apply the second processing strategy to other instances, generating a differentiated strategy set, and tilting resource allocation towards instances with high spillover probability.

[0139] The instance impact probability is the likelihood of each database instance being affected by an anomaly, calculated during the anomaly impact scope assessment. When generating a differentiated strategy set based on the instance impact probabilities from the impact scope assessment, all database instances are traversed, and their impact probabilities are compared to preset values. For instances with impact probabilities exceeding the preset value, a first processing strategy is applied, such as increasing resource allocation and performing more in-depth query optimization; for instances with impact probabilities below the preset value, a second processing strategy is applied, such as performing resource checks and query adjustments. Regarding resource allocation, adjustments are made based on the instance's impact probability, allocating more resources, such as CPU, memory, and storage, to instances with high impact probabilities. The instances applying different processing strategies and their corresponding strategies are then compiled to generate a differentiated strategy set.

[0140] Step S540: Analyze the resource requirements in the differentiated strategy set, calculate the adjustment amount of CPU, memory, storage and network resources, determine the direction and specific value of resource adjustment based on the current resource usage status and historical resource adjustment effect data, and generate a resource scheduling requirement list, which includes resource type, target instance and adjustment range.

[0141] When analyzing resource requirements within the differentiated strategy set, a detailed analysis of the processing strategy for each instance in the set is conducted to clarify the resource adjustment needs involved. For each instance, based on its strategy, it is determined whether adjustments to CPU, memory, storage, and network resources are necessary, and the general direction of these adjustments (increase or decrease).

[0142] When calculating CPU resource adjustments, consider the instance's current CPU utilization, workload, and CPU resource requirements in the policy. For example, if an instance's workload has increased significantly recently, and the policy requires improving the instance's processing performance, the required number of CPU cores or the percentage increase in CPU utilization can be estimated based on historical data showing the relationship between workload growth and CPU utilization. For memory resources, analyze the instance's memory usage patterns, such as cache hit rate and page swapping frequency. If the cache hit rate is low, and the policy aims to improve caching effectiveness, the required increase in memory can be calculated based on historical memory usage data and business characteristics. For storage resources, calculate the adjustment based on the instance's storage access characteristics, capacity usage characteristics, and data growth characteristics. If storage access is frequent and capacity is nearing its limit, while the policy requires ensuring fast data access, it may be necessary to increase storage capacity or adjust storage access priority. For network resources, calculate based on the instance's network bandwidth usage characteristics, transmission latency characteristics, and data packet loss characteristics. If network bandwidth utilization is high and transmission latency is significant, and the policy requires optimizing network performance, the required increase in bandwidth can be calculated.

[0143] Current resource usage status can be obtained in real time through system monitoring tools to understand whether the usage of each resource is within the normal range. Historical resource adjustment effect data records the changes in database performance after past resource adjustments. By comparing this data, it can be determined whether the direction of resource adjustments is correct and whether the adjustment range is appropriate. If adding a certain amount of CPU resources in the past has achieved good results, then in the current situation, the previous adjustment range can be referenced, and appropriate adjustments can be made based on current business needs.

[0144] When generating the resource scheduling requirement list, the calculated resource adjustment information is organized. The resource types in the list are explicitly defined as CPU, memory, storage, and network; the target instance is the database instance that needs resource adjustment; and the adjustment range records the specific value or proportion that each resource needs to be increased or decreased.

[0145] Step S550: Convert the resource scheduling requirement list into standardized resource scheduling instructions. These instructions cover operation types such as resource allocation, release, and migration, and include execution time windows, operation steps, and verification indicators. The execution time window is selected during off-peak business periods.

[0146] In one implementation, step S550 may specifically include the following steps S551-S556: Step S551: Parse the resource type field in the resource scheduling requirement list, distinguish between CPU, memory and storage and network resources, and generate resource scheduling sub-lists by resource type. Each sub-list contains the adjustment requirements for that type of resource. The adjustment requirements include the target instance, the current resource quantity and the target resource quantity.

[0147] Each sub-list contains adjustment requirements including the target instance, current resource quantity, and target resource quantity. The target instance is the database instance requiring resource adjustments; the current resource quantity reflects the instance's current resource usage; and the target resource quantity is the adjusted resource quantity determined by the resource scheduling requirement list. When parsing the resource type field in the resource scheduling requirement list, string matching or regular expressions are used to identify records with resource types of CPU, memory, storage, and network. For each identified record, information such as the target instance, current resource quantity, and target resource quantity is extracted. After generating resource scheduling sub-lists grouped by resource type, each sub-list is organized and sorted. Sort can be done by the name of the target instance or the priority of resource adjustments for easier subsequent operations.

[0148] Step S552: For CPU resource scheduling requirements, calculate the adjustment amount based on the number of CPU cores and utilization rate of the target instance and the number of business threads to generate CPU resource allocation or release instructions. The instructions include the CPU core number adjustment value, the main frequency setting and the affinity configuration. The affinity configuration realizes the binding of CPU resource allocation with the physical cores of the instance deployment.

[0149] When calculating adjustments, these factors should be considered comprehensively. If the number of business threads is large and CPU utilization is close to its limit, it indicates that the current CPU resources may be insufficient to meet business needs, requiring an increase in the number of CPU cores. The required number of cores can be estimated based on the ratio of business threads to CPU cores. A higher clock speed can improve CPU processing speed, but it also increases power consumption. The clock speed should be set reasonably based on business needs and system cooling capabilities. Affinity configuration binds CPU resource allocation to the physical cores deployed on the instance, ensuring that the instance's business threads can run on the specified physical cores, improving CPU resource utilization efficiency. When generating CPU resource allocation or release instructions, the calculated CPU core adjustment value, clock speed setting, and affinity configuration information should be included in the instruction.

[0150] Step S553: ​​For memory resource scheduling requirements, calculate the adjustment amount based on the instance's memory usage, cache hit rate, and page swapping frequency, and generate memory resource allocation or release instructions. The instructions include memory size adjustment values, cache policy settings, and memory limit thresholds.

[0151] When calculating memory resource adjustments, if memory usage is nearing its limit, cache hit rate is low, and page swapping frequency is high, it indicates that current memory resources may be insufficient, requiring additional memory. The required increase in memory can be estimated based on historical data showing the relationship between memory usage and workload, as well as trends in cache hit rate and page swapping frequency.

[0152] Caching strategies include LRU (Least Recently Used) and LFU (Least Frequently Used). The appropriate caching strategy should be selected based on the instance's business characteristics and data access patterns. For example, for business scenarios that frequently access recent data, the LRU caching strategy is more suitable.

[0153] Memory limit thresholds are upper limits set to prevent instances from overusing memory. These thresholds should be set appropriately based on the overall memory resources of the system and the needs of other instances. When generating memory allocation or deallocation commands, the calculated memory size adjustment value, caching policy settings, and memory limit thresholds should be included in the command.

[0154] Step S554: For storage resource scheduling requirements, calculate the adjustment amount based on the instance's storage access characteristics, capacity usage characteristics, and data growth characteristics, and generate storage resource allocation or migration instructions. The allocation instructions include storage capacity adjustment values ​​and access priority settings, while the migration instructions include source storage path, target storage path, and migration rate control.

[0155] When calculating storage resource adjustments, if storage access is frequent and capacity is nearing its limit, while data growth is evident, it indicates that current storage resources may not meet future business needs, requiring increased storage capacity or storage migration. For storage capacity adjustments, the required increase in storage capacity is estimated based on data growth characteristics and current capacity usage. For storage migration operations, the source and target storage paths are determined. The source storage path is the current location of the data on the storage device; the target storage path is the new location of the data to be migrated to. Migration rate control is implemented to avoid excessive impact on business operations during data migration by setting a reasonable migration rate. When generating storage resource allocation or migration instructions, the calculated storage capacity adjustment value, access priority settings, source storage path, target storage path, and migration rate control information are included in the instruction.

[0156] Step S555: Based on the network resource scheduling requirements, calculate the adjustment amount according to the network bandwidth usage characteristics, transmission delay characteristics and data packet loss characteristics of the instance, and generate a network resource allocation instruction. The instruction includes the bandwidth adjustment value, service quality priority setting and traffic control strategy.

[0157] When calculating network resource adjustments, if network bandwidth usage is nearing its limit, transmission latency is high, and data packet loss rate is high, it indicates that current network resources may not meet business needs, and increased network bandwidth is required. The required increase in bandwidth can be estimated based on the business's network traffic growth trend and current usage.

[0158] Quality of Service (QoS) priority settings are used to ensure the quality of network transmission for critical business data. For services with high real-time requirements, such as online transactions and real-time monitoring, a higher QoS priority can be set to ensure that this business data is transmitted over the network preferentially. Traffic control policies are used to avoid network congestion and allocate network resources rationally. For example, token bucket or leaky bucket algorithms can be used to limit the network traffic of each instance. When generating network resource allocation instructions, the calculated bandwidth adjustment value, QoS priority settings, and traffic control policies are included in the instructions.

[0159] Step S556: Standardize the format of each resource scheduling instruction, including the instruction unique identifier, target instance identifier, resource type and adjustment parameters, execution time window and verification indicators, and generate standardized resource scheduling instructions that conform to the application programming interface specification of the database management platform. The verification indicators are used for the effect evaluation after resource scheduling.

[0160] Standardizing the format of all resource scheduling commands ensures that they can be correctly identified and executed on the database management platform. Unique command identifiers distinguish different resource scheduling commands, facilitating management and tracking. Target instance identifiers clearly define the object of the command's operation, i.e., the database instance that requires resource scheduling.

[0161] The resource type and adjustment parameters include specific resource scheduling information, such as CPU core count adjustments and memory size adjustments. The execution time window specifies the time period for instruction execution; selecting off-peak periods can reduce the impact on business operations. Validation metrics are used to evaluate the effectiveness of resource scheduling operations and ensure that resource adjustments achieve the expected goals.

[0162] When standardizing the format, the information in each resource scheduling instruction is organized according to the prescribed format. For example, the instruction's unique identifier can be generated using a UUID (Universally Unique Identifier); the target instance identifier uses the database instance's name or number; and the resource type and adjustment parameters are combined according to the format "Resource Type: Adjustment Parameters".

[0163] When generating standardized resource scheduling instructions that conform to the database management platform application programming interface (API) specifications, the organized information is encapsulated according to the API specification requirements. For example, if the API specification requires instructions to be transmitted in JSON format, then the instruction information is converted into a JSON string.

[0164] Verification metrics should be specific and measurable. For CPU resource scheduling instructions, verification metrics could include whether CPU utilization stabilizes within the target range after adjustment and whether business response time has been shortened. For memory resource scheduling instructions, verification metrics could include whether cache hit rate has improved and page swapping frequency has decreased. By evaluating verification metrics, problems in resource scheduling operations can be identified in a timely manner, allowing for adjustments and optimizations.

[0165] Step S560: Extract query optimization requirements from the differentiated strategy set, combine them with the query anomaly feature set from the anomaly diagnosis results, generate query statement rewriting suggestions, index optimization schemes, and execution plan adjustment strategies. The query optimization suggestions include expected performance improvement evaluation and resource scheduling instructions, and are combined to generate an operation and maintenance optimization scheme.

[0166] In one implementation, step S560 may specifically include the following steps S561-S566: Step S561: Extract abnormal query identifiers from the abnormal feature set of the abnormal diagnosis results. This includes query identifiers whose execution frequency fluctuations exceed the historical baseline range, query types whose time consumption ratios deviate from the historical distribution range, and query statements whose result set size deviates from the historical average level by a set proportion. Generate an abnormal query list, which includes the database instance to which the query belongs and the execution time range.

[0167] Abnormal query indicators are key information for identifying query anomalies. Fluctuations in execution frequency exceeding the historical baseline indicate unstable query execution frequency, potentially suggesting business anomalies or problems with the query statement itself.

[0168] A deviation of the execution time percentage from the historical distribution range indicates an anomaly in query execution time. If the execution time of a certain query type has historically been a relatively stable percentage of the total query time, but this percentage has recently increased or decreased suddenly, it may be due to changes in the complexity of the query statement or changes in the database environment. A result set size deviating from the historical average by more than a set percentage indicates an anomaly in the amount of data returned by the query.

[0169] When extracting abnormal query identifiers, a detailed analysis of the query anomaly feature set in the anomaly diagnosis results is performed. Data filtering and comparison methods are used to identify query identifiers that meet the above anomaly conditions. For each abnormal query identifier, its associated database instance and execution time range are recorded. The execution time range can be obtained from query logs or monitoring data, which helps determine the time period in which the abnormal query occurred, facilitating further analysis and troubleshooting. The extracted abnormal query identifiers, their associated database instances, and execution time ranges are then compiled into an abnormal query list.

[0170] Step S562: Analyze the execution plan of each query statement in the abnormal query list, identify inefficient execution paths using the database's built-in execution plan analysis tool, including execution paths where the proportion of full table scan operations exceeds the historical normal proportion, execution paths where the number of nested join operations exceeds the historical average level, execution paths where the data volume of sorting operations exceeds the memory buffer capacity, and execution paths where the frequency of temporary table creation is higher than the historical level. Record the type, location, and resource consumption ratio of inefficient paths, and generate an execution plan defect report.

[0171] Excessive nested join operations increase query complexity and execution time. If the number of nested join operations exceeds the historical average, query optimization may be necessary, such as adjusting the join order or using a more efficient join method.

[0172] If the data volume for sorting operations exceeds the memory buffer capacity, the sorting operation will need to use temporary disk storage, significantly reducing query performance. If this occurs, you should consider optimizing the sorting operation, such as increasing the memory buffer capacity or adjusting the sorting algorithm.

[0173] Creating temporary tables more frequently than historically average can waste system resources, as the creation and destruction of temporary tables consume time and resources. If the frequency of temporary table creation is too high, it's necessary to analyze the query statements for unnecessary temporary table creation operations and optimize them. When parsing the execution plan of each query in the abnormal query list, use the database's built-in execution plan analysis tools, such as MySQL's EXPLAIN statement or Oracle's EXPLAINPLAN statement. Perform a detailed analysis of the execution plan to identify inefficient execution paths.

[0174] For each inefficient execution path, record its type (e.g., full table scan, nested join), location (specific position in the query statement), and resource consumption percentage. The resource consumption percentage can be estimated using information provided by execution plan analysis tools, which helps assess the impact of each inefficient execution path on query performance. Compile the recorded inefficient execution path information into an execution plan defect report.

[0175] Step S563: Based on the execution plan defect report, query the preset query optimization template library, match the optimization template corresponding to the inefficient execution path. The optimization template includes syntax transformation rules for converting subqueries into join queries, rewriting rules for placing constant filtering conditions at the beginning of conditional clauses, and adjustment rules for aligning grouping fields and index fields in aggregation operations. Each template is associated with specific syntax transformation steps.

[0176] The pre-defined query optimization template library is a collection of predefined templates containing various common query optimization methods and rules. The syntax transformation rules for converting subqueries to join queries in the optimization templates can transform complex subqueries into simpler and more efficient join queries. The rewriting rule for moving constant filter conditions to the beginning of the conditional clause reduces unnecessary data scans by moving constant filter conditions to the front of the query statement. The adjustment rule for aligning grouping fields with index fields in aggregation operations ensures that the grouping fields in aggregation operations are consistent with the index fields in the database table, improving the performance of aggregation operations. If the grouping field does not have a corresponding index, the database may need to perform a full table scan to complete the aggregation operation, which is inefficient.

[0177] Step S564: Adjust and optimize the template based on the characteristics of the database type. For relational databases, focus on optimizing transaction processing strategies and lock management mechanisms. Reduce resource locking duration by splitting long transactions into multiple short transactions and adjust lock wait timeout parameters to avoid transaction blocking accumulation. For document-oriented databases, focus on optimizing sharding strategies and index design to ensure that filter fields in query conditions match sharding key fields and improve the coverage ratio of indexes on query results. For time-series databases, focus on optimizing data retention strategies and compression mechanisms. Set tiered storage periods based on data access frequency and adopt compression algorithms that match the time distribution characteristics of the data.

[0178] Different types of databases have different characteristics and performance bottlenecks, so the optimization template needs to be adjusted according to the characteristics of the database type.

[0179] Document-oriented databases differ from relational databases in their data storage structure and query methods. Sharding strategies and index design are key to optimizing their performance. Ensuring that filter fields in query conditions match the shard key field improves query efficiency because the database can quickly locate the relevant data shard based on the shard key. Increasing the coverage of indexes on query results reduces database disk I / O operations, thereby improving query performance.

[0180] Time-series databases are primarily used for processing time-series data. Data retention strategies and compression mechanisms are crucial factors affecting their storage efficiency and query performance. Setting tiered storage periods based on data access frequency allows for the rational allocation of storage resources, storing frequently accessed data on high-speed storage devices and less frequently accessed data on lower-speed devices. Employing compression algorithms that match the data's temporal distribution characteristics can effectively reduce storage space and improve read / write performance. For example, for time-series data with obvious periodic characteristics, period-based compression algorithms can be used.

[0181] When adjusting and optimizing templates based on database type characteristics, select the appropriate optimization focus according to the database type. For relational databases, add suggestions for adjusting transaction processing strategies and lock management mechanisms to the optimization template; for document-oriented databases, add optimization content for sharding strategies and index design; for time-series databases, add optimization measures for data retention strategies and compression mechanisms.

[0182] Step S565: Based on the optimization template and database type characteristics, generate specific query statement rewriting suggestions, including suggestions for adjusting the table join order in the syntax structure, suggestions for rearranging the priority of filtering conditions in the condition logic, and suggestions for swapping the positions of aggregation operations and filtering operations in the operation order. Each suggestion includes the original statement text, the rewritten statement text, and effect evaluation data based on historical optimization cases.

[0183] When generating specific query rewriting suggestions, first determine the optimization direction for the query statement based on the optimization template and database type characteristics. Regarding adjusting the table join order, analyze the relationships and data volume between the tables in the query statement. It is recommended to place tables with smaller data volumes earlier in the join order. This can reduce the size of intermediate result sets and improve the efficiency of join operations.

[0184] When prioritizing filter conditions, place constant filter conditions and highly selective filter conditions first. Constant filter conditions can filter out a large amount of data that does not meet the criteria at the beginning of the query, while highly selective filter conditions can reduce the amount of data processed in subsequent operations. Swapping the positions of aggregation and filtering operations is also a common optimization method. If the filtering operation can be completed before the aggregation operation, the amount of data processed in the aggregation operation can be reduced, thus improving the performance of the aggregation operation.

[0185] The various algorithms involved in the above embodiments of the present invention, such as the Euclidean distance algorithm and the cosine distance algorithm, can all be obtained from relevant content in the prior art. To save space, they will not be elaborated on further.

[0186] When implementing this invention, those skilled in the art can supplement the details based on common knowledge in the art. For example, they can use normalization to eliminate dimensional conflicts before feature fusion, use interpolation to eliminate dimensional differences, combine historical data, experience or business scenario requirements to reasonably set the threshold, train the model based on a general model training method, set the number of layers in the model structure based on actual needs, select the activation function, etc.

[0187] For example, in step S230, numerical data such as connection duration and data transmission volume can be independently regularized to eliminate dimensional differences and generate corresponding regularized numerical features. Then, the classification coding vector is concatenated with each regularized numerical feature to generate a connection feature vector. As another example, in step S460, during specific execution, vector deviation, distribution deviation, and tensor deviation can be regularly regularized and mapped to a unified scaling range. Then, based on the regularized deviation values, a weighted calculation is performed using the performance impact factors of each database instance to generate a single-instance comprehensive deviation value for each database instance. The single-instance comprehensive deviation values ​​of all instances are aggregated to obtain the overall comprehensive deviation value of the system. When the overall comprehensive deviation value of the system exceeds a preset abnormal threshold, the abnormal pattern recognition process is triggered.

[0188] Such issues will not be elaborated upon further in this invention.

[0189] Please see Figure 2 , Figure 2This is a schematic diagram of the structure of a computer system 100 provided in an embodiment of the present invention. The computer system 100 includes at least a processor 101, a communication interface 102, and a memory 103. The processor 101, communication interface 102, and memory 103 are connected via a bus or other means. The processor 101 is the computing and control core of the computer system, capable of parsing various instructions and processing various data within the computer system. The communication interface 102 may optionally include a standard wired interface or a wireless interface, and under the control of the processor 101, it can be used for sending and receiving data; the communication interface 102 can also be used for data transmission and interaction within the computer system. The memory 103 is used to store programs and data. The memory 103 may include the computer system's built-in memory or, of course, extended memory supported by the computer system. The memory 103 provides storage space, which stores the computer system's operating system; this invention does not limit this. In one embodiment, the processor 101 executes the service-oriented multi-type database unified real-time monitoring and operation and maintenance optimization method provided above in the embodiment of the present invention by running the computer program in the memory 103.

Claims

1. A method for unified real-time monitoring and operation and maintenance optimization of service-oriented multi-type databases, characterized in that, The method includes: Establish a distributed data acquisition node network for relational, document-based, and time-series databases, and receive the runtime status data stream of each database instance through a protocol adaptation interface. The runtime status data stream includes database connection session records, structured query statement execution traces, and storage resource usage snapshots. The running status data stream is subjected to cross-database type standardization transformation processing to generate a standardized status data sequence with a unified data format and timestamp alignment. The standardized status data sequence includes a connection feature vector, a query behavior matrix, and a resource consumption tensor. A database performance correlation graph is constructed based on the standardized state data sequence. The performance impact factor of each database instance is calculated by the dependency weight between graph nodes. The performance impact factor is used to characterize the propagation path of performance disturbances between different database instances. Based on the historical baseline data of the performance impact factors and standardized state data sequences, database anomaly pattern recognition is performed to generate anomaly diagnosis results that include anomaly occurrence probability distribution and impact range assessment. Based on the probability distribution and impact range assessment of anomalies in the anomaly diagnosis results, a preset operation and maintenance strategy generation rule base is invoked to generate an operation and maintenance optimization plan containing resource scheduling instructions and query optimization suggestions. The operation and maintenance optimization plan is then pushed to the database management platform for automated operation and maintenance.

2. The method according to claim 1, characterized in that, The process of constructing a database performance correlation graph based on the standardized state data sequence, and calculating the performance impact factor of each database instance through the dependency weights between graph nodes, includes: Extract the database instance metadata from the standardized state data sequence, including the instance's unique identifier, deployment node address, and service port information. Create an initial association graph with the database instance as the core node and data interaction relationships as edges. The node attributes of the initial association graph include database type labels and deployment physical location information. Analyze the cross-database query records in the query behavior matrix to identify the interaction relationship between the query initiating instance and the target instance, count the number of cross-database query requests, average response latency, and data transmission volume per unit time, and generate query dependency weights that characterize the coupling strength between instances. The query dependency weights are positively correlated with the cross-database query frequency and negatively correlated with the average response latency. The resource contention relationship in the resource consumption tensor is analyzed, the set of database instances sharing storage resources is identified, and the resource contention coefficient and the resource waiting queue length are calculated to generate resource dependency weights based on the resource contention relationship. The resource dependency weights are positively correlated with the resource contention coefficient. By combining the query dependency weight and resource dependency weight, the comprehensive dependency weight of each edge connection in the initial association graph is calculated. Based on the comprehensive dependency weights, the initial association graph is edge-filtered to remove weak dependency connections with weight values ​​below a preset threshold, generating a simplified database performance association graph. The node influence of the database performance correlation graph is calculated. Starting from the initial node, the influence value is propagated according to the edge weight. After multiple rounds of iteration, the performance influence factor of each database instance is generated. The performance influence factor represents the degree of influence of the instance on the overall system performance.

3. The method according to claim 2, characterized in that, The analysis of cross-database query records in the query behavior matrix identifies the interaction relationship between the query initiating instance and the target instance, and statistically analyzes the number of cross-database query requests, average response latency, and data transmission volume per unit time, including: Traverse each row of the query behavior matrix, identify query operation records containing cross-database identifiers, extract the target database instance name and access data table information from the query statement to determine the bidirectional interaction relationship between the query initiating instance and the target instance, and generate a query interaction triplet containing the initiator, receiver and interaction type. Using the system sampling period as the time unit, the query interaction triplet is divided into time windows, and the number of query requests between the same initiator and receiver within each time window is counted to generate a query request frequency sequence. The length of the time window is the same as the time granularity of the standardized state data sequence. Calculate the response latency of a single query in each query interaction triplet, where the response latency is the time interval from the time the query is initiated to the time the result is returned. Calculate the mean and variance of the response latency of all queries within the time window to generate an average response latency sequence. Parse the data volume field in the query response results, count the number of records returned and the data size for each query, calculate the total data transmission volume and the average single transmission volume within the time window, and generate a data transmission volume sequence. By associating the query request frequency sequence, average response latency sequence, and data transmission volume sequence, the correlation coefficient of the three sequences is calculated using a sliding window. The positive correlation between query request frequency and response latency, and the positive correlation between data transmission volume and response latency are identified to generate a cross-database query interaction feature set. Based on the cross-database query interaction feature set, a query dependency evaluation model is constructed. The query request frequency, average response latency, and data transmission volume are used as input parameters to generate query dependency weights through linear combination.

4. The method according to claim 2, characterized in that, The step of calculating the node influence of the database performance correlation graph, propagating the influence value according to the edge weight starting from the initial node, includes: Initialize the influence value of all nodes in the database performance correlation graph, set the initial influence value of each node to a uniform baseline value, and match the sum of the initial influence values ​​with the total number of nodes in the graph; Select the node with the largest sum of edge weights in the graph as the initial propagation source node. Distribute the influence value of the initial propagation source node to the adjacent nodes according to the weight ratio of the connecting edges. The influence value obtained by each adjacent node is equal to the current influence value of the propagation source node multiplied by the ratio of the corresponding edge weight to the sum of all edge weights of the propagation source node. In each propagation iteration, all nodes simultaneously receive influence allocation values ​​from their neighboring nodes and add the received values ​​to their current influence values ​​to obtain a new influence value. During the propagation iteration process, the influence value of each node is attenuated. After each iteration, the current influence value of the node is multiplied by the attenuation coefficient. The attenuation coefficient is set according to the characteristics of the database type of the node. The attenuation coefficient of relational database is lower than that of document-type and time-series database. When the change in node influence value between two consecutive iterations is less than the preset convergence threshold, the propagation iteration stops, and the final influence value of each node is recorded. The final influence value is normalized by dividing it by the sum of the influence values ​​of all nodes, converting it into a relative value between 0 and 1. This relative value is then used as the performance impact factor for each database instance.

5. The method according to claim 1, characterized in that, The step involves identifying database anomaly patterns based on the performance impact factors and historical baseline data of the standardized state data sequence, generating anomaly diagnostic results that include anomaly occurrence probability distribution and impact range assessment, including: Retrieve the standardized status data sequence history from the database historical operation data storage system, filter the historical data within the historical operation period where the system has no abnormal alarms and the performance indicators are in the normal range, divide it into multiple baseline periods according to the time period, and generate a historical baseline data set by each baseline period containing a complete standardized status data sequence. Each baseline period of the historical baseline data set is associated with the corresponding system configuration version information. Baseline data identical to the current database instance and deployed in the same environment are extracted from the historical baseline data set. The current standardized state data sequence and the historical baseline data are divided into segments according to the time granularity alignment principle. Each segment contains the same number of time units to generate a comparison data pair between the current data segment and the baseline data segment. The vector space difference is calculated for the connection feature vectors in the comparison data pair. The cosine value of the angle between the current vector and the baseline vector is converted into a similarity measure value, and then converted into vector deviation by (1-similarity measure value). The distribution difference of the query behavior matrix in the comparison data pair is calculated, the matrix is ​​normalized by row to become a probability distribution matrix, the cross-entropy value of the current distribution matrix and the baseline distribution matrix is ​​calculated, and the cross-entropy value is converted into the distribution deviation. Tensor space difference calculation is performed on the resource consumption tensor in the comparison data pair. The three-dimensional tensor is converted into a matrix sequence through tensor decomposition. The tensor norm difference between the current matrix sequence and the baseline matrix sequence is calculated and converted into tensor deviation. The vector deviation, distribution deviation, and tensor deviation are combined and calculated in multiple dimensions with the performance impact factors of each database instance to generate a comprehensive deviation value that reflects the overall performance deviation. When the comprehensive deviation value exceeds a preset abnormal threshold, the abnormal pattern recognition process is triggered. The abnormal threshold is dynamically adjusted according to the deviation distribution characteristics of historical baseline data.

6. The method according to claim 5, characterized in that, The process involves retrieving standardized status data sequences from the historical data storage system of the database, filtering historical data from periods during which the system had no abnormal alarms and whose performance indicators were within the normal range, including: Connect to the database historical data archiving service, send a query request containing historical running cycles and data types, and receive the returned historical data file set. The historical data file set is compressed and stored according to time units, wherein each file contains a standardized state data sequence of all database instances within the corresponding time unit. The metadata information of the historical data file set is parsed, and the system operation status tag corresponding to each file is extracted to filter historical data files that are in normal operation. The normal operation status is determined by the system having no alarm records, performance indicators being within a preset range, and stable business transaction volume. Performance metrics data, including database response characteristics, query success characteristics, and resource utilization characteristics, are extracted from the filtered historical data files. It is then verified whether these metrics are all within a preset normal range, which is determined by statistically analyzing the distribution characteristics of historical performance metrics. The verified historical data is segmented according to time period to generate multiple baseline periods. Each baseline period contains a standardized state data sequence of all normal operating states within the corresponding period. A baseline period index table is established to record the time range, data integrity check value and associated system configuration version of each period. The standardized state data sequence for each baseline period is checked. Data with a missing proportion below the preset limit is supplemented using a prediction and filling method based on historical data from similar databases. Outliers are corrected using the mean replacement method of the same period, thus generating a complete set of historical baseline data. According to the preset cycle, incremental additions of new normal operation status data are made periodically, the baseline cycle is redefined, and the baseline cycle index table is updated.

7. The method according to claim 5, characterized in that, The vector deviation, distribution deviation, and tensor deviation are fused together and combined with the performance impact factors of each database instance for multi-dimensional calculation to generate a comprehensive deviation value reflecting the overall performance deviation, including: The three deviation values ​​are normalized and converted into relative deviation values ​​in the range of 0 to 1 by dividing them by their respective historical maximum values. The performance impact factor of each database instance is extracted as the weight coefficient for the deviation calculation. The database instance with the higher the performance impact factor value, the higher the proportion of the deviation value in the combined calculation. The normalized vector deviation, distribution deviation, and tensor deviation are combined in multiple dimensions to generate a single instance comprehensive deviation value. The weight coefficient of the vector deviation is determined based on the contribution analysis of each deviation in historical anomaly cases. The overall deviation value of the system is calculated by summing the single-instance comprehensive deviation values ​​of all database instances and combining them from multiple dimensions. The weight of each instance is its performance impact factor. The overall deviation value of the system is subjected to time-series filtering. The window size is set to multiple time units, and the average deviation value within the window is calculated as the filtered overall deviation value at the current moment. The filtered overall deviation value is compared with a dynamic anomaly threshold, which is dynamically adjusted based on the distribution characteristics of the overall deviation of historical baseline data. When the filtered overall deviation value exceeds the dynamic anomaly threshold, it is determined that the system currently has a performance anomaly, triggering the anomaly pattern recognition process.

8. The method according to claim 5, characterized in that, When the overall deviation value exceeds a preset anomaly threshold, the anomaly pattern recognition process is triggered, including: Extract standardized state data sequence fragments when an anomaly is triggered, including complete data from multiple time windows before and after the anomaly occurs. The length of the time window is consistent with the fragment length used in the comprehensive deviation calculation. Generate an anomaly analysis data window, which contains performance change process data before and after the anomaly occurs. The trend of connection feature vector changes in the anomaly analysis data window is analyzed, the first difference of connection feature vectors within a continuous time unit is calculated, the time point when the absolute value of the difference exceeds the historical baseline fluctuation range is identified as the connection status change point, the deviation rate of the fluctuation amplitude of the number of connections within a unit time window from the historical fluctuation range is calculated, the deviation of the connection type proportion value from the historical distribution interval is calculated, and the change point, fluctuation deviation rate and proportion deviation degree are integrated to generate a connection anomaly feature set. The query behavior matrix in the anomaly analysis data window is analyzed, the probability density function of the query type distribution is calculated and the cumulative distribution difference of the historical distribution model is calculated, the query type with the distribution difference exceeding the set threshold is identified as the distribution change type, the deviation of the time consumption ratio of each query type from the historical ratio interval is calculated, the ratio of the sliding window variance of the execution frequency sequence to the historical variance benchmark is calculated, and the distribution change type, the time consumption ratio deviation and the frequency variance ratio are integrated to generate a query anomaly feature set. The resource consumption tensor in the anomaly analysis data window is analyzed, the growth rate of the resource usage sequence is calculated and the residual of the historical growth trend model is calculated, the resource types with residuals exceeding the confidence interval are identified as resources with changing usage, the ratio of the frequency of resource competition events to the historical average frequency is calculated, the coefficient of variation of the resource utilization rate sequence is calculated and the deviation from the historical stable interval is calculated, and the resources with changing usage, the competition frequency ratio and the utilization rate deviation are integrated to generate a resource anomaly feature set. The connection anomaly feature set, query anomaly feature set, and resource anomaly feature set are input into a preset anomaly pattern matching library. The anomaly pattern matching library contains feature templates of historical anomaly cases. The degree of matching between the current anomaly and historical cases is identified by feature similarity comparison to generate a preliminary anomaly type candidate list. The candidate list contains multiple anomaly types sorted by matching degree. Based on the preliminary candidate list of anomalies, an anomaly probability calculation model is constructed. The feature items in the connection, query and resource anomaly feature set are used as input variables. The probability of occurrence of each candidate anomaly type is calculated by the correlation between the features of historical anomaly cases and the anomaly type, and an anomaly occurrence probability distribution is generated. Based on the database performance correlation graph and the probability distribution of anomalies, the scope of anomaly impact is assessed. Starting from the instance where the anomaly occurred, the impact is propagated to adjacent instances according to the edge weights in the graph. The probability of each instance being affected by the anomaly is calculated, and the probability of being affected is equal to the probability of anomaly occurrence multiplied by the comprehensive dependency weight between instances. Summarize the probability distribution of anomalies and the assessment results of their impact range, and generate anomaly diagnosis results that include anomaly types, probability of occurrence, a list of impact instances, and propagation paths.

9. The method according to claim 1, characterized in that, The assessment, based on the probability distribution and impact range of the anomaly in the anomaly diagnosis results, calls a preset operation and maintenance strategy generation rule base to generate an operation and maintenance optimization plan containing resource scheduling instructions and query optimization suggestions, including: The probability distribution of abnormal occurrence in the abnormal diagnosis results is analyzed, and the abnormal type with the highest probability value is extracted as the dominant abnormal type probability value, and the next highest probability value is extracted as the associated abnormal type. The abnormal types are sorted from high to low probability values ​​to generate an abnormal type priority sequence. Based on the priority sequence of the anomaly types, a preset operation and maintenance strategy is queried to generate a rule base, and the core processing rules for the dominant anomaly type and the auxiliary processing rules for the associated anomaly type are matched. The core processing rules include key strategies to directly alleviate the anomaly, and the auxiliary processing rules include supplementary strategies to prevent the development of associated anomalies. Based on the instance spillover probability in the impact range assessment, a first processing strategy is applied to instances with a spillover probability exceeding a preset value, and a second processing strategy is applied to other instances, generating a differentiated strategy set, and resource allocation is tilted towards instances with high spillover probability. Analyze the resource requirements in the differentiated strategy set, calculate the adjustment amount of CPU, memory, storage and network resources, determine the direction and specific value of resource adjustment based on the current resource usage status and historical resource adjustment effect data, and generate a resource scheduling requirement list, which includes resource type, target instance and adjustment range; The resource scheduling requirement list is converted into standardized resource scheduling instructions. These instructions cover the operation types of resource allocation, release, and migration, and include execution time windows, operation steps, and verification indicators. The execution time window is selected during off-peak business periods. Extract query optimization requirements from the differentiated strategy set, combine them with the query anomaly feature set from the anomaly diagnosis results, and generate query statement rewriting suggestions, index optimization schemes, and execution plan adjustment strategies. The query optimization suggestions include expected performance improvement effect assessments and resource scheduling instructions, which are then combined to generate an operation and maintenance optimization scheme.

10. A computer system, characterized in that, include: A memory, wherein a computer program is stored; A processor is used to load the computer program to implement the service-oriented multi-type database unified real-time monitoring and operation and maintenance optimization method as described in any one of claims 1-9.

Images