Database analysis method and apparatus, electronic device, and storage medium

Abstract

The application relates to the technical field of data processing, in particular to a database analysis method and device, electronic equipment and storage medium, which are used for efficiently completing log analysis of each database in a distributed cluster according to different analysis requirements. The method comprises the following steps: acquiring a database analysis requirement, wherein the database analysis requirement represents an analysis requirement for a database table in a distributed database cluster; screening each target database node corresponding to the database analysis requirement from each database node of the distributed database cluster; analyzing, by using each target database node, a database log of each target database node based on the database analysis requirement, to obtain each node analysis result; and determining an analysis result corresponding to the database analysis requirement according to the each node analysis result.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a database analysis method, apparatus, electronic device and storage medium. Background Technology

[0002] Database logs are a fundamental file type in the underlying storage of a database. Analyzing database logs is one of the most common ways to diagnose database problems and analyze data performance. In large-scale distributed system environments, due to the extremely large amount of system data, hundreds or even thousands of physical machines are needed to form a distributed database cluster to provide database services to the upper-layer systems.

[0003] For this large distributed database cluster, the logs generated by the database are scattered across hundreds or thousands of different physical machines, making traditional database log analysis solutions inefficient. Summary of the Invention

[0004] This application provides a database analysis method, apparatus, electronic device, and storage medium for efficiently performing log analysis on various databases within a distributed cluster according to different analysis needs.

[0005] In a first aspect, embodiments of this application provide a database analysis method, the method comprising:

[0006] Obtain database analysis requirements, wherein the database analysis requirements represent the analysis requirements for database tables in a distributed database cluster;

[0007] From the database nodes of the distributed database cluster, select the target database nodes that correspond to the database analysis requirements;

[0008] Based on the aforementioned database analysis requirements, the database logs of each target database node are analyzed to obtain the analysis results of each node.

[0009] Based on the analysis results of each node, the analysis results corresponding to the database analysis requirements are determined.

[0010] In some possible embodiments, selecting target database nodes corresponding to the database analysis requirements from the database nodes of the distributed database cluster includes:

[0011] Determine the database tables corresponding to the database analysis requirements;

[0012] Based on the pre-configured information of each database node in the distributed database cluster, target database nodes corresponding to the database table are selected from the database nodes.

[0013] In some possible embodiments, based on the database analysis requirements, the step of analyzing the database logs of each target database node to obtain the analysis results of each node includes:

[0014] The database analysis requirements are broken down into multiple node requirements, and these multiple node requirements are sent to their respective target database nodes, wherein each node requirement corresponds to a target database node.

[0015] The system receives node analysis results sent by each target database node, wherein the node analysis results sent by each target database node are obtained by analyzing their respective database logs.

[0016] In some possible embodiments, sending multiple node requests to their respective target database nodes also includes:

[0017] Based on the information of each database node in the pre-configured distributed database cluster, the interference data corresponding to each target database node is determined, wherein the interference data represents the data that causes interference when performing anomaly analysis on the database logs.

[0018] Each piece of interference data is sent to its corresponding target database node so that the target database node can filter out the interference data when performing database log analysis.

[0019] In some possible embodiments, sending multiple node requests to their respective target database nodes, and sending each piece of interference data to its respective target database node, includes:

[0020] Based on the node requirements and interference data corresponding to each target database node, generate the program script corresponding to each target database node;

[0021] Send the program scripts corresponding to each target database node to their respective target database nodes.

[0022] In some possible embodiments, breaking down the database analysis requirement into multiple node requirements includes:

[0023] Based on the information of each database node in the pre-configured distributed database cluster, determine the requirements in the database analysis requirements that correspond to each target database node;

[0024] Based on the requirements corresponding to each target database node, the database analysis requirements are broken down into multiple node requirements.

[0025] In some possible embodiments, determining the analysis result corresponding to the database analysis requirement based on the analysis results of each node includes:

[0026] The analysis results of each node are merged, summarized, statistically analyzed, or subjected to at least one of the following operations to determine the analysis results corresponding to the database analysis requirements.

[0027] In some possible embodiments, the information of each database node in the pre-configured distributed database cluster includes any one or more of the following:

[0028] Cluster identifier;

[0029] Database node IP address;

[0030] Database users;

[0031] Database log storage address;

[0032] The database's metadata information.

[0033] Secondly, this application provides a data analysis system, including a client and multiple database nodes located in a distributed database cluster, wherein:

[0034] The multiple database nodes are configured to analyze their respective database logs to obtain the analysis results of each node;

[0035] The client is configured to: acquire database analysis requirements, wherein the database analysis requirements represent the analysis requirements for database tables in a distributed database cluster; select target database nodes corresponding to the database analysis requirements from the database nodes of the distributed database cluster; analyze the database logs of each target database node based on the database analysis requirements to obtain the analysis results of each node; and determine the analysis results corresponding to the database analysis requirements based on the analysis results of each node.

[0036] Thirdly, this application provides a database analysis apparatus, the apparatus comprising:

[0037] The requirement acquisition unit is used to acquire database analysis requirements, wherein the database analysis requirements represent the analysis requirements for database tables in the distributed database cluster.

[0038] The filtering node unit is used to filter out the target database nodes corresponding to the database analysis requirements from the database nodes of the distributed database cluster.

[0039] The node analysis unit is used to analyze the database logs of each target database node based on the database analysis requirements, and obtain the analysis results of each node.

[0040] The comprehensive analysis unit is used to determine the analysis results corresponding to the database analysis requirements based on the analysis results of each node.

[0041] In some possible embodiments, the filtering node unit is specifically used for:

[0042] Determine the database tables corresponding to the database analysis requirements;

[0043] Based on the pre-configured information of each database node in the distributed database cluster, target database nodes corresponding to the database table are selected from the database nodes.

[0044] In some possible embodiments, the node analysis unit is specifically used for:

[0045] The database analysis requirements are broken down into multiple node requirements, and these multiple node requirements are sent to their respective target database nodes, wherein each node requirement corresponds to a target database node.

[0046] The system receives node analysis results sent by each target database node, wherein the node analysis results sent by each target database node are obtained by analyzing their respective database logs.

[0047] In some possible embodiments, the node analysis unit is further configured to:

[0048] Based on the information of each database node in the pre-configured distributed database cluster, the interference data corresponding to each target database node is determined, wherein the interference data represents the data that causes interference when performing anomaly analysis on the database logs.

[0049] Each piece of interference data is sent to its corresponding target database node so that the target database node can filter out the interference data when performing database log analysis.

[0050] In some possible embodiments, the node analysis unit is specifically used for:

[0051] Based on the node requirements and interference data corresponding to each target database node, generate the program script corresponding to each target database node;

[0052] Send the program scripts corresponding to each target database node to their respective target database nodes.

[0053] In some possible embodiments, the node analysis unit is specifically used for:

[0054] Based on the information of each database node in the pre-configured distributed database cluster, determine the requirements in the database analysis requirements that correspond to each target database node;

[0055] Based on the requirements corresponding to each target database node, the database analysis requirements are broken down into multiple node requirements.

[0056] In some possible embodiments, the comprehensive analysis unit is specifically used for:

[0057] The analysis results of each node are merged, summarized, statistically analyzed, or subjected to at least one of the following operations to determine the analysis results corresponding to the database analysis requirements.

[0058] In some possible embodiments, the information of each database node in the pre-configured distributed database cluster includes any one or more of the following:

[0059] Cluster identifier;

[0060] Database node IP address;

[0061] Database users;

[0062] Database log storage address;

[0063] The database's metadata information.

[0064] Fourthly, this application provides an electronic device, comprising:

[0065] Memory, used to store program instructions;

[0066] A processor is configured to invoke program instructions stored in the memory and execute the steps of the method described in any one of the first aspects according to the obtained program instructions.

[0067] Fifthly, this application provides a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a computer, cause the computer to perform the method described in any one of the first aspects.

[0068] In a sixth aspect, this application provides a computer program product comprising: computer program code, which, when run on a computer, causes the computer to perform the method described in any one of the first aspects.

[0069] The method provided in this application has at least the following beneficial effects:

[0070] In this embodiment, based on different database analysis needs, each database node in the distributed database cluster is filtered to obtain the target database node corresponding to the database analysis needs. Each target database node is used to analyze its own database logs first, and then the final analysis result is determined based on the analysis results of each node. This is suitable for large distributed database systems with a large number of database nodes, and can flexibly and efficiently complete the database log analysis needs.

[0071] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0072] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0073] Figure 1 A flowchart illustrating the implementation of a database analysis method provided in this application embodiment;

[0074] Figure 2 A flowchart illustrating the implementation of a database log analysis method provided in this application embodiment;

[0075] Figure 3 A schematic diagram of a database analysis system provided in an embodiment of this application;

[0076] Figure 4 An interactive flowchart of a database log analysis system provided in this application embodiment;

[0077] Figure 5 A schematic diagram of a data analysis system architecture provided in this application embodiment;

[0078] Figure 6 A schematic diagram of a database analysis device provided in an embodiment of this application;

[0079] Figure 7 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0080] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0081] The terms "first" and "second" in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising" and any variations thereof are intended to cover non-exclusive protection. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. The term "multiple" in this application can mean at least two, for example, two, three, or more, and the embodiments of this application do not impose limitations.

[0082] The data collection, dissemination, and use in this application all comply with relevant national laws and regulations.

[0083] In this embodiment of the invention, the term "distributed system" refers to a software system built on a network. Due to the characteristics of software, distributed systems exhibit high cohesion and transparency. The difference between a network and a distributed system lies more in the high-level software (especially the operating system) than in the hardware.

[0084] In this embodiment of the invention, the term "distributed database cluster" usually refers to a logical database cluster consisting of multiple physical database machine nodes.

[0085] In this embodiment of the invention, the term "database log" refers to a type of underlying file used for database storage, and is commonly used for database problem diagnosis and database performance analysis.

[0086] Before introducing the database analysis method provided in the embodiments of this application, for ease of understanding, the technical background of the embodiments of this application will be described in detail below.

[0087] Database logs are a fundamental file type stored at the underlying level of a database. Analyzing database logs is one of the most common ways to diagnose database problems and analyze data performance. In large-scale distributed system environments, due to the extremely large volume of system data, hundreds or even thousands of physical machines are needed to form a distributed database cluster to provide database services to upper-layer systems. For this large distributed database cluster, the logs generated by the database are scattered across hundreds or thousands of different physical machines, making traditional database log analysis methods inefficient.

[0088] Database logs are typically written to files in chronological order. When the time span is long, there are many log files to analyze, and the content is highly repetitive. Currently, the Percona Toolkit tool pt-query-digest can be used for database log analysis. pt-query-digest is a tool for analyzing slow queries in MySQL (relational database management system). It can analyze binlog, general log, and slowlog, specify time ranges, and group and statistically analyze various queries to obtain the execution count, time, and percentage of each query. The analysis results can be used to identify problems and optimize the database. However, pt-query-digest is only suitable for small, single-machine database systems. Large distributed database systems have many machines, and log data is scattered. Database tables are horizontally partitioned across multiple machines in a cluster environment. There are no flexible and efficient methods for database location and analysis. Even if analysis is performed machine by machine, the data analysis results are relatively scattered, which is not conducive to clearly diagnosing database problems. Currently, there is no log analysis solution applicable to large-scale distributed multi-database cluster database systems.

[0089] In view of this, the embodiments of this application provide a database analysis method applied to a large-scale distributed multi-database cluster. The database analysis method provided by this application filters each database node of the distributed database cluster according to different database analysis requirements to obtain the target database node corresponding to the database analysis requirements. After analyzing the database logs of each target database node, the final analysis result is determined based on the analysis results of each node. It can be applied to large-scale distributed database systems with a large number of database nodes, and can flexibly and efficiently complete the database log analysis requirements.

[0090] This embodiment provides a database analysis method that can be executed by a terminal, such as a client.

[0091] The database analysis method of this application embodiment will be described below with reference to the accompanying drawings and specific embodiments.

[0092] Reference Figure 1 As shown in the figure, the database analysis method provided in this application embodiment specifically includes the following steps:

[0093] Step 100: Obtain database analysis requirements, wherein the database analysis requirements represent the analysis requirements for database tables in the distributed database cluster;

[0094] In practice, database analysis requirements can include information such as the names and identifiers of database tables, as well as the specific content of the analysis. For example, database analysis requirements may include whether the execution response time of a certain transaction's SQL (Structured Query Language) statement is normal.

[0095] Optionally, the database analysis requirement is a database analysis requirement received from user input.

[0096] Step 101: Select the target database nodes that correspond to the database analysis requirements from the database nodes of the distributed database cluster;

[0097] In some embodiments, after obtaining the database analysis requirements, the database nodes of the distributed database cluster are filtered according to the database analysis requirements. The specific filtering steps are as follows:

[0098] Step a) Determine the database table corresponding to the database analysis requirements;

[0099] During implementation, upon receiving a database analysis request, the first step is to determine which database table the database log analysis is for, i.e., to identify the database table to be analyzed.

[0100] Step b) Based on the pre-configured information of each database node in the distributed database cluster, select the target database nodes corresponding to the database table from the database nodes.

[0101] During implementation, after identifying the database table to be analyzed, the information of each database node in the pre-configured distributed database cluster is read to determine the location of the database logs required by the database table within the distributed database cluster, as well as the database's metadata information. The location of the database logs within the distributed database cluster can be understood as the database node's IP address, the database log storage address within the database node, and other similar information.

[0102] It should be noted that the database node in this embodiment can specifically be a database server.

[0103] Optionally, a cluster configuration file is pre-configured, which includes information about each database node in the distributed database cluster. When a database analysis request is received, the cluster configuration file is read to determine the target database nodes and metadata information of each target database corresponding to the database analysis request.

[0104] In some embodiments, the information of each database node in the pre-configured distributed database cluster includes any one or more of the following:

[0105] Cluster identifier; database node IP address; database user; database log storage address; database metadata information.

[0106] In practice, the system receives database analysis requests from users and, based on the pre-configured information of each database node in the distributed database cluster, determines which database nodes in the distributed database cluster contain the database table to be analyzed, thereby identifying the target database nodes corresponding to the database table.

[0107] In some embodiments, interference data for each target database node can be pre-determined for use in subsequent analysis to filter out interference data, thereby improving the efficiency and accuracy of database log analysis. The specific implementation steps are as follows:

[0108] Based on the information of each database node in the pre-configured distributed database cluster, the interference data corresponding to each target database node is determined, wherein the interference data represents the data that causes interference when performing anomaly analysis on the database logs.

[0109] For example, based on the information of each database node in the pre-configured distributed database cluster, it is known that there is a normal SQL statement on the database table TABLE_A that runs for a long time and consumes a lot of time, which is likely to interfere with log analysis. Therefore, this normal SQL statement is used as the interference data corresponding to each target database node.

[0110] Optionally, after determining the interference data corresponding to each target database node, notify each target database node so that it can filter out the interference data and obtain more accurate analysis results when performing its own database log analysis.

[0111] Step 102: Based on the database analysis requirements, analyze the database logs of each target database node to obtain the analysis results of each node;

[0112] In some embodiments, the analysis results for each node are obtained through the following process:

[0113] Process 1) Break down the database analysis requirements into multiple node requirements, and send the multiple node requirements to their respective target database nodes, wherein each node requirement corresponds to the target database node;

[0114] In some embodiments, the database analysis requirement is broken down into multiple node requirements through the following steps:

[0115] Based on the information of each database node in the pre-configured distributed database cluster, determine the requirements corresponding to each target database node in the database analysis requirements; based on the requirements corresponding to each target database node, break down the database analysis requirements into multiple node requirements.

[0116] In practice, each target database node corresponds to a node requirement. Each node requirement is sent to its corresponding target database node so that each target database node can perform preliminary analysis of the database logs based on the received node requirements and obtain the analysis results of each node.

[0117] Optionally, the node requirements of different target database nodes can be the same or different, depending on the information of each database node and the database analysis requirements. This embodiment does not impose too many restrictions on this.

[0118] In some embodiments, this embodiment may also send interference data to each target database node, and the specific steps are as follows:

[0119] Based on the information of each database node in the pre-configured distributed database cluster, the interference data corresponding to each target database node is determined, wherein the interference data represents the data that causes interference when performing anomaly analysis on the database log; each interference data is sent to its corresponding target database node so that the target database node can filter out the interference data when performing database log analysis.

[0120] In some embodiments, when sending multiple node requests to their respective target database nodes, and sending various interference data to their respective target database nodes, the specific steps are as follows:

[0121] Based on the node requirements and interference data corresponding to each target database node, generate the program script corresponding to each target database node; and send the program script corresponding to each target database node to its respective target database node.

[0122] During implementation, after receiving the program script, each target database node filters out interfering data from its own database logs and performs a preliminary analysis of the filtered database logs based on the node's requirements to obtain the node analysis results.

[0123] Step 2) Receive the node analysis results sent by each target database node, wherein the node analysis results sent by each target database node are obtained by each target database node analyzing its own database logs.

[0124] Step 103: Based on the analysis results of each node, determine the analysis results corresponding to the database analysis requirements.

[0125] In some embodiments, at least one of the following operations is performed on the analysis results of each node: merging, summarizing, statistical analysis, and data analysis, to determine the analysis results corresponding to the database analysis requirements.

[0126] During implementation, the analysis results are used to determine whether each target database node is abnormal, for example, including any of the following analysis results:

[0127] Analysis Result 1) In the analysis results, the execution time of the SQL statement TABLE_A on target database node 2 is significantly longer than that on other target database nodes. Therefore, target database node 2 is determined to be abnormal. Further analysis reveals relevant core data on target database node 2, including CPU, memory, IO, and network metrics.

[0128] Analysis Result 2) In the analysis results, the node analysis results of the database logs returned by each target database node are similar, confirming that there are no anomalies in each target database node. The analysis results of each node can be merged, summarized, and statistically analyzed. Furthermore, some indicators can be recalculated, including the total number of SQL executions and the average time consumption of SQL statements on various indicators such as disk I / O and locks.

[0129] This application's embodiments efficiently locate the storage location of the log data to be analyzed in the distributed database cluster according to data analysis needs, process and select the corresponding data, and provide analysis results by presenting the analysis data scattered across multiple data nodes in a logically unified manner.

[0130] like Figure 2 As shown in the figure, this application embodiment provides an implementation flow of a database log analysis method, applied to a distributed database cluster. The specific implementation flow is as follows:

[0131] Step 200: Pre-configure the cluster configuration file, which includes information about each database node in the distributed database cluster;

[0132] The information of each database node includes, but is not limited to, any one or more of the following:

[0133] Cluster identifier; database node IP address; database user; database log storage address; database metadata information.

[0134] Step 201: Obtain database analysis requirements, which represent the analysis requirements for database tables in the distributed database cluster;

[0135] Step 202: Determine the database table corresponding to the database analysis requirements;

[0136] Step 203: Based on the cluster configuration file, select the target database nodes corresponding to the database table from each database node, as well as the metadata information of the database in each target database node;

[0137] Step 204: Based on the cluster configuration file, determine the requirements corresponding to each target database node in the database analysis requirements, and break down the database analysis requirements into multiple node requirements based on the requirements corresponding to each target database node.

[0138] The node requirements and target database nodes are in a one-to-one correspondence.

[0139] Step 205: Determine the interference data corresponding to each target database node according to the cluster configuration file;

[0140] The interference data refers to the data that causes interference when performing anomaly analysis on the database logs;

[0141] Step 206: Generate the program script corresponding to each target database node based on the node requirements and interference data corresponding to each target database node;

[0142] Step 207: Send the program scripts corresponding to each target database node to their respective target database nodes.

[0143] Step 208: Receive the node analysis results sent by each target database node;

[0144] Step 209: Perform at least one of the following operations on the analysis results of each node: merging, summarizing, statistical analysis, and data analysis, to determine the analysis results corresponding to the database analysis requirements.

[0145] This application can identify the storage strategy or sharding scheme of database tables based on the configured cluster configuration information, enabling flexible and efficient completion of database log analysis needs in environments with a large number of database nodes in large distributed database systems. In multi-database node log analysis scenarios, it can provide analysis results from a unified view of scattered log data, according to the application's perspective.

[0146] Based on the same inventive concept, this application provides a database analysis system, please refer to... Figure 3The system includes: a client 300 and multiple database nodes 301 located in a distributed database cluster, wherein:

[0147] The multiple database nodes 301 are configured to analyze their respective database logs and obtain the analysis results of each node;

[0148] The client 300 is configured to perform the following steps:

[0149] Obtain database analysis requirements, wherein the database analysis requirements represent the analysis requirements for database tables in a distributed database cluster;

[0150] From the database nodes of the distributed database cluster, select the target database nodes that correspond to the database analysis requirements;

[0151] Based on the aforementioned database analysis requirements, the database logs of each target database node are analyzed to obtain the analysis results of each node.

[0152] Based on the analysis results of each node, the analysis results corresponding to the database analysis requirements are determined.

[0153] In one possible embodiment, the client 300 is specifically configured to perform:

[0154] Determine the database tables corresponding to the database analysis requirements;

[0155] Based on the pre-configured information of each database node in the distributed database cluster, target database nodes corresponding to the database table are selected from the database nodes.

[0156] In one possible embodiment, the client 300 is specifically configured to perform:

[0157] The database analysis requirements are broken down into multiple node requirements, and these multiple node requirements are sent to their respective target database nodes, wherein each node requirement corresponds to a target database node.

[0158] The system receives node analysis results sent by each target database node, wherein the node analysis results sent by each target database node are obtained by analyzing their respective database logs.

[0159] In one possible embodiment, multiple node requests are sent to their respective target database nodes, and the client 300 is further configured to execute:

[0160] Based on the information of each database node in the pre-configured distributed database cluster, the interference data corresponding to each target database node is determined, wherein the interference data represents the data that causes interference when performing anomaly analysis on the database logs.

[0161] Each piece of interference data is sent to its corresponding target database node so that the target database node can filter out the interference data when performing database log analysis.

[0162] In one possible embodiment, the client 300 is specifically configured to perform:

[0163] Based on the node requirements and interference data corresponding to each target database node, generate the program script corresponding to each target database node;

[0164] Send the program scripts corresponding to each target database node to their respective target database nodes.

[0165] In one possible embodiment, the client 300 is specifically configured to perform:

[0166] Based on the information of each database node in the pre-configured distributed database cluster, determine the requirements in the database analysis requirements that correspond to each target database node;

[0167] Based on the requirements corresponding to each target database node, the database analysis requirements are broken down into multiple node requirements.

[0168] In one possible embodiment, the client 300 is specifically configured to perform:

[0169] The analysis results of each node are merged, summarized, statistically analyzed, or subjected to at least one of the following operations to determine the analysis results corresponding to the database analysis requirements.

[0170] In one possible embodiment, the information of each database node in the pre-configured distributed database cluster includes any one or more of the following:

[0171] Cluster identifier; database node IP address; database user; database log storage address; database metadata information.

[0172] like Figure 4 As shown in the illustration, this application also provides an interaction flow for a database log analysis system. This system includes a client and multiple database nodes and can be applied to distributed database clusters. The specific interaction flow is as follows:

[0173] Step 400: The client pre-configures the cluster configuration file;

[0174] The cluster configuration file contains information about each database node in the distributed database cluster.

[0175] Step 401: The client receives the database analysis request input by the user;

[0176] Database analysis requirements represent the need for analysis of database tables in a distributed database cluster.

[0177] Step 402: The client determines the database table corresponding to the database analysis requirements;

[0178] Step 403: The client selects the target database nodes corresponding to the database table from each database node according to the cluster configuration file, as well as the metadata information of the database in each target database node;

[0179] Step 404: The client determines the requirements corresponding to each target database node in the database analysis requirements according to the cluster configuration file, and splits the database analysis requirements into multiple node requirements according to the requirements corresponding to each target database node.

[0180] Step 405: The client determines the interference data corresponding to each target database node based on the cluster configuration file;

[0181] Step 406: Generate the program script corresponding to each target database node based on the node requirements and interference data corresponding to each target database node;

[0182] Step 407: The client sends the program scripts corresponding to each target database node to their respective target database nodes.

[0183] Step 408: Each target database node analyzes its own database logs according to the program script to obtain the corresponding node analysis results;

[0184] Step 409: Each target database node sends its node analysis results to the client;

[0185] Step 410: The client performs at least one of the following operations on the analysis results of each node: merging, summarizing, statistical analysis, and data analysis, to determine the analysis results corresponding to the database analysis requirements.

[0186] like Figure 5As shown in the figure, this application embodiment also provides a schematic diagram of a data analysis system architecture, wherein the client pre-configures a cluster configuration file, receives database analysis requirements for input, and splits the database analysis requirements into multiple node requirements according to the cluster configuration file, and sends them to the corresponding target database nodes, that is, each node requirement is sent to the corresponding database node A, database node N, and database node K. Each target database node performs a preliminary analysis of its own database logs based on the node requirements, obtains its corresponding node analysis results, and sends the node analysis results to the client, which then performs further analysis to obtain the final analysis results.

[0187] Based on the same inventive concept, embodiments of this application provide a database analysis device, please refer to... Figure 6 The device includes:

[0188] The requirement acquisition unit 600 is used to acquire database analysis requirements, wherein the database analysis requirements represent the analysis requirements for database tables in the distributed database cluster.

[0189] The filtering node unit 601 is used to filter out each target database node corresponding to the database analysis requirements from each database node of the distributed database cluster.

[0190] The node analysis unit 602 is used to analyze the database logs of each target database node based on the database analysis requirements, and obtain the analysis results of each node.

[0191] The comprehensive analysis unit 603 is used to determine the analysis results corresponding to the database analysis requirements based on the analysis results of each node.

[0192] In some possible embodiments, the filtering node unit 601 is specifically used for:

[0193] Determine the database tables corresponding to the database analysis requirements;

[0194] Based on the pre-configured information of each database node in the distributed database cluster, target database nodes corresponding to the database table are selected from the database nodes.

[0195] In some possible embodiments, the node analysis unit 602 is specifically used for:

[0196] The database analysis requirements are broken down into multiple node requirements, and these multiple node requirements are sent to their respective target database nodes, wherein each node requirement corresponds to a target database node.

[0197] The system receives node analysis results sent by each target database node, wherein the node analysis results sent by each target database node are obtained by analyzing their respective database logs.

[0198] In some possible embodiments, the node analysis unit 602 is further configured to:

[0199] Based on the information of each database node in the pre-configured distributed database cluster, the interference data corresponding to each target database node is determined, wherein the interference data represents the data that causes interference when performing anomaly analysis on the database logs.

[0200] Each piece of interference data is sent to its corresponding target database node so that the target database node can filter out the interference data when performing database log analysis.

[0201] In some possible embodiments, the node analysis unit 602 is specifically used for:

[0202] Based on the node requirements and interference data corresponding to each target database node, generate the program script corresponding to each target database node;

[0203] Send the program scripts corresponding to each target database node to their respective target database nodes.

[0204] In some possible embodiments, the node analysis unit 602 is specifically used for:

[0205] Based on the information of each database node in the pre-configured distributed database cluster, determine the requirements in the database analysis requirements that correspond to each target database node;

[0206] Based on the requirements corresponding to each target database node, the database analysis requirements are broken down into multiple node requirements.

[0207] In some possible embodiments, the comprehensive analysis unit 603 is specifically used for:

[0208] The analysis results of each node are merged, summarized, statistically analyzed, or subjected to at least one of the following operations to determine the analysis results corresponding to the database analysis requirements.

[0209] In some possible embodiments, the information of each database node in the pre-configured distributed database cluster includes any one or more of the following:

[0210] Cluster identifier;

[0211] Database node IP address;

[0212] Database users;

[0213] Database log storage address;

[0214] The database's metadata information.

[0215] Based on the same inventive concept, this application provides an electronic device that can realize the functions of the database analysis device described above. Please refer to... Figure 7 The device includes a processor 701 and a memory 702, the memory 702 being used to store program instructions; the processor 701 is used to call the program instructions stored in the memory 702 and execute the steps included in any of the database analysis methods in the above embodiments according to the obtained program instructions.

[0216] This application does not limit the specific connection medium between the memory 702 and the processor 701. For example, the memory 702 and the processor 701 are connected via a bus, which can be divided into an address bus, a data bus, a control bus, etc.

[0217] The memory 702 may include read-only memory (ROM) and random access memory (RAM), and may also include non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0218] The processor 701 mentioned above can be a general-purpose processor, including a central processing unit, a network processor (NP), etc.; it can also be a digital signal processing (DSP), an application-specific integrated circuit, a field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc.

[0219] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium. The computer program product includes computer program code, which, when executed on a computer, causes the computer to perform any of the database analysis methods discussed above. Since the principle by which the computer-readable storage medium solves the problem is similar to that of the database analysis methods, the implementation of the computer-readable storage medium can be found in the implementation of the methods, and repeated details will not be elaborated further.

[0220] In specific implementation, computer storage media can include: Universal Serial Bus Flash Drive (USB), portable hard drive, Read-Only Memory (ROM), Random Access Memory (RAM), magnetic disk or optical disk, and other storage media that can store program code.

[0221] Based on the same inventive concept, this application also provides a computer program product, which includes computer program code. When the computer program code is run on a computer, it causes the computer to execute any of the database analysis methods discussed above. Since the principle by which the above-described computer program product solves the problem is similar to that of the database analysis method, the implementation of the above-described computer program product can be referred to the implementation of the method, and repeated details will not be described again.

[0222] Computer program products may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0223] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0224] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0225] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0226] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of user-operated steps to be executed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0227] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A database analysis method, characterized in that, include: Obtain database analysis requirements, wherein the database analysis requirements characterize the analysis needs of database tables in a distributed database cluster; the database analysis requirements include whether the SQL statement execution response time is normal; The database table corresponding to the database analysis requirement is determined; based on the pre-configured information of each database node in the distributed database cluster, target database nodes corresponding to the database table are selected from the database nodes; the pre-configured information of each database node in the distributed database cluster includes: cluster identifier; database node IP address; database user; database log storage address; database metadata information; the database node includes a database server; The database analysis requirements are broken down into multiple node requirements, each corresponding to a target database node. Based on the information of each database node in a pre-configured distributed database cluster, interference data corresponding to each target database node is determined, where the interference data represents data that causes interference during anomaly analysis of the database logs. Based on the node requirements and interference data corresponding to each target database node, a program script corresponding to each target database node is generated. The program scripts corresponding to each target database node are sent to their respective target database nodes so that the target database nodes filter out interference data during database log analysis. The node analysis results sent by each target database node are received, where the node analysis results are obtained by each target database node through analysis of its own database logs. The analysis results of each node are merged, summarized, statistically analyzed, and the total number of SQL executions and the average time consumption of SQL statements on disk I / O and locks are recalculated to determine the analysis results corresponding to the database analysis requirements; based on the analysis results, it is determined whether each target database node is abnormal.

2. The method according to claim 1, characterized in that, The process of breaking down the database analysis requirements into multiple node requirements includes: Based on the information of each database node in the pre-configured distributed database cluster, determine the requirements in the database analysis requirements that correspond to each target database node; Based on the requirements corresponding to each target database node, the database analysis requirements are broken down into multiple node requirements.

3. A data analysis system, characterized in that, This includes the client and multiple database nodes located in a distributed database cluster, among which: The multiple database nodes are configured to analyze their respective database logs to obtain the analysis results of each node; The client is configured to acquire database analysis requirements, wherein the database analysis requirements represent the analysis needs of database tables in a distributed database cluster; the database analysis requirements include whether the SQL statement execution response time is normal; determine the database table corresponding to the database analysis requirements; and, based on pre-configured information of each database node in the distributed database cluster, filter out each target database node corresponding to the database table from the database nodes; the pre-configured information of each database node in the distributed database cluster includes: cluster identifier; database node IP address; database user; database log storage address; database metadata information; the database node includes a database server; split the database analysis requirements into multiple node requirements, wherein each node requirement corresponds to a target database node; and determine each target database node based on the pre-configured information of each database node in the distributed database cluster. The system generates interference data corresponding to each target database node, where the interference data represents data that causes interference during anomaly analysis of the database logs. Based on the node requirements and interference data for each target database node, it generates program scripts for each target database node. These program scripts are then sent to their respective target database nodes to filter out interference data during database log analysis. The system receives node analysis results from each target database node, obtained by analyzing their respective database logs. The system merges, summarizes, statistically analyzes, and recalculates the total number of SQL executions and the average time consumed by SQL statements on disk I / O and locks to determine the analysis results corresponding to the database analysis requirements. Finally, it determines whether each target database node is abnormal based on the analysis results.

4. A database analysis device, characterized in that, include: The requirement acquisition unit is used to acquire database analysis requirements, wherein the database analysis requirements represent the analysis requirements for database tables in the distributed database cluster. The database analysis requirements include whether the SQL statement execution response time is normal; A filtering node unit is used to determine the database table corresponding to the database analysis requirement; based on the pre-configured information of each database node in the distributed database cluster, it filters out each target database node corresponding to the database table from the database nodes; the pre-configured information of each database node in the distributed database cluster includes: cluster identifier; database node IP address; database user; database log storage address; database metadata information; the database node includes a database server; The node analysis unit is used to break down the database analysis requirements into multiple node requirements, each corresponding to a target database node; determine the interference data corresponding to each target database node based on the information of each database node in a pre-configured distributed database cluster, wherein the interference data represents the data that causes interference when performing anomaly analysis on the database logs; generate program scripts corresponding to each target database node based on the node requirements and interference data; send the program scripts corresponding to each target database node to their respective target database nodes so that the target database nodes can filter out interference data when performing database log analysis; and receive the node analysis results sent by each target database node, wherein the node analysis results sent by each target database node are obtained by each target database node analyzing its own database logs. The comprehensive analysis unit is used to merge, summarize, statistically analyze, and perform data analysis on the analysis results of each node, recalculate the total number of SQL executions, the average time consumption of SQL statements on disk I / O and locks, determine the analysis results corresponding to the database analysis requirements, and determine whether each target database node is abnormal based on the analysis results.

5. An electronic device, characterized in that, include: Memory, used to store program instructions; A processor is configured to invoke program instructions stored in the memory and execute the steps included in the method of claim 1 or 2 according to the obtained program instructions.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a computer, cause the computer to perform the method as described in claim 1 or 2.

7. A computer program product, characterized in that, The computer program product includes: computer program code, which, when run on a computer, causes the computer to perform the method described in claim 1 or 2.

Images