Database cluster log collection method, system, equipment and medium

Abstract

The present application discloses a database cluster log collection method, system, device and medium, which are applied to the field of database technology to solve the problem of low efficiency in fault cause analysis of database clusters in the prior art. Specifically, based on a first configuration file, according to the type of fault to be checked, the database node information to be checked corresponding to the database cluster and the log type information to be checked corresponding to the database node information to be checked are determined; based on a second configuration file, according to the log type information to be checked, the target log collection component information is determined; the target log collection component corresponding to the target log collection component information is called, and based on the log type information to be checked, the log file of the database node to be checked represented by the database node information to be checked is collected, thereby realizing automatic log collection when the database cluster fails, improving the log collection efficiency when the database cluster fails, and further improving the efficiency of fault cause analysis of the database cluster.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a database cluster log collection method, system, device and medium. Background Art

[0002] A database cluster typically includes multiple database nodes, each of which contains one or more database instances. Each database instance corresponds to one or more log files. In the prior art, when a database cluster failure occurs, it is often necessary to manually review the log files of all database instances across the multiple database nodes within the cluster to analyze the cause of the failure. When there are a large number of database nodes, manually reviewing the log files of all database instances on each database node is time-consuming, resulting in inefficient database cluster failure analysis. Summary of the Invention

[0003] This application provides a database cluster log collection method, system, device, and medium to address the low efficiency of existing technologies in analyzing the cause of database cluster failures. The technical solutions provided by this application are as follows:

[0004] In one aspect, the present application provides a database cluster log collection method, comprising:

[0005] Get the type of fault to be investigated in the database cluster;

[0006] Based on the first configuration file, according to the type of fault to be checked, determining the database node information to be checked corresponding to the database cluster and the log type information to be checked corresponding to the database node information to be checked; wherein the first configuration file is configured with database node information corresponding to different fault types and log type information corresponding to the database node information;

[0007] Based on the second configuration file, according to the log type information to be checked corresponding to the database node information to be checked, determining the target log collection component information corresponding to the database node information to be checked; wherein the second configuration file is configured with log collection component information corresponding to different log type information;

[0008] The target log collection component corresponding to the target log collection component information is called to collect the log file of the database node to be checked represented by the database node to be checked information based on the log type information to be checked.

[0009] Optionally, based on the first configuration file and according to the type of the fault to be checked, determining the database node information to be checked corresponding to the database cluster and the log type information to be checked corresponding to the database node information to be checked includes:

[0010] Calling the parameter parsing component to parse the database node information corresponding to the fault type to be checked and the log type information corresponding to the database node information from the first configuration file;

[0011] The parsed database node information and the log type information corresponding to the database node information are determined as the to-be-checked database node information corresponding to the database cluster and the to-be-checked log type information corresponding to the to-be-checked database node information.

[0012] Optionally, based on the second configuration file and according to the log type information to be checked corresponding to the database node information to be checked, determining target log collection component information corresponding to the database node information to be checked includes:

[0013] Calling the parameter parsing component to parse the log collection component information corresponding to the log type information to be checked from the second configuration file;

[0014] The parsed log collection component information is determined as the target log collection component information corresponding to the database node information to be checked.

[0015] Optionally, the target log collection component corresponding to the target log collection component information is called to collect log files of the database node to be checked represented by the database node to be checked information based on the log type information to be checked, including:

[0016] Call the task generation component to generate a log collection task based on the database node information and the log type information to be checked corresponding to the database node information;

[0017] Call the task execution component to execute the log collection task, and in the process of executing the log collection task, call the target log collection component corresponding to the target log collection component information to collect the log file corresponding to the log type information to be checked from the database node to be checked represented by the database node information to be checked.

[0018] Optionally, a task generation component is called to generate a log collection task based on the database node information and the log type information to be checked corresponding to the database node information, including:

[0019] The task generation component is called to obtain the user login account information, and a log collection task is generated based on the user login account information, the database node information, and the log type information to be checked corresponding to the database node information.

[0020] Optionally, after calling the target log collection component corresponding to the target log collection component information to collect log files corresponding to the log type information to be checked from the database node to be checked represented by the database node to be checked information, the method further includes:

[0021] Calling the task execution component to save the log file to the temporary output directory of the database node to be checked represented by the information of the database node to be checked, and compressing all the log files in the temporary output directory to obtain a log file compression package;

[0022] When it is determined that the log file downloading conditions are met, the log file compressed package is downloaded from the temporary output directory to the local output directory, and the log file compressed package in the local output directory is decompressed to obtain the log file of the database node to be checked represented by the information of the database node to be checked.

[0023] Optionally, after calling the target log collection component corresponding to the target log collection component information and collecting the log file of the database node to be checked represented by the database node to be checked information based on the log type information to be checked, the method further includes:

[0024] Generates and outputs a log collection result statistical report based on the component execution logs during the log file collection process.

[0025] On the other hand, the present application provides a database cluster log collection system, including:

[0026] The database cluster fault monitoring component is used to obtain the type of fault to be investigated in the database cluster;

[0027] A database parameter acquisition component is configured to determine, based on a first configuration file and according to the type of fault to be checked, the database node information to be checked corresponding to the database cluster and the log type information to be checked corresponding to the database node information to be checked; wherein the first configuration file is configured with database node information corresponding to different fault types and log type information corresponding to the database node information;

[0028] A tool parameter acquisition component is configured to determine target log collection component information corresponding to the database node information to be checked based on the second configuration file and in accordance with the log type information to be checked corresponding to the database node information to be checked; wherein the second configuration file is configured with log collection component information corresponding to different log type information;

[0029] The log collection execution component is used to call the target log collection component corresponding to the target log collection component information, and based on the log type information to be checked, collect the log file of the database node to be checked represented by the database node information to be checked.

[0030] On the other hand, the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned database cluster log collection method is implemented.

[0031] On the other hand, the present application provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed by a processor, the above-mentioned database cluster log collection method is implemented.

[0032] The beneficial effects of this application are as follows:

[0033] The present application determines, based on a first configuration file, information of a database node to be checked for a type of fault to be checked and information of a log type to be checked corresponding to the information of the database node to be checked, and determines, based on a second configuration file, information of a target log collection component corresponding to the information of the database node to be checked. Then, the target log collection component corresponding to the target log collection component information is called, and based on the log type information to be checked, the log file of the database node to be checked represented by the information of the database node to be checked is collected. This enables automated log collection when a database cluster fails, thereby improving the efficiency and accuracy of log collection when a database cluster fails, and further improving the efficiency and accuracy of failure cause analysis of the database cluster.

[0034] Other features and advantages of the present application will be described in the following description, and in part, will become apparent from the description or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures particularly pointed out in the written description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The schematic diagrams and descriptions of the present application are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings:

[0036] Figure 1 This is a schematic diagram of the overall framework of the database cluster log collection method in an embodiment of the present application;

[0037] Figure 2 This is a schematic diagram of the overview of the database cluster log collection method in an embodiment of the present application;

[0038] Figure 3 Schematic diagram of the model structure of the fault node association prediction model in the embodiment of the present application;

[0039] Figure 4 This is a schematic diagram of a specific process of the database cluster log collection method in an embodiment of the present application;

[0040] Figure 5 This is a schematic diagram of the structure of the database cluster log collection system in an embodiment of the present application;

[0041] Figure 6 Schematic diagram of the hardware structure of the electronic device in the embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and beneficial effects of this application more clearly understood, the technical solutions of this application will be clearly and completely described below in conjunction with the embodiments and drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of this application.

[0043] In order to facilitate those skilled in the art to better understand this application, the technical terms involved in this application are briefly introduced below.

[0044] Database node information includes node attribute information such as node identification information, network address information, database version, and database deployment mode corresponding to the database node.

[0045] Log type information, which includes log type, log storage address, and other log attribute information for the log to be checked corresponding to the database node. In this application, different deployment forms and / or versions of database nodes correspond to different logs to be checked. For example, the logs to be checked corresponding to active-standby database nodes include DN logs and CM logs, while the logs to be checked corresponding to distributed database nodes include DN logs, CN logs, GTM logs, and etcd logs.

[0046] Log collection component information includes tool attribute information such as the component name and calling method of the log collection component.

[0047] The first configuration file is a file configured with database node information corresponding to different fault types and log type information corresponding to the database node information.

[0048] The second configuration file is a file configured with log collection component information corresponding to different log type information.

[0049] It should be noted that the terms "first", "second", etc. mentioned in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be interchanged where appropriate so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "and / or" mentioned in this application describe the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent three situations: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0050] After introducing the technical terms involved in this application, the application scenarios and design concepts of this application are briefly introduced.

[0051] Currently, database cluster fault location typically requires personnel to manually log in to each database node in the cluster and access the directories of multiple database instances under each node to view log files for fault location and cause analysis. However, when the number of database nodes exceeds three, the efficiency of viewing log files decreases rapidly, impacting the efficiency of database cluster fault location and cause analysis.

[0052] For this purpose, in this application, see Figure 1 As shown, the first stage is the parameter configuration stage, which includes: (1) configuring the first configuration file (such as Figure 1 Cluster information in): The first configuration file includes at least the first correspondence between the database version and database deployment form and the log type information and the log collection range; wherein the log collection range includes at least the fault type, and may also include one of the log file time range and the percentage of the number of log files in the total number; that is, the log file time range and the percentage of the number of log files in the total number can be configured by selecting one or more configurations; (2) Configure the second configuration file (such as Figure 1 The second configuration file includes at least the second correspondence between the log type information and the log collection component information; wherein the log type information includes at least the following log types and the log storage address of each log type and other log attribute information: logs under the database log main directory; transaction log information; serial number log information recording transaction submission; operating system log information; stack information after program crash; database main configuration file; database client authentication configuration file; database main process file; database current running status information; data table information, including database, system table, system view, ordinary table and ordinary view; the directory where the database main process file is located; correspondingly, the log collection component information includes at least the following component names and the calling method of the log collection component represented by each component name and other tool attribute information: database log collection component; database transaction log collection component; database record transaction serial number collection component; operating system log collection component; program stack information file component; database configuration file collection component; database client authentication configuration file; system database and business database collection component; system table and business table collection component; system view and business view collection component; database executable program collection component; (3) configure user login account information (such as Figure 1Node information in): If collecting operating system related log content, configure the operating system super administrator user name and password as the user login account information; if collecting database related log content, configure the database installation administrator user name and password as the user login account information; (4) Configure the output path (such as Figure 1 ): temporary output directory and local output directory. The second stage is the log collection stage, including: calling the parameter parsing component to obtain the tool parameters from the second configuration file and the database parameters from the first configuration file; calling the task generation component to generate a task list, each task in the task list contains information such as the directory for which logs need to be collected, the node network address, and the login account; using the task execution component to execute each task in the task list in a multi-threaded manner to collect the log files of the database nodes, generating a log collection result statistical report based on the component execution log during the log file collection process, and outputting the log collection result statistical report, thereby realizing automatic log collection when the database cluster fails, improving the log collection efficiency when the database cluster fails, and further improving the efficiency of fault location and fault cause analysis of the database cluster.

[0053] After introducing the application scenarios and design ideas of this application, the technical solutions provided by this application are described in detail below.

[0054] The present invention provides a database cluster log collection method, which is applicable to electronic devices such as computers and servers. Figure 2 As shown, the overview process of the database cluster log collection method provided by the embodiment of the present application is as follows:

[0055] Step 201: Obtain the type of fault to be checked for the database cluster.

[0056] In an embodiment of the present application, when a database cluster fault is detected by the database cluster fault monitoring component, the fault code of the database cluster is obtained, and the type of fault to be investigated is determined based on the fault code of the database cluster. For example, the fault code of the database cluster is used as the type of fault to be investigated.

[0057] Step 202: Based on the first configuration file, determine the database node information to be checked corresponding to the database cluster and the log type information to be checked corresponding to the database node information to be checked according to the fault type to be checked; wherein the first configuration file is configured with database node information corresponding to different fault types and log type information corresponding to the database node information.

[0058] In the embodiment of the present application, the first configuration file is pre-generated based on the database node information corresponding to different fault types and the log type information corresponding to the database node information. The database nodes corresponding to different fault types and the log types corresponding to the database nodes can be predicted based on the fault node association prediction model. Specifically, when predicting the correspondence between the fault type in the database cluster and the database node and the log type of the database node based on the fault node association prediction model, the following methods can be used but are not limited to:

[0059] Step 1: Collect each historical fault case, and perform text recognition and feature extraction on each historical fault case to obtain each training sample; wherein, each training sample includes the historical fault type, the associated database node identifier, and the historical fault type and historical feature data corresponding to the associated database node identifier. Specifically including:

[0060] (1) Collect historical failure cases of the database cluster; each historical failure case includes a failure type label (such as lock contention, query timeout, node downtime, etc.), associated database node identifier, database node log text, and failure period performance indicators (CPU, memory, network, etc.).

[0061] (2) Based on NLP technology, the associated database node identifier and the historical log data, historical fault data and historical feature data corresponding to the associated database node identifier are extracted from each historical fault case; among them, the historical log data includes slow query logs, error logs, transaction logs, lock wait logs, etc.; the historical fault data includes error codes, exception stacks, SQL statement patterns, etc. in the database node log text; the historical feature data includes structured features and topological features. The structured features include the time series data of performance indicators such as resource utilization rate, transaction throughput, lock wait time, etc. of the database node; the topological features include the database node dependency graph, and the database node dependency graph includes the call link weights between database nodes.

[0062] (3) According to the timestamp, the associated database node identifier is aligned with the historical log data, historical fault data and historical feature data to obtain various training samples.

[0063] Step 2: Based on each training sample, iteratively train the fault node association prediction model; wherein the fault node association prediction model includes the following: Figure 3 The fault classification sub-model, node location sub-model, log correlation sub-model, and fusion analysis sub-model are shown. Specifically, they include:

[0064] (1) Through the fault classification sub-model, a supervised learning network (such as XGBoost, LSTM, etc.) is used to predict the probability distribution of fault types based on the historical log data and historical feature data corresponding to the associated database node identifier, and output the predicted fault type based on the fault type probability distribution; for example, when "Deadlock detected" frequently appears in the historical log data of a certain associated database node and the lock waiting time exceeds the threshold, it is predicted as a "deadlock fault".

[0065] (2) Through the node positioning sub-model, a graph neural network (such as GNN) is used to identify the associated database node that caused the historical fault based on the historical feature data and historical fault data corresponding to the associated database node identifier, that is, the first mapping relationship between the fault type and the associated database node; for example, by analyzing the sudden increase trend of the number of connections between each associated database node and the query queue depth, the hotspot node with the highest load is located as the associated database node of the "query peak" fault.

[0066] (3) Through the log association sub-model, a language analysis network (such as BERT) is used to predict the second mapping relationship between the fault type and the required log based on the historical log data and historical fault data corresponding to the associated database node identifier. The second mapping relationship between the fault type and the required log is shown in Table 1:

[0067] Table 1.

[0068]

[0069] (4) Through the fusion analysis sub-model, a multimodal fusion analysis network (such as a knowledge graph reasoning network, a multimodal joint analysis network based on an attention mechanism, a dual-mode fusion network of a knowledge graph and an attention mechanism, an integrated learning network, etc.) is used to perform fusion analysis on the predicted fault type output by the fault classification sub-model, the first mapping relationship between the fault type output by the node positioning sub-model and the associated database node, and the second mapping relationship between the fault type output by the log association sub-model and the required log, thereby obtaining the associated database node corresponding to the fault type and the log type of the associated database node. In the embodiment of the present application, the multimodal fusion analysis network is a dual-mode fusion network of a knowledge graph and an attention mechanism as a preferred embodiment. For example, the multimodal fusion analysis network constructs the fault type, database node, and log type as a triple of the knowledge graph, dynamically calculates the weight of each modal feature through a cross-attention mechanism (such as focusing on the node positioning result in a scenario with high fault type confidence), and generates a joint probability distribution of the fused fault type, database node, and log type through a graph traversal algorithm (such as a bidirectional breadth-first search) to determine a complete association chain.

[0070] (5) Based on the associated database node identifiers and historical log data corresponding to the historical fault data in the training samples, as well as the associated database nodes and log types of the associated database nodes corresponding to the predicted fault types, the parameters of the fault node association prediction model are updated until the iteration termination condition is met. The fault node association prediction model after the last parameter update is determined as the trained fault node association prediction model.

[0071] Step 3: Perform fault node association prediction based on the trained fault node association prediction model to obtain database nodes and log types corresponding to different fault types.

[0072] Step 4: Generate a first configuration file based on the database nodes and log types corresponding to different fault types; for example: (deadlock fault, associated node, gbase node) → (analysis basis, required logs, lock wait logs).

[0073] Furthermore, based on the first configuration file and according to the fault type to be checked, when determining the database node information to be checked corresponding to the database cluster and the log type information to be checked corresponding to the database node information to be checked, the following methods may be used but are not limited to:

[0074] First, the parameter parsing component is called to parse the database node information corresponding to the fault type to be checked and the log type information corresponding to the database node information from the first configuration file.

[0075] Then, the parsed database node information and the log type information corresponding to the database node information are determined as the to-be-checked database node information corresponding to the database cluster and the to-be-checked log type information corresponding to the to-be-checked database node information.

[0076] Step 203: Based on the second configuration file, target log collection component information corresponding to the database node information to be checked is determined according to the log type information to be checked corresponding to the database node information to be checked; wherein the second configuration file is configured with log collection component information corresponding to different log type information.

[0077] In the embodiment of the present application, the second configuration file may be a file that is manually configured and generated. Based on the second configuration file, when determining the target log collection component information corresponding to the database node information to be checked according to the log type information to be checked corresponding to the database node information to be checked, the following methods may be used, but are not limited to:

[0078] First, the parameter parsing component is called to parse the log collection component information corresponding to the log type information to be checked from the second configuration file.

[0079] Then, the parsed log collection component information is determined as the target log collection component information corresponding to the database node information to be checked.

[0080] Step 204: calling the target log collection component corresponding to the target log collection component information, and collecting the log files of the database node to be checked represented by the database node to be checked information based on the log type information to be checked.

[0081] In the embodiment of the present application, when calling the target log collection component corresponding to the target log collection component information, based on the log type information to be checked, the log file of the database node to be checked represented by the database node information to be checked can be collected in the following manner, but not limited to:

[0082] First, the task generation component is called to generate a log collection task based on the database node information and the log type information to be checked corresponding to the database node information. Specifically, the task generation component is called to obtain the user login account information and generate a log collection task based on the user login account information, the database node information, and the log type information to be checked corresponding to the database node information.

[0083] Then, the task execution component is called to execute the log collection task, and in the process of executing the log collection task, the target log collection component corresponding to the target log collection component information is called to collect the log files corresponding to the log type information to be checked from the database node to be checked represented by the database node information to be checked. Specifically, first, the task generation component is called to generate a log collection task list, wherein each log collection task in the log collection task list contains information such as the log type information to be checked (such as the log directory to be collected), database node information (such as the node network address), and user login account information; then, the task execution component is called to execute each log collection task in the log collection task list in parallel using multiple threads; wherein, when executing the log collection task, each thread logs into the database node represented by the database node information based on the user login account information, and collects the log files of the database node from the logged-in database node based on the log type information to be checked.

[0084] In addition, in an embodiment of the present application, when calling the target log collection component corresponding to the target log collection component information and collecting the log files corresponding to the log type information to be checked from the database node to be checked represented by the database node information to be checked, it can also be combined with the log file time range and / or the percentage of the number of log files in the total number parsed from the first configuration file. Specifically, first, the task generation component is called to generate a log collection task list; wherein, each log collection task in the log collection task list includes the log type information to be checked (for example, the log directory to be collected), database node information (for example, the node network address), log collection range information (for example, the log file time range and / or the percentage of the log file number in the total number), user login account information and other information; the task execution component is called to use multiple threads to execute each log collection task in the log collection task list in parallel; wherein, when each thread executes the log collection task, based on the user login account information, logs in to the database node represented by the database node information, and based on the log type information to be checked, collects the log files of the database node from the logged-in database node according to the log collection range information (for example, the log file time range and / or the percentage of the log file number in the total number).

[0085] In the implementation of this application, in order to further improve the efficiency of log file collection, the target log collection component corresponding to the target log collection component information is called, and after collecting the log file corresponding to the log type information to be checked from the database node to be checked represented by the database node information to be checked, the task execution component can also be called first to save the log file to the temporary output directory of the database node to be checked represented by the database node information to be checked, and compress all log files in the temporary output directory to obtain a log file compressed package. Afterwards, when it is determined that the log file download condition is met, the log file compressed package is downloaded from the temporary output directory to the local output directory, and the log file compressed package in the local output directory is decompressed to obtain the log file of the database node to be checked represented by the database node information to be checked; wherein, the log file download condition includes at least one of the following conditions: reaching the set download cycle (for example, downloading once every 2 minutes), receiving the download instruction, and all log files of the log type to be checked for all database nodes to be checked corresponding to the fault type to be checked are collected. In this way, by first saving the log file to a temporary output directory and then compressing the log file through a compression file tool, the execution efficiency of the log collection task can be improved while reducing the file size of the log file transmission, thereby saving data transmission resources and improving the efficiency of log file collection.

[0086] Furthermore, the target log collection component corresponding to the target log collection component information is called, and based on the log type information to be checked, the log file of the database node to be checked represented by the database node information to be checked is collected. Then, based on the component execution log in the log file collection process, a log collection result statistical report can be generated, and the log collection result statistical report can be output so that the log file collection process can be traced later, which is convenient for the optimization and upgrade of the database cluster log collection system.

[0087] Next, the database cluster log collection method provided by the embodiment of the present application is further described in detail. Figure 4 As shown, the specific process of the database cluster log collection method provided in the embodiment of the present application is as follows:

[0088] Step 301: When a fault occurs in a database cluster, the fault monitoring component obtains a fault code of the database cluster and determines the type of fault to be investigated based on the fault code of the database cluster.

[0089] Step 302: Call the parameter parsing component to parse the database node information to be checked and the log type information to be checked corresponding to the database node information to be checked from the first configuration file according to the fault type to be checked, and parse the target log collection component information corresponding to the database node information to be checked from the second configuration file according to the log type information to be checked corresponding to the database node information to be checked.

[0090] Step 303: calling the task generation component to generate a log collection task based on the user login account information, the database node information, and the log type information to be checked corresponding to the database node information.

[0091] Step 304: Call the task execution component to execute the log collection task, and in the process of executing the log collection task, call the target log collection component corresponding to the target log collection component information to collect the log file corresponding to the log type information to be checked from the database node to be checked represented by the database node information to be checked.

[0092] Step 305: calling the task execution component to save the log files collected by the target log collection component to the temporary output directory of the database node to be checked represented by the database node information to be checked, and compressing all log files in the temporary output directory to obtain a log file compression package.

[0093] Step 306: Call the task execution component to determine that when all log files of all log types to be checked for all database nodes to be checked corresponding to the fault type to be checked have been collected, download the log file compression package from the temporary output directory of each database node to be checked to the local output directory, and decompress each log file compression package in the local output directory to obtain all log files of all database nodes to be checked corresponding to the fault type to be checked.

[0094] Step 307: calling the task execution component, generating a log collection result statistical report based on the component execution log in the log file collection process, and outputting the log collection result statistical report.

[0095] In the database cluster log collection provided by the embodiment of the present application, by configuring the log collection range options, for example, the fault type, the time range of the log file, and the percentage of the number of log files in the folder to the total number, etc., log collection can be performed according to the fault requirements, thereby improving the flexibility of log collection. Moreover, by first saving the log file to a temporary output directory and then compressing the log file through a compression file tool, the file size of the log file transmission can be reduced while improving the execution efficiency of the log collection task, thereby saving data transmission resources and improving the collection efficiency of the log file. In addition, when locating a database cluster fault, it is possible to view all log files of all database nodes related to the fault type in one stop, thereby improving the efficiency of fault locating and providing effective data support for database operation and maintenance.

[0096] Based on the above embodiments, the present application provides a database cluster log collection system. Figure 5 As shown, the database cluster log collection system 400 provided in this embodiment of the application includes at least:

[0097] The database cluster fault monitoring component 401 is used to obtain the type of fault to be checked in the database cluster;

[0098] The database parameter acquisition component 402 is configured to determine, based on the first configuration file and according to the fault type to be checked, the database node information to be checked corresponding to the database cluster and the log type information to be checked corresponding to the database node information to be checked; wherein the first configuration file is configured with database node information corresponding to different fault types and log type information corresponding to the database node information;

[0099] The tool parameter acquisition component 403 is configured to determine target log collection component information corresponding to the database node information to be checked based on the second configuration file and the log type information to be checked corresponding to the database node information to be checked; wherein the second configuration file is configured with log collection component information corresponding to different log type information;

[0100] The log collection execution component 404 is used to call the target log collection component 405 corresponding to the target log collection component information, and collect the log files of the database node to be checked represented by the database node information to be checked based on the log type information to be checked.

[0101] In one possible implementation, the database parameter acquisition component 402 is used to parse the database node information corresponding to the fault type to be checked and the log type information corresponding to the database node information from the first configuration file; and determine the parsed database node information and the log type information corresponding to the database node information as the database node information to be checked corresponding to the database cluster and the log type information to be checked corresponding to the database node information to be checked.

[0102] In a possible implementation, the tool parameter acquisition component 403 is used to parse the log collection component information corresponding to the log type information to be checked from the second configuration file; and determine the parsed log collection component information as the target log collection component information corresponding to the database node information to be checked.

[0103] In a possible implementation, the log collection execution component 404 includes a task generation component 4041 and a task execution component 4042;

[0104] The task generation component 4041 is used to generate a log collection task based on the database node information and the log type information to be checked corresponding to the database node information;

[0105] The task execution component 4042 is used to execute the log collection task, and in the process of executing the log collection task, calls the target log collection component 405 corresponding to the target log collection component information to collect the log file corresponding to the log type information to be checked from the database node to be checked represented by the database node information to be checked.

[0106] In a possible implementation, the task generation component 4041 is configured to obtain user login account information and generate a log collection task based on the user login account information, database node information, and information on the type of log to be checked corresponding to the database node information.

[0107] In one possible implementation, the task execution component 4042 is used to save the log file to a temporary output directory of the database node to be checked represented by the information of the database node to be checked, and compress all the log files in the temporary output directory to obtain a log file compression package; when it is determined that the log file download condition is met, the log file compression package is downloaded from the temporary output directory to the local output directory, and the log file compression package in the local output directory is decompressed to obtain the log file of the database node to be checked represented by the information of the database node to be checked.

[0108] In a possible implementation, the task execution component 4042 is further configured to generate a log collection result statistical report based on the component execution logs during the log file collection process, and output the log collection result statistical report.

[0109] It should be noted that the principle of solving the technical problem by the database cluster log collection system 400 provided in the embodiment of the present application is similar to that of the database cluster log collection method provided in the embodiment of the present application. Therefore, the implementation of the database cluster log collection system 400 provided in the embodiment of the present application can refer to the implementation of the database cluster log collection method provided in the embodiment of the present application, and the repeated parts will not be repeated.

[0110] After introducing the database cluster log collection method and system provided by the embodiments of the present application, the electronic device provided by the embodiments of the present application is briefly introduced next.

[0111] See Figure 6 As shown, the electronic device 500 provided in the embodiment of the present application includes at least a processor 501, a memory 502, and a computer program stored in the memory 502 and executable on the processor 501. When the processor 501 executes the computer program, the database cluster log collection method provided in the embodiment of the present application is implemented.

[0112] The electronic device 500 provided in the embodiment of the present application may further include a bus 503 connecting different components (including the processor 501 and the memory 502). The bus 503 represents one or more of several types of bus structures, including a memory bus, a peripheral bus, a local bus, etc.

[0113] Memory 502 may include a readable storage medium in the form of volatile memory, such as random access memory (RAM) 5021 and / or cache memory 5022, and may further include read-only memory (ROM) 5023. Memory 502 may also include a program tool 5025 having a set (at least one) of program modules 5024. Program modules 5024 include, but are not limited to, an operating subsystem, one or more application programs, other program modules, and program data. Each of these examples, or some combination thereof, may include an implementation of a network environment.

[0114] Processor 501 can be a single processing element or a collective term for multiple processing elements. For example, processor 501 can be a central processing unit (CPU) or one or more integrated circuits configured to implement the database cluster log collection method provided in the embodiments of the present application. Specifically, processor 501 can be a general-purpose processor, including but not limited to a CPU, an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0115] The electronic device 500 can communicate with one or more external devices 504 (e.g., keyboard, remote control, etc.), and can also communicate with one or more devices that enable a user to interact with the electronic device 500 (e.g., mobile phone, computer, etc.), and / or communicate with a device that enables the electronic device 500 to communicate with one or more other electronic devices 500 (e.g., router, modem, etc.). Such communication can be performed through an input / output (I / O) interface 505. In addition, the electronic device 500 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN) and / or public network, such as the Internet) through a network adapter 506. Figure 6 As shown, the network adapter 506 communicates with other modules of the electronic device 500 via the bus 503. Figure 6Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 500, including but not limited to microcode, device drivers, redundant processors, external disk drive arrays, disk arrays (Redundant Arrays of Independent Disks, RAID) subsystems, tape drives, and data backup storage subsystems.

[0116] It should be noted that Figure 6 The electronic device 500 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0117] The following describes the computer-readable storage medium provided in the embodiments of the present application. The computer-readable storage medium provided in the embodiments of the present application stores computer instructions that, when executed by a processor, implement the database cluster log collection method provided in the embodiments of the present application. Specifically, the computer instructions may be built into or installed in the processor, so that the processor implements the database cluster log collection method provided in the embodiments of the present application by executing the built-in or installed computer instructions.

[0118] In addition, the database cluster log collection method provided in the embodiment of the present application can also be implemented as a computer program product, which includes program code. When the program code is run on a processor, it implements the database cluster log collection method provided in the embodiment of the present application.

[0119] The computer program product provided in the embodiments of the present application may adopt one or more computer-readable storage media, and the computer-readable storage medium may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. Specifically, more specific examples of computer-readable storage media (a non-exhaustive list) include an electrical connection with one or more wires, a portable disk, a hard disk, RAM, ROM, Erasable Programmable Read Only Memory (EPROM), optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.

[0120] The computer program product provided in the embodiments of the present application may be a CD-ROM and include program code, and may also be run on electronic devices such as servers and computers. However, the computer program product provided in the embodiments of the present application is not limited thereto. In the embodiments of the present application, the computer-readable storage medium may be any tangible medium that contains or stores program code, and the program code may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0121] It should be noted that although several units or subunits of the device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, depending on the embodiment of the application, the features and functions of two or more units described above can be embodied in a single unit. Conversely, the features and functions of a single unit described above can be further divided and embodied by multiple units.

[0122] Furthermore, although the operations of the method of the present application are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all illustrated operations must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0123] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0124] Obviously, those skilled in the art may make various changes and modifications to the embodiments of the present application without departing from the spirit and scope of the embodiments of the present application. Thus, if these modifications and variations of the embodiments of the present application fall within the scope of the claims of the present application and their equivalents, the present application also intends to include such modifications and variations.

Claims

1. A database cluster log collection method, characterized in that: include: Get the type of fault to be investigated in the database cluster; Based on the first configuration file, according to the fault type to be checked, determining the database node information to be checked corresponding to the database cluster and the log type information to be checked corresponding to the database node information to be checked; wherein the first configuration file is configured with database node information corresponding to different fault types and log type information corresponding to the database node information; Based on the second configuration file, according to the log type information to be checked corresponding to the database node information to be checked, determining the target log collection component information corresponding to the database node information to be checked; wherein the second configuration file is configured with log collection component information corresponding to different log type information; Calling the target log collection component corresponding to the target log collection component information, and collecting the log file of the database node to be checked represented by the database node to be checked information based on the log type information to be checked; Wherein, based on the first configuration file and according to the fault type to be checked, determining the database node information to be checked corresponding to the database cluster and the log type information to be checked corresponding to the database node information to be checked, includes: Calling a parameter parsing component to parse the first configuration file to obtain database node information corresponding to the fault type to be checked and log type information corresponding to the database node information; The parsed database node information and the log type information corresponding to the database node information are determined as the to-be-checked database node information corresponding to the database cluster and the to-be-checked log type information corresponding to the to-be-checked database node information.

2. The database cluster log collection method according to claim 1, wherein: Determining target log collection component information corresponding to the database node information to be checked based on the second configuration file and according to the log type information to be checked corresponding to the database node information to be checked, including: Calling a parameter parsing component to parse the second configuration file to obtain log collection component information corresponding to the log type information to be checked; The parsed log collection component information is determined as the target log collection component information corresponding to the database node information to be checked.

3. The database cluster log collection method according to claim 1, wherein: Calling the target log collection component corresponding to the target log collection component information, and collecting the log file of the database node to be checked represented by the database node to be checked information based on the log type information to be checked, including: Calling a task generation component to generate a log collection task based on the database node information and the log type information to be checked corresponding to the database node information; Call the task execution component to execute the log collection task, and in the process of executing the log collection task, call the target log collection component corresponding to the target log collection component information to collect the log file corresponding to the log type information to be checked from the database node to be checked represented by the database node information to be checked.

4. The database cluster log collection method according to claim 3, wherein: Calling a task generation component to generate a log collection task based on the database node information and the log type information to be checked corresponding to the database node information, including: The task generation component is called to obtain user login account information, and the log collection task is generated based on the user login account information, the database node information, and the log type information to be checked corresponding to the database node information.

5. The database cluster log collection method according to claim 3, wherein: After calling the target log collection component corresponding to the target log collection component information and collecting the log file corresponding to the log type information to be checked from the database node to be checked represented by the database node information to be checked, the method further includes: Calling a task execution component to save the log file to a temporary output directory of the database node to be checked represented by the information of the database node to be checked, and compressing all log files in the temporary output directory to obtain a log file compression package; When it is determined that the log file downloading conditions are met, the log file compressed package is downloaded from the temporary output directory to the local output directory, and the log file compressed package in the local output directory is decompressed to obtain the log file of the database node to be checked represented by the information of the database node to be checked.

6. The database cluster log collection method according to any one of claims 1 to 5, characterized in that: After calling the target log collection component corresponding to the target log collection component information and collecting the log file of the database node to be checked represented by the database node to be checked information based on the log type information to be checked, the method further includes: Generate a log collection result statistical report based on the component execution logs during the log file collection process, and output the log collection result statistical report.

7. A database cluster log collection system, characterized in that: include: The database cluster fault monitoring component is used to obtain the type of fault to be investigated in the database cluster; a database parameter acquisition component, configured to determine, based on a first configuration file and in accordance with the type of fault to be checked, database node information to be checked corresponding to the database cluster and log type information to be checked corresponding to the database node information to be checked; wherein the first configuration file is configured with database node information corresponding to different fault types and log type information corresponding to the database node information; a tool parameter acquisition component for determining target log collection component information corresponding to the database node information to be checked based on a second configuration file and in accordance with the log type information to be checked corresponding to the database node information to be checked; wherein the second configuration file is configured with log collection component information corresponding to different log type information; a log collection execution component, configured to call a target log collection component corresponding to the target log collection component information, and collect log files of the database node to be checked represented by the database node to be checked information based on the log type information to be checked; Among them, the database parameter acquisition component is used to call the parameter parsing component to parse out the database node information corresponding to the fault type to be checked and the log type information corresponding to the database node information from the first configuration file; the parsed database node information and the log type information corresponding to the database node information are determined as the database node information to be checked corresponding to the database cluster and the log type information to be checked corresponding to the database node information to be checked.

8. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for collecting database cluster logs according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the database cluster log collection method according to any one of claims 1 to 6 is implemented.

Images