Log analysis method and system based on openGauss database

Abstract

The invention discloses a log analysis method and system based on an openGauss database, belongs to the technical field of database operation, and aims to solve the technical problem of how to perform real-time analysis and running state monitoring on logs of the openGauss database and improve the analysis efficiency and accuracy of the logs of the database. According to the technical scheme, the method comprises the steps that multi-dimensional log preprocessing is conducted, log content generated based on an openGauss database serves as a data source, a virtual machine connected with an openGauss database instance is configured, log file data is collected, the obtained log data is cleaned, denoised and formatted, and preprocessed log data is obtained; extracting features; pattern recognition; real-time monitoring is realized; trend analysis; and querying a result.

Description

Technical Field

[0001] The present invention relates to the technical field of database operation and management, and in particular to a log analysis method and system based on an openGauss database. Background Art

[0002] As a core component of modern information technology, database technology is responsible for the storage, management, processing, and analysis of data, supporting the operation of various systems from enterprise applications to internet services. As the foundational infrastructure for data management and analysis, database technology is becoming increasingly important.

[0003] The openGauss database, a high-performance, highly available, and easy-to-manage open-source database system, has become the preferred choice for many enterprises and institutions processing large-scale data sets. It not only provides powerful data processing capabilities but also meets the growing demand for data storage, management, and analysis through continuous technological innovation. Existing openGauss database log analysis relies on manual analysis, which is inefficient and lacks real-time performance. This makes it difficult to accurately identify and analyze large amounts of logs in complex scenarios.

[0004] Therefore, how to conduct real-time analysis and operation status monitoring of the openGauss database logs and improve the analysis efficiency and accuracy of the database logs is a technical problem that needs to be solved urgently. Summary of the Invention

[0005] The technical task of the present invention is to provide a log analysis method and system based on the openGauss database to solve the problem of how to perform real-time analysis and operation status monitoring on the logs of the openGauss database and improve the analysis efficiency and accuracy of the database logs.

[0006] The technical task of the present invention is achieved in the following manner: a log analysis method based on the openGauss database, the method is as follows:

[0007] Multi-dimensional log preprocessing: Based on the log content generated by the openGauss database as the data source, by configuring a virtual machine connected to the openGauss database instance, log file data is collected, and the obtained log data is cleaned, denoised, and formatted to obtain preprocessed log data for subsequent analysis;

[0008] Feature extraction: Label the preprocessed log data, convert the unstructured log data into structured data, extract features from the structured data, identify elements whose importance for describing the database state is greater than a set threshold, and perform standardization and normalization on the identified elements whose importance for describing the database state is greater than the set threshold to obtain the extracted features and eliminate the dimensional differences between different data;

[0009] Pattern recognition: Perform model training and state recognition on the data after feature extraction for pattern recognition. By identifying and classifying different patterns in the log data, describe the running state of the database system, and use supervised learning algorithms to train the normal operation patterns and abnormal database behaviors in pattern recognition. The trained model can learn the relationship between the data after feature extraction and the known patterns, so as to accurately classify new log data;

[0010] Real-time monitoring: Continuously perform feature extraction and pattern recognition on the log data so that abnormal behaviors of the database can be detected in a timely manner; Once an anomaly is detected, immediately generate an alarm message, send the alarm message, provide relevant log information and possible reasons to help the operation and maintenance personnel discover problems and troubleshoot them as soon as possible;

[0011] Trend analysis: Identify long-term trends from historical log data, and obtain the change trend of database performance by analyzing the time series of log data; At the same time, predict the performance indicators and potential fault points of the database through time series analysis and prediction models;

[0012] Result query: Query the log data and analysis results through an intuitive interface, give reasonable and effective measure suggestions, and perform queries through multiple query methods such as keyword-based search, time range filtering, and custom report generation. At the same time, through the log analysis dashboard function, customize the analysis report according to your own needs, so that users can obtain information that meets their needs and quickly transfer the log information and take corresponding measures as soon as possible; Among them, the content of the customized report includes log summaries, detailed descriptions of abnormal events, and charts of performance trends.

[0013] Preferably, in the pattern recognition process, the support vector machine algorithm is used to improve the recognition accuracy of the model through supervised learning.

[0014] More preferably, the log data includes system logs, operation logs, black box logs, audit logs, WAL logs, and performance logs;

[0015] Among them, the audit logs are recorded in the OS files, and during preprocessing, the audit logs are desensitized.

[0016] More preferably, the preprocessing of the system logs is specifically as follows:

[0017] The system log generated during the operation of the database system process records the exception information of the database system process. Among them, the system log file is saved in the $GAUSSLOG / pg_log directory. The GAUSSLOG variable is the configured log path of the openGauss database. The log file format is postgresql-creation time.log. When crossing days or restarting the database, a new log file is generated.

[0018] During the preprocessing of the system log, key information is extracted from the openGauss database system log using regular expressions. The key information includes the printing time, log level, database name, operation type, and log content.

[0019] More preferably, the operation log refers to the log generated when the database administrator operates the database using tools and when the tools are called by openGauss, which is used to trace the operation history of users or administrators operating the database and analyze the change situation of the database. The operation log file is saved in the $GAUSSLOG / bin directory. The log file format is tool name-log creation time-current.log. The operation log includes the gs_ctl tool log, gs_guc tool log, and gs_dump tool log. During preprocessing, the operation information is analyzed for the operation logs of different tools. The operation information analyzed for the operation logs of different tools includes the operation tool, operation time, operation level, and operation content.

[0020] More preferably, the black box log refers to the log that dumps the heap, stack, and register information of processes and threads when the database system crashes, which is used for fault location. When the preprocessing discovers the existence of the black box log, the occurrence time and log information are simply parsed, and an alarm message is sent.

[0021] More preferably, the WAL log is the Write-Ahead Log (WAL), which is an important data log file and is used to restore database data when necessary. It is located in the pg_xlog folder of the data directory. During preprocessing, the remaining disk space of the database system and the occupied space information of the WAL log are obtained, and the platform component is notified to back up the WAL log when necessary.

[0022] More preferably, the performance log is a log that records the running status of physical resources and performs performance detection when accessing external resources, including access detection information of external resources such as disks. When there are performance problems, the performance log can conveniently locate the cause of the problem and improve the problem-solving efficiency. Among them, the performance log file is saved in the $GAUSSLOG / gs_profile directory, and the log file format is postgresql-creation time.prf.

[0023] During preprocessing, regular expressions are used to match and extract information from performance logs. The information extracted by regular expression matching includes printing time, host name, instance name, thread number, and log content;

[0024] After preprocessing the performance log information, noise data has been removed to obtain the performance log information with noise removed. Among them, the performance log information with noise removed includes execution progress, repeated processes, and irrelevant operations.

[0025] A log analysis system based on the openGauss database, the system includes:

[0026] A multi-dimensional log collection module for collecting and preprocessing log data from openGauss database instances;

[0027] A model training module for feature extraction and pattern recognition of the preprocessed log data to obtain a training model;

[0028] A real-time monitoring module for continuously performing feature extraction and pattern recognition on log data;

[0029] A trend analysis module for storing historical log data and identifying long-term trends from the historical log data;

[0030] A visualization interface for querying the pattern recognition structure, capable of generating custom reports and displaying custom log analysis dashboards.

[0031] Preferably, the log data includes system logs, operation logs, black box logs, audit logs, WAL logs, and performance logs;

[0032] Among them, the preprocessing of system logs is specifically as follows: System logs generated during the operation of the database system process record the abnormal information of the database system process. Among them, the system log files are saved in the $GAUSSLOG / pg_log directory, the GAUSSLOG variable is the configured openGauss database log path, the log file format is postgresql-creation time.log, and new log files are generated when crossing days or when the database is restarted. When preprocessing the system logs of the openGauss database, regular expressions are used to extract key information, and the key information includes printing time, log level, database name, operation type, and log content;

[0033] The operation log refers to the log generated when the database administrator operates the database using tools and when the tools are called by openGauss, which is used to trace the operation history of users or administrators operating the database and analyze the changes in the database. The operation log file is saved in the $GAUSSLOG / bin directory, and the log file format is tool name - log creation time - current.log. The operation log includes the gs_ctl tool log, gs_guc tool log, and gs_dump tool log. During preprocessing, the operation information is analyzed for the operation logs of different tools. The operation information analyzed from the operation logs of different tools includes the operation tool, operation time, operation level, and operation content.

[0034] The black box log refers to the log that dumps the heap, stack, and register information of processes and threads when the database system crashes, which is used for fault location. When the preprocessing discovers the existence of the black box log, the occurrence time and log information are simply parsed, and an alarm message is sent.

[0035] The audit log is recorded in the OS file, and during preprocessing, the audit log is desensitized.

[0036] The WAL log is the Write-Ahead Log (WAL), which is an important data log file and is used to restore database data when necessary. It is located in the pg_xlog folder in the data directory. During preprocessing, the remaining disk space of the database system and the occupied space information of the WAL log are obtained, and the platform component is notified to back up the WAL log when necessary.

[0037] The performance log is a log that records the running status of physical resources and performs performance detection when accessing external resources, including access detection information of external resources such as disks. When there are performance problems, the performance log can conveniently locate the cause of the problem and improve the efficiency of problem solving. Among them, the performance log file is saved in the $GAUSSLOG / gs_profile directory, and the log file format is postgresql - creation time.prf. During preprocessing, regular expression matching is used to extract information from the performance log. The information extracted by regular expression matching includes the printing time, host name, instance name, thread number, and log content. After preprocessing the performance log information, the noise data has been removed, and the performance log information with noise removed is obtained. Among them, the performance log information with noise removed includes the execution progress, repeated processes, and irrelevant operations.

[0038] The log analysis method and system based on the openGauss database of the present invention have the following advantages:

[0039] (1) Through multiple processes such as multi-dimensional log preprocessing, feature extraction, pattern recognition, real-time monitoring, trend analysis, and result query, the present invention provides a comprehensive, efficient, and intelligent log analysis solution for users and operation and maintenance personnel, solving the problems of long time-consuming manual log troubleshooting and easy omission of key information. By utilizing the analysis and extraction capabilities of the model, the efficiency of database operation and maintenance is greatly improved, the reliability and security of the database system are enhanced, and it provides convenience for users and operation and maintenance personnel to operate and troubleshoot the running status of the database;

[0040] (2) Using the logs of the openGauss database as the data source, the present invention realizes real-time monitoring and trend analysis of the running status of the database from the perspective of logs through supervised learning. Compared with the traditional method of manually troubleshooting and analyzing logs, the accuracy of log analysis is greatly improved, the efficiency of database operation and maintenance is increased, the time for troubleshooting log information after a database failure is reduced, the fault repair time is shortened, potential problems are discovered in advance through trend analysis, the reliability and security of the database system are enhanced, and it provides convenience for users and operation and maintenance personnel to operate and troubleshoot the running status of the database;

[0041] (3) Using the openGauss database logs as the data source, the present invention realizes real-time monitoring and trend analysis of the running status of the database from the perspective of logs through supervised learning, greatly improving the accuracy of log analysis;

[0042] (4) Through steps such as multi-dimensional log preprocessing, feature extraction, pattern recognition, real-time monitoring, trend analysis, and result query, the present invention realizes in-depth analysis of the openGauss database logs; at the same time, it can realize real-time analysis of the openGauss database logs and monitoring of the running status;

[0043] (5) The present invention uses supervised learning algorithms to process log data, automatically identify and classify patterns in logs without manual intervention, thereby real-time monitoring the running status of the database, predicting potential performance problems, and responding to security threats in a timely manner;

[0044] (6) The present invention also includes a user-friendly query interface, enabling users to easily retrieve and analyze log data, significantly improving the convenience and efficiency of database operation and maintenance; through the present invention, database administrators can more precisely understand the running status of the database, optimize system performance, and ensure data security. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The present invention will be further described below with reference to the accompanying drawings.

[0046] ATTACHED Figure 1 is a flowchart of a log analysis method based on the openGauss database. Detailed implementation manners

[0047] The log analysis method and system based on the openGauss database of the present invention will be described in detail below with reference to the accompanying drawings of the specification and specific embodiments.

[0048] Embodiment 1:

[0049] As shown in the Figure 1 accompanying drawings, this embodiment provides a log analysis method based on the openGauss database, and the method is as follows:

[0050] S1. Multi-dimensional log preprocessing: Using the log content generated by the openGauss database as the data source, collecting log file data by configuring a virtual machine connected to the openGauss database instance, cleaning, denoising, and formatting the obtained log data to obtain preprocessed log data for subsequent analysis;

[0051] S2. Feature extraction: Labeling the preprocessed log data to convert unstructured log data into structured data so that it can be effectively recognized and processed by algorithms. Log data is usually a large amount of text information, and there is a lot of information redundancy and noise when directly used for analysis. Therefore, feature extraction is performed on the structured data to identify elements whose importance for describing the database state is greater than a set threshold, and standardization and normalization processing are performed on the identified elements whose importance for describing the database state is greater than the set threshold to obtain the extracted features and eliminate the dimensional difference between different data;

[0052] S3. Pattern recognition: Training the model and identifying the state of the data after feature extraction for pattern recognition. By identifying and classifying different patterns in the log data, the running state of the database system is described, and a supervised learning algorithm is used to train the normal operation mode and abnormal database behavior of pattern recognition. The trained model can learn the relationship between the data after feature extraction and the known patterns, so as to accurately classify new log data;

[0053] S4. Real-time monitoring: Continuously performing feature extraction and pattern recognition on the log data so that abnormal behaviors of the database can be detected in time; once an abnormality is detected, an alarm message is immediately generated and the alarm message is sent, providing relevant log information and possible reasons to help the operation and maintenance personnel discover problems and troubleshoot them as soon as possible;

[0054] S5. Trend Analysis: Identify long-term trends from historical log data and obtain the changing trends of database performance by analyzing the time series of log data, such as the increase in response time or the fluctuation of transaction processing volume. This information can be used for pattern analysis to help operation and maintenance personnel optimize database configuration and allocate resources; at the same time, predict database performance metrics and potential failure points through time series analysis and prediction models, and the prediction results can help database administrators make more informed decisions to improve the overall performance and reliability of the system;

[0055] S6. Result Query: To make the analysis results more user and operation and maintenance personnel-friendly, a powerful result query function is provided. Query log data and analysis results through an intuitive interface, and at the same time give reasonable and effective measure suggestions to help users improve database operation efficiency and quickly solve potential failures; give reasonable and effective measure suggestions, and perform queries through multiple query methods such as keyword-based search, time range filtering, and custom report generation. At the same time, through the log analysis dashboard function, customize the analysis report according to your own needs, so that users can obtain information that meets their needs and quickly transmit log information and take corresponding measures as soon as possible; among them, the content of the customized report includes log summary, detailed description of abnormal events, and charts of performance trends.

[0056] In the pattern recognition process of step S3 in this embodiment, the support vector machine algorithm is used to improve the recognition accuracy of the model through supervised learning, improving the accuracy and robustness of pattern recognition;

[0057] Pattern recognition can not only help us discover common problems in the database, such as performance bottlenecks or configuration errors, but also reveal unusual behaviors that are difficult to detect by analyzing a large number of abnormal behavior patterns. These behaviors may be caused by system failures or security threats.

[0058] The log data in step S1 of this embodiment includes system logs, operation logs, black box logs, audit logs, WAL logs, and performance logs; [[ID=,14]]

[0059] Among them, the audit logs are recorded in OS files, and during preprocessing, the audit logs are desensitized.

[0060] In this embodiment, the system log preprocessing is specifically as follows:

[0061] The system logs generated during the operation of the database system process record the abnormal information of the database system process; among them, the system log files are saved in the $GAUSSLOG / pg_log directory, the GAUSSLOG variable is the configured openGauss database log path, and the log file format is postgresql-creation time.log. When crossing days or restarting the database, new log files are generated;

[0062] During the preprocessing of system logs, key information is extracted from the openGauss database system logs using regular expressions. The key information includes the printing time, log level, database name, operation type, and log content.

[0063] The operation logs in this embodiment refer to the logs generated when the database administrator operates the database using tools and when the tools are called by openGauss, which are used to trace the operation history of users or administrators operating the database and analyze the changes in the database. The operation log files are saved in the $GAUSSLOG / bin directory, and the log file format is tool name - log creation time - current.log. The operation logs include gs_ctl tool logs, gs_guc tool logs, and gs_dump tool logs. During preprocessing, the operation information is analyzed for the operation logs of different tools. The operation information analyzed for the operation logs of different tools includes the operation tool, operation time, operation level, and operation content.

[0064] The black box logs in this embodiment refer to the logs that contain the heap, stack, and register information of processes and threads dumped when the database system crashes, which are used for fault location. When the preprocessing discovers the existence of black box logs, the occurrence time and log information are simply parsed, and an alarm message is sent.

[0065] The WAL logs in this embodiment are Write-Ahead Logs (WAL), which are important data log files and are used to restore database data when necessary. They are located in the pg_xlog folder in the data directory. During preprocessing, the remaining disk space of the database system and the occupied space information of the WAL logs are obtained, and the platform components are notified to back up the WAL logs when necessary.

[0066] The performance logs in this embodiment are logs that record the running status of physical resources and perform performance detection when accessing external resources, including access detection information of external resources such as disks. When performance problems occur, the performance logs can conveniently locate the cause of the problem and improve the efficiency of problem-solving. Among them, the performance log files are saved in the $GAUSSLOG / gs_profile directory, and the log file format is postgresql - creation time.prf. During preprocessing, regular matching is performed on the performance logs to extract information. The information extracted by regular matching includes the printing time, host name, instance name, thread number, and log content. After preprocessing the performance log information, the noise data has been removed, and the performance log information with noise removed is obtained. Among them, the performance log information with noise removed includes the execution progress, repeated processes, and irrelevant operations.

[0067] Embodiment 2:

[0068] This embodiment provides a log analysis system based on the openGauss database, which includes:

[0069] A multi-dimensional log collection module for collecting and preprocessing log data from the openGauss database instance;

[0070] A model training module for extracting features and pattern recognition from the preprocessed log data to obtain a training model;

[0071] A real-time monitoring module for continuously extracting features and pattern recognition from the log data;

[0072] A trend analysis module for storing historical log data and identifying long-term trends from the historical log data;

[0073] A visualization interface for querying the pattern recognition structure, capable of generating custom reports and displaying custom log analysis dashboards.

[0074] The log data in this embodiment includes system logs, operation logs, black box logs, audit logs, WAL logs, and performance logs;

[0075] Among them, the preprocessing of system logs is specifically as follows: System logs generated during the operation of the database system process record the abnormal information of the database system process. Among them, the system log files are saved in the $GAUSSLOG / pg_log directory, and the GAUSSLOG variable is the configured openGauss database log path. The log file format is postgresql-creation time.log. When crossing days or restarting the database, new log files are generated. When preprocessing the system logs, key information is extracted from the openGauss database system logs using regular expressions. The key information includes print time, log level, database name, operation type, and log content;

[0076] Operation logs refer to the logs generated when the database administrator uses tools to operate the database and when the tools are called by openGauss, which are used to trace the operation history of users or administrators operating the database and analyze the change situation of the database. The operation log files are saved in the $GAUSSLOG / bin directory, and the log file format is tool name-log creation time-current.log. The operation logs include gs_ctl tool logs, gs_guc tool logs, and gs_dump tool logs. When preprocessing, the operation information is analyzed for the operation logs of different tools. The operation information analyzed for the operation logs of different tools includes operation tools, operation time, operation level, and operation content;

[0077] The black box log refers to the log dumped by the database system when a crash occurs, which contains the heap, stack, and register information of processes and threads and is used for fault location. When the preprocessing discovers the existence of the black box log, the occurrence time and log information are simply parsed, and an alarm message is sent.

[0078] The audit log is recorded into the OS file, and during preprocessing, the audit log is desensitized.

[0079] The WAL log is the Write-Ahead Log (WAL), which is an important data log file and is used to restore database data when necessary. It is located in the pg_xlog folder in the data directory. During preprocessing, the remaining disk space of the database system and the occupied space information of the WAL log are obtained, and when necessary, the platform component is notified to back up the WAL log.

[0080] The performance log is a log that records the running status of physical resources and is used for performance detection when accessing external resources, including access detection information of external resources such as disks. When performance problems occur, the performance log can conveniently locate the cause of the problem and improve the efficiency of problem-solving. Among them, the performance log file is saved in the $GAUSSLOG / gs_profile directory, and the log file format is postgresql - creation time.prf. During preprocessing, regular matching is performed on the performance log to extract information, and the information extracted by regular matching includes the printing time, host name, instance name, thread number, and log content. After preprocessing the performance log information, the noise data has been removed, and the performance log information with noise removed is obtained. Among them, the performance log information with noise removed includes the execution progress, repeated processes, and irrelevant operations.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A log analysis method based on the openGauss database, characterized in that, The method is as follows: Multi-dimensional log preprocessing: Based on the log content generated by the openGauss database as the data source, by configuring a virtual machine to connect to the openGauss database instance, collect log file data, and perform cleaning, denoising, and formatting on the obtained log data to obtain preprocessed log data; Feature extraction: Label the preprocessed log data, convert the unstructured log data into structured data, perform feature extraction on the structured data, identify elements whose importance for describing the database state is greater than the set threshold, and perform standardization and normalization on the identified elements whose importance for describing the database state is greater than the set threshold to obtain the extracted features and eliminate the dimensional differences between different data; Pattern recognition: Perform model training and state recognition on the data after feature extraction for pattern recognition. By identifying and classifying different patterns in the log data, describe the running state of the database system, and use supervised learning algorithms to train the normal operation patterns and abnormal database behaviors in pattern recognition. The trained model can learn the relationship between the data after feature extraction and the known patterns, so as to accurately classify new log data; Real-time monitoring: Continuously perform feature extraction and pattern recognition on the log data so that abnormal behaviors of the database can be detected in time; Once an anomaly is detected, immediately generate an alarm message, send the alarm message, and provide relevant log information and possible reasons to help the operation and maintenance personnel discover problems and troubleshoot them as soon as possible; Trend analysis: Identify long-term trends from historical log data, and obtain the change trend of database performance by analyzing the time series of log data; At the same time, predict the performance indicators and potential fault points of the database through time series analysis and prediction models; Result query: Query log data and analysis results through an intuitive interface, give reasonable and effective measure suggestions, and perform queries through multiple query methods such as keyword-based search, time range filtering, and custom report generation. At the same time, through the log analysis dashboard function, customize the analysis report according to your own needs, so that users can obtain information that meets their needs and quickly transmit log information and take corresponding measures as soon as possible; Among them, the content of the customized report includes log summaries, detailed descriptions of abnormal events, and charts of performance trends.

2. The log analysis method and system based on the openGauss database according to claim 1, characterized in that During the pattern recognition process, the support vector machine algorithm is used to improve the recognition accuracy of the model through supervised learning.

3. The log analysis method based on the openGauss database according to claim 1 or 2, characterized in that, Log data includes system logs, operation logs, black box logs, audit logs, WAL logs, and performance logs; Among them, audit logs are recorded in OS files, and during preprocessing, the audit logs are desensitized.

4. The log analysis method based on the openGauss database according to claim 3, wherein The system log preprocessing is as follows: The system logs generated during the operation of the database system process record the exception information of the database system process. Among them, the system log files are saved in the $GAUSSLOG / pg_log directory. The GAUSSLOG variable is the configured log path of the openGauss database. The log file format is postgresql-creation time.log. When crossing days or restarting the database, new log files are generated. During the preprocessing of the system logs, key information is extracted from the openGauss database system logs using regular expressions. The key information includes the printing time, log level, database name, operation type, and log content.

5. The log analysis method based on the openGauss database according to claim 3, wherein The operation logs refer to the logs generated when the database administrator operates the database using tools and when the tools are called by openGauss, which are used to trace the operation history of users or administrators operating the database and analyze the change situation of the database. The operation log files are saved in the $GAUSSLOG / bin directory. The log file format is tool name-log creation time-current.log. The operation logs include the gs_ctl tool logs, gs_guc tool logs, and gs_dump tool logs. During preprocessing, the operation information is analyzed for the operation logs of different tools. The operation information analyzed for the operation logs of different tools includes the operation tool, operation time, operation level, and operation content.

6. The log analysis method based on the openGauss database according to claim 3, wherein The black box logs refer to the logs dumped when the database system crashes, which contain the heap, stack, and register information of processes and threads and are used for fault location. When preprocessing discovers the existence of black box logs, the occurrence time and log information are simply parsed, and an alarm message is sent.

7. The log analysis method based on the openGauss database according to claim 3, characterized in that The WAL logs are the Write-Ahead Logs (WAL), which are important data log files and are used to restore database data when necessary. They are located in the pg_xlog folder of the data directory. During preprocessing, the remaining disk space of the database system and the occupied space information of the WAL logs are obtained, and the platform components are notified to back up the WAL logs when necessary.

8. The log analysis method and system based on the openGauss database according to claim 3, characterized in that The performance logs record the running status of physical resources and perform performance detection during access to external resources, including access detection information of external resources such as disks. When there are performance problems, the performance logs can conveniently locate the cause of the problem and improve the efficiency of problem solving. Among them, the performance log files are saved in the $GAUSSLOG / gs_profile directory. The log file format is postgresql-creation time.prf. During preprocessing, information is extracted from the performance logs through regular matching. The information extracted through regular matching includes the printing time, host name, instance name, thread number, and log content. After preprocessing the performance log information, the noise data has been removed, and the performance log information with noise removed is obtained. Among them, the performance log information with noise removed includes the execution progress, repeated processes, and irrelevant operations.

9. A log analysis system based on the openGauss database, characterized in that, This system includes: A multi-dimensional log collection module for collecting and preprocessing log data from openGauss database instances. A model training module for performing feature extraction and pattern recognition on the preprocessed log data to obtain a training model; A real-time monitoring module for continuously performing feature extraction and pattern recognition on the log data; A trend analysis module for storing historical log data and identifying long-term trends from the historical log data; A visualization interface for querying the pattern recognition structure, capable of generating custom reports and displaying custom log analysis dashboards.

10. The log analysis system based on the openGauss database according to claim 9, characterized in that, The log data includes system logs, operation logs, black box logs, audit logs, WAL logs, and performance logs; Among them, the preprocessing of system logs is specifically as follows: System logs generated during the operation of the database system process record the abnormal information of the database system process. The system log files are saved in the $GAUSSLOG / pg_log directory, where the GAUSSLOG variable is the configured log path of the openGauss database. The log file format is postgresql-creation time.log. When crossing days or restarting the database, new log files are generated. During the preprocessing of system logs, key information is extracted from the openGauss database system logs using regular expressions. The key information includes printing time, log level, database name, operation type, and log content; Operation logs refer to the logs generated when database administrators use tools to operate the database and when the tools are called by openGauss, which are used to trace the operation history of users or administrators operating the database and analyze the changes in the database. The operation log files are saved in the $GAUSSLOG / bin directory, and the log file format is tool name-log creation time-current.log. The operation logs include gs_ctl tool logs, gs_guc tool logs, and gs_dump tool logs. During preprocessing, the operation information of different tool operation logs is analyzed separately. The operation information analysis of different tool operation logs includes operation tool, operation time, operation level, and operation content; Black box logs refer to the logs containing the heap, stack, and register information of processes and threads dumped when the database system crashes, which are used for fault location. When preprocessing discovers the existence of black box logs, the occurrence time and log information are simply parsed, and an alarm message is sent; Audit logs are recorded in OS files. During preprocessing, the audit logs are desensitized; WAL logs are write-ahead logs WAL, which are important data log files and are used to restore database data when necessary. They are located in the pg_xlog folder of the data directory. During preprocessing, the remaining disk space of the database system and the occupied space information of the WAL logs are obtained, and the platform components are notified to back up the WAL logs when necessary; The performance log is a log that records the running status of physical resources, including performance detection information during access to external resources, such as access detection information for external resources like disks. When performance issues occur, the performance log can facilitate the location of the cause of the problem and improve the efficiency of problem-solving. Among them, the performance log file is saved in the $GAUSSLOG / gs_profile directory, and the log file format is postgresql-creation time.prf. During preprocessing, regular matching is performed on the performance log to extract information, and the information extracted by regular matching includes the printing time, host name, instance name, thread number, and log content. After preprocessing the performance log information, the noise data has been removed to obtain the performance log information with noise removed. Among them, the performance log information with noise removed includes the execution progress, repeated processes, and irrelevant operations.

Images