Log analysis and storage method based on message queue and columnar database

Abstract

The invention particularly relates to a log analysis and storage method based on a message queue and a columnar database. According to the log analysis and storage method based on the message queue and the columnar database, each business system takes log data as a producer to access a kafka message queue; defining consumers of the kafka message queue, and consuming the log data sent by each business system; the method comprises the following steps of: inserting log data of consumption into a column database ClickHouse; a JDBC drive program is adopted to read log data in a column database ClickHouse, and the log data is analyzed according to a user IP, operation time, an operation type and an operation module. According to the log analysis and storage method based on the message queue and the columnar database, decoupling between system components is allowed, the response performance and stability of the system are improved, large-scale data can be efficiently processed, high availability and reliability in the data processing process can be guaranteed, and the log analysis and storage method is suitable for application and popularization.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud computing, and in particular to a log analysis and storage method based on a message queue and a column-based database. Background Art

[0002] In early computer systems, inter-process communication mainly relied on shared memory and direct calls, but with the development of distributed systems, these communication methods became complex and fragile.

[0003] The original intention of designing a columnar database is to optimize data storage and retrieval efficiency for large-scale data analysis scenarios. Unlike traditional row-based databases, columnar databases store data by column, which is more suitable for scenarios that require efficient reading of large amounts of data.

[0004] Previously, distributed systems have used message queues to collect and process system logs. However, for large-scale log storage and analysis, traditional row-based databases store data by row. Even if only a certain column needs to be queried, the entire row of data must be read, which results in a lot of I / O overhead when processing large data sets. For log data query and analysis, traditional row-based databases usually require additional indexes to speed up queries, but these indexes become larger and more complex as the amount of data increases, increasing maintenance costs and other issues.

[0005] Based on the above problems, the present invention proposes a log analysis and storage method based on a message queue and a column-based database. Summary of the invention

[0006] In order to overcome the defects of the prior art, the present invention provides a simple and efficient log analysis and storage method based on message queue and column database.

[0007] The present invention is achieved through the following technical solutions:

[0008] A log analysis and storage method based on a message queue and a column database comprises the following steps:

[0009] Step S1, each business system accesses the log data as a producer to the Kafka message queue;

[0010] In step S1, the implementation process is as follows:

[0011] Step S1.1, set a topic for the log data generated by each business system, and use the set topic as a unique identifier to distinguish the log data of each business system; at the same time, set log rules, including storing the user's IP, operation time, operation type, operation module and operation content, for subsequent log analysis;

[0012] Step S1.2, define the KafkaProducer integrated message queue, configure the IP address, port and topic of Kafka, and serialize the relevant configuration;

[0013] Step S1.3: Send the log data generated by each business system to the topic specified by the Kafka message queue through the KafkaProducer.send() method.

[0014] Step S2: define the consumers of the Kafka message queue to consume the log data sent by each business system;

[0015] In step S2, the implementation process is as follows:

[0016] Step S2.1, define DefaultKafkaConsumerFactory as the message queue consumer, configure the IP address, port and topic of Kafka, and serialize the relevant configuration;

[0017] Step S2.2: Define @KafkaListener to monitor the log data in ConsumerRecord in real time to consume the log data sent by each business system in real time.

[0018] Step S3: Insert the consumed log data into the column-based database ClickHouse;

[0019] In step S3, the implementation process is as follows:

[0020] Step S3.1, define the data structure, create the log data table structure and entity data;

[0021] Step S3.2: The log storage and analysis system is persistently integrated with the column-based database ClickHouse, and relevant configurations are implemented;

[0022] Step S3.3, execute the save() method to persist the log data to the column-based database ClickHouse.

[0023] Step S4: Analyze and process the log data in the column-based database ClickHouse.

[0024] The JDBC driver is used to read the log data in the column-based database ClickHouse, and the log data is analyzed according to user IP, operation time, operation type and operation module.

[0025] In step S4, the implementation process is as follows:

[0026] Step S4.1: Analyze the user's location based on the user IP address, and provide corresponding regional services for areas with a high number of operating users;

[0027] Step S4.2: Analyze the peak hours of users using the business system according to the operation time, and reinforce and flexibly expand the business system during the peak hours.

[0028] Step S4.3, analyzing the frequency of use of each module by the user according to the operation module, analyzing the high-frequency module, and improving the performance of the corresponding high-frequency module in a targeted manner to ensure system stability;

[0029] Step S4.4: Use echarts visualization chart technology to display relevant log analysis data according to custom log rules, and intuitively reflect the usage and operation of the business system.

[0030] A log analysis storage system based on message queue and column database, including Kafka message queue, column database ClickHous and analysis module;

[0031] The Kafka message queue is responsible for accessing the log data generated by each business system, and inserting the log data into the columnar database ClickHouse by consuming the accessed log data;

[0032] The column-based database ClickHous is responsible for storing log data;

[0033] The analysis module is responsible for reading log data in the column-based database ClickHouse using the JDBC driver, and analyzing the log data based on user IP, operation time, operation type and operation module.

[0034] The Kafka message queue accesses the log data generated by each business system, and the implementation process is as follows:

[0035] Step S1.1, set a topic for the log data generated by each business system, and use the set topic as a unique identifier to distinguish the log data of each business system; at the same time, set log rules, including storing the user's IP, operation time, operation type, operation module and operation content, for subsequent log analysis;

[0036] Step S1.2, define the KafkaProducer integrated message queue, configure the IP address, port and topic of Kafka, and serialize the relevant configuration;

[0037] Step S1.3: Send the log data generated by each business system to the topic specified by the Kafka message queue through the KafkaProducer.send() method.

[0038] The Kafka message queue consumes the accessed log data and inserts the log data into the columnar database ClickHouse. The implementation process is as follows:

[0039] Step S2.1, define DefaultKafkaConsumerFactory as the message queue consumer, configure the IP address, port and topic of Kafka, and serialize the relevant configuration;

[0040] Step S2.2, define @KafkaListener to monitor the log data in ConsumerRecord in real time to consume the log data sent by each business system in real time;

[0041] Step S2.3: Insert the consumed log data into the column-based database ClickHouse.

[0042] The column-based database ClickHous stores log data, and the implementation process is as follows:

[0043] Step S3.1, define the data structure, create the log data table structure and entity data;

[0044] Step S3.2: The log storage and analysis system is persistently integrated with the column-based database ClickHouse, and relevant configurations are implemented;

[0045] Step S3.3, execute the save() method to persist the log data to the column-based database ClickHouse.

[0046] The analysis module analyzes the log data, and the implementation process is as follows:

[0047] Step S4.1: Analyze the user's location based on the user IP address, and provide corresponding regional services for areas with a high number of operating users;

[0048] Step S4.2: Analyze the peak hours of users using the business system according to the operation time, and reinforce and flexibly expand the business system during the peak hours.

[0049] Step S4.3, analyzing the frequency of use of each module by the user according to the operation module, analyzing the high-frequency module, and improving the performance of the corresponding high-frequency module in a targeted manner to ensure system stability;

[0050] Step S4.4: Use echarts visualization chart technology to display relevant log analysis data according to custom log rules, and intuitively reflect the usage and operation of the business system.

[0051] A log analysis storage device based on a message queue and a columnar database, characterized in that it includes a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the above method steps when executing the computer program.

[0052] A readable storage medium, characterized in that: a computer program is stored on the readable storage medium, and the computer program implements the above method steps when executed by a processor.

[0053] The beneficial effects of the present invention are as follows: the log analysis and storage method based on message queues and columnar databases allows decoupling between system components, improves the response performance and stability of the system, can efficiently process large-scale data, and ensure high availability and reliability during data processing, and is suitable for promotion and application. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0055] Attached Figure 1 The diagram is a schematic diagram of a log analysis and storage method based on a message queue and a column-based database according to the present invention. DETAILED DESCRIPTION

[0056] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0057] The emergence of message queues is to solve the communication problem between components in distributed systems. As a middleware, message queues provide a more flexible and scalable solution, allowing multiple producers to send messages to multiple consumers, thereby achieving decoupling between producers and consumers.

[0058] The log analysis and storage method based on message queue and column database includes the following steps:

[0059] Step S1, each business system accesses the log data as a producer to the Kafka message queue;

[0060] In step S1, the implementation process is as follows:

[0061] Step S1.1, set a topic for the log data generated by each business system, and use the set topic as a unique identifier to distinguish the log data of each business system; at the same time, set log rules, including storing the user's IP, operation time, operation type, operation module and operation content, for subsequent log analysis;

[0062] Step S1.2, define the KafkaProducer integrated message queue, configure the IP address, port and topic of Kafka, and serialize the relevant configuration;

[0063] Step S1.3: Send the log data generated by each business system to the topic specified by the Kafka message queue through the KafkaProducer.send() method.

[0064] Step S2: define the consumers of the Kafka message queue to consume the log data sent by each business system;

[0065] In step S2, the implementation process is as follows:

[0066] Step S2.1, define DefaultKafkaConsumerFactory as the message queue consumer, configure the IP address, port and topic of Kafka, and serialize the relevant configuration;

[0067] Step S2.2: Define @KafkaListener to monitor the log data in ConsumerRecord in real time to consume the log data sent by each business system in real time.

[0068] Step S3: Insert the consumed log data into the column-based database ClickHouse;

[0069] In step S3, the implementation process is as follows:

[0070] Step S3.1, define the data structure, create the log data table structure and entity data;

[0071] Step S3.2: The log storage and analysis system is persistently integrated with the column-based database ClickHouse, and relevant configurations are implemented;

[0072] driver-class-name:com.clickhouse.jdbc.ClickHouseDriver,

[0073] url:jdbc:ch: / / 127.0.0.1 / default? serverTimezone=Asia / S,

[0074] password:xxxx

[0075] Step S3.3, execute the save() method to persist the log data to the column-based database ClickHouse.

[0076] Step S4: Analyze and process the log data in the column-based database ClickHouse.

[0077] The JDBC driver is used to read the log data in the column-based database ClickHouse, and the log data is analyzed according to user IP, operation time, operation type and operation module.

[0078] In step S4, the implementation process is as follows:

[0079] Step S4.1: Analyze the user's location based on the user IP address, and provide corresponding regional services for areas with a high number of operating users;

[0080] Step S4.2: Analyze the peak hours of users using the business system according to the operation time, and reinforce and flexibly expand the business system during the peak hours.

[0081] Step S4.3, analyzing the frequency of use of each module by the user according to the operation module, analyzing the high-frequency module, and improving the performance of the corresponding high-frequency module in a targeted manner to ensure system stability;

[0082] Step S4.4: Use echarts visualization chart technology to display relevant log analysis data according to custom log rules, and intuitively reflect the usage and operation of the business system.

[0083] The log analysis storage system based on message queue and column database includes Kafka message queue, column database ClickHous and analysis module;

[0084] The Kafka message queue is responsible for accessing the log data generated by each business system, and inserting the log data into the columnar database ClickHouse by consuming the accessed log data;

[0085] The column-based database ClickHous is responsible for storing log data;

[0086] The analysis module is responsible for reading log data in the column-based database ClickHouse using the JDBC driver, and analyzing the log data based on user IP, operation time, operation type and operation module.

[0087] The Kafka message queue accesses the log data generated by each business system, and the implementation process is as follows:

[0088] Step S1.1, set a topic for the log data generated by each business system, and use the set topic as a unique identifier to distinguish the log data of each business system; at the same time, set log rules, including storing the user's IP, operation time, operation type, operation module and operation content, for subsequent log analysis;

[0089] Step S1.2, define the KafkaProducer integrated message queue, configure the IP address, port and topic of Kafka, and serialize the relevant configuration;

[0090] Step S1.3: Send the log data generated by each business system to the topic specified by the Kafka message queue through the KafkaProducer.send() method.

[0091] The Kafka message queue consumes the accessed log data and inserts the log data into the columnar database ClickHouse. The implementation process is as follows:

[0092] Step S2.1, define DefaultKafkaConsumerFactory as the message queue consumer, configure the IP address, port and topic of Kafka, and serialize the relevant configuration;

[0093] Step S2.2, define @KafkaListener to monitor the log data in ConsumerRecord in real time to consume the log data sent by each business system in real time;

[0094] Step S2.3: Insert the consumed log data into the column-based database ClickHouse.

[0095] The column-based database ClickHous stores log data, and the implementation process is as follows:

[0096] Step S3.1, define the data structure, create the log data table structure and entity data;

[0097] Step S3.2: The log storage and analysis system is persistently integrated with the column-based database ClickHouse, and relevant configurations are implemented;

[0098] driver-class-name:com.ClickHouse.jdbc.ClickHouseDriver,

[0099] url:jdbc:ch: / / 127.0.0.1 / default? serverTimezone=Asia / S,

[0100] password:xxxx

[0101] Step S3.3, execute the save() method to persist the log data to the column-based database ClickHouse.

[0102] The analysis module analyzes the log data, and the implementation process is as follows:

[0103] Step S4.1: Analyze the user's location based on the user IP address, and provide corresponding regional services for areas with a high number of operating users;

[0104] Step S4.2: Analyze the peak hours of users using the business system according to the operation time, and reinforce and flexibly expand the business system during the peak hours.

[0105] Step S4.3, analyzing the frequency of use of each module by the user according to the operation module, analyzing the high-frequency module, and improving the performance of the corresponding high-frequency module in a targeted manner to ensure system stability;

[0106] Step S4.4: Use echarts visualization chart technology to display relevant log analysis data according to custom log rules, and intuitively reflect the usage and operation of the business system.

[0107] The log analysis storage device based on message queue and column database includes a memory and a processor; the memory is used to store computer programs, and the processor is used to implement the above method steps when executing the computer program.

[0108] The readable storage medium stores a computer program, and the computer program implements the above method steps when executed by a processor.

[0109] Compared with the existing technology, the log analysis and storage method based on message queue and column database has the following characteristics:

[0110] High performance: The message queue is used to ensure high performance through asynchronous communication, distributed deployment, data caching and other functions. At the same time, the column-based database is used to store and process column data, which can quickly locate the required log data, thereby improving query efficiency and making log analysis more convenient.

[0111] Decoupling: Logs and businesses can be decoupled, so that each business system can focus on business scenarios and send log data to the message queue in a unified manner without having to perform other operations on the log data.

[0112] Unified management: Logs of different business systems can be managed in a unified manner. By setting different topic consumption, the log data can be fully centralized, which facilitates the unified processing and analysis of subsequent logs.

[0113] Integrated analysis: Through the integrated analysis of logs, the system usage and operation status are analyzed as a whole, which improves the performance of high-frequency applications and ensures system stability.

[0114] The embodiment described above is only one specific implementation of the present invention. Common changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the protection scope of the present invention.

Claims

1. A log analysis and storage method based on a message queue and a column-based database, characterized in that: The following steps are involved: Step S1, each business system accesses the log data as a producer to the Kafka message queue; Step S2: define the consumers of the Kafka message queue to consume the log data sent by each business system; Step S3: Insert the consumed log data into the column-based database ClickHouse; Define the data structure, create the log data table structure and entity data, integrate the column-based database ClickHouse for persistence, and implement related configurations; execute the save() method to persist the consumed log data into the column-based database ClickHouse; Step S4: Analyze and process the log data in the column-based database ClickHouse; The JDBC driver is used to read the log data in the column-based database ClickHouse, and the log data is analyzed according to user IP, operation time, operation type and operation module.

2. The log analysis and storage method based on message queue and column database according to claim 1 is characterized in that: In step S1, the implementation process is as follows: Step S1.1, set a topic for the log data generated by each business system, and use the set topic as a unique identifier to distinguish the log data of each business system; at the same time, set log rules, including storing the user's IP, operation time, operation type, operation module and operation content, for subsequent log analysis; Step S1.2, define the KafkaProducer integrated message queue, configure the IP address, port and topic of Kafka, and serialize the relevant configuration; Step S1.3: Send the log data generated by each business system to the topic specified by the Kafka message queue through the KafkaProducer.send() method.

3. The log analysis and storage method based on message queue and column database according to claim 1 is characterized in that: In step S2, the implementation process is as follows: Step S2.1, define DefaultKafkaConsumerFactory as the message queue consumer, configure the IP address, port and topic of kafka, and serialize the relevant configuration; Step S2.12: Define @KafkaListener to monitor the log data in ConsumerRecord in real time to consume the log data sent by each business system in real time.

4. The log analysis and storage method based on message queue and column database according to claim 1 is characterized in that: In step S4, the implementation process is as follows: Step S4.1: Analyze the user's location based on the user IP address, and provide corresponding regional services for areas with a high number of operating users; Step S4.2: Analyze the peak hours of users using the business system according to the operation time, and reinforce and flexibly expand the business system during the peak hours; Step S4.3, analyzing the frequency of use of each module by the user according to the operation module, analyzing the high-frequency module, and improving the performance of the corresponding high-frequency module in a targeted manner to ensure system stability; Step S4.4: Use echarts visualization chart technology to display relevant log analysis data according to custom log rules, and intuitively reflect the usage and operation of the business system.

5. A log analysis storage system based on message queue and column database, characterized in that: Including Kafka message queue, column database ClickHous and analysis module; The Kafka message queue is responsible for accessing the log data generated by each business system, and inserting the log data into the columnar database ClickHouse by consuming the accessed log data; The column-based database ClickHous is responsible for storing log data; The analysis module is responsible for reading log data in the column-based database ClickHouse using the JDBC driver, and analyzing the log data based on user IP, operation time, operation type and operation module.

6. The log analysis storage system based on message queue and column database according to claim 5, characterized in that: The Kafka message queue accesses the log data generated by each business system, and the implementation process is as follows: Step S1.1, set a topic for the log data generated by each business system, and use the set topic as a unique identifier to distinguish the log data of each business system; at the same time, set log rules, including storing the user's IP, operation time, operation type, operation module and operation content, for subsequent log analysis; Step S1.2, define the KafkaProducer integrated message queue, configure the IP address, port and topic of Kafka, and serialize the relevant configuration; Step S1.3: Send the log data generated by each business system to the topic specified by the Kafka message queue through the KafkaProducer.send() method.

7. The log analysis storage system based on message queue and column database according to claim 5, characterized in that: The Kafka message queue consumes the accessed log data and inserts the log data into the columnar database ClickHouse. The implementation process is as follows: Step S2.1, define DefaultKafkaConsumerFactory as the message queue consumer, configure the IP address, port and topic of kafka, and serialize the relevant configuration; Step S2.12, define @KafkaListener to monitor the log data in ConsumerRecord in real time, so as to consume the log data sent by each business system in real time; Step S2.3, insert the consumed log data into the column-based database ClickHouse; The column-based database ClickHous stores log data, and the implementation process is as follows: Step S3.1, define the data structure, create the log data table structure and entity data; Step S3.2: The log storage and analysis system is persistently integrated with the column-based database ClickHouse, and relevant configurations are implemented; Step S3.3, execute the save() method to persist the log data to the column-based database ClickHouse.

8. The log analysis storage system based on message queue and column database according to claim 5, characterized in that: The analysis module analyzes the log data, and the implementation process is as follows: Step S4.1: Analyze the user's location based on the user IP address, and provide corresponding regional services for areas with a high number of operating users; Step S4.2: Analyze the peak hours of users using the business system according to the operation time, and reinforce and flexibly expand the business system during the peak hours; Step S4.3, analyzing the frequency of use of each module by the user according to the operation module, analyzing the high-frequency module, and improving the performance of the corresponding high-frequency module in a targeted manner to ensure system stability; Step S4.4: Use echarts visualization chart technology to display relevant log analysis data according to custom log rules, and intuitively reflect the usage and operation of the business system.

9. A log analysis storage device based on a message queue and a column database, characterized in that: The method comprises a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the method according to any one of claims 1 to 4 when executing the computer program.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images