Intelligent analysis platform for log big data

Abstract

The invention relates to the technical field of computer data processing, in particular to a log big data intelligent analysis platform. Comprising the following steps: S1, a client sends data to a collector module for data acquisition Agent; s2, after the collected data are accessed, the collected data firstly enter a message queue Kafka component to be temporarily stored; s3, carrying out ETL (Extract Transform Load) cleaning and Logriver on the temporarily stored data; s4, the temporary storage data subjected to ETF cleaning and path combining are subjected to Beaver storage; s5, performing search statistics SPLServer on grammar analysis and task scheduling of the search processing language; s6, on the basis of search statistics, displaying layer functions for all log big data intelligent analysis platforms; s7, executing AI operation on the index data and the log data; and S8, carrying out operation maintenance on the system cluster. According to the method, the problem of log big data transmission is solved, a method about timestamps, sources, data and the like in a processing process is solved, and safe, efficient and high-speed data transmission is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of computer data processing, and in particular to a log big data intelligent analysis platform. Background Art

[0002] 1. Logs are messages generated by computer systems, devices, software, etc. in response to certain stimuli to indicate that something has happened. The basic content of a typical log message includes at least three parts: timestamp, source, and data. Log analysis is to process and analyze logs to get their meaning; a sound log record and a flexible log analysis system are the basis for the normal operation, optimization, and incident response of IT systems.

[0003] In the 21st century, with the advent of the Internet and big data era, Chinese enterprises are facing unprecedented challenges. Various systems in the computer room have been generating logs continuously, but when there are problems or hidden dangers in the system, can we analyze clues from the logs? When the system suffers from serious attacks and damage, can the log analysis system help us get out of the quagmire? This is a test of the actual working ability of the entire IT department.

[0004] In addition, the frequent changes in the logs themselves also bring great uncertainty and difficulty to the analysis. According to statistics: 1.1. Logging is common in software development. On average, one line of code out of every 30 lines is a log. 1.2. Log information is very helpful for debugging the actual deployment system, and the acceleration ratio of shortening the debugging time is; 1.3. The update frequency of the log code is about 1 times faster than other codes; 1.4. About a quarter of log modifications are to write new program variables to the log; 1.5. About half of the log modifications are modifications to the static text of the log message.

[0005] 2. Limitations of general technology implementation Log analysis technology has gone through several generations of development over the years, and various technical means and tools have emerged. While these tools can solve some problems, they also have certain limitations.

[0006] 2.1. Command line tools such as grep / sed / awk: After an operation and maintenance failure occurs, some companies' operation and maintenance engineers log in to each server and use Linux script tools such as grep / sed / awk to find the cause of the failure in the log. This takes a long time to troubleshoot and may not find the root cause of the failure in time. Awk can be used in conjunction with sort, uniq, etc. for preliminary statistical analysis, but the terminal table type output is very unintuitive and lacks visual visualization of charts, making it impossible to find abnormal points in the first place.

[0007] 2.2. Distributed systems such as Hadoop / Spark / Storm Hadoop, Storm, and Spark are all development frameworks. Users need to develop separate Hadoop, Storm, or Spark programs to process logs. The usage threshold is high, and it is not easy to recruit excellent distributed development engineers.

[0008] In addition, Hadoop is a batch processing, and its real-time performance is poor: many companies that use Hadoop to process logs usually process the day's logs every night and produce statistical reports the next day. Some companies do better, but they can only see log analysis from a few hours ago. Some users have a more standardized log structure design and can use Hive or Pig under the Hadoop framework to query logs, but the delay may still reach tens of minutes.

[0009] Storm is a stream processing framework that can do real-time processing very well, but it lacks the ability to review historical data, even historical data that is not too long ago, which makes Storm only suitable for fast statistics and display on cumulative indicators such as PV statistics. Generally speaking, only individual key values ​​such as transaction amounts on the large screen need to be accurately calculated in milliseconds in real time.

[0010] 2.3. Column storage such as Druid / ClickHouse / Kylin Using a columnar storage system for log storage and analysis is essentially an extension of the idea of ​​using MySQL for log storage in the early days. This method requires a clear understanding of the log format and sufficient grasp of the purpose of analysis, so that the logs can be structured and processed into multi-dimensional statistical data tables in advance through pre-aggregation for subsequent high-speed queries. Configuring pre-aggregation itself also requires research, design, and debugging time, and cannot be fully automated and efficient.

[0011] Columnar storage is not enough for operation and maintenance and security scenarios with clear full-text search requirements; just like the early MySQL solution required additional Sphinx indexing, this method is generally only suitable for business analysis scenarios of website access logs.

[0012] 2.4. Elastic Stack / Graylog and other projects Elastic Stack is currently the most popular choice for log analysis in the open source field. Although Elasticsearch itself is not developed specifically for log analysis needs, Elastic Stack does provide the ability to collect, transmit, process, store, query, count, and display logs. However, the overly fragmented product line has led to a certain threshold for its deployment and management, and its open source part lacks common advanced features such as design, permissions, alarms, and management. In fact, starting from version 6.0, most of the new non-Lucene features of Elastic Stack are released under the ElasticLicense license.

[0013] In addition, the internal API of Elastic Stack changes frequently, and it is extremely difficult to conduct in-depth secondary development on it. Graylog is a model for integrating and constructing open source log analysis systems on the Elasticsearch system. Different components and processes can be easily managed and used on a unified interface. Due to its single architecture, it is a wise choice to use Graylog to build an entry-level log analysis system when the log volume is not large, the latency requirements are not high, and the analysis requirements are not complex. However, due to the limitations of Elasticsearch's capabilities, Graylog can only provide query and simple statistical functions for fields that have been extracted and stored, and cannot provide secondary statistics and high-end data mining capabilities based on the results.

[0014] 2.5. Grafana / Fluentd / Loki and other cloud-native projects In cloud-native projects, Grafana is often used to display monitoring indicators and statistics of container and microservice environments. Some of the data comes from logs. Grafana also launched the Loki open source project in 2019, which is specially used for the storage and query of container environment logs in conjunction with Fluentd collection. However, the design premise of Loki is that on the container platform, the microservice splitting is already very fine-grained, and each log can be fully located by querying kubernetes labels or dockertags. Users only need to page up and down to read the original log text. This means that loki does not provide index query capabilities for the original log text, nor does it provide log field extraction and statistical analysis capabilities. Since there is no full-text index, keyword filtering is performed through concurrent grep, and the Loki project is still in its early stages, with poor fault tolerance. If the time span is slightly larger, it will enter a dead state. Summary of the invention The purpose of the present invention is to provide a log big data intelligent analysis platform to solve the problems raised in the above background technology; In order to achieve the above object, the present invention adopts the following technical solutions: The log big data intelligent analysis platform includes the following steps: S1: The client sends data to the collector module for data collection; S2: After the collected data is accessed, it will first enter the message queue Kafka component for temporary storage; S3: ETL cleaning and routing of temporary data LogRiver; S4: Store Beaver for the temporary data after ETF cleaning and routing; S5: Search statistics SPLServer for syntax parsing and task scheduling of search processing language; S6: Based on search statistics, the intelligent analysis platform for all log big data presents layer functions; S7: Perform AI operations on indicator data and log data; S8: Perform operation and maintenance on the system cluster itself; The log big data analysis system includes: collector module, data cleaning component LogRiver, backup component Archiver, intelligent operation and maintenance component KPIMonitor, data factory component DataFlow, LogRiver component, Beaver index storage component, SPLServer search statistics module, name service component Zookeeper, shared storage component MongoDB GridFS, ZZZWeb component, KPIAnalyzer algorithm service component, LogAnalyzer algorithm service component, Spark component, Manager component and InfluxDB component.

[0016] The data collection step Agent includes: sending data from the client to the collector module, mainly through the dedicated Agent provided by the log big data intelligent analysis platform, and can also use the standard syslog protocol (RFC5424), HTTP (S) and protobuf methods.

[0017] The message queue step Kafka includes: after the collected data is connected, it enters the message queue component for temporary storage. Some open source / commercial log analysis products do not introduce message queue components to prevent the subsequent indexing process from reaching a bottleneck when the data volume suddenly increases due to business reasons, causing the entire performance to fluctuate and even data loss.

[0018] The data cleaning and routing steps LogRiver described include: ETL cleaning of data, which runs downstream of the message queue component and can perform stateless horizontal cluster expansion. LogRiver focuses on streaming single-line log parsing.

[0019] The index storage step Beaver includes: using the Beaver storage engine to comprehensively process data, which is the core technology of the log big data intelligent analysis platform.

[0020] The search statistics step SPLServer includes: using the SPL (Search Processing Language) syntax uniquely designed by the search statistics module SPLServer log big data intelligent analysis platform, and undertaking the syntax parsing and task scheduling of SPL.

[0021] The functions of the presentation and interface ZZZWeb component include: based on search statistics, all log big data intelligent analysis platform presentation layer functions, including search, statistics, visualization, dashboard, report, alarm, etc.

[0022] The Analyzer algorithm service component includes: using KPIAnalyzer and LogAnalyzer components to perform AI operations on intelligent operation and maintenance of indicator data.

[0023] The management and maintenance Manager component includes: independently completing the operation and maintenance of the system cluster itself, the deployment of each component, configuration changes, performance monitoring alarms, and version upgrades.

[0024] Compared with the prior art, the present invention has the following beneficial effects: This system is based on the data collection Agent step, the message queue Kafka step, the index storage Beaver step, the search statistics SPLServer step, the display and interface ZZZWeb step, the algorithm service Analyzer step, the management and maintenance Manager step and the various functional modules and components mentioned above. It solves the problems of log big data transmission, display, security, effectiveness, transmission speed in the processing process, and realizes safe, efficient and high-speed data transmission.

[0025] Its differentiated features are: 1. View all logs on one interface, greatly improving operation and maintenance efficiency.

[0026] 2. For interrelated logs, unified processing and analysis of logs at the collective unit level can compare and correlate logs from different sources, remove noise, and reflect the real situation.

[0027] 3. When hackers invade servers or network devices, they often delete logs to erase evidence. Unified uploading and management of logs can detect intrusions in a timely manner, monitor alarms, and store logs for a long time to facilitate post-event security audits.

[0028] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 The flowchart of the overall architecture of the platform of the present invention is as follows; Figure 2 This is a schematic diagram comparing the CPU usage of the Agent of the present invention and the mainstream open source Agent at the same sending rate; Figure 3 It is a schematic diagram comparing the maximum sending rates of the Agent of the present invention and the mainstream open source Agent under the same resource consumption; Figure 4 It is a comparison diagram of the reading and writing performance test of Beaver of the present invention and the mainstream open source solution; Figure 5 It is a schematic diagram of the product architecture of the present invention; Figure 6 This is a data flow diagram of the intelligent operation and maintenance module of the present invention. DETAILED DESCRIPTION

[0031] The principles and features of the present invention are described below in conjunction with the accompanying drawings.

[0032] like Figure 1 As shown in the figure, the log big data intelligent analysis platform processes data in the following steps: S1: The client sends data to the collector module for data collection; S2: After the collected data is accessed, it will first enter the message queue Kafka component for temporary storage; S3: ETL cleaning and routing of temporary data LogRiver; S4: Store Beaver for the temporary data after ETF cleaning and routing; S5: Search statistics SPLServer for syntax parsing and task scheduling of search processing language; S6: Based on search statistics, the intelligent analysis platform for all log big data presents layer functions; S7: Perform AI operations on indicator data and log data; S8: Perform operation and maintenance on the system cluster itself.

[0033] Furthermore, the log big data analysis system is constructed to include: collector module, data cleaning component LogRiver, backup component Archiver, intelligent operation and maintenance component KPIMonitor, data factory component DataFlow, LogRiver component, Beaver index storage component, SPLServer search statistics module, name service component Zookeeper, shared storage component MongoDB GridFS, ZZZWeb component, KPIAnalyzer algorithm service component, LogAnalyzer algorithm service component, Spark component, Manager component and InfluxDB component.

[0034] Furthermore, the data collection step Agent includes: the client sends data to the collector module, mainly through the dedicated Agent provided by the log big data intelligent analysis platform, and can also use the standard syslog protocol (RFC5424), HTTP (S) and protobuf methods.

[0035] Furthermore, the message queue step Kafka includes: after the collected data is connected, it is entered into the message queue component for temporary storage. Some open source / commercial log analysis products do not introduce message queue components to prevent the subsequent indexing process from reaching a bottleneck when the data volume suddenly increases due to business reasons, causing the entire performance to fluctuate until data is lost.

[0036] From the perspective of data reliability, the message queue is an indispensable component in the log analysis system. In addition, the message queue component also facilitates data distribution and consumption for different purposes. In the log big data intelligent analysis platform, including the data cleaning component LogRiver, the backup component Archiver, the intelligent operation and maintenance component KPIMonitor, the data factory component DataFlow, etc., can consume data from specific topics in the message queue. The log big data intelligent analysis platform also allows other third-party applications other than its own to consume data.

[0037] Furthermore, the data cleaning and routing steps of LogRiver include: ETL cleaning of data, which runs downstream of the message queue component and can perform stateless horizontal cluster expansion. LogRiver focuses on streaming single-line log parsing.

[0038] Furthermore, the index storage step Beaver includes: using the Beaver storage engine to perform comprehensive processing on the data, which is the core technology of the log big data intelligent analysis platform.

[0039] In addition to data cleaning, LogRiver is also responsible for data forwarding and routing management. Currently, the log big data intelligent analysis platform supports forwarding cleaned log data to other topics of the message queue component or writing it to other indexes of the index storage component according to specified rules. Tests have shown that the LogRiver component is an order of magnitude faster than the data cleaning and storage speed based on open source Spark Streaming, with the same resource consumption.

[0040] Furthermore, the search statistics step SPLServer includes: using the SPL (Search Processing Language) syntax uniquely designed by the search statistics module SPLServer log big data intelligent analysis platform, and undertaking the syntax parsing and task scheduling of SPL.

[0041] Among them, SPL syntax instructions are divided into streaming instructions and centralized instructions. For streaming instructions, SPLServer can automatically distribute them to Beaver storage components for distributed execution after parsing. Centralized instructions are executed locally on SPLServer.

[0042] SPLServer is responsible for scheduling two types of tasks. One is the query level. SPLServer will automatically perform time-based sharding on all SPL queries and perform fair round-robin scheduling on the sharded subtasks to ensure that queries from different users can get the response they deserve. This avoids the phenomenon in mainstream open source solutions where a large-scale query hangs up the entire cluster. In addition, SPLServer can automatically convert queries for specific scenarios (such as accurate statistics of massive grouped data) into Spark tasks according to user needs and distribute them to Spark clusters for execution.

[0043] Avoid the result errors and OutOfMemory failures of massive group statistics in mainstream open source solutions.

[0044] The other is the result processing level, where SPLServer is responsible for the execution plan scheduling of all query tasks that need to be executed offline in the background of the log big data intelligent analysis platform, including offline tasks, download tasks, scheduled tasks, alarm tasks, report tasks, etc.

[0045] SPLServer can also be expanded horizontally in clusters, where the task status and task result data will rely on the name service component Zookeeper and the shared storage component MongoDB GridFS.

[0046] Furthermore, the steps of displaying and interfacing ZZZWeb include: based on search statistics, all log big data intelligent analysis platform display layer functions, including but not limited to: search, statistics, visualization, dashboard, report, alarm, etc. are provided by ZZZWeb components. In addition, ZZZWeb components are also responsible for external API interfaces for various functional services such as permission configuration, resource management and data query.

[0047] The ZZZWeb component can also be expanded horizontally in clusters and forwarded through the outermost load balancing component Nginx.

[0048] Furthermore, the management and maintenance Manager component includes: independently completing the operation and maintenance of the system cluster itself, the deployment of each component, configuration changes, performance monitoring alarms, and version upgrades.

[0049] The log big data intelligent analysis platform supports multiple data access methods, such as the standard syslog protocol (RFC5424), HTTP(S) and protobuf. The more important method is through the dedicated Agent provided by the log big data intelligent analysis platform.

[0050] The log big data intelligent analysis platform provides two Agent implementations. One is a lightweight implementation written in Golang language, with full functions and good performance. The other is a Java language developed for UNIX platforms that do not support Golang language, which only supports basic functions. In general, users are recommended to use the Golang version of Agent.

[0051] In order to solve the network security problem of cross-computer room transmission, the log big data intelligent analysis platform also provides a dedicated Agent-Proxy to implement it. The data in the same computer room can be transferred through the Agent-Proxy to facilitate network policy settings.

[0052] At the data collection level, there are many options in the open source community, and their functions and performance vary. The functional comparison between the log big data intelligent analysis platform agent and the mainstream open source agents is shown in Table 1.

[0053] Table 1: Open Source Agent Function Comparison

[0054] The CPU usage of the Log Big Data Intelligent Analysis Platform Agent and the mainstream open source Agent at the same sending rate is shown in Figure 2 ,The data of the Agent of the present invention is obviously better than that of other mainstream open source Agents.

[0055] For a comparison of the maximum sending rates of the Log Big Data Intelligent Analysis Platform Agent and mainstream open source Agents under the same resource consumption, see Figure 3 The maximum sending rate data of the Agent of the present invention is obviously better than that of other mainstream open source Agents.

[0056] Furthermore, the message queue Kafka step includes: after the collected data is connected, it is entered into the message queue component for temporary storage. Some open source / commercial log analysis products do not introduce message queue components to prevent the subsequent indexing process from reaching a bottleneck when the data volume suddenly increases due to business reasons, causing the performance of the entire cluster to fluctuate until data is lost.

[0057] From the perspective of data reliability, the message queue is an indispensable component in the log analysis system. In addition, the message queue component also facilitates data distribution and consumption for different purposes. In the log big data intelligent analysis platform, including the data cleaning component LogRiver, the backup component Archiver, the intelligent operation and maintenance component KPIMonitor, the data factory component DataFlow, etc., can consume data from specific topics in the message queue. The log big data intelligent analysis platform also allows other third-party applications other than its own to consume data.

[0058] The data cleaning and routing LogRiver steps include: ETL cleaning of data, which runs downstream of the message queue component and can perform stateless horizontal cluster expansion. LogRiver focuses on streaming single-line log parsing.

[0059] In addition to data cleaning, LogRiver is also responsible for data forwarding and routing management. Currently, the log big data intelligent analysis platform supports forwarding cleaned log data to other topics of the message queue component or writing it to other indexes of the index storage component according to specified rules. Tests have shown that the LogRiver component is an order of magnitude faster than the data cleaning and storage speed based on open source Spark Streaming, with the same resource consumption.

[0060] Furthermore, the index storage Beaver step includes: using the Beaver storage engine to perform comprehensive data processing, which is the core technology of the log big data intelligent analysis platform. Based on the years of experience accumulated by the Lucene open source community, the log big data intelligent analysis platform uses the Beaver index storage component in C++ language according to the characteristics of log data. Compared with mainstream open source solutions, it provides many functional and performance advantages in log analysis scenarios. See Table 2 for the comparison.

[0061] Table 2: Comparison of the advantages and disadvantages of the solutions

[0062] For a comparison of Beaver's read and write performance with mainstream open source solutions, see Figure 4 , According to a comprehensive comparison, under the same storage and query conditions, Beaver saves more than half of the hardware costs compared to mainstream open source solutions.

[0063] Furthermore, the search statistics SPLServer step includes: using the SPL (Search Processing Language) syntax uniquely designed by the search statistics module SPLServer log big data intelligent analysis platform, and undertaking the syntax parsing and task scheduling of SPL. Among them, SPL syntax instructions are divided into streaming instructions and centralized instructions. For the streaming instruction part, SPLServer can automatically distribute it to the Beaver storage component for distributed execution after parsing. Centralized instructions are executed locally on SPLServer.

[0064] SPLServer is responsible for scheduling two types of tasks. One is the query level. SPLServer will automatically perform time-based sharding on all SPL queries, and perform fair polling scheduling on the sharded subtasks to ensure that queries from different users can get the response they deserve. This avoids the phenomenon in mainstream open source solutions where a large-scale query hangs the entire cluster. In addition, SPLServer can automatically convert queries for specific scenarios (such as accurate statistics of massive grouped data) into Spark tasks according to user needs, and distribute them to Spark clusters for execution. This avoids the result errors and OutOfMemory failures in mainstream open source solutions for massive grouped statistics.

[0065] The other is the result processing level, where SPLServer is responsible for the execution plan scheduling of all query tasks that need to be executed offline in the background of the log big data intelligent analysis platform, including offline tasks, download tasks, scheduled tasks, alarm tasks, report tasks, etc.

[0066] SPLServer can also be expanded horizontally in clusters, where the task status and task result data will rely on the name service component Zookeeper and the shared storage component MongoDB GridFS.

[0067] Furthermore, the steps of displaying and interfacing ZZZWeb include: based on search statistics, all log big data intelligent analysis platform display layer functions, including but not limited to: search, statistics, visualization, dashboard, report, alarm, etc. are provided by ZZZWeb components. In addition, ZZZWeb components are also responsible for external API interfaces for various functional services such as permission configuration, resource management and data query.

[0068] The ZZZWeb component can also be expanded horizontally in clusters and forwarded through the outermost load balancing component Nginx.

[0069] Furthermore, the algorithm service Analyzer step includes: using KPIAnalyzer and LogAnalyzer components to perform AI operations on the intelligent operation and maintenance of indicator data. The components provide various AI algorithms customized and developed by the log big data intelligent analysis platform for the needs of intelligent operation and maintenance. The algorithm service component itself does not provide data processing and product function level encapsulation. The consumption processing and algorithm call of indicator data are completed by KPIMonitor. The consumption processing and algorithm call of log data will generate SparkStreaming tasks to be completed on the Spark component.

[0070] Furthermore, the steps of managing and maintaining the Manager include: completing the operation and maintenance of the system cluster itself, the deployment of each component, configuration changes, performance monitoring alarms, and version upgrades through independent Manager components; although the platform itself can be independently operated without the Manager component, for long-term operation and maintenance considerations, it is recommended that users use the Manager module to manage and maintain the log big data intelligent analysis platform cluster. The performance monitoring indicator data of each component is also stored in an independent InfluxDB component.

[0071] Technical principle Log big data intelligent analysis revolves around the Beaver log search engine platform, providing simple configuration, powerful and easy-to-use log collection, processing, analysis and management functions to help users conduct real-time monitoring of online business, locate causes of business anomalies, conduct statistical analysis of business log data, and conduct security and compliance audits.

[0072] Technical principles and features of the main modules.

[0073] 1. Architecture diagram: For the product architecture diagram of the log big data intelligent analysis platform, see Figure 5 .

[0074] 2. Data flow diagram: The log big data intelligent analysis platform adopts a distributed service architecture, with different modules responsible for different stages of data processing. Figure 6 The straight line from left to right in the center of the figure is the main process of reading and writing log data in the log big data intelligent analysis platform.

[0075] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any ordinary technician in the industry can smoothly implement the present invention as shown in the drawings and described above. However, any equivalent changes, modifications and evolutions made by technicians familiar with the profession without departing from the scope of the technical solution of the present invention using the technical content disclosed above are all equivalent embodiments of the present invention. At the same time, any equivalent changes, modifications and evolutions made to the above embodiments based on the essential technology of the present invention are still within the protection scope of the technical solution of the present invention.

Claims

1. Log big data intelligent analysis platform, characterized by: The following steps are involved: S1: The client sends data to the collector module for data collection Agent; S2: After the collected data is accessed, it will first enter the message queue Kafka component for temporary storage; S3: ETL cleaning and routing LogRiver for temporary data; S4: Store Beaver for the temporary data after ETF cleaning and routing; S5: Search statistics SPLServer for syntax parsing and task scheduling of search processing language; S6: Based on search statistics, the intelligent analysis platform for all log big data presents layer functions; S7: Perform AI operations on indicator data and log data; S8: Perform operation and maintenance on the system cluster itself.

2. The log big data intelligent analysis platform according to claim 1 is characterized in that: The log big data analysis system includes: collector module, data cleaning component LogRiver, backup component Archiver, intelligent operation and maintenance component KPIMonitor, data factory component DataFlow, LogRiver component, Beaver index storage component, SPLServer search statistics module, name service component Zookeeper, shared storage component MongoDB GridFS, ZZZWeb component, KPIAnalyzer algorithm service component, LogAnalyzer algorithm service component, Spark component, Manager component and InfluxDB component.

3. The log big data intelligent analysis platform according to claim 1 is characterized in that: The data collection step Agent includes: sending data from the client to the collector module, mainly through the exclusive Agent provided by the log big data intelligent analysis platform, and can also use the standard syslog protocol (RFC5424), HTTP (S) and protobuf methods.

4. The log big data intelligent analysis platform according to claim 1 is characterized in that: The message queue step Kafka includes: after the collected data is connected, it enters the message queue component for temporary storage. Some open source / commercial log analysis products do not introduce message queue components to prevent the subsequent indexing process from reaching a bottleneck when the data volume suddenly increases due to business reasons, causing the entire performance to fluctuate and even data loss.

5. The log big data intelligent analysis platform according to claim 1 is characterized in that: The data cleaning and routing steps LogRiver includes: ETL cleaning of data, which runs downstream of the message queue component and can perform stateless horizontal cluster expansion. LogRiver focuses on streaming single-line log parsing.

6. The log big data intelligent analysis platform according to claim 1 is characterized in that: The index storage step Beaver includes: using the Beaver storage engine to comprehensively process data, which is the core technology of the log big data intelligent analysis platform.

7. The log big data intelligent analysis platform according to claim 1 is characterized in that: The search statistics step SPLServer includes: using the SPL (Search Processing Language) grammar uniquely designed by the search statistics module SPLServer log big data intelligent analysis platform, and undertaking the grammar parsing and task scheduling of SPL.

8. The log big data intelligent analysis platform according to claim 2 is characterized in that: The functions of the presentation and interface ZZZWeb component include: based on search statistics, all log big data intelligent analysis platform presentation layer functions, including search, statistics, visualization, dashboard, report, alarm, etc.

9. The log big data intelligent analysis platform according to claim 1, characterized in that: The Analyzer algorithm service component includes: using KPIAnalyzer and LogAnalyzer components to perform AI operations on intelligent operation and maintenance of indicator data.

10. The log big data intelligent analysis platform according to claim 1, characterized in that: The management and maintenance Manager component includes: independently completing the operation and maintenance of the system cluster itself, the deployment of each component, configuration changes, performance monitoring alarms, and version upgrades.

Images