Log monitoring method based on big data platform

Abstract

The invention provides a log monitoring method based on a big data platform, and belongs to the field of big data, the log monitoring method collects data of each component through a log collection tool and transmits the data to a server, the server processes the data and stores the data, and a visual management platform calling interface can be designed for logs to realize various functions. And new components can be added on the management platform by modifying configuration. Component logs can be captured and analyzed in real time, the system state is fed back in real time, and operation and maintenance personnel are helped to rapidly position and solve problems. And meanwhile, retrieval of historical data for many years and various retrieval conditions are supported, so that troubleshooting is more efficient.

Description

Technical Field

[0001] The present invention relates to the field of big data, and in particular to a log monitoring method based on a big data platform. Background Art

[0002] With the rapid advancement of big data technology, many companies have introduced big data platforms to integrate and efficiently utilize various big data components. However, the traditional log collection and monitoring process has obvious deficiencies in the centralized management of logs, processing efficiency, and retrieval convenience. These deficiencies make it difficult for operation and maintenance personnel to quickly locate and solve problems when they occur, which in turn affects the stability and efficiency of the system. In the existing log monitoring system, different big data components operate independently and use independent log collection methods to generate a large number of scattered log files. These log files are not only difficult to manage in a unified manner, but also when problems occur, operation and maintenance personnel need to manually retrieve and analyze a large amount of log data, which is extremely inefficient. Summary of the invention

[0003] In order to solve the above technical problems, the present invention provides a log monitoring method based on a big data platform.

[0004] The technical solution of the present invention is:

[0005] A log monitoring method based on a big data platform. The log collection tool collects data from each component and transmits it to the server. The server processes and stores the data, and can design a visual management platform to call the interface to implement various functions. The configuration can be modified and new components can be added on the management platform.

[0006] Further, it includes

[0007] (1) Real-time log data collection: Through a dedicated real-time log data collection tool, the log data of various components in the big data platform can be collected in real time, and the collected log data can be transmitted to the server for storage and subsequent processing.

[0008] (2) Real-time log monitoring and retrieval: Build a log management platform that can monitor stored log data in real time and provide retrieval functions. Users can view log data from the past five years through the platform, and it also supports accurate retrieval based on time, component type, importance, and other conditions.

[0009] (3) New component log monitoring: For newly installed components in the big data platform, configuration files are added to specify the installation location and log location of the new components, so that the new component logs can be automatically captured and analyzed, ensuring that the new component logs can also be monitored and retrieved in real time.

[0010] (4) Intelligent log processing: Integrate multiple log files into text files and perform preprocessing such as word segmentation, stop word removal, and part-of-speech tagging. Then, use NLP technology to perform text mining and semantic analysis to extract key information and contextual relationships. By removing unimportant features and continuously optimizing the model, the efficiency of subsequent processing is improved. In this way, log data can be understood and used more effectively.

[0011] Going further,

[0012] Real-time monitoring also includes log analysis tools and technologies, which can parse, aggregate and visualize log data in real time, provide users with an intuitive and clear display of log information, and help users quickly locate the source of the problem.

[0013] Going further,

[0014] In the log data collection step, log collectors are deployed on the data source to achieve real-time collection of various log data. These log collectors can collect data in real time and send it to the server to ensure the timeliness and integrity of the data.

[0015] Going further,

[0016] Intelligent log processing can integrate log files and centrally process and analyze them, while optimizing the model based on the analysis results.

[0017] Going further,

[0018] Log search provides a variety of search conditions, including but not limited to time range, component type, importance, log level, etc. Users can combine and filter according to actual needs to quickly locate the required log information.

[0019] The present invention has good scalability, and users can implement the expansion of new components by simply modifying configuration files without large-scale modification or reconstruction of the platform itself, thus reducing maintenance costs and expansion difficulty.

[0020] The beneficial effects of the present invention are

[0021] This log management platform significantly improves the efficiency and accuracy of operation and maintenance work in a big data environment through functions such as real-time monitoring, efficient retrieval, and flexible expansion. It can capture and analyze component logs in real time, provide instant feedback on system status, and help operation and maintenance personnel quickly locate and solve problems. At the same time, it supports the retrieval of many years of historical data and multiple retrieval conditions, making troubleshooting more efficient. In addition, the platform also has flexible scalability and can adapt to changing business needs, providing strong support for the stable operation and troubleshooting of enterprises. By deeply mining log data, the platform can also provide enterprises with more valuable data insights and promote the mining and application of data value. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a schematic diagram of the overall technical architecture of the present invention;

[0023] Figure 2 It is a schematic diagram of the working function flow of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0025] The present invention provides a log monitoring method based on a big data platform, which collects data of each component through a log collection tool and transmits it to a server, which processes and stores the data, and can design a visual management platform to call an interface to implement various functions. New components can be added by modifying the configuration on the management platform.

[0026] 1. Log collection

[0027] The present invention designs an efficient log collection tool. Through a pre-written installation script, the user can simply install the tool on the server. We can modify the configuration file according to the actual situation to clarify the components to be collected and the log storage location. When started, it will automatically locate the storage location of the log data according to the configuration file. Once the target log is found, the tool will immediately start a dedicated collector, which can read the new content in the log in real time and use kafka to efficiently transmit the new data to the server. The whole process realizes the accurate capture and instant transmission of the log, greatly improving the efficiency and accuracy of log management.

[0028] 2. Real-time monitoring

[0029] The present invention designs a log management platform, one of the core functions of which is real-time monitoring, which ensures uninterrupted tracking of the logs of each component in a big data environment. The system can capture and analyze these logs in real time, and present the current status, operating conditions, and potential problems of the components to the user in an intuitive form. This not only helps users to understand the health status of the entire big data platform at any time, but also can promptly discover and solve potential problems to ensure the stable operation of the platform. The real-time monitoring function can parse, aggregate, and visualize log data in real time by integrating advanced log analysis tools and technologies, helping users to quickly locate the source of the problem. Whether it is a server anomaly, a data processing error, or a network delay, it can be quickly detected through the real-time monitoring function, providing users with timely feedback and solutions.

[0030] 3. Log retrieval

[0031] In order to meet the user's query needs for historical logs, the log management platform supports viewing log data for up to the past five years. This means that users can trace back log information from any time period in the past as needed to conduct a comprehensive audit and troubleshooting of the system. Log retrieval and visualization can be achieved through elasticsearch and kibana. In addition to the time range, the platform also provides a variety of search conditions, such as component type, importance, log level, etc., to help users locate the required logs more accurately. These search conditions can be combined and filtered according to the user's actual needs, greatly improving the flexibility and efficiency of retrieval. Through the log retrieval function, users can easily find log entries of interest and conduct in-depth analysis and mining. Whether it is performance optimization, security auditing or troubleshooting, it can be strongly supported by the log retrieval function.

[0032] 4. Intelligent log processing

[0033] Users can pre-set rules, including keywords, component types, operation types, etc. The platform will process the selected log files. Unify multiple log files into text files, perform pre-processing work such as word segmentation, stop word removal, and part-of-speech tagging on the text data in the logs, and then use NLP technology to perform text mining and semantic analysis to extract key information and contextual relationships in the logs. Remove unimportant features based on the analysis results, continuously optimize the model, and improve the efficiency of subsequent processing.

[0034] 5. Component extension

[0035] With the continuous development of big data technology and the continuous expansion of enterprise business, new big data components and tools continue to emerge. In order to meet users' needs for new component log monitoring, the log management platform supports component extension functions. Users can add newly installed components to the platform's monitoring list through simple configuration operations. This includes specifying the installation location of the component, the log path, and other relevant configuration information. Once the configuration is complete, the platform can automatically capture and analyze the log data of the new component to provide users with real-time monitoring and alarm services. The introduction of component extension functions makes the log management platform more flexible and scalable. Whether it is the upgrade of existing components or the introduction of new components, log monitoring and management can be achieved through simple configuration operations, significantly reducing the user's operation and maintenance costs.

[0036] The present invention integrates the capabilities of real-time log data collection, flexible storage and in-depth analysis. Through advanced technical architecture and algorithm optimization, the data processing efficiency is significantly improved, and the processing delay is greatly reduced, thereby providing enterprises with more timely and accurate log information. In terms of real-time monitoring, the platform has the function of real-time capture and analysis of log data, enabling enterprises to quickly discover potential risks and problems, and immediately take corresponding countermeasures, thereby effectively preventing or reducing losses. This real-time monitoring capability greatly improves the risk response efficiency and decision-making speed of enterprises. At the same time, a large number of log files can be intelligently analyzed and processed to improve operation and maintenance efficiency. In addition, the platform also provides intuitive visualization display tools, and users can quickly understand the log status of each component through intuitive methods such as charts and reports. This visual display form not only makes complex log data easier to understand and analyze, but also greatly improves the user experience. Through the above scheme, the log monitoring efficiency is greatly optimized, and the needs under various complex scenarios can be met.

[0037] The above description is only a preferred embodiment of the present invention, which is only used to illustrate the technical solution of the present invention, and is not used to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A log monitoring method based on a big data platform, characterized in that: The data of each component is collected through the log collection tool and transmitted to the server. The server processes and stores the data, and designs a visual management platform call interface for the log to implement various functions. The configuration can be modified and new components can be added on the management platform.

2. The method according to claim 1, characterized in that include: (1) Real-time log data collection: Log data of various components in the big data platform are collected in real time and transmitted to the server for storage and subsequent processing; (2) Real-time log monitoring and retrieval: Build a log management platform that monitors stored log data in real time and provides retrieval functions; (3) New component log monitoring: For newly installed components in the big data platform, add configuration files to specify the installation location and log location of the new components, automatically capture and analyze the new component logs, and ensure that the new component logs can also be monitored and retrieved in real time; (4) Intelligent log processing: Integrate several log files into text files and perform preprocessing. Then use NLP technology to perform text mining and semantic analysis to extract key information and contextual relationships.

3. The method according to claim 2, characterized in that Monitoring also includes log analysis tools and technologies that parse, aggregate and visualize log data in real time, providing users with an intuitive and clear display of log information to help users quickly locate the source of the problem.

4. The method according to claim 2, characterized in that: In step (1), log collectors are deployed on the data source to realize real-time collection of various log data. These log collectors collect data in real time and send it to the server to ensure the timeliness and integrity of the data.

5. The method according to claim 2, characterized in that: In step 2), you can view log data for the past five years through the platform, and support retrieval based on time, component type, and importance.

6. The method according to claim 2, characterized in that Integrate log files, process and analyze them centrally, and optimize the model based on the analysis results.

7. The method according to claim 2, characterized in that Log retrieval includes time range, component type, importance, and log level. Users can combine and filter according to actual needs.

8. The method according to claim 2, characterized in that: Users can pre-set rules, including keywords, component types, and operation types; process selected log files; unify multiple log files into text files, and perform pre-processing work on the text data in the logs, such as word segmentation, stop word removal, and part-of-speech tagging.

Images