A big data processing system for massive log analysis and early warning

Abstract

The present invention relates to the technical field of big data processing, and in particular to a big data processing system for massive log analysis and early warning, comprising a log information collection module, a transmission module, a preprocessing module, a log information analysis module, an early warning module, a management module, an early warning information generation module, a GSM module, a client and a push module, wherein the transmission module is connected to the log information collection module, the preprocessing module is connected to the transmission module, the log information analysis module is connected to the preprocessing module, the early warning modules are all connected to the log information analysis module and the management module, and the early warning information generation module is connected to the early warning module, thereby solving the technical problem that in the actual use of the log analysis and early warning system in the prior art, due to the wide sources and various formats of log data, the log data contains a large amount of redundant, erroneous and incomplete data, resulting in more complicated data processing and reduced accuracy of analysis results.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing, and in particular to a big data processing system for massive log analysis and early warning. Background Art

[0002] With the rapid development of technologies such as smart grid, Internet of Things, and cloud computing, the power industry has generated a large amount of log data in daily operations and monitoring. These data contain important information such as power grid operation, equipment status, user behavior, and potential safety risks. They are of great significance to the stable operation and intelligent management of the power system. In the information construction of the power industry, log data has become an important basis for power grid operation monitoring, fault warning, safety auditing, and business optimization.

[0003] However, in actual use, the existing log analysis and early warning systems have a wide range of log data sources and various formats. Log data contains a large amount of redundant, erroneous and incomplete data, which makes data processing more complicated and reduces the accuracy of analysis results. Summary of the invention

[0004] The purpose of the present invention is to provide a big data processing system for massive log analysis and early warning, aiming to solve the technical problems in the actual use of the log analysis and early warning system in the prior art, that is, the log data comes from a wide range of sources and has various formats, and the log data contains a large amount of redundant, erroneous and incomplete data, which makes data processing more complicated and the accuracy of the analysis results decreased.

[0005] To achieve the above-mentioned purpose, the present invention adopts a big data processing system for massive log analysis and early warning, including a log information collection module, a transmission module, a preprocessing module, a log information analysis module, an early warning module, a management module, an early warning information generation module, a GSM module, a client and a push module, the transmission module is connected to the log information collection module, the preprocessing module is connected to the transmission module, the log information analysis module is connected to the preprocessing module, the early warning module is connected to the log information analysis module and the management module, the early warning information generation module is connected to the early warning module, the GSM module is connected to the early warning information generation module, and the client and the push module are both connected to the GSM module;

[0006] The management module is used to construct an early warning model in the early warning module;

[0007] The log information collection module is used to collect log information;

[0008] The transmission module is used to efficiently transmit the data collected by the log information collection module to the preprocessing module;

[0009] The preprocessing module is used to preprocess the received log data to eliminate redundancy, errors and incomplete information in the data and convert it into a unified structured format;

[0010] The log information analysis module is used to perform in-depth analysis and mining on the pre-processed log data;

[0011] The early warning module triggers an early warning based on the constructed early warning model combined with the data obtained by the log information analysis module;

[0012] The warning information generation module is used to convert the warning signal generated by the warning module into specific warning information;

[0013] The GSM module is used to transmit the warning information generated by the warning information generation module to the client and the push module, and the push module sends the warning information to relevant managers or users using text messages, emails or other communication methods.

[0014] Wherein, the preprocessing module includes a data cleaning unit, a format conversion unit, a data deduplication unit and a data compression unit, the data cleaning unit is connected to the transmission module, the data deduplication unit is connected to the format conversion unit, and the data compression unit is connected to the data deduplication unit;

[0015] The data cleaning unit uses regular expressions to match and remove irrelevant information in the log data, and processes missing values ​​by filling default values, deleting records, or estimating based on other fields, and uses statistical methods (such as mean, standard deviation, box plot, etc.) to detect abnormal values ​​and correct or delete them;

[0016] The format conversion unit parses the log data into structured data using text parsing technology (such as regular expressions, word segmenters, etc.) according to the format definition of the log data, and converts the log data stored in JSON or XML format into structured data, and then performs standardization processing according to a unified format;

[0017] The data deduplication unit uses a hash algorithm or similarity calculation to remove duplicate log records;

[0018] The data compression unit is used to compress the data and then transmit it to the log information analysis module.

[0019] The log information collection module includes a log information collection unit and a log information storage unit, and the log information collection unit is connected to the transmission module and the log information storage unit.

[0020] Wherein, the log information collection unit includes an automatic acquisition subunit, an authorization subunit and a scanning and recording subunit, the automatic acquisition subunit and the scanning subunit are both connected to the log information storage unit, and the authorization subunit is connected to the automatic acquisition subunit;

[0021] The authorization subunit is used for staff authorization, the automatic acquisition subunit automatically acquires log information from the device after receiving the authorization instruction, and the scanning and inputting subunit is used for scanning and inputting log information on paper documents.

[0022] The big data processing system for massive log analysis and early warning further comprises a database and an automatic modeling module, and the automatic modeling module is connected to the database and the early warning module.

[0023] Among them, the big data processing system for massive log analysis and early warning also includes a collection module, and the collection module is connected to the database.

[0024] Among them, the big data processing system for massive log analysis and early warning also includes an enhancement module, and the enhancement module is connected to the scanning and input subunit.

[0025] The invention discloses a big data processing system for massive log analysis and early warning. First, the management module constructs an early warning model in the early warning module; then the log information collection module collects log information; the transmission module efficiently transmits the data collected by the log information collection module to the preprocessing module; the preprocessing module performs preprocessing operations on the received log data to eliminate redundant, erroneous and incomplete information in the data and converts it into a unified structured format; then the log information analysis module is used to perform in-depth analysis and mining on the preprocessed log data; then the early warning module triggers an early warning based on the constructed early warning model combined with the data obtained by the log information analysis module; and the early warning information generation module is used to convert the early warning signal generated by the early warning module into specific early warning information; finally, the GSM module is used to transmit the early warning information generated by the early warning information generation module to the client and the push module, and the push module sends the early warning information to relevant managers or users by SMS, email or other communication methods, thereby solving the technical problem that the log analysis and early warning system in the prior art is more complicated in actual use due to the wide sources and various formats of log data, and the log data contains a large amount of redundant, erroneous and incomplete data, resulting in more complicated data processing and reduced accuracy of analysis results.

[0026] The present invention realizes the rapid collection and efficient transmission of massive log data through the collaborative work of the log information collection module and the transmission module. The transmission module adopts efficient data transmission protocols and technologies (such as TCP / IP, HTTP, etc.), as well as possible compression and encryption technologies, to ensure the rapid and stable transmission of log data. The preprocessing module performs preprocessing operations such as cleaning, deduplication, and format conversion on the received log data, effectively eliminating redundancy, errors, and incomplete information in the data, reducing the complexity of data processing, and thus significantly improving data processing efficiency.

[0027] At the same time, association rule mining, cluster analysis and other technologies are used to extract valuable information and features from log data. Classification, regression, anomaly detection and other machine learning algorithms are used to perform pattern recognition and prediction on log data. The algorithm uses decision trees, random forests, support vector machines, neural networks, etc. to conduct in-depth analysis and mining of pre-processed log data. The log information analysis module can extract key information and features from log data and provide accurate data support for the early warning module. The early warning module is based on the constructed early warning model and the results of the log information analysis module to determine in real time whether the system status meets the early warning conditions, thereby improving the accuracy and reliability of the analysis results;

[0028] Moreover, the GSM module and the push module enable rapid transmission and reception of warning information. The client uses a graphical user interface (GUI) or a command line interface (CLI) and other technologies to display warning information to managers and users in an intuitive manner, making it convenient for managers and users to view warning information, understand system status, and take corresponding countermeasures. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0030] Figure 1 It is a principle block diagram of the first embodiment of the present invention.

[0031] Figure 2 It is a principle block diagram of the second embodiment of the present invention.

[0032] Figure 3 It is a principle block diagram of the third embodiment of the present invention.

[0033] 101-log information collection module, 102-transmission module, 103-preprocessing module, 104-log information analysis module, 105-warning module, 106-management module, 107-warning information generation module, 108-GSM module, 109-client, 110-push module, 111-data cleaning unit, 112-format conversion unit, 113-data deduplication unit, 114-data compression unit, 115-log information collection unit, 116-log information storage unit, 117-automatic acquisition subunit, 118-authorization subunit, 119-scanning and entry subunit, 201-database, 202-automatic modeling module, 203-acquisition module, 204-enhancement module, 301-log archiving module, 302-query module, 303-performance monitoring module. DETAILED DESCRIPTION

[0034] Embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, but should not be construed as limiting the present invention.

[0035] The first embodiment of the present application is:

[0036] See also Figure 1 , Figure 1 It is a principle block diagram of the first embodiment of the present invention.

[0037] The present invention provides a big data processing system for massive log analysis and early warning, comprising a log information collection module 101, a transmission module 102, a preprocessing module 103, a log information analysis module 104, an early warning module 105, a management module 106, an early warning information generation module 107, a GSM module 108, a client 109 and a push module 110, wherein the preprocessing module 103 comprises a data cleaning unit 111, a format conversion unit 112, a data deduplication unit 113 and a data compression unit 114, the log information collection module 101 comprises a log information collection unit 115 and a log information storage unit 116, the log information collection unit 115 comprises an automatic acquisition subunit 117, an authorization subunit 118 and a scanning and entry subunit 119, the above-mentioned scheme solves the technical problem that in the actual use of the log analysis and early warning system in the prior art, due to the wide sources and various formats of log data, the log data contains a large amount of redundant, erroneous and incomplete data, resulting in more complicated data processing and reduced accuracy of analysis results.

[0038] For this specific implementation, the management module 106 is used to construct an early warning model in the early warning module 105;

[0039] The log information collection module 101 is used to collect log information;

[0040] The transmission module 102 is used to efficiently transmit the data collected by the log information collection module 101 to the pre-processing module 103;

[0041] The preprocessing module 103 is used to preprocess the received log data to eliminate redundancy, errors and incomplete information in the data and convert it into a unified structured format;

[0042] The log information analysis module 104 is used to perform in-depth analysis and mining on the pre-processed log data;

[0043] The warning module 105 triggers a warning based on the constructed warning model combined with the data obtained by the log information analysis module 104;

[0044] The warning information generation module 107 is used to convert the warning signal generated by the warning module 105 into specific warning information;

[0045] The GSM module 108 is used to transmit the warning information generated by the warning information generation module 107 to the client 109 and the push module 110. The push module 110 sends the warning information to relevant managers or users via SMS, email or other communication methods.

[0046] Among them, the transmission module 102 is connected to the log information collection module 101, the preprocessing module 103 is connected to the transmission module 102, the log information analysis module 104 is connected to the preprocessing module 103, the early warning module 105 is connected to the log information analysis module 104 and the management module 106, the early warning information generation module 107 is connected to the early warning module 105, the GSM module 108 is connected to the early warning information generation module 107, the client 109 and the push module 110 are connected to the GSM module 108, firstly, the early warning model is constructed in the early warning module 105 through the management module 106; then the log information collection module 101 collects the log information; the transmission module 102 efficiently transmits the data collected by the log information collection module 101 to the preprocessing module 103; the preprocessing module 103 performs preprocessing operations on the received log data to eliminate redundancy, errors and incomplete information, and convert it into a unified structured format; then the log information analysis module 104 is used to conduct in-depth analysis and mining of the pre-processed log data; then the early warning module 105 triggers an early warning based on the constructed early warning model combined with the data obtained by the log information analysis module 104; and the early warning information generation module 107 is used to convert the early warning signal generated by the early warning module 105 into specific early warning information; finally, the GSM module 108 is used to transmit the early warning information generated by the early warning information generation module 107 to the client 109 and the push module 110, and the push module 110 sends the early warning information to relevant managers or users using SMS, email or other communication methods, thereby solving the technical problem that in the actual use of the log analysis and early warning system in the prior art, due to the wide range of log data sources and various formats, the log data contains a large amount of redundant, erroneous and incomplete data, resulting in more complicated data processing and reduced accuracy of the analysis results.

[0047] The present invention realizes the rapid collection and efficient transmission of massive log data through the collaborative work of the log information collection module 101 and the transmission module 102. The transmission module 102 adopts efficient data transmission protocols and technologies (such as TCP / IP, HTTP, etc.), as well as possible compression and encryption technologies to ensure the rapid and stable transmission of log data. The preprocessing module 103 performs preprocessing operations such as cleaning, deduplication, and format conversion on the received log data, effectively eliminating redundancy, errors, and incomplete information in the data, reducing the complexity of data processing, and thus significantly improving data processing efficiency.

[0048] At the same time, association rule mining, cluster analysis and other technologies are used to extract valuable information and features from log data. Classification, regression, anomaly detection and other machine learning algorithms are used to perform pattern recognition and prediction on log data. The algorithm uses decision trees, random forests, support vector machines, neural networks, etc. to conduct in-depth analysis and mining of pre-processed log data. The log information analysis module 104 can extract key information and features from the log data and provide accurate data support for the early warning module 105. The early warning module 105 is based on the constructed early warning model and the results of the log information analysis module 104 to determine in real time whether the system status meets the early warning conditions, thereby improving the accuracy and reliability of the analysis results;

[0049] Furthermore, the rapid transmission and reception of warning information is achieved through the GSM module 108 and the push module 110. The client 109 uses a graphical user interface (GUI) or a command line interface (CLI) and other technologies to display the warning information to managers and users in an intuitive manner, so that managers and users can view the warning information, understand the system status, and take corresponding countermeasures.

[0050] Secondly, the data cleaning unit 111 is connected to the transmission module 102, the data deduplication unit 113 is connected to the format conversion unit 112, and the data compression unit 114 is connected to the data deduplication unit 113;

[0051] The data cleaning unit 111 uses regular expressions to match and remove irrelevant information in the log data, and processes missing values ​​by filling default values, deleting records, or estimating based on other fields, and uses statistical methods (such as mean, standard deviation, box plot, etc.) to detect abnormal values ​​and correct or delete them;

[0052] The format conversion unit 112 parses the log data into structured data using text parsing techniques (such as regular expressions, word segmenters, etc.) according to the format definition of the log data, and converts the log data stored in JSON or XML format into structured data, and then performs standardization processing according to a unified format;

[0053] The data deduplication unit 113 uses a hash algorithm or similarity calculation to remove duplicate log records;

[0054] The data compression unit 114 is used to compress the data and transmit it to the log information analysis module 104.

[0055] At the same time, the log information collection unit 115 is connected to the transmission module 102 and the log information storage unit 116;

[0056] The automatic acquisition subunit 117 and the scanning subunit are both connected to the log information storage unit 116, and the authorization subunit 118 is connected to the automatic acquisition subunit 117;

[0057] The authorization subunit 118 is used for staff authorization. After receiving the authorization instruction, the automatic acquisition subunit 117 automatically obtains log information from the device. The scanning and inputting subunit 119 is used to scan and input log information from paper documents.

[0058] For device logs, use log collection tools such as Flume to configure Agent and Collector to achieve real-time collection and transmission of logs. Flume supports multiple data sources and can efficiently collect and transmit log data to a specified storage unit.

[0059] For paper logs, OCR (Optical Character Recognition) technology is used in combination with scanning equipment to convert paper files into electronic log information. The scanning and input subunit 119 can integrate an OCR engine to achieve automatic recognition and input of log information.

[0060] A big data processing system for massive log analysis and early warning using the present embodiment first constructs an early warning model in the early warning module 105 through the management module 106; then the log information collection module 101 collects log information; the transmission module 102 efficiently transmits the data collected by the log information collection module 101 to the preprocessing module 103; the preprocessing module 103 performs preprocessing operations on the received log data to eliminate redundancy, errors and incomplete information in the data, and converts it into a unified structured format; then the log information analysis module 104 is used to perform in-depth analysis and mining on the preprocessed log data; then the early warning module 105 combines the log information with the early warning model constructed to generate an early warning model. The data obtained by the information analysis module 104 is used to trigger the early warning; and the early warning information generation module 107 is used to convert the early warning signal generated by the early warning module 105 into specific early warning information; finally, the GSM module 108 is used to transmit the early warning information generated by the early warning information generation module 107 to the client 109 and the push module 110, and the push module 110 sends the early warning information to relevant managers or users by SMS, email or other communication methods. In this way, the log analysis and early warning system in the prior art is solved in actual use. Due to the wide range of log data sources and various formats, the log data contains a large amount of redundant, erroneous and incomplete data, which makes data processing more complicated and the accuracy of the analysis results reduced. Technical problems.

[0061] The present invention realizes the rapid collection and efficient transmission of massive log data through the collaborative work of the log information collection module 101 and the transmission module 102. The transmission module 102 adopts efficient data transmission protocols and technologies (such as TCP / IP, HTTP, etc.), as well as possible compression and encryption technologies to ensure the rapid and stable transmission of log data. The preprocessing module 103 performs preprocessing operations such as cleaning, deduplication, and format conversion on the received log data, effectively eliminating redundancy, errors, and incomplete information in the data, reducing the complexity of data processing, and thus significantly improving data processing efficiency.

[0062] At the same time, association rule mining, cluster analysis and other technologies are used to extract valuable information and features from log data. Classification, regression, anomaly detection and other machine learning algorithms are used to perform pattern recognition and prediction on log data. The algorithm uses decision trees, random forests, support vector machines, neural networks, etc. to conduct in-depth analysis and mining of pre-processed log data. The log information analysis module 104 can extract key information and features from the log data and provide accurate data support for the early warning module 105. The early warning module 105 is based on the constructed early warning model and the results of the log information analysis module 104 to determine in real time whether the system status meets the early warning conditions, thereby improving the accuracy and reliability of the analysis results;

[0063] Furthermore, the rapid transmission and reception of warning information is achieved through the GSM module 108 and the push module 110. The client 109 uses a graphical user interface (GUI) or a command line interface (CLI) and other technologies to display the warning information to managers and users in an intuitive manner, so that managers and users can view the warning information, understand the system status, and take corresponding countermeasures.

[0064] The second embodiment of the present application is:

[0065] Based on the first embodiment, please refer to Figure 2 , Figure 2 It is a principle block diagram of the second embodiment of the present invention.

[0066] The present invention provides a big data processing system for massive log analysis and early warning, which also includes a database 201, an automatic modeling module 202, a collection module 203 and an enhancement module 204.

[0067] According to this specific implementation, the automatic modeling module 202 is connected to the database 201 and the early warning module 105 , and the automatic modeling module 202 can automatically build an early warning model according to the characteristics of the historical log data in the database 201 and business requirements.

[0068] The acquisition module 203 is connected to the database 201 , and the acquisition module 203 is used to acquire historical log data.

[0069] Secondly, the enhancement module 204 is connected to the scanning and recording subunit 119, and the enhancement module 204 can work together with other modules such as the scanning and recording subunit 119 and the database 201 to form a complete data processing and analysis process, which helps to achieve seamless data flow and efficient processing.

[0070] Using a big data processing system for massive log analysis and early warning of this embodiment, the acquisition module 203 is used to collect historical log data and store it in the database 201, the automatic modeling module 202 can automatically build an early warning model according to the characteristics and business requirements of the historical log data in the database 201, and the enhancement module 204 can work in coordination with other modules such as the scanning and input subunit 119 and the database 201 to form a complete data processing and analysis process. This helps to achieve seamless flow and efficient processing of data.

[0071] The third embodiment of the present application is:

[0072] Based on the second embodiment, please refer to Figure 3 , Figure 3 It is a principle block diagram of the third embodiment of the present invention.

[0073] The present invention provides a big data processing system for massive log analysis and early warning, which also includes a log archiving module 301, a query module 302 and a performance monitoring module 303.

[0074] For this specific implementation, the log archiving module 301 is connected to the log information analysis module 104, the query module 302 is connected to the log information analysis module 104, the log archiving module 301 is used to archive and store the processed log data, and the query module 302 is used to query the data in the log archiving module 301.

[0075] The performance monitoring module 303 is connected to the early warning module 105 , and the performance monitoring module 303 is used to monitor the performance of the early warning module 105 .

[0076] A big data processing system for massive log analysis and early warning is used in this embodiment. The log archiving module 301 is used to archive and store processed log data, the query module 302 is used to query the data in the log archiving module 301, and the performance monitoring module 303 is used to monitor the performance of the early warning module 105.

[0077] What is disclosed above is only a preferred embodiment of the present invention, and it certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A big data processing system for massive log analysis and early warning, characterized in that: It includes a log information collection module, a transmission module, a preprocessing module, a log information analysis module, an early warning module, a management module, an early warning information generation module, a GSM module, a client and a push module, wherein the transmission module is connected to the log information collection module, the preprocessing module is connected to the transmission module, the log information analysis module is connected to the preprocessing module, the early warning module is connected to the log information analysis module and the management module, the early warning information generation module is connected to the early warning module, the GSM module is connected to the early warning information generation module, and the client and the push module are both connected to the GSM module; The management module is used to construct an early warning model in the early warning module; The log information collection module is used to collect log information; The transmission module is used to efficiently transmit the data collected by the log information collection module to the preprocessing module; The preprocessing module is used to preprocess the received log data to eliminate redundancy, errors and incomplete information in the data and convert it into a unified structured format; The log information analysis module is used to perform in-depth analysis and mining on the pre-processed log data; The early warning module triggers an early warning based on the constructed early warning model combined with the data obtained by the log information analysis module; The warning information generation module is used to convert the warning signal generated by the warning module into specific warning information; The GSM module is used to transmit the warning information generated by the warning information generation module to the client and the push module, and the push module sends the warning information to relevant managers or users using text messages, emails or other communication methods.

2. The big data processing system for massive log analysis and early warning according to claim 1, characterized in that: The preprocessing module includes a data cleaning unit, a format conversion unit, a data deduplication unit and a data compression unit, the data cleaning unit is connected to the transmission module, the data deduplication unit is connected to the format conversion unit, and the data compression unit is connected to the data deduplication unit; The data cleaning unit uses regular expressions to match and remove irrelevant information in the log data, and processes missing values ​​by filling default values, deleting records, or estimating based on other fields, and uses statistical methods to detect abnormal values ​​and correct or delete them; The format conversion unit parses the log data into structured data using text parsing technology according to the format definition of the log data, and converts the log data stored in JSON or XML format into structured data, and then performs standardization processing according to a unified format; The data deduplication unit uses a hash algorithm or similarity calculation to remove duplicate log records; The data compression unit is used to compress the data and then transmit it to the log information analysis module.

3. The big data processing system for massive log analysis and early warning as claimed in claim 2, characterized in that: The log information collection module includes a log information collection unit and a log information storage unit, and the log information collection unit is connected to the transmission module and the log information storage unit.

4. The big data processing system for massive log analysis and early warning as claimed in claim 3, characterized in that: The log information collection unit includes an automatic acquisition subunit, an authorization subunit and a scanning and recording subunit, the automatic acquisition subunit and the scanning subunit are both connected to the log information storage unit, and the authorization subunit is connected to the automatic acquisition subunit; The authorization subunit is used for staff authorization, the automatic acquisition subunit automatically acquires log information from the device after receiving the authorization instruction, and the scanning and inputting subunit is used for scanning and inputting log information on paper documents.

5. The big data processing system for massive log analysis and early warning as claimed in claim 4, characterized in that: The big data processing system for massive log analysis and early warning also includes a database and an automatic modeling module, and the automatic modeling module is connected to the database and the early warning module.

6. The big data processing system for massive log analysis and early warning as claimed in claim 5, characterized in that: The big data processing system for massive log analysis and early warning also includes a collection module, and the collection module is connected to the database.

7. The big data processing system for massive log analysis and early warning according to claim 6, characterized in that: The big data processing system for massive log analysis and early warning also includes an enhancement module, and the enhancement module is connected to the scanning and input subunit.

Images