Big data processing system for analysis and early warning of massive logs

By designing a big data processing system, including data acquisition, preprocessing, classification, packaging and in-depth analysis modules, the problems of slow speed and insufficient depth when processing massive data are solved by traditional log analysis methods, and efficient processing of massive log data is achieved and early warning notifications meet the needs of the company.

CN118916254AInactive Publication Date: 2024-11-08GANSU ELECTRIC POWER INFORMATION COMM
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411113817.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-14
Publication Date
2024-11-08
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Traditional log analysis methods have slow processing speed, insufficient analysis depth and lag in early warning when processing massive data, which is difficult to meet the growing demand of enterprises.

Method used

A big data processing system including data acquisition, preprocessing, classification, packaging, unpacking, analysis and early warning modules was designed. Through real-time data acquisition, preprocessing, classification, packaging and in-depth analysis, it can achieve efficient use of massive log data. Processing and early warning.

Benefits of technology

It realizes efficient processing and in-depth analysis of massive log data, which can meet the growing needs of enterprises, improves processing speed and analysis depth, and promptly triggers early warning notifications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916254A_ABST
    Figure CN118916254A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of log analysis and big data, in particular to a big data processing system for analysis and early warning of massive logs. Comprising a data acquisition module, a first data buffer module, a transmission module, a preprocessing module, a second data buffer module, a data classification module, a packing module, an unpacking module, an analysis module, a client, a display module and an early warning module, the pre-processing module pre-processes the collected original log data, the second data buffer module temporarily stores the pre-processed log data, the analysis module deeply analyzes the stored log data, and the early warning module triggers an alarm according to the analyzed log data when an early warning rule is reached and informs a manager. By means of the mode, efficient processing and deep analysis of mass log data are achieved, and the increasing requirements of enterprises can be met.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of log analysis and big data technology, and in particular to a big data processing system for massive log analysis and early warning. Background Art

[0002] With the rapid development of information technology, the amount of log data generated by daily business operations has exploded. These log data contain important information such as system operation status, user behavior, transaction records, etc., and are key resources for evaluating system health, discovering potential problems, and optimizing business processes.

[0003] However, traditional log analysis methods have the problems of slow processing speed, insufficient analysis depth and delayed warning when processing massive amounts of data, resulting in inefficient processing and difficulty in meeting the growing needs of enterprises. Summary of the invention

[0004] The purpose of the present invention is to provide a big data processing system for massive log analysis and early warning, aiming to solve the technical problems that traditional log analysis methods have slow processing speed, insufficient analysis depth and delayed early warning when processing massive data, resulting in inefficient processing and difficulty in meeting the growing needs of enterprises.

[0005] To achieve the above-mentioned purpose, the present invention adopts a big data processing system for massive log analysis and early warning, including a data acquisition module, a first data buffer module, a transmission module, a preprocessing module, a second data buffer module, a data classification module, a packaging module, an unpacking module, an analysis module, a client, a display module and an early warning module, wherein the early warning module and the analysis module are both connected to the client, the analysis module, the unpacking module, the packaging module, the data classification module, the second data buffer module, the transmission module, the first data buffer module and the data acquisition module are connected in sequence, and the preprocessing module is connected to the first data buffer module;

[0006] The client is used to configure, monitor and schedule log data;

[0007] The display module is used to display the processing process of log data, which is convenient for users to understand;

[0008] The data collection module is used to collect log data from the data source in real time;

[0009] The first data buffer module is used to receive original log data from a data source;

[0010] The preprocessing module is used to preprocess the collected raw log data;

[0011] The transmission module is used to transmit the data in the first data buffer module to the second data buffer module;

[0012] The second data buffer module is used to temporarily store the preprocessed log data;

[0013] The data classification module is used to classify the stored log data;

[0014] The packaging module is used to package the classified log data according to a specified format;

[0015] The unpacking module is used to unpack the packaged log data into a processable format and transmit it to the analysis module;

[0016] The analysis module is used to perform in-depth analysis on the stored log data to extract valuable information and patterns;

[0017] The warning module is used to trigger an alarm based on the analysis of log data and to notify management personnel when the warning rules are met.

[0018] Wherein, the preprocessing module includes a real-time unit, a sorting unit and a screening unit, and the real-time unit, the real-time unit and the screening unit are all connected to the first data buffer module;

[0019] The real-time unit is used to assign real-time time to the log data, and collect and process the log data in real time;

[0020] The sorting unit is used to sort the log data according to the time before and after.

[0021] The screening unit is used to filter and clean the log data to remove data that is irrelevant to the current analysis task and does not meet the requirements.

[0022] Wherein, the data classification module includes a plurality of data storage subunits and a plurality of processing subunits, the plurality of data storage subunits are connected to the second data buffer module, and each of the processing subunits is respectively connected to the packaging module and the corresponding data storage subunit;

[0023] The data storage subunit is used to receive the log data transmitted from the second data buffer module;

[0024] The processing subunit is used to further process and classify the log data obtained from the data storage subunit.

[0025] The analysis module includes a division unit, an identification unit, a plurality of storage subunits, a plurality of transmission subunits and a data analysis unit, the division unit is connected to the unpacking module, the identification unit is connected to the division unit, a plurality of storage subunits are connected to the division unit, the data analysis unit is connected to the client, and each transmission subunit is respectively connected to the data analysis unit and the corresponding storage subunit;

[0026] The division unit is used to classify the received log data according to the rules and divide it into multiple log data blocks;

[0027] The identification unit is used to identify each divided log data block separately;

[0028] The storage subunit is used to store the divided log data blocks in storage media respectively for subsequent analysis and query;

[0029] The transmission subunit is used to transmit the log data blocks in the storage subunit to the data analysis unit;

[0030] The data analysis unit is used to perform in-depth analysis and mining on the transmitted log data to extract valuable information.

[0031] Wherein, the early warning module includes an abnormality setting unit, an abnormality monitoring unit, an alarm unit and a trend prediction unit, and the abnormality setting unit, the abnormality monitoring unit, the alarm unit and the trend prediction unit are all connected to the client;

[0032] The abnormality setting unit is used to set warning rules and trigger thresholds;

[0033] The abnormality monitoring unit is used to perform abnormality monitoring on the processed log data;

[0034] The warning unit is used to send abnormal notifications to management personnel;

[0035] The trend prediction unit is used to predict the trend of the system operation status according to historical data and discover potential risks in advance.

[0036] The big data processing system for massive log analysis and early warning further includes a login module, a verification module and an administrator operation module, wherein the administrator operation module is connected to the client, the verification module is connected to the administrator operation module, and the login module is connected to the verification module;

[0037] The login module is used by administrators to enter the administrator operation module;

[0038] The verification module is used to perform login verification when the administrator enters the administrator operation module;

[0039] The administrator operation module is used for the administrator to configure, monitor and schedule the log data in the client.

[0040] Wherein, the verification module includes a face verification unit and an account verification unit, the face verification unit is connected to the login module and the administrator operation module respectively, and the account verification unit is connected to the login module and the administrator operation module respectively;

[0041] The face verification unit is used for the administrator to log in to the administrator operation module through face recognition verification;

[0042] The account verification unit is used for the administrator to log in to the administrator operation module through account identification verification.

[0043] The present invention discloses a big data processing system for massive log analysis and early warning. The client configures, monitors and schedules log data. The display module displays the processing process of log data for the convenience of users. The data acquisition module collects log data from a data source in real time, such as a server log, an application log, a network device log, etc. The first data buffer module receives the original log data from the data source. The preprocessing module preprocesses the collected original log data. The transmission module transmits the data in the first data buffer module to the second data buffer module. The second data buffer module temporarily stores the preprocessed log data. The data classification module classifies the stored log data. The packaging module packages the classified log data according to a specified format. The unpacking module unpacks the packaged log data into a processable format and transmits it to the analysis module. The analysis module performs in-depth analysis on the stored log data to extract valuable information and rules. The early warning module triggers an alarm according to the analysis of the log data to meet the early warning rules and notifies the management personnel. Through the above method, efficient processing and in-depth analysis of massive log data are achieved, which can meet the growing needs of enterprises. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1It is a schematic diagram of the principle of a big data processing system for massive log analysis and early warning of the present invention.

[0046] Figure 2 It is a schematic diagram of the principle of the data classification module of the present invention.

[0047] Figure 3 It is a schematic diagram of the principle of the analysis module of the present invention.

[0048] 101-data acquisition module, 102-first data buffer module, 103-transmission module, 104-second data buffer module, 105-packaging module, 106-unpacking module, 107-client, 108-display module, 109-login module, 110-administrator operation module, 111-real-time unit, 112-sorting unit, 113-screening unit, 114-data storage subunit, 115-processing subunit, 116-division unit, 117-identification unit, 118-storage subunit, 119-transmission subunit, 120-data analysis unit, 121-abnormal setting unit, 122-abnormal monitoring unit, 123-warning unit, 124-trend prediction unit, 125-face verification unit, 126-account verification unit, 127-cache module, 128-historical query module. DETAILED DESCRIPTION

[0049] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0050] See also Figure 1 to Figure 3 The present invention provides a big data processing system for massive log analysis and early warning, comprising a data acquisition module 101, a first data buffer module 102, a transmission module 103, a preprocessing module, a second data buffer module 104, a data classification module, a packaging module 105, an unpacking module 106, an analysis module, a client 107, a display module 108 and an early warning module, wherein the early warning module and the analysis module are both connected to the client 107, the analysis module, the unpacking module 106, the packaging module 105, the data classification module, the second data buffer module 104, the transmission module 103, the first data buffer module 102 and the data acquisition module 101 are connected in sequence, and the preprocessing module is connected to the first data buffer module 102;

[0051] The client 107 is used to configure, monitor and schedule log data;

[0052] The display module 108 is used to display the processing process of log data, which is convenient for users to understand;

[0053] The data collection module 101 is used to collect log data from the data source in real time;

[0054] The first data buffer module 102 is used to receive original log data from a data source;

[0055] The preprocessing module is used to preprocess the collected raw log data;

[0056] The transmission module 103 is used to transmit the data in the first data buffer module 102 to the second data buffer module 104;

[0057] The second data buffer module 104 is used to temporarily store the pre-processed log data;

[0058] The data classification module is used to classify the stored log data;

[0059] The packaging module 105 is used to package the classified log data according to a specified format;

[0060] The unpacking module 106 is used to unpack the packaged log data into a processable format and transmit it to the analysis module;

[0061] The analysis module is used to perform in-depth analysis on the stored log data to extract valuable information and patterns;

[0062] The warning module is used to trigger an alarm based on the analysis of log data and to notify management personnel when the warning rules are met.

[0063] In this embodiment, the client 107 configures, monitors and schedules the log data, the display module 108 displays the processing process of the log data, which is convenient for users to understand, the data acquisition module 101 collects log data from the data source in real time, such as server logs, application logs, network device logs, etc., receives the original log data from the data source through the first data buffer module 102, the preprocessing module preprocesses the collected original log data, the transmission module 103 transmits the data in the first data buffer module 102 to the second data buffer module 104, and the second data buffer module 10 4 temporarily stores the preprocessed log data, the data classification module classifies the stored log data, the packaging module 105 packages the classified log data according to a specified format, the unpacking module 106 unpacks the packaged log data into a processable format and transmits it to the analysis module, the analysis module performs in-depth analysis on the stored log data to extract valuable information and rules, the early warning module triggers an alarm according to the analysis of the log data, and notifies the management personnel. Through the above method, efficient processing and in-depth analysis of massive log data are achieved, which can meet the growing needs of enterprises.

[0064] Further, the preprocessing module includes a real-time unit 111, a sorting unit 112 and a screening unit 113, and the real-time unit 111, the real-time unit 111 and the screening unit 113 are all connected to the first data buffer module 102;

[0065] The real-time unit 111 is used to assign real-time time to the log data, and collect and process the log data in real time;

[0066] The sorting unit 112 is used to sort the log data according to the time before and after;

[0067] The screening unit 113 is used to filter and clean the log data to remove data that is irrelevant to the current analysis task and does not meet the requirements.

[0068] In this embodiment, the real-time unit 111 assigns real-time time to the log data, collects and processes the log data in real time, the sorting unit 112 sorts and processes the log data according to the time before and after the log data, and the screening unit 113 filters and cleans the log data to remove data that is irrelevant to the current analysis task and does not meet the requirements.

[0069] Further, the data classification module includes a plurality of data storage subunits 118114 and a plurality of processing subunits 115, the plurality of data storage subunits 118114 are connected to the second data buffer module 104, and each of the processing subunits 115 is respectively connected to the packaging module 105 and the corresponding data storage subunit 118114;

[0070] The data storage subunit 118114 is used to receive the log data transmitted from the second data buffer module 104;

[0071] The processing subunit 115 is used to further process and classify the log data obtained from the data storage subunits 118114.

[0072] In this embodiment, the data storage subunit 118114 receives the log data transmitted from the second data buffer module 104 , and the processing subunit 115 further processes and classifies the log data obtained from the data storage subunit 118114 .

[0073] Further, the analysis module includes a division unit 116, an identification unit 117, a plurality of storage subunits 118, a plurality of transmission subunits 119 and a data analysis unit 120, the division unit 116 is connected to the unpacking module 106, the identification unit 117 is connected to the division unit 116, a plurality of the storage subunits 118 are all connected to the division unit 116, the data analysis unit 120 is connected to the client 107, and each of the transmission subunits 119 is respectively connected to the data analysis unit 120 and the corresponding storage subunit 118;

[0074] The division unit 116 is used to classify the received log data according to the rules and divide it into multiple log data blocks;

[0075] The identification unit 117 is used to identify each divided log data block separately;

[0076] The storage subunit 118 is used to store the divided log data blocks in storage media for subsequent analysis and query;

[0077] The transmission subunit 119 is used to transmit the log data blocks in the storage subunit 118 to the data analysis unit 120;

[0078] The data analysis unit 120 is used to perform in-depth analysis and mining on the transmitted log data to extract valuable information.

[0079] In this embodiment, the division unit 116 classifies the received log data according to the rules and divides it into multiple log data blocks. The identification unit 117 separately identifies each divided log data block. The storage subunit 118 stores the divided log data blocks in storage media respectively for subsequent analysis and query. The transmission subunit 119 transmits the log data blocks in the storage subunit 118 to the data analysis unit 120. The data analysis unit 120 performs in-depth analysis and mining on the transmitted log data to extract valuable information.

[0080] Further, the early warning module includes an abnormality setting unit 121, an abnormality monitoring unit 122, an alarm unit 123 and a trend prediction unit 124, and the abnormality setting unit 121, the abnormality monitoring unit 122, the alarm unit 123 and the trend prediction unit 124 are all connected to the client 107;

[0081] The abnormality setting unit 121 is used to set warning rules and trigger thresholds;

[0082] The abnormality monitoring unit 122 is used to monitor the processed log data for abnormalities;

[0083] The warning unit 123 is used to send abnormal notifications to management personnel;

[0084] The trend prediction unit 124 is used to predict the trend of the system operation status according to historical data and discover potential risks in advance.

[0085] In this embodiment, the management personnel use the abnormal setting unit 121 to set warning rules and trigger thresholds, the abnormal monitoring unit 122 performs abnormal monitoring on the processed log data, the warning unit 123 sends abnormal notifications to the management personnel, and the trend prediction unit 124 is used to predict the trend of the system operation status based on historical data to discover potential risks in advance.

[0086] Further, the big data processing system for massive log analysis and early warning also includes a login module 109, a verification module and an administrator operation module 110, the administrator operation module 110 is connected to the client 107, the verification module is connected to the administrator operation module 110, and the login module 109 is connected to the verification module;

[0087] The login module 109 is used by the administrator to enter the administrator operation module 110;

[0088] The verification module is used to perform login verification when the administrator enters the administrator operation module 110;

[0089] The administrator operation module 110 is used for the administrator to configure, monitor and schedule log data in the client 107 .

[0090] In this embodiment, the administrator enters the administrator operation module 110 through the login module 109, and the verification module performs login verification on the administrator when entering the administrator operation module 110. The administrator uses the administrator operation module 110 to configure, monitor and schedule log data in the client 107.

[0091] Further, the verification module includes a face verification unit 125 and an account verification unit 126, the face verification unit 125 is connected to the login module 109 and the administrator operation module 110 respectively, and the account verification unit 126 is connected to the login module 109 and the administrator operation module 110 respectively;

[0092] The face verification unit 125 is used for the administrator to log in to the administrator operation module 110 through face recognition verification;

[0093] The account verification unit 126 is used for the administrator to log in to the administrator operation module 110 through account identification verification.

[0094] In this embodiment, the administrator logs into the administrator operation module 110 through the face verification unit 125 and the account verification unit 126 for verification.

[0095] Furthermore, the big data processing system for massive log analysis and early warning further includes a cache module 127 and a history query module 128, wherein the cache module 127 is connected to the client 107, and the history query module 128 is connected to the cache module 127;

[0096] The cache module 127 is used to temporarily cache log data;

[0097] The history query module 128 is used for management personnel to view historical log data.

[0098] In this embodiment, the cache module 127 temporarily caches the log data, and the administrator checks the historical log data through the history query module 128 .

[0099] What is disclosed above is only a preferred embodiment of the present invention, and it certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A big data processing system for massive log analysis and early warning, characterized in that: It includes a data acquisition module, a first data buffer module, a transmission module, a preprocessing module, a second data buffer module, a data classification module, a packaging module, an unpacking module, an analysis module, a client, a display module and an early warning module, wherein the early warning module and the analysis module are both connected to the client, the analysis module, the unpacking module, the packaging module, the data classification module, the second data buffer module, the transmission module, the first data buffer module and the data acquisition module are connected in sequence, and the preprocessing module is connected to the first data buffer module; The client is used to configure, monitor and schedule log data; The display module is used to display the processing process of log data, which is convenient for users to understand; The data collection module is used to collect log data from the data source in real time; The first data buffer module is used to receive original log data from a data source; The preprocessing module is used to preprocess the collected raw log data; The transmission module is used to transmit the data in the first data buffer module to the second data buffer module; The second data buffer module is used to temporarily store the preprocessed log data; The data classification module is used to classify the stored log data; The packaging module is used to package the classified log data according to a specified format; The unpacking module is used to unpack the packaged log data into a processable format and transmit it to the analysis module; The analysis module is used to perform in-depth analysis on the stored log data to extract valuable information and patterns; The warning module is used to trigger an alarm based on the analysis of log data and to notify management personnel when the warning rules are met.

2. The big data processing system for massive log analysis and early warning according to claim 1, characterized in that: The preprocessing module includes a real-time unit, a sorting unit and a screening unit, and the real-time unit, the real-time unit and the screening unit are all connected to the first data buffer module; The real-time unit is used to assign real-time time to the log data, and collect and process the log data in real time; The sorting unit is used to sort the log data according to the time before and after; The screening unit is used to filter and clean the log data to remove data that is irrelevant to the current analysis task and does not meet the requirements.

3. The big data processing system for massive log analysis and early warning as claimed in claim 2, characterized in that: The data classification module includes a plurality of data storage subunits and a plurality of processing subunits, wherein the plurality of data storage subunits are connected to the second data buffer module, and each of the processing subunits is respectively connected to the packaging module and the corresponding data storage subunit; The data storage subunit is used to receive the log data transmitted from the second data buffer module; The processing subunit is used to further process and classify the log data obtained from the data storage subunit.

4. The big data processing system for massive log analysis and early warning as claimed in claim 3, characterized in that: The analysis module includes a division unit, an identification unit, a plurality of storage subunits, a plurality of transmission subunits and a data analysis unit, wherein the division unit is connected to the unpacking module, the identification unit is connected to the division unit, a plurality of storage subunits are connected to the division unit, the data analysis unit is connected to the client, and each transmission subunit is respectively connected to the data analysis unit and the corresponding storage subunit; The division unit is used to classify the received log data according to the rules and divide it into multiple log data blocks; The identification unit is used to identify each divided log data block separately; The storage subunit is used to store the divided log data blocks in storage media respectively for subsequent analysis and query; The transmission subunit is used to transmit the log data blocks in the storage subunit to the data analysis unit; The data analysis unit is used to perform in-depth analysis and mining on the transmitted log data to extract valuable information.

5. The big data processing system for massive log analysis and early warning as claimed in claim 4, characterized in that: The early warning module includes an abnormality setting unit, an abnormality monitoring unit, an alarm unit and a trend prediction unit, and the abnormality setting unit, the abnormality monitoring unit, the alarm unit and the trend prediction unit are all connected to the client; The abnormality setting unit is used to set warning rules and trigger thresholds; The abnormality monitoring unit is used to perform abnormality monitoring on the processed log data; The warning unit is used to send abnormal notifications to management personnel; The trend prediction unit is used to predict the trend of the system operation status according to historical data and discover potential risks in advance.

6. The big data processing system for massive log analysis and early warning according to claim 1, characterized in that: The big data processing system for massive log analysis and early warning also includes a login module, a verification module and an administrator operation module, the administrator operation module is connected to the client, the verification module is connected to the administrator operation module, and the login module is connected to the verification module; The login module is used by administrators to enter the administrator operation module; The verification module is used to perform login verification when the administrator enters the administrator operation module; The administrator operation module is used for the administrator to configure, monitor and schedule the log data in the client.

7. The big data processing system for massive log analysis and early warning according to claim 6, characterized in that: The verification module includes a face verification unit and an account verification unit, the face verification unit is connected to the login module and the administrator operation module respectively, and the account verification unit is connected to the login module and the administrator operation module respectively; The face verification unit is used for the administrator to log in to the administrator operation module through face recognition verification; The account verification unit is used for the administrator to log in to the administrator operation module through account identification verification.