Big data processing system for analysis and early warning of massive logs

Abstract

The invention relates to the technical field of data processing, in particular to a big data processing system for analysis and early warning of massive logs, which comprises a log collection module, a transmission module, a preprocessing module, an analysis module, a monitoring unit, an early warning generation module, a GSM (Global System for Mobile Communications) module and a display module, the preprocessing module is connected with the transmission module, the analysis module is connected with the preprocessing module, the monitoring unit and the early warning generation module are both connected with the analysis module, the GSM module is connected with the early warning generation module, the display module is connected with the GSM module, and the analysis module comprises at least two processors. In this way, the technical problems that in the prior art, when a real-time analysis and early warning system encounters a processor fault, the risk of data loss is often faced, the continuity of log analysis is affected, fault early warning may be delayed, and then the safety of a power grid is threatened are solved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a big data processing system for massive log analysis and early warning. Background Art

[0002] The power system generates a large amount of information and data during operation. These information and data are important bases for people to understand the network operating environment and the operating status of various equipment. Therefore, communication equipment is needed to transmit this information to the information processing platform through the use of communication technology for people to refer to and analyze, so as to fully control the operation of the system, conduct real-time supervision and control of certain key operations and key equipment, and prevent faults from causing greater adverse consequences. Among them, real-time analysis and early warning of massive equipment operation logs are the core links to ensure the stable operation of the power grid. These log data not only reflect the operating status of the equipment, but also contain potential fault information, which is crucial for timely discovery and handling of problems in the power grid.

[0003] Existing real-time analysis and early warning systems often face the risk of data loss when encountering processor failures, which not only affects the continuity of log analysis, but may also delay fault warnings, thereby posing a threat to power grid security. Summary of the invention

[0004] The purpose of the present invention is to provide a big data processing system for massive log analysis and early warning, aiming to solve the technical problem that the real-time analysis and early warning system in the prior art often faces the risk of data loss when encountering a processor failure, which not only affects the continuity of log analysis, but may also delay fault warning, thereby posing a threat to power grid security.

[0005] To achieve the above-mentioned purpose, the present invention adopts a big data processing system for massive log analysis and early warning, including a log collection module, a transmission module, a preprocessing module, an analysis module, a monitoring unit, an early warning generation module, a GSM module and a display module, wherein the transmission module is connected to the log collection module, the preprocessing module is connected to the transmission module, the analysis module is connected to the preprocessing module, the monitoring unit and the early warning generation module are both connected to the analysis module, the GSM module is connected to the early warning generation module, the display module is connected to the GSM module, and the analysis module includes at least two processors;

[0006] The log collection module is used to automatically and in real time collect operation log data from various devices in the power system and transmit it to the pre-processing module via the transmission module;

[0007] The preprocessing module is used to clean, format and perform preliminary analysis on the received log data in preparation for in-depth analysis;

[0008] The analysis module uses advanced algorithm models to conduct in-depth analysis of pre-processed log data to identify equipment status trends and potential failure modes;

[0009] The warning generation module is used to automatically generate warning information based on the data analyzed by the analysis module, and send it to the designated recipient through the GSM module;

[0010] The display module is used to receive the warning information and analysis report transmitted by the GSM module, and display it to the operator through a graphical interface, which is convenient for intuitive understanding and rapid response;

[0011] The monitoring unit is used to perform all-round monitoring and management of the analysis module. By integrating multiple professional modules, it ensures that the analysis module can perceive, respond to and handle abnormal situations in real time, thereby maintaining the stable operation of the system.

[0012] Wherein, the monitoring unit includes a state monitoring module, an abnormality identification module, a line switching module, a control module, a data recovery module and a data backup module, the data backup module is connected to the preprocessing module, the state detection module is connected to the analysis module, the abnormality identification module is connected to the state monitoring module, the control module is connected to the abnormality identification module, the line switching module and the data recovery module are both connected to the control module, the data recovery module is also connected to the data backup module, and the line switching module is also connected to the analysis module;

[0013] Firstly, the data processed by the preprocessing module is backed up regularly by the data backup module;

[0014] The status monitoring module is used to monitor the operating status of the analysis module in real time; it evaluates the health and efficiency of the module by collecting and analyzing the performance data of the module (such as processing speed, resource usage, etc.). Once any abnormal signs are found, the status monitoring module will immediately trigger an alarm to provide early warning for subsequent abnormal processing.

[0015] The abnormality identification module is used to receive data from the condition monitoring module and use advanced algorithms and technologies to identify potential abnormal patterns or behaviors. Once an abnormality is identified, the abnormality identification module will quickly pass the information to the control module for subsequent processing;

[0016] The control module is used to receive information from the abnormality identification module, make decisions according to preset strategies and rules, switch processors at the same time, and request the data recovery module to restore data from the data backup module, and then restore the restored data to the switched processor for further processing.

[0017] Among them, the big data processing system for massive log analysis and early warning also includes an encryption module, and the encryption module is connected to the transmission module.

[0018] Among them, the big data processing system for massive log analysis and early warning also includes a chart generation module, and the chart generation module is implanted in the display module.

[0019] Among them, the big data processing system for massive log analysis and early warning also includes a sensing module and a self-adjusting module. The sensing module is also implanted in the display module, and the self-adjusting module is connected to the sensing module and the display module.

[0020] Among them, the big data processing system for massive log analysis and early warning also includes a calibration module, and the calibration module is connected to the log collection module.

[0021] Among them, the big data processing system for massive log analysis and early warning also includes a fault isolation module, and the fault isolation module is connected to the control module.

[0022] The present invention discloses a big data processing system for massive log analysis and early warning. When used specifically, the log collection module automatically and in real time collects operation log data from various devices in the power system. These data may include the operation status, operation records, abnormal information, etc. of the equipment. The collected log data is transmitted to the preprocessing module through the transmission module. The encryption module is used in the transmission process to ensure the security and transmission efficiency of the data. The preprocessing module cleans, unifies the format, and performs preliminary analysis on the received log data. The cleaning process includes operations such as removing duplicate data, filling missing values, and correcting erroneous data. The analysis module uses advanced algorithm models to perform in-depth analysis on the preprocessed log data. The algorithm model includes machine learning algorithms and data mining algorithms. The early warning generation module automatically generates early warning information based on the data analyzed by the analysis module. The early warning information may include early warning level, early warning content, recommended measures, etc. The generated early warning information is sent to the designated recipient through the GSM module. The recipients include managers and maintenance personnel of the power system. The display module receives the warning information and analysis report transmitted by the GSM module and displays it to the operators. In this way, the real-time analysis and warning system in the existing technology often faces the risk of data loss when encountering a processor failure. This not only affects the continuity of log analysis, but may also delay fault warnings, thereby posing a threat to the safety of the power grid. Technical problems.

[0023] The present invention has at least two processors built into the analysis module. This design ensures that when a processor fails, another processor can immediately take over the work, effectively preventing data loss and ensuring the continuity and integrity of log analysis.

[0024] The monitoring unit performs all-round monitoring on the analysis module, can sense and respond to processor failures in real time, and quickly take remedial measures to further reduce the risk of data loss, thus providing a solid guarantee for power grid security;

[0025] The log collection module can automatically and in real time collect operation log data from various devices in the power system, ensuring the timeliness and accuracy of the data and providing a solid foundation for early warning generation.

[0026] The analysis module uses advanced algorithm models to conduct in-depth analysis of pre-processed log data, accurately identifying equipment status trends and potential failure modes, making early warning information more accurate and reliable.

[0027] The warning generation module automatically generates warning information according to the analysis results, and quickly sends it to the designated recipient through the GSM module, which greatly shortens the warning response time and improves the efficiency of power grid fault processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0029] Figure 1 It is a principle block diagram of the first embodiment of the present invention.

[0030] Figure 2 It is a principle block diagram of the second embodiment of the present invention.

[0031] Figure 3 It is a principle block diagram of the third embodiment of the present invention.

[0032] 101-log collection module, 102-transmission module, 103-preprocessing module, 104-analysis module, 105-monitoring unit, 106-warning generation module, 107-GSM module, 108-display module, 109-encryption module, 110-chart generation module, 111-status monitoring module, 112-abnormal identification module, 113-line switching module, 114-control module, 115-data recovery module, 116-data backup module, 201-sensing module, 202-self-adjustment module, 203-calibration module, 204-fault isolation module, 301-security protection module, 302-management module, 303-login module. DETAILED DESCRIPTION

[0033] Embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, but should not be construed as limiting the present invention.

[0034] The first embodiment of the present application is:

[0035] See also Figure 1 , Figure 1 It is a principle block diagram of the first embodiment of the present invention.

[0036] The present invention provides a big data processing system for massive log analysis and early warning, comprising a log collection module 101, a transmission module 102, a preprocessing module 103, an analysis module 104, a monitoring unit 105, an early warning generation module 106, a GSM module 107, a display module 108, an encryption module 109 and a chart generation module 110, wherein the monitoring unit 105 comprises a state monitoring module 111, an abnormality identification module 112, a line switching module 113, a control module 114, a data recovery module 115 and a data backup module 116. The above-mentioned scheme solves the technical problem that the real-time analysis and early warning system in the prior art often faces the risk of data loss when encountering a processor 117 failure, which not only affects the continuity of log analysis, but also may delay fault early warning, thereby posing a threat to power grid security.

[0037] According to this specific implementation, the log collection module 101 is used to automatically and in real time collect operation log data from various devices in the power system, and transmit the data to the pre-processing module 103 via the transmission module 102;

[0038] The pre-processing module 103 is used to clean, format and preliminarily analyze the received log data in preparation for in-depth analysis;

[0039] The analysis module 104 uses advanced algorithm models to conduct in-depth analysis of the pre-processed log data to identify equipment status trends and potential failure modes;

[0040] The warning generation module 106 is used to automatically generate warning information based on the data analyzed by the analysis module 104, and send it to a designated recipient through the GSM module 107;

[0041] The display module 108 is used to receive the warning information and analysis report transmitted by the GSM module 107, and display it to the operator through a graphical interface to facilitate intuitive understanding and rapid response;

[0042] The monitoring unit 105 is used to perform all-round monitoring and management on the analysis module 104. By integrating multiple professional modules, it ensures that the analysis module 104 can sense, respond to and handle abnormal situations in real time, thereby maintaining the stable operation of the system.

[0043] Among them, the transmission module 102 is connected to the log collection module 101, the preprocessing module 103 is connected to the transmission module 102, the analysis module 104 is connected to the preprocessing module 103, the monitoring unit 105 and the warning generation module 106 are both connected to the analysis module 104, the GSM module 107 is connected to the warning generation module 106, the display module 108 is connected to the GSM module 107, the analysis module 104 includes at least two processors 117, the encryption module 109 is connected to the transmission module 102, and when used specifically, the log collection module 101 automatically and in real time collects operation log data from various devices in the power system, which may include the operation status, operation records, abnormal information, etc. of the equipment. The collected log data is transmitted to the preprocessing module 103 through the transmission module 102, and the encryption module 109 is used in the transmission process to ensure the security and transmission efficiency of the data. The preprocessing module 103 cleans, unifies the format and performs preliminary analysis on the received log data. The cleaning process includes operations such as removing duplicate data, filling missing values, and correcting erroneous data. The analysis module 104 uses advanced algorithm models to conduct in-depth analysis of the preprocessed log data. The algorithm model includes machine learning algorithms and data mining algorithms. The warning generation module 106 automatically generates warning information based on the data analyzed by the analysis module 104. The warning information may include warning level, warning content, recommended measures, etc. The generated warning information is sent to the designated recipient through the GSM module 107. The recipients include managers and maintenance personnel of the power system. The display module 108 receives the warning information and analysis report transmitted by the GSM module 107 and displays it to the operator. In this way, the real-time analysis and warning system in the prior art often faces the risk of data loss when encountering a processor 117 failure. This not only affects the continuity of log analysis, but also may delay fault warnings, thereby posing a threat to the safety of the power grid.

[0044] Secondly, the data backup module 116 is connected to the preprocessing module 103, the state detection module is connected to the analysis module 104, the abnormality identification module 112 is connected to the state monitoring module 111, the control module 114 is connected to the abnormality identification module 112, the line switching module 113 and the data recovery module 115 are both connected to the control module 114, the data recovery module 115 is also connected to the data backup module 116, and the line switching module 113 is also connected to the analysis module 104;

[0045] First, the data processed by the pre-processing module 103 is backed up regularly by the data backup module 116;

[0046] The state monitoring module 111 is used to monitor the operating state of the analysis module 104 in real time; it evaluates the health and efficiency of the module by collecting and analyzing the performance data (such as processing speed, resource usage, etc.) of the module 104. Once any abnormal signs are found, the state monitoring module 111 will immediately trigger an alarm to provide early warning for subsequent abnormal processing.

[0047] The abnormality identification module 112 is used to receive data from the state monitoring module 111 and use advanced algorithms and technologies to identify potential abnormal patterns or behaviors. Once an abnormality is identified, the abnormality identification module 112 will quickly pass the information to the control module 114 for subsequent processing;

[0048] The control module 114 is used to receive information from the exception identification module 112, and make decisions according to preset strategies and rules, while switching the processor 117 and requesting the data recovery module 115 to restore data from the data backup module 116, and then restore the restored data to the switched processor 117 for further processing.

[0049] Meanwhile, the chart generation module 110 is embedded in the display module 108 , and the chart generation module 110 can display the display data in the form of charts.

[0050] A big data processing system for massive log analysis and early warning using this embodiment, when used specifically, the log collection module 101 automatically and in real time collects operation log data from various devices in the power system, and these data may include the operation status, operation records, abnormal information, etc. of the equipment. The collected log data is transmitted to the preprocessing module 103 through the transmission module 102, and the encryption module 109 is used in the transmission process to ensure the security and transmission efficiency of the data. The preprocessing module 103 cleans, unifies the format and performs preliminary analysis on the received log data. The cleaning process includes operations such as removing duplicate data, filling missing values, and correcting erroneous data. The analysis module 104 uses advanced algorithm models to perform in-depth analysis on the preprocessed log data. The algorithm model includes machine learning algorithms and data mining algorithms. The early warning generation module 106 automatically generates early warning information based on the data analyzed by the analysis module 104. The early warning information may include early warning level, early warning content, recommended measures, etc. The generated early warning information is sent to the designated recipient through the GSM module 107. The recipients include managers and maintenance personnel of the power system. The display module 108 receives the warning information and analysis report transmitted by the GSM module 107 and displays it to the operators. In this way, the real-time analysis and warning system in the prior art often faces the risk of data loss when encountering a processor 117 failure. This not only affects the continuity of log analysis, but may also delay fault warnings, thereby posing a threat to the safety of the power grid. Technical problems.

[0051] The second embodiment of the present application is:

[0052] Based on the first embodiment, please refer to Figure 2 , Figure 2 It is a principle block diagram of the second embodiment of the present invention.

[0053] The present invention provides a big data processing system for massive log analysis and early warning, which also includes a sensing module 201, a self-adjusting module 202, a calibration module 203 and a fault isolation module 204.

[0054] For this specific implementation, the sensing module 201 is also implanted in the display module 108, and the self-adjusting module 202 is connected to the sensing module 201 and the display module 108. When in use, the sensing module 201 is used to sense whether someone is using the display module 108, and transmit the information to the self-adjusting module 202, and the self-adjusting module 202 automatically adjusts the brightness of the display module 108.

[0055] The calibration module 203 is connected to the log collection module 101, and can receive data from the log collection module 101 in real time and calibrate the data according to a preset calibration rule or algorithm. This can ensure the accuracy and consistency of the data and reduce false positives or negatives caused by data errors.

[0056] Secondly, the fault isolation module 204 is connected to the control module 114. The fault isolation module 204 is used to quickly isolate the faulty part when an abnormality or fault is found, so as to prevent the fault from spreading to the entire system.

[0057] A big data processing system for massive log analysis and early warning using this embodiment, when in use, the sensing module 201 is used to sense whether someone is using the display module 108, and transmits the information to the self-adjusting module 202, the self-adjusting module 202 automatically adjusts the brightness of the display module 108, the calibration module 203 can receive data from the log collection module 101 in real time, and calibrate the data according to a preset calibration rule or algorithm. This can ensure the accuracy and consistency of the data, reduce false alarms or omissions caused by data errors, and the fault isolation module 204 is used to quickly isolate the faulty part when an abnormality or fault is found to prevent the fault from spreading to the entire system.

[0058] The third embodiment of the present application is:

[0059] Based on the second embodiment, please refer to Figure 3 , Figure 3 It is a principle block diagram of the third embodiment of the present invention.

[0060] The present invention provides a big data processing system for massive log analysis and early warning, which also includes a security protection module 301, a management module 302 and a login module 303.

[0061] For this specific implementation, the security protection module 301 is connected to the analysis module 104. The security protection module 301 protects the analysis module 104 from network attacks, data leakage and other security threats. The security protection module 301 can use various security technologies, such as firewalls, encryption, identity authentication, etc., to ensure the security and reliability of the analysis module 104.

[0062] The management module 302 is connected to the security protection module 301 , and the login module 303 is connected to the management module 302 . The login module 303 is used for the administrator to log in to the management module 302 , and then manage the security protection module 301 .

[0063] A big data processing system for massive log analysis and early warning is used in this embodiment. The security protection module 301 protects the analysis module 104 from network attacks, data leakage and other security threats. The security protection module 301 can use various security technologies, such as firewalls, encryption, identity authentication, etc. to ensure the security and reliability of the analysis module 104. When used specifically, the login module 303 is used for managers to log in to the management module 302, and then manage the security protection module 301.

[0064] The present invention has at least two processors 117 built into the analysis module 104. This design ensures that when a processor 117 fails, another processor 117 can immediately take over the work, effectively preventing data loss and ensuring the continuity and integrity of log analysis.

[0065] The monitoring unit 105 performs all-round monitoring on the analysis module 104, can sense and respond to the failure of the processor 117 in real time, and quickly take remedial measures to further reduce the risk of data loss, thus providing a solid guarantee for the safety of the power grid;

[0066] The log collection module 101 can automatically and in real time collect operation log data from various devices in the power system, ensuring the timeliness and accuracy of the data and providing a solid foundation for early warning generation.

[0067] The analysis module 104 uses advanced algorithm models to conduct in-depth analysis on the pre-processed log data, accurately identifying equipment status trends and potential failure modes, making the early warning information more accurate and reliable.

[0068] The warning generation module 106 automatically generates warning information according to the analysis results, and quickly sends it to the designated recipient through the GSM module 107, which greatly shortens the warning response time and improves the efficiency of power grid fault processing.

[0069] What is disclosed above is only a preferred embodiment of the present invention, and it certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A big data processing system for massive log analysis and early warning, characterized in that: It includes a log collection module, a transmission module, a preprocessing module, an analysis module, a monitoring unit, an early warning generation module, a GSM module and a display module, wherein the transmission module is connected to the log collection module, the preprocessing module is connected to the transmission module, the analysis module is connected to the preprocessing module, the monitoring unit and the early warning generation module are both connected to the analysis module, the GSM module is connected to the early warning generation module, the display module is connected to the GSM module, and the analysis module includes at least two processors; The log collection module is used to automatically and in real time collect operation log data from various devices in the power system, and transmit the data to the pre-processing module via the transmission module; The preprocessing module is used to clean, unify the format and perform preliminary analysis on the received log data in preparation for in-depth analysis; The analysis module uses advanced algorithm models to conduct in-depth analysis of pre-processed log data to identify equipment status trends and potential failure modes; The warning generation module is used to automatically generate warning information based on the data analyzed by the analysis module, and send it to the designated recipient through the GSM module; The display module is used to receive the warning information and analysis report transmitted by the GSM module and display them to the operator; The monitoring unit is used to perform all-round monitoring and management of the analysis module. By integrating multiple professional modules, it ensures that the analysis module can perceive, respond to and handle abnormal situations in real time, thereby maintaining the stable operation of the system.

2. The big data processing system for massive log analysis and early warning according to claim 1, characterized in that: The monitoring unit includes a state monitoring module, an abnormality identification module, a line switching module, a control module, a data recovery module and a data backup module, wherein the data backup module is connected to the preprocessing module, the state detection module is connected to the analysis module, the abnormality identification module is connected to the state monitoring module, the control module is connected to the abnormality identification module, the line switching module and the data recovery module are both connected to the control module, the data recovery module is also connected to the data backup module, and the line switching module is also connected to the analysis module; Firstly, the data processed by the preprocessing module is backed up regularly by the data backup module; The state monitoring module is used to monitor the operating state of the analysis module in real time; The abnormality identification module is used to receive data from the condition monitoring module and use advanced algorithms and technologies to identify potential abnormal patterns or behaviors. Once an abnormality is identified, the abnormality identification module will quickly pass the information to the control module for subsequent processing; The control module is used to receive information from the abnormality identification module, make decisions according to preset strategies and rules, switch processors at the same time, and request the data recovery module to restore data from the data backup module, and then restore the restored data to the switched processor for further processing.

3. The big data processing system for massive log analysis and early warning as claimed in claim 2, characterized in that: The big data processing system for massive log analysis and early warning also includes an encryption module, which is connected to the transmission module.

4. The big data processing system for massive log analysis and early warning as claimed in claim 3, characterized in that: The big data processing system for massive log analysis and early warning also includes a chart generation module, and the chart generation module is embedded in the display module.

5. The big data processing system for massive log analysis and early warning as claimed in claim 4, characterized in that: The big data processing system for massive log analysis and early warning also includes a sensing module and a self-adjusting module. The sensing module is also implanted in the display module, and the self-adjusting module is connected to the sensing module and the display module.

6. The big data processing system for massive log analysis and early warning as claimed in claim 5, characterized in that: The big data processing system for massive log analysis and early warning also includes a calibration module, which is connected to the log collection module.

7. The big data processing system for massive log analysis and early warning according to claim 6, characterized in that: The big data processing system for massive log analysis and early warning also includes a fault isolation module, which is connected to the control module.

Images