Big data processing system for analysis and early warning of massive logs

Abstract

The invention relates to the technical field of data processing, in particular to a big data processing system for mass log analysis and early warning, which comprises a log reading module, an encryption transmission module, a preprocessing module, two processor platforms, an optimization strategy generation module, a pushing unit, an execution module, a balancing unit and an early warning module, the preprocessing module is connected with the log reading module through the encryption transmission module, the main control module is connected with the preprocessing module, the balancing unit is connected with the preprocessing module, the two processor platforms are connected with the balancing unit, and the optimization strategy generation module is connected with the two processor platforms. In this way, the technical problems that when a uniprocessor platform adopted in the prior art processes huge data, if the processor platform breaks down, data processing is interrupted, analysis and early warning cannot be continued, and potential risks are brought to safe operation of a power grid are solved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a big data processing system for massive log analysis and early warning. Background Art

[0002] With the continuous development of information technology and the in-depth promotion of smart grid construction, power companies are increasingly dependent on information and communication systems. Information and communication systems not only carry the task of real-time transmission of power grid operation data, but also shoulder the responsibility of monitoring equipment status, collecting important information such as user electricity consumption behavior, etc. In this context, the generation and analysis of massive log data has become a key link in the operation and maintenance management of information and communication systems. By efficiently and accurately processing and analyzing these log data, power companies can promptly discover and solve potential problems in power grid operation, thereby ensuring the stability and security of the power grid; however, with the rapid development of information technology, the demand for processing massive data has shown explosive growth, which has posed unprecedented challenges to the performance and reliability of data processing systems.

[0003] When the traditional single processor platform processes such a large amount of data, if the processor platform fails, data processing will be interrupted and analysis and early warning will not be able to continue, posing potential risks to the safe operation of the power grid. Summary of the invention

[0004] The purpose of the present invention is to provide a big data processing system for massive log analysis and early warning, aiming to solve the technical problem that when a single processor platform used in the prior art processes such a large amount of data, if the processor platform fails, it will lead to data processing interruption and inability to continue analysis and early warning, which brings potential risks to the safe operation of the power grid.

[0005] To achieve the above-mentioned purpose, the present invention adopts a big data processing system for massive log analysis and early warning, including a log reading module, an encryption transmission module, a preprocessing module, two processor platforms, an optimization strategy generation module, a push unit, an execution module, a balancing unit and an early warning module, wherein the preprocessing module is connected to the log reading module through the encryption transmission module, the balancing unit is connected to the preprocessing module, the two processor platforms are connected to the balancing unit, the optimization strategy generation module is connected to the two processor platforms, the push unit and the execution module are connected to the optimization strategy generation module, and the early warning module is connected to the two processor platforms;

[0006] The log reading module is responsible for reading original log data from various log sources (such as server log files, database logs, application logs, etc.);

[0007] The encryption transmission module is used to encrypt the acquired log data and transmit it to the pre-processing module through a secure channel;

[0008] The pre-processing module decrypts the encrypted and transmitted log data and performs preliminary cleaning, formatting and classification processing;

[0009] The balancing unit dynamically allocates tasks to two processor platforms according to the load conditions of the processor platforms;

[0010] The processor platform is used for computing tasks of in-depth analysis of log data and early warning strategies;

[0011] The optimization strategy generation module generates targeted early warning strategies and optimization suggestions based on the analysis results of the processor platform; the early warning module receives and analyzes the early warning information generated by the processor platform in real time, and triggers the early warning mechanism in time when potential problems are detected;

[0012] The push unit pushes the warning information and optimization suggestions generated by the optimization strategy generation module to relevant personnel or systems;

[0013] After the management personnel determine, the execution module automatically executes the corresponding early warning response measures according to the instructions generated by the optimization strategy generation module.

[0014] Among them, the balancing unit includes a load monitoring module, a main control module, a balanced distribution module and a fault switching module. The load monitoring module is connected to the main control module and the two processor platforms. The balanced distribution module and the fault switching module are also connected to the main control module. The two processor platforms are connected to the balanced distribution module and the fault switching module.

[0015] Wherein, the push unit includes a message prompt module and a vibration prompt module, and both the message push module and the vibration prompt module are connected to the optimization strategy generation module.

[0016] Among them, the big data processing system for massive log analysis and early warning also includes a storage module, and the storage module is connected to the two processor platforms.

[0017] The big data processing system for massive log analysis and early warning further includes a monitoring module and a recording module. The monitoring module is connected to the balancing unit, and the recording module is connected to the monitoring module.

[0018] Among them, the big data processing system for massive log analysis and early warning also includes a data backup module, the data backup module and the optimization strategy generation module.

[0019] Among them, the big data processing system for massive log analysis and early warning also includes a compression module, and the compression module is connected to the data backup module.

[0020] Wherein, the big data processing system for massive log analysis and early warning further includes a self-learning optimization module, and the self-learning optimization module is connected to the early warning module;

[0021] The self-learning optimization module analyzes historical and real-time warning data based on a machine learning algorithm, and continuously adjusts model parameters to improve the accuracy of the warning model in the warning module.

[0022] The big data processing system for massive log analysis and early warning of the present invention, when used specifically, first constructs an early warning model in the early warning module based on a machine learning algorithm, the log reading module is responsible for reading original log data from various log sources; the encryption transmission module is used to encrypt the acquired log data and transmit it to the preprocessing module through a secure channel; the preprocessing module decrypts the encrypted and transmitted log data and performs preliminary cleaning, formatting and classification processing; the balancing unit dynamically allocates tasks to two processor platforms according to the load conditions of the processor platforms; the processor platform is used for in-depth analysis of log data and calculation tasks of early warning strategies; the optimization strategy generation module is based on Based on the analysis results of the processor platform, targeted early warning strategies and optimization suggestions are generated; the early warning module receives and analyzes the early warning information generated by the processor platform in real time, and triggers the early warning mechanism in time when potential problems are detected; the push unit pushes the early warning information and optimization suggestions generated by the optimization strategy generation module to relevant personnel or systems; after the management personnel confirm, the execution module automatically executes the corresponding early warning response measures according to the instructions generated by the optimization strategy generation module, thereby solving the technical problem that when a single processor platform used in the prior art processes such a large amount of data, if the processor platform fails, the data processing will be interrupted, and analysis and early warning cannot be continued, which brings potential risks to the safe operation of the power grid.

[0023] The present invention realizes dynamic allocation of tasks and load balancing by introducing the above-mentioned balancing unit and the two processor platforms. When a processor platform fails, the balancing unit can quickly transfer tasks to other healthy processor platforms to ensure the continuity of data processing and analysis, effectively avoiding system paralysis and data processing interruption caused by single point failure, thereby greatly improving the stability and reliability of the system.

[0024] At the same time, the parallel processing capability of the redundantly set processor platform greatly improves the processing speed of massive log data. At the same time, through the cleaning, formatting and classification processing of the pre-processing module and the in-depth analysis of the processor platform, the accuracy and availability of data are also significantly improved. This helps power companies to discover potential problems in power grid operation more quickly and accurately, and take early warning measures in a timely manner.

[0025] In addition, the early warning module can receive and analyze the early warning information generated by the processor platform in real time, and immediately trigger the early warning mechanism once a potential problem is detected. This real-time early warning and response mechanism enables power companies to take quick action, effectively prevent problems from expanding, and ensure the safe and stable operation of the power grid. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 It is a principle block diagram of the first embodiment of the present invention.

[0028] Figure 2 It is a principle block diagram of the second embodiment of the present invention.

[0029] Figure 3 It is a principle block diagram of the third embodiment of the present invention.

[0030] 101-log reading module, 102-encryption transmission module, 103-preprocessing module, 104-processor platform, 105-optimization strategy generation module, 106-push unit, 107-execution module, 108-balancing unit, 109-early warning module, 110-storage module, 111-load monitoring module, 112-main control module, 113-balanced distribution module, 114-fault switching module, 115-message prompt module, 116-vibration prompt module, 201-monitoring module, 202-recording module, 203-data backup module, 204-compression module, 205-self-learning optimization module, 301-maintenance module, 302-login module, 303-verification module. DETAILED DESCRIPTION

[0031] Embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, but should not be construed as limiting the present invention.

[0032] The first embodiment of the present application is:

[0033] See also Figure 1 Figure 1 It is a principle block diagram of the first embodiment of the present invention.

[0034] The present invention provides a big data processing system for massive log analysis and early warning, comprising a log reading module 101, an encryption transmission module 102, a preprocessing module 103, two processor platforms 104, an optimization strategy generation module 105, a push unit 106, an execution module 107, a balancing unit 108, an early warning module 109 and a storage module 110, wherein the balancing unit 108 comprises a load monitoring module 111, a main control module 112, a balanced distribution module 113 and a fault switching module 114, and the push unit 106 comprises a message prompt module 115 and a vibration prompt module 116. The above-mentioned scheme solves the problem.

[0035] For this specific implementation, the log reading module 101 is responsible for reading original log data from various log sources (such as server log files, database logs, application logs, etc.);

[0036] The encryption transmission module 102 is used to encrypt the acquired log data and transmit it to the pre-processing module 103 through a secure channel;

[0037] The pre-processing module 103 decrypts the encrypted and transmitted log data, and performs preliminary cleaning, formatting and classification processing;

[0038] The balancing unit 108 dynamically allocates tasks to two processor platforms 104 according to the load conditions of the processor platforms 104;

[0039] The processor platform 104 is used for computing tasks of in-depth analysis of log data and early warning strategies;

[0040] The optimization strategy generation module 105 generates targeted early warning strategies and optimization suggestions based on the analysis results of the processor platform 104; the early warning module 109 receives and analyzes the early warning information generated by the processor platform 104 in real time, and triggers the early warning mechanism in time when a potential problem is detected;

[0041] The push unit 106 pushes the warning information and optimization suggestions generated by the optimization strategy generation module 105 to relevant personnel or systems;

[0042] After the management personnel make the determination, the execution module 107 automatically executes the corresponding early warning response measures according to the instructions generated by the optimization strategy generation module 105 .

[0043] Among them, the preprocessing module 103 is connected to the log reading module 101 through the encryption transmission module 102, the balancing unit 108 is connected to the preprocessing module 103, the two processor platforms 104 are connected to the balancing unit 108, the optimization strategy generation module 105 is connected to the two processor platforms 104, the push unit 106 and the execution module 107 are connected to the optimization strategy generation module 105, and the early warning module 109 is connected to the two processor platforms 104. When used specifically, the early warning model in the early warning module 109 is first constructed based on the machine learning algorithm. The log reading module 101 is responsible for reading the original log data from various log sources; the encryption transmission module 102 is used to encrypt the acquired log data and transmit it to the preprocessing module 103 through a secure channel; the preprocessing module 103 decrypts the encrypted and transmitted log data, and performs preliminary cleaning, formatting and classification processing; the balancing unit 108 The processor platform 104 is used for in-depth analysis of log data and calculation of early warning strategies; the optimization strategy generation module 105 generates targeted early warning strategies and optimization suggestions based on the analysis results of the processor platform 104; the early warning module 109 receives and analyzes the early warning information generated by the processor platform 104 in real time, and triggers the early warning mechanism in time when potential problems are detected; the push unit 106 pushes the early warning information and optimization suggestions generated by the optimization strategy generation module 105 to relevant personnel or systems; after the management personnel confirms, the execution module 107 automatically executes the corresponding early warning response measures according to the instructions generated by the optimization strategy generation module 105, thereby solving the technical problem that when the single processor platform 104 used in the prior art processes such a large amount of data, if the processor platform 104 fails, the data processing will be interrupted, and the analysis and early warning cannot be continued, which brings potential risks to the safe operation of the power grid.

[0044] Secondly, the load monitoring module 111 is connected to the main control module 112 and the two processor platforms 104, the balanced distribution module 113 and the fault switching module 114 are also connected to the main control module 112, and the two processor platforms 104 are connected to the balanced distribution module 113 and the fault switching module 114;

[0045] The load monitoring module 111 is responsible for monitoring the load conditions of the two processor platforms 104, including key indicators such as CPU usage and memory occupancy. The load monitoring module 111 is connected to the main control module 112 and the two processor platforms 104 to obtain and process load information in real time. Through regular or real-time load monitoring, the load monitoring module 111 can ensure that the system can maintain stable operation under high load conditions.

[0046] The main control module 112 issues corresponding instructions according to the information transmitted by the load monitoring module 111 to adjust the operating state of the system;

[0047] The balanced allocation module 113 is responsible for dynamically allocating tasks to the two processor platforms 104 according to the load conditions of the processor platforms 104; through an intelligent balanced allocation algorithm, the balanced allocation module 113 can ensure load balancing between the two processor platforms 104 and improve the overall performance of the system;

[0048] Once the fault switching module 114 detects that a certain processor platform 104 fails, the fault switching module 114 will immediately issue an instruction to switch the task to another processor platform 104 to ensure the continuous operation of the system.

[0049] At the same time, the message push module and the vibration prompt module 116 are both connected to the optimization strategy generation module 105. The message push module is used to push the optimization strategy generated by the optimization strategy generation module 105. If the management personnel does not respond within the specified time, the vibration prompt module 116 will vibrate to prompt.

[0050] In addition, the storage module 110 is connected to the two processor platforms 104 , and the storage module 110 is used to store historical information for the convenience of the processor platform 104 to obtain.

[0051] A big data processing system for massive log analysis and early warning using the present embodiment is used. When in use, first, an early warning model in the early warning module 109 is constructed based on a machine learning algorithm. The log reading module 101 is responsible for reading original log data from various log sources; the encryption transmission module 102 is used to encrypt the acquired log data and transmit it to the preprocessing module 103 through a secure channel; the preprocessing module 103 decrypts the encrypted and transmitted log data, and performs preliminary cleaning, formatting and classification processing; the balancing unit 108 dynamically allocates tasks to two processor platforms 104 according to the load of the processor platform 104; the processor platform 104 is used for in-depth analysis of log data and calculation tasks of early warning strategies; the optimization strategy generation module 108 is used for in-depth analysis of log data and calculation tasks of early warning strategies; the optimization strategy generation module 101 is used for in-depth analysis of log data and calculation tasks of early warning strategies; the optimization strategy generation module 102 is used for in-depth analysis of log data and calculation tasks of early warning strategies; the optimization strategy generation module 103 ...1 is used for in-depth analysis of log data and calculation tasks of early warning strategies; the optimization strategy generation module 101 is used for in-depth analysis of log data and calculation tasks of early warning strategies; the optimization strategy generation module 101 is used for in-depth analysis of log data and calculation tasks of early warning strategies; the optimization strategy generation module 101 is used for in-depth analysis of log data and calculation tasks of early 05 Based on the analysis results of the processor platform 104, targeted early warning strategies and optimization suggestions are generated; the early warning module 109 receives and analyzes the early warning information generated by the processor platform 104 in real time, and triggers the early warning mechanism in time when a potential problem is detected; the push unit 106 pushes the early warning information and optimization suggestions generated by the optimization strategy generation module 105 to relevant personnel or systems; after the management personnel confirm, the execution module 107 automatically executes the corresponding early warning response measures according to the instructions generated by the optimization strategy generation module 105, thereby solving the technical problem that when the single processor platform 104 used in the prior art processes such a large amount of data, if the processor platform 104 fails, it will cause data processing to be interrupted, and analysis and early warning cannot be continued, which brings potential risks to the safe operation of the power grid.

[0052] The present invention realizes dynamic allocation of tasks and load balancing by introducing the above-mentioned balancing unit 108 and the two processor platforms 104. When a processor platform 104 fails, the balancing unit 108 can quickly transfer tasks to other healthy processor platforms 104, ensuring the continuity of data processing and analysis, effectively avoiding system paralysis and data processing interruption caused by single point failure, thereby greatly improving the stability and reliability of the system.

[0053] At the same time, the parallel processing capability of the redundantly set processor platform 104 greatly improves the processing speed of massive log data, and at the same time, the accuracy and availability of data are also significantly improved through the cleaning, formatting and classification processing of the preprocessing module 103 and the in-depth analysis of the processor platform 104. This helps power companies to discover potential problems in power grid operation more quickly and accurately, and take early warning measures in a timely manner.

[0054] Furthermore, the early warning module 109 can receive and analyze the early warning information generated by the processor platform 104 in real time, and immediately trigger the early warning mechanism once a potential problem is detected. This real-time early warning and response mechanism enables the power company to take prompt action, effectively prevent the problem from expanding, and ensure the safe and stable operation of the power grid.

[0055] The second embodiment of the present application is:

[0056] Based on the first embodiment, please refer to Figure 2 , Figure 2 It is a principle block diagram of the second embodiment of the present invention.

[0057] The present invention provides a big data processing system for massive log analysis and early warning, which also includes a monitoring module 201, a recording module 202, a data backup module 203, a compression module 204 and a self-learning optimization module 205.

[0058] According to this specific implementation, the monitoring module 201 is connected to the balancing unit 108 , and the recording module 202 is connected to the monitoring module 201 . The monitoring module 201 is used to monitor the operating status of the balancing unit 108 and record it through the recording module 202 .

[0059] Among them, the data backup module 203 and the optimization strategy generation module 105, the data backup module 203 is used to back up the optimization strategy generated by the optimization strategy generation module 105 to avoid loss.

[0060] Secondly, the compression module 204 is connected to the data backup module 203 , and the compression module 204 is used to compress the data in the data backup module 203 .

[0061] In addition, the self-learning optimization module 205 is connected to the early warning module 109;

[0062] The self-learning optimization module 205 analyzes historical and real-time warning data based on a machine learning algorithm, and continuously adjusts model parameters to improve the accuracy of the warning model in the warning module 109.

[0063] A big data processing system for massive log analysis and early warning is used in this embodiment. The monitoring module 201 is used to monitor the operating status of the balancing unit 108 and record it through the recording module 202. The data backup module 203 is used to back up the optimization strategy generated by the optimization strategy generation module 105 to avoid loss. The self-learning optimization module 205 analyzes historical and real-time early warning data based on a machine learning algorithm, and continuously adjusts model parameters to improve the accuracy of the early warning model in the early warning module 109.

[0064] The third embodiment of the present application is:

[0065] Based on the second embodiment, please refer to Figure 3 , Figure 3 It is a principle block diagram of the third embodiment of the present invention.

[0066] The present invention provides a big data processing system for massive log analysis and early warning, which also includes a maintenance module 301, a login module 302 and a verification module 303.

[0067] According to this specific implementation, the maintenance module 301 performs maintenance on the balancing unit 108 , the login module 302 is connected to the maintenance module 301 , the maintenance module 301 is logged in through the login module 302 , and the balancing unit 108 can be maintained through the maintenance module 301 .

[0068] The verification module 303 is connected to the login module 302 , and the verification module 303 can authenticate the identity of the administrator who logs into the maintenance module 301 .

[0069] A big data processing system for massive log analysis and early warning is used in this embodiment. The maintenance module 301 is logged in through the login module 302, and the balancing unit 108 can be maintained through the maintenance module 301. The verification module 303 can authenticate the identity of the administrator who logs in to the maintenance module 301.

[0070] What is disclosed above is only a preferred embodiment of the present invention, and it certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A big data processing system for massive log analysis and early warning, characterized in that: It includes a log reading module, an encryption transmission module, a preprocessing module, two processor platforms, an optimization strategy generation module, a push unit, an execution module, a balancing unit and an early warning module, wherein the preprocessing module is connected to the log reading module through the encryption transmission module, the balancing unit is connected to the preprocessing module, the two processor platforms are connected to the balancing unit, the optimization strategy generation module is connected to the two processor platforms, the push unit and the execution module are connected to the optimization strategy generation module, and the early warning module is connected to the two processor platforms; The log reading module is responsible for reading original log data from various log sources; The encryption transmission module is used to encrypt the acquired log data and transmit it to the pre-processing module through a secure channel; The pre-processing module decrypts the encrypted and transmitted log data and performs preliminary cleaning, formatting and classification processing; The balancing unit dynamically allocates tasks to two processor platforms according to the load conditions of the processor platforms; The processor platform is used for computing tasks of in-depth analysis of log data and early warning strategies; The optimization strategy generation module generates targeted early warning strategies and optimization suggestions based on the analysis results of the processor platform; The warning module receives and analyzes the warning information generated by the processor platform in real time, and triggers the warning mechanism in time when a potential problem is detected; The push unit pushes the warning information and optimization suggestions generated by the optimization strategy generation module to relevant personnel or systems; After the management personnel determine, the execution module automatically executes the corresponding early warning response measures according to the instructions generated by the optimization strategy generation module.

2. The big data processing system for massive log analysis and early warning according to claim 1, characterized in that: The balancing unit includes a load monitoring module, a main control module, a balanced distribution module and a fault switching module. The load monitoring module is connected to the main control module and the two processor platforms. The balanced distribution module and the fault switching module are also connected to the main control module. The two processor platforms are connected to the balanced distribution module and the fault switching module.

3. The big data processing system for massive log analysis and early warning as claimed in claim 2, characterized in that: The push unit includes a message prompt module and a vibration prompt module, and both the message push module and the vibration prompt module are connected to the optimization strategy generation module.

4. The big data processing system for massive log analysis and early warning as claimed in claim 3, characterized in that: The big data processing system for massive log analysis and early warning also includes a storage module, and the storage module is connected to the two processor platforms.

5. The big data processing system for massive log analysis and early warning as claimed in claim 4, characterized in that: The big data processing system for massive log analysis and early warning also includes a monitoring module and a recording module. The monitoring module is connected to the balancing unit, and the recording module is connected to the monitoring module.

6. The big data processing system for massive log analysis and early warning as claimed in claim 5, characterized in that: The big data processing system for massive log analysis and early warning also includes a data backup module, which is connected to the optimization strategy generation module.

7. The big data processing system for massive log analysis and early warning according to claim 6, characterized in that: The big data processing system for massive log analysis and early warning also includes a compression module, which is connected to the data backup module.

8. The big data processing system for massive log analysis and early warning according to claim 7, characterized in that: The big data processing system for massive log analysis and early warning further includes a self-learning optimization module, which is connected to the early warning module; The self-learning optimization module analyzes historical and real-time warning data based on a machine learning algorithm, and continuously adjusts model parameters to improve the accuracy of the warning model in the warning module.

Images