Big data processing system for analysis and early warning of massive logs

Abstract

The invention relates to the technical field of big data processing, in particular to a big data processing system for mass log analysis and early warning, which comprises a log data acquisition module, a transmission module, an encryption module, an analysis module, an alarm module, a database, a maintenance module, a feedback module and a management platform, the encryption module is connected with the log data acquisition module and the analysis module, the alarm module is connected with the analysis module and the database, the maintenance module is connected with the database, the management platform is connected with the maintenance module and the alarm module, and the feedback module is also connected with the alarm module. In this way, the technical problem that in the prior art, when a log data analysis early warning system is actually used, data leakage is extremely likely to be caused, and then safe and stable operation of a power grid is seriously threatened is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing, and in particular to a big data processing system for massive log analysis and early warning. Background Art

[0002] In the electric power information and communication industry, with the rapid development of smart grids and the deepening of digital transformation, the amount of log data generated by various business systems has increased dramatically. These log data contain key information such as equipment operation logs, network traffic logs, and user operation logs, which are of immeasurable value in monitoring the operation status of the power grid, predicting fault trends, and ensuring information security. However, traditional log processing systems have obvious limitations when dealing with massive log data.

[0003] The processed log data often contains sensitive information, such as equipment configuration parameters and user identity information. The existing log data analysis and early warning system is very likely to cause data leakage when used in practice, which in turn poses a serious threat to the safe and stable operation of the power grid. Summary of the invention

[0004] The purpose of the present invention is to provide a big data processing system for massive log analysis and early warning, aiming to solve the technical problem that the log data analysis and early warning system in the prior art is very likely to cause data leakage when used in actual use, thereby posing a serious threat to the safe and stable operation of the power grid.

[0005] To achieve the above-mentioned purpose, the present invention adopts a big data processing system for massive log analysis and early warning, including a log data collection module, a transmission module, an encryption module, an analysis module, an alarm module, a database, a maintenance module, a feedback module and a management platform, wherein the transmission module is connected to the log data collection module, the encryption module is connected to the log data collection module and the analysis module, the alarm module is connected to the analysis module and the database, the maintenance module is connected to the database, the management platform is connected to the maintenance module and the alarm module, and the feedback module is also connected to the alarm module;

[0006] The log data collection module is used to capture log data from various servers, application systems and network devices in real time;

[0007] The encryption module is used to encrypt the captured log data;

[0008] The transmission module is used to transmit the encrypted data to the analysis module;

[0009] The analysis module decrypts the encrypted data and then performs analysis and processing;

[0010] The alarm module generates alarm information based on the analysis results and the rules in the database, and displays it on the management platform;

[0011] The feedback module feeds back the generated alarm information to the corresponding staff;

[0012] The maintenance module is used to perform regular maintenance and management on the database, including data backup, recovery, and optimization.

[0013] Wherein, the encryption module includes a key management unit, a key generation unit, an encryption unit and a decryption unit, the encryption unit and the decryption unit are connected to the log data collection module and the analysis module respectively, the key management unit is connected to the encryption unit and the decryption unit, and the key management unit is connected to the key generation unit;

[0014] The key generation unit is used to generate a key of a specified type according to the instruction of the key management unit and store it in a safe place;

[0015] The key management unit is used to request and manage keys, and the key management unit securely distributes keys to the encryption unit (sender) and the decryption unit (receiver);

[0016] The encryption unit encrypts the original data from the log data collection module using the key (a symmetric key or a public key of the recipient of asymmetric encryption) provided by the key management unit and an encryption algorithm (such as AES-GCM or RSA-OAEP) to generate a ciphertext, and transmits it to the analysis module via the transmission module;

[0017] The decryption unit decrypts the ciphertext from the transmission module using the key provided by the key management unit (a symmetric key or a private key of the recipient of asymmetric encryption) and a corresponding decryption algorithm to decrypt the data.

[0018] Wherein, the log data collection module includes a distributed data collection unit and a preprocessing unit, and the preprocessing unit is connected to the distributed data collection unit and the transmission module;

[0019] The distributed data collection unit adopts distributed data collection technology to collect log data from multiple data sources in real time through API interface, web crawler and log file reading method;

[0020] The preprocessing unit implements data cleaning, deduplication, format conversion and standardization based on the big data processing framework of Hadoop and Spark.

[0021] Wherein, the big data processing system for massive log analysis and early warning further includes a storage module, and the storage module is connected to the alarm module;

[0022] The storage module adopts a block encryption and chain storage method, and the specific process is as follows:

[0023] First, the log data is divided into blocks of fixed size, and each block of data is encrypted using the AES encryption algorithm;

[0024] Then, the encrypted data blocks are stored on the disk in chronological order, and a pointer to the next data block is stored in the front of each data block to form a chain structure.

[0025] Among them, the big data processing system for massive log analysis and early warning also includes a sensing module and a self-adjusting module. The sensing module is connected to the management platform, and the self-adjusting modules are connected to the sensing module and the management platform.

[0026] Among them, the big data processing system for massive log analysis and early warning also includes a chart generation module and a chart display module, and the chart display module is connected to the alarm module through the chart generation module.

[0027] Among them, the big data processing system for massive log analysis and early warning also includes a review module, and the review module is connected to the storage module.

[0028] The big data processing system for massive log analysis and early warning of the present invention, when used specifically, firstly, the log data acquisition module captures log data from various servers, application systems and network devices in real time; and after the captured log data is encrypted by the encryption module, the encrypted data is transmitted to the analysis module based on the transmission module; the analysis module decrypts the encrypted data and performs analysis and processing; then the alarm module generates alarm information based on the analysis results and the rules in the database, and displays it on the management platform; then the feedback module feeds back the generated alarm information to the corresponding staff; the maintenance module is used to perform regular maintenance and management of the database, including data backup, recovery and optimization, thereby solving the technical problem that the log data analysis and early warning system in the prior art is very likely to cause data leakage when actually used, thereby posing a serious threat to the safe and stable operation of the power grid.

[0029] The present invention introduces the encryption module to encrypt the log data captured in real time from various servers, application systems and network devices, effectively preventing the risk of data leakage during transmission and storage, thereby ensuring the data confidentiality required for the safe and stable operation of the power grid;

[0030] At the same time, the use of efficient data transmission and the analysis module can realize the rapid collection, encryption, transmission and decryption of massive log data under the premise of ensuring data security, and then perform accurate analysis and processing; this not only improves the timeliness of data analysis, but also reduces system resource consumption and improves overall operating efficiency;

[0031] In combination with the analysis module and the rules in the database, the present invention can automatically generate alarm information and display it to relevant personnel in a timely manner through the management platform. At the same time, through the feedback module, the alarm information can be quickly fed back to the corresponding staff, so that they can take timely measures to deal with potential safety risks, thereby improving the safety warning and emergency response capabilities of the power grid. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 It is a principle block diagram of the first embodiment of the present invention.

[0034] Figure 2 It is a principle block diagram of the second embodiment of the present invention.

[0035] Figure 3 It is a principle block diagram of the third embodiment of the present invention.

[0036] 101-log data collection module, 102-transmission module, 103-encryption module, 104-analysis module, 105-alarm module, 106-database, 107-maintenance module, 108-feedback module, 109-management platform, 110-storage module, 111-key management unit, 112-key generation unit, 113-encryption unit, 114-decryption unit, 115-distributed data collection unit, 116-preprocessing unit, 201-sensing module, 202-self-adjustment module, 203-chart generation module, 204-chart display module, 205-reference module, 301-login module, 302-identity authentication module, 303-face recognition unit, 304-account password verification unit. DETAILED DESCRIPTION

[0037] Embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, but should not be construed as limiting the present invention.

[0038] The first embodiment of the present application is:

[0039] See also Figure 1 , Figure 1 It is a principle block diagram of the first embodiment of the present invention.

[0040] The present invention provides a big data processing system for massive log analysis and early warning, comprising a log data collection module 101, a transmission module 102, an encryption module 103, an analysis module 104, an alarm module 105, a database 106, a maintenance module 107, a feedback module 108, a management platform 109 and a storage module 110, wherein the encryption module 103 comprises a key management unit 111, a key generation unit 112, an encryption unit 113 and a decryption unit 114, and the log data collection module 101 comprises a distributed data collection unit 115 and a preprocessing unit 116. The above scheme solves the technical problem that the log data analysis and early warning system in the prior art is very likely to cause data leakage when actually used, thereby posing a serious threat to the safe and stable operation of the power grid.

[0041] For this specific implementation, the log data collection module 101 is used to capture log data from various servers, application systems and network devices in real time;

[0042] The encryption module 103 is used to encrypt the captured log data;

[0043] The transmission module 102 is used to transmit the encrypted data to the analysis module 104;

[0044] The analysis module 104 decrypts the encrypted data and performs analysis and processing;

[0045] The alarm module 105 generates alarm information based on the analysis results and the rules in the database 106, and displays it on the management platform 109;

[0046] The feedback module 108 feeds back the generated alarm information to the corresponding staff;

[0047] The maintenance module 107 is used to perform regular maintenance and management on the database 106, including data backup, recovery, and optimization.

[0048] The transmission module 102 is connected to the log data collection module 101, the encryption module 103 is connected to the log data collection module 101 and the analysis module 104, the alarm module 105 is connected to the analysis module 104 and the database 106, the maintenance module 107 is connected to the database 106, the management platform 109 is connected to the maintenance module 107 and the alarm module 105, and the feedback module 108 is also connected to the alarm module 105. When used, the log data collection module 101 first captures log data from various servers, application systems and network devices in real time; and the captured log data is encrypted by the encryption module 103. After encryption, the transmission module 102 is used to transmit the encrypted data to the analysis module 104; the analysis module 104 decrypts the encrypted data and performs analysis and processing; then the alarm module 105 generates alarm information based on the analysis results and the rules in the database 106, and displays it on the management platform 109; then the feedback module 108 feeds back the generated alarm information to the corresponding staff; the maintenance module 107 is used to perform regular maintenance and management of the database 106, including data backup, recovery, and optimization. In this way, the technical problem that the log data analysis and early warning system in the prior art is very likely to cause data leakage when actually used, thereby posing a serious threat to the safe and stable operation of the power grid is solved.

[0049] The present invention introduces the encryption module 103 to encrypt the log data captured in real time from various servers, application systems and network devices, effectively preventing the risk of data leakage during transmission and storage, thereby ensuring the data confidentiality required for the safe and stable operation of the power grid;

[0050] At the same time, by adopting efficient data transmission and the analysis module 104, it is possible to realize the rapid collection, encryption, transmission and decryption of massive log data under the premise of ensuring data security, and then to perform accurate analysis and processing; this not only improves the timeliness of data analysis, but also reduces system resource consumption and improves the overall operation efficiency;

[0051] In combination with the analysis module 104 and the rules in the database 106, the present invention can automatically generate alarm information and timely display it to relevant personnel through the management platform 109. At the same time, through the feedback module 108, the alarm information can be quickly fed back to the corresponding staff, so that they can take measures to deal with potential safety risks in a timely manner, thereby improving the safety warning and emergency response capabilities of the power grid.

[0052] Secondly, the encryption unit 113 and the decryption unit 114 are connected to the log data collection module 101 and the analysis module 104 respectively, the key management unit 111 is connected to the encryption unit 113 and the decryption unit 114, and the key management unit 111 is connected to the key generation unit 112;

[0053] The key generation unit 112 is used to generate a key of a specified type according to the instruction of the key management unit 111 and store it in a safe place;

[0054] The key management unit 111 is used to request and manage keys, and the key management unit 111 securely distributes keys to the encryption unit 113 (sender) and the decryption unit 114 (receiver);

[0055] The encryption unit 113 encrypts the original data from the log data collection module 101 using the key (symmetric key or asymmetric encryption recipient's public key) provided by the key management unit 111 and an encryption algorithm (such as AES-GCM or RSA-OAEP) to generate a ciphertext, and transmits it to the analysis module 104 via the transmission module 102;

[0056] The decryption unit 114 decrypts the ciphertext from the transmission module 102 using the key (a symmetric key or a private key of the recipient of asymmetric encryption) provided by the key management unit 111 and a corresponding decryption algorithm.

[0057] At the same time, the distributed data collection unit 115 adopts distributed data collection technology to collect log data from multiple data sources in real time through API interface, web crawler, and log file reading;

[0058] The preprocessing unit 116 implements data cleaning, deduplication, format conversion and standardization based on the big data processing framework of Hadoop and Spark.

[0059] In addition, the storage module 110 is connected to the alarm module 105;

[0060] The storage module 110 adopts a block-based encryption and chain storage method, and the specific process is as follows:

[0061] First, the log data is divided into blocks of fixed size, and each block of data is encrypted using the AES encryption algorithm;

[0062] Then, the encrypted data blocks are stored on the disk in chronological order, and a pointer to the next data block is stored in the front of each data block to form a chain structure.

[0063] A big data processing system for massive log analysis and early warning using the present embodiment is used in specific use. First, the log data collection module 101 captures log data from various servers, application systems and network devices in real time; the captured log data is encrypted by the encryption module 103, and then the encrypted data is transmitted to the analysis module 104 based on the transmission module 102; the analysis module 104 decrypts the encrypted data and performs analysis and processing; then the alarm module 105 generates alarm information based on the analysis results and the rules in the database 106, and displays it on the management platform 109; then the feedback module 108 feeds back the generated alarm information to the corresponding staff; the maintenance module 107 is used to perform regular maintenance and management of the database 106, including data backup, recovery, and optimization. In this way, the technical problem that the log data analysis and early warning system in the prior art is very likely to cause data leakage when actually used, thereby posing a serious threat to the safe and stable operation of the power grid is solved.

[0064] The present invention introduces the encryption module 103 to encrypt the log data captured in real time from various servers, application systems and network devices, effectively preventing the risk of data leakage during transmission and storage, thereby ensuring the data confidentiality required for the safe and stable operation of the power grid;

[0065] At the same time, by adopting efficient data transmission and the analysis module 104, it is possible to realize the rapid collection, encryption, transmission and decryption of massive log data under the premise of ensuring data security, and then to perform accurate analysis and processing; this not only improves the timeliness of data analysis, but also reduces system resource consumption and improves the overall operation efficiency;

[0066] In combination with the analysis module 104 and the rules in the database 106, the present invention can automatically generate alarm information and timely display it to relevant personnel through the management platform 109. At the same time, through the feedback module 108, the alarm information can be quickly fed back to the corresponding staff, so that they can take measures to deal with potential safety risks in a timely manner, thereby improving the safety warning and emergency response capabilities of the power grid.

[0067] The second embodiment of the present application is:

[0068] Based on the first embodiment, please refer to Figure 2 , Figure 2 It is a principle block diagram of the second embodiment of the present invention.

[0069] The present invention provides a big data processing system for massive log analysis and early warning, which also includes a sensing module 201, a self-adjusting module 202, a chart generating module 203, a chart display module 204 and a review module 205.

[0070] For this specific implementation, the sensing module 201 is connected to the management platform 109, and the self-adjusting module 202 is connected to both the sensing module 201 and the management platform 109. When in use, the sensing module 201 senses whether the management platform 109 is being used, and the self-adjusting module 202 automatically adjusts the brightness of the management platform 109 based on the sensing result.

[0071] The chart display module 204 is connected to the alarm module 105 through the chart generation module 203. The chart display module 204 can display the alarm information generated by the alarm module 105 in the form of a chart using the chart generation module 203. This visual display method enables operation and maintenance personnel to more intuitively understand the key information such as the type, quantity, and occurrence time of the alarm information. At the same time, through chart analysis, operation and maintenance personnel can also discover the rules and trends of the alarm information, providing strong support for formulating more effective response strategies.

[0072] Secondly, the review module 205 is connected to the storage module 110 , and the review module 205 allows operation and maintenance personnel to conveniently query and browse the historical log data and alarm information stored in the storage module 110 .

[0073] When using a big data processing system for massive log analysis and early warning according to the present embodiment, the sensing module 201 senses whether the management platform 109 is used by someone, and the self-adjusting module 202 automatically adjusts the brightness of the management platform 109 according to the sensing result. The chart display module 204 can use the chart generation module 203 to display the alarm information generated by the alarm module 105 in the form of a chart. This visual display method enables operation and maintenance personnel to more intuitively understand the key information such as the type, quantity, and occurrence time of the alarm information. At the same time, through chart analysis, operation and maintenance personnel can also discover the rules and trends of alarm information, providing strong support for formulating more effective response strategies. The reference module 205 allows operation and maintenance personnel to easily query and browse the historical log data and alarm information stored in the storage module 110.

[0074] The third embodiment of the present application is:

[0075] Based on the second embodiment, please refer to Figure 3 , Figure 3 It is a principle block diagram of the third embodiment of the present invention.

[0076] The present invention provides a big data processing system for massive log analysis and early warning, which also includes a login module 301 and an identity authentication module 302. The identity authentication module 302 includes a face recognition unit 303 and an account password verification unit 304.

[0077] For this specific implementation, the login module 301 is connected to the management platform 109, and the identity authentication module 302 is connected to the login module 301. The management personnel can log in to the management platform 109 through the login module 301, and the identity authentication unit is used to authenticate the personnel who log in to the management platform 109.

[0078] The face recognition unit 303 and the account password verification unit 304 are both connected to the login module 301 , and the administrator can use face recognition and account password verification to authenticate the identity.

[0079] Using a big data processing system for massive log analysis and early warning of this embodiment, managers can log in to the management platform 109 through the login module 301. The identity authentication unit is used to authenticate the personnel who log in to the management platform 109. Managers can use face recognition and account password verification to authenticate their identities.

[0080] What is disclosed above is only a preferred embodiment of the present invention, and it certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A big data processing system for massive log analysis and early warning, characterized in that: It includes a log data collection module, a transmission module, an encryption module, an analysis module, an alarm module, a database, a maintenance module, a feedback module and a management platform, wherein the transmission module is connected to the log data collection module, the encryption module is connected to the log data collection module and the analysis module, the alarm module is connected to the analysis module and the database, the maintenance module is connected to the database, the management platform is connected to the maintenance module and the alarm module, and the feedback module is also connected to the alarm module; The log data collection module is used to capture log data from various servers, application systems and network devices in real time; The encryption module is used to encrypt the captured log data; The transmission module is used to transmit the encrypted data to the analysis module; The analysis module decrypts the encrypted data and then performs analysis and processing; The alarm module generates alarm information based on the analysis results and the rules in the database, and displays it on the management platform; The feedback module feeds back the generated alarm information to the corresponding staff; The maintenance module is used to perform regular maintenance and management on the database, including data backup, recovery, and optimization.

2. The big data processing system for massive log analysis and early warning according to claim 1, characterized in that: The encryption module includes a key management unit, a key generation unit, an encryption unit and a decryption unit, the encryption unit and the decryption unit are connected to the log data collection module and the analysis module respectively, the key management unit is connected to the encryption unit and the decryption unit, and the key management unit is connected to the key generation unit; The key generation unit is used to generate a key of a specified type according to the instruction of the key management unit and store it in a safe place; The key management unit is used to request and manage keys, and the key management unit securely distributes keys to the encryption unit and the decryption unit; The encryption unit encrypts the original data from the log data collection module using the key provided by the key management unit and the encryption algorithm to generate ciphertext, and transmits the ciphertext to the analysis module via the transmission module; The decryption unit decrypts the ciphertext from the transmission module using the key provided by the key management unit and the corresponding decryption algorithm.

3. The big data processing system for massive log analysis and early warning as claimed in claim 2, characterized in that: The log data collection module includes a distributed data collection unit and a preprocessing unit, and the preprocessing unit is connected to the distributed data collection unit and the transmission module; The distributed data collection unit adopts distributed data collection technology to collect log data from multiple data sources in real time through API interface, web crawler and log file reading method; The preprocessing unit implements data cleaning, deduplication, format conversion and standardization based on the big data processing framework of Hadoop and Spark.

4. The big data processing system for massive log analysis and early warning as claimed in claim 3, characterized in that: The big data processing system for massive log analysis and early warning further includes a storage module, which is connected to the alarm module; The storage module adopts a block encryption and chain storage method, and the specific process is as follows: First, the log data is divided into blocks of fixed size, and each block of data is encrypted using the AES encryption algorithm; Then, the encrypted data blocks are stored on the disk in chronological order, and a pointer to the next data block is stored in the front of each data block to form a chain structure.

5. The big data processing system for massive log analysis and early warning as claimed in claim 4, characterized in that: The big data processing system for massive log analysis and early warning also includes a sensing module and a self-adjusting module. The sensing module is connected to the management platform, and the self-adjusting modules are connected to the sensing module and the management platform.

6. The big data processing system for massive log analysis and early warning as claimed in claim 5, characterized in that: The big data processing system for massive log analysis and early warning also includes a chart generation module and a chart display module, and the chart display module is connected to the alarm module through the chart generation module.

7. The big data processing system for massive log analysis and early warning according to claim 6, characterized in that: The big data processing system for massive log analysis and early warning also includes a review module, and the review module is connected to the storage module.

Images