Full-link intelligent root cause analysis method and system based on power grid big data

Abstract

The invention provides a full-link intelligent root cause analysis method and system based on power grid big data. The method comprises the following steps: deploying a probe to a target system, and capturing required data; compressing and encrypting the data; a pyhive database is used for being connected to a Hive database; writing the compressed and encrypted data into a Hive table; data in the Hive database is cleaned through Spark; analyzing the cleaned data by applying a Python algorithm, and returning the analysis data and the date of the data; displaying the cleaned data, the analysis data and the date of the data in the form of a data table and a histogram through Excel; and setting an alarm mechanism according to a preset alarm rule and a threshold value. According to the scheme provided by the invention, data islands can be broken, and unified alarm is realized. Bidirectional driving of the service and the IT is realized, the service continuity is ensured, and the service efficiency is improved. Business problems can be quickly positioned, and business health conditions can be continuously improved.

Description

Technical Field

[0001] The present invention belongs to the field of electric power industry, and in particular relates to a full-link intelligent root cause analysis method and system based on power grid big data. Background Art

[0002] As the power industry has made progress, the growth rate of trade data in the power industry is also increasing, which also means that the power industry is gradually entering the era of big data. The application of energy data in the big data environment has a decisive impact on the development of the energy industry. Only by in-depth research on the application of energy data can we understand the development prospects of the energy industry. Especially due to the growth of electricity demand and the development of science and technology, a large amount of data plays an important role in supporting network design and creating conditions for safe operation and stable smart grids.

[0003] With the continuous development of the economy, the ideas and methods of power grid design have become outdated. Therefore, power companies should actively explore network planning systems that match the background of power big data. Companies need to actively seek power grid planning systems that match their large-scale knowledge background to ensure that they are invincible in the highly competitive market. The design system usually consists of three parts, namely data collection, data processing, and application programs. The core of its operation is to vigorously carry out a series of planning work, including smart grid programming, user interaction, and efforts to improve the actual value of smart grid deployment, by flexibly using large amounts of data to meet network deployment needs and operation capabilities. As a result, business risks can be avoided.

[0004] The use and development of smart grids have greatly improved the electrical stability during transportation and appropriately compensated for the shortcomings of previous power grids. The actual operation of smart networks often causes many problems, such as large coverage and fuzzy coverage areas, so stakeholders need to gradually optimize these problems in future work. It can be seen that in practice, it is of practical significance to discuss power grid planning in the context of power big data.

[0005] Prior art

[0006] The data scale is relatively large, and traditional methods are difficult to process quickly. The power operation monitoring system involves many aspects such as data collection, information transmission and analysis, equipment fault troubleshooting and diagnosis, and fault impact risk assessment. The amount of data collected is huge, and there are relatively many standardized indicators. The current traditional data analysis software and hardware are difficult to process quickly, and must rely on cloud computing, distributed processing systems and other information parallel processing technologies that meet big data conditions.

[0007] The low degree of data resource sharing and integration affects the accuracy of the industry's overall planning and analysis. As the current power operation monitoring systems are decentralized and independent, there are differences in the communication standards between the data control center and the basic nodes, the degree of data sharing and utilization is relatively low, and there are multi-source heterogeneous barriers, which makes it difficult to meet the requirements of modern intelligent and digital integration.

[0008] Disadvantages of existing technology

[0009] The data scale is relatively large, and traditional methods are difficult to process quickly. The power operation monitoring system involves many aspects such as data collection, information transmission and analysis, equipment fault troubleshooting and diagnosis, and fault impact risk assessment. The amount of data collected is huge, and there are relatively many standardized indicators. The current traditional data analysis software and hardware are difficult to process quickly, and must rely on cloud computing, distributed processing systems and other information parallel processing technologies that meet big data conditions.

[0010] The low degree of data resource sharing and integration affects the accuracy of the industry's overall planning and analysis. As the current power operation monitoring systems are decentralized and independent, there are differences in the communication standards between the data control center and the basic nodes, the degree of data sharing and utilization is relatively low, and there are multi-source heterogeneous barriers, which makes it difficult to meet the requirements of modern intelligent and digital integration. Summary of the invention

[0011] In order to solve the above technical problems, the present invention proposes a technical solution of a full-link intelligent root cause analysis method based on power grid big data to solve the above technical problems.

[0012] The first aspect of the present invention discloses a full-link intelligent root cause analysis method based on power grid big data, the method comprising:

[0013] Step S1: Determine the requirements and scope of full-link monitoring by analyzing business requirements;

[0014] Step S2: write Python probe code according to the requirements and scope, and deploy the probe to the target system to capture the required data;

[0015] Step S3, compressing and encrypting the data; connecting to the Hive database using the pyhive library; writing the compressed and encrypted data into the Hive table;

[0016] Step S4: Clean the data in the Hive database using Spark; apply the Python algorithm to analyze the cleaned data and return the analyzed data and the date of the data;

[0017] Step S5: display the cleaned data, analysis data and data date in the form of a data table and a bar chart through Excel; and set an alarm mechanism according to preset alarm rules and thresholds.

[0018] According to the method of the first aspect of the present invention, in step S3, compressing the data includes:

[0019] The data is compressed using the zlib compression algorithm.

[0020] According to the method of the first aspect of the present invention, in step S3, encrypting the data includes:

[0021] The data is encrypted using the AES encryption algorithm.

[0022] According to the method of the first aspect of the present invention, in step S3, the method further includes:

[0023] In Hive, indexes are created based on data access patterns, and queries are optimized through partitioning, bucketing, and sorting.

[0024] According to the method of the first aspect of the present invention, in step S4, the cleaning of data in the Hive database by Spark includes deleting invalid data:

[0025] Loop through the current data and record the column field where the data is located. The column fields of the Hive database are I1, I2, I3...I96, where I1 is the beginning and I96 is the end. During the traversal process, null values ​​and non-positive values ​​are recorded as invalid points. First, the data of the invalid points are assigned to null. If the invalid points have exceeded 5 invalid values, no subsequent judgment is required, and the loop is directly exited, and the current data is recorded as invalid data and deleted.

[0026] According to the method of the first aspect of the present invention, in step S4, the cleaning of data in the Hive database by Spark further includes data deduplication:

[0027] Concatenate each piece of data in the Hive database into a string and record it in an array to determine whether the data in the array is repeated. If the string data of two records is the same, they are considered to be duplicate records.

[0028] According to the method of the first aspect of the present invention, in step S4, the cleaning of data in the Hive database by Spark further includes data completion:

[0029] Get the column index of the invalid point, and take the average of the previous non-empty data and the next non-empty data of the indexed column to complete the data of the invalid point.

[0030] The second aspect of the present invention discloses a full-link intelligent root cause analysis system based on power grid big data, the system comprising:

[0031] A first processing module is configured to deploy a probe to a target system to capture required data;

[0032] The second processing module is configured to compress and encrypt the data; connect to the Hive database using the pyhive library; and write the compressed and encrypted data into the Hive table;

[0033] The third processing module is configured to clean the data in the Hive database through Spark; apply the Python algorithm to analyze the cleaned data, and return the analyzed data and the date of the data;

[0034] The fourth processing module is configured to display the cleaned data, analysis data and data date in the form of a data table and a bar chart through Excel; and set an alarm mechanism according to preset alarm rules and thresholds.

[0035] The third aspect of the present invention discloses an electronic device. The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the steps in any one of the full-link intelligent root cause analysis methods based on power grid big data in the first aspect of the present disclosure are implemented.

[0036] The fourth aspect of the present invention discloses a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any one of the full-link intelligent root cause analysis methods based on power grid big data in the first aspect of the present disclosure are implemented.

[0037] In summary, the solution proposed in this invention can break data silos, realize unified alarm, and build a three-dimensional IT monitoring and operation and maintenance management system. From the business perspective, it can perceive the business and system operation status in real time, realize the two-way drive of business and IT, ensure business continuity, and improve business efficiency. It can make full use of cloud technology, build an intelligent operation and maintenance management platform, quickly locate business problems, and continuously improve business health. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 Flow chart of a full-link intelligent root cause analysis method based on power grid big data according to an embodiment of the present invention;

[0040] Figure 2 A structural diagram of a full-link intelligent root cause analysis system based on power grid big data according to an embodiment of the present invention;

[0041] Figure 3 The figure is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0043] The first aspect of the present invention discloses a full-link intelligent root cause analysis method based on power grid big data. Figure 1 : is a flow chart of a full-link intelligent root cause analysis method based on power grid big data according to an embodiment of the present invention, such as Figure 1 As shown, the method includes:

[0044] Step S1: Determine the requirements and scope of full-link monitoring by analyzing business requirements;

[0045] Step S2: write Python probe code according to the requirements and scope, and deploy the probe to the target system to capture the required data;

[0046] Step S3, compressing and encrypting the data; connecting to the Hive database using the pyhive library; writing the compressed and encrypted data into the Hive table;

[0047] Step S4: Clean the data in the Hive database using Spark; apply the Python algorithm to analyze the cleaned data and return the analyzed data and the date of the data;

[0048] Step S5: display the cleaned data, analysis data and data date in the form of a data table and a bar chart through Excel; and set an alarm mechanism according to preset alarm rules and thresholds.

[0049] In step S1, the requirements and scope of full-link monitoring are determined by analyzing business requirements.

[0050] Specifically, it includes determining the business processes, key performance indicators (KPIs), service interfaces, etc. that need to be monitored. At the same time, it is also necessary to determine the granularity and frequency of monitoring to meet business needs and the accuracy of analysis.

[0051] In step S2, Python probe code is written according to the requirements and scope, and the probe is deployed to the target system to capture the required data.

[0052] Specifically, these probes can collect system operation data in real time, including but not limited to user request information, service call chain data, performance indicators, etc.

[0053] According to business needs, obtain information such as psutil (used to obtain system information), requests (if data needs to be obtained through network requests), time (used for timing), and logging (used for logging).

[0054] CPU usage: Use the psutil.cpu_percent() function to get the current CPU usage. This function can receive a time interval as a parameter to smooth the result, but it is usually set to None or 0 to get the instant usage.

[0055] Memory usage: Use the psutil.virtual_memory() function to obtain memory usage, including total memory, available memory, usage rate, etc.

[0056] Disk I / O: The psutil.disk_io_counters() function can obtain information such as the number and time of disk read and write operations.

[0057] Network bandwidth: The psutil.net_io_counters() function provides the number of bytes sent and received by the network interface, which can be used to estimate the network bandwidth usage.

[0058] Error logs: You can capture error logs by reading the system's log files. You can also use a logging library (such as Python's logging module) to log errors directly in the application code.

[0059] Timed collection: Use the time.sleep() function in combination with a loop to collect the above data at a fixed time. Integrate the probe code into the target system, which may include modifying the application code, configuring services, or deploying it as an independent process.

[0060] In step S3, the data is compressed and encrypted; the pyhive library is used to connect to the Hive database; and the compressed and encrypted data is written into the Hive table.

[0061] In some embodiments, in step S3, compressing the data includes:

[0062] The data is compressed using the zlib compression algorithm.

[0063] Encrypting the data includes:

[0064] The data is encrypted using the AES encryption algorithm.

[0065] The method further comprises:

[0066] In Hive, indexes are created based on data access patterns, and queries are optimized through partitioning, bucketing, and sorting.

[0067] Specifically, prepare the configuration information required for Hive connection, such as host name, port, user name, database, etc. Use the pyhive library to connect to the Hive database. After connecting to the database, you can implement SQL queries and operations through the cursor() method of the connection object. This method returns a cursor object that can be used to execute queries and obtain results.

[0068] Use the zlib compression algorithm to compress data and the AES encryption algorithm to encrypt data.

[0069] Data compression: Use zlib's ˋcompress()ˋ or ˋcompressobj()ˋ method to compress data. ˋcompress()ˋ is suitable for compressing small data blocks that need to be completed at one time, while ˋcompressobj()ˋ is more suitable for scenarios where large amounts of data need to be processed step by step. After compression, a byte string (bytes) will be obtained, which contains the compressed data and possible zlib header information.

[0070] Data encryption: Use a library such as pycryptodome or cryptography to perform AES encryption. AES encryption requires a key and an initialization vector (IV). The key length is usually 128, 192, or 256 bits, and the IV length must be the same as the block size (for AES, the block size is 128 bits). Use an encryption mode such as CBC, ECB, and decide how to handle data padding. Encrypt the data using the selected key, IV, encryption mode, and padding. After encryption, you will get an encrypted byte string that contains the encrypted data, but may also contain some additional information (such as IV, depending on the encryption method and subsequent usage requirements).

[0071] Write the encrypted (and compressed) data to the Hive table. Note that if the data is encrypted and compressed, it needs to be decrypted and decompressed first. Use the hive.Connection class of pyhive to create a connection to the Hive database. Configure the host name, port, user name, and other information. Call SQL statements to create a database and data table, and insert the collected data into the data table.

[0072] In Hive, indexes are created based on data access patterns, and queries are optimized to improve retrieval efficiency and accuracy. Hive does not support a wide range of index types like traditional relational databases, but it can optimize query performance through partitioning, bucketing, and sorting.

[0073] In step S4, the data in the Hive database is cleaned by Spark; the cleaned data is analyzed by applying the Python algorithm, and the analyzed data and the date of the data are returned.

[0074] In some embodiments, in step S4, cleaning the data in the Hive database by using Spark includes deleting invalid data:

[0075] Loop through the current data and record the column field where the data is located. The column fields of the Hive database are I1, I2, I3...I96, where I1 is the beginning and I96 is the end. During the traversal process, null values ​​and non-positive values ​​are recorded as invalid points. First, the data of the invalid points are assigned to null. If the invalid points have exceeded 5 invalid values, no subsequent judgment is required, and the loop is directly exited, and the current data is recorded as invalid data and deleted.

[0076] The data cleaning of the Hive database by Spark also includes data deduplication:

[0077] Concatenate each piece of data in the Hive database into a string and record it in an array to determine whether the data in the array is repeated. If the string data of two records is the same, they are considered to be duplicate records.

[0078] The data cleaning of the Hive database through Spark also includes data completion:

[0079] The column index of the invalid point is obtained, and the average value of the previous non-empty data and the next non-empty data of the index column is taken to complete the data of the invalid point.

[0080] Specifically, Spark uses in-memory computing technology to efficiently process data in memory. Compared with traditional disk-based computing systems, Spark has faster computing speed and higher performance. This feature makes Spark particularly efficient when processing large-scale data sets. Spark is not only a data processing framework, but also has a series of powerful components, such as Spark SQL, Spark Streaming, MLlib, and GraphX. These components can be seamlessly integrated and provide a one-stop solution platform to meet various data processing and analysis needs.

[0081] The Python algorithm analyzes the data and returns the analyzed data and the data date. The data analysis process uses APIs such as Sequential, ModelCheckpoint, Dense, LSTM, Dropout, Activation, and MinMaxScaler.

[0082] Python has significant advantages in implementing neural network algorithms, including:

[0083] 1. Python has a large number of libraries and frameworks designed specifically for neural networks, such as TensorFlow,

[0084] PyTorch, Keras, Caffe, etc. These libraries and frameworks provide efficient tensor calculation, automatic differentiation, model training and optimization functions, which greatly simplify the design and implementation process of neural networks.

[0085] 2. Python is a high-level programming language that provides flexible data structures and control flow statements, allowing developers to freely define the structure and training process of neural networks.

[0086] 3. With the help of Python's object-oriented programming features, developers can easily create and manage complex neural network models.

[0087] 4. Python supports multiple programming paradigms, including procedural, object-oriented, and functional programming. This diversity makes Python highly scalable in the implementation of neural network algorithms.

[0088] 5. Python can be easily integrated with other programming languages ​​and tools, such as C++, Java, etc., to achieve more efficient computing or utilize specific features of other languages.

[0089] LSTM neural network algorithm: In the LSTM neural network algorithm, the forget gate, input gate, output gate, cell state, and hidden state are interrelated components that each play a different role.

[0090] The input gate determines which new information should be added to the cell state. It also reads the hidden state of the previous moment (ht-1) and the input of the current moment (xt), and uses the sigmoid function to determine which information needs to be updated. At the same time, it also generates a new candidate value vector (c~t) through the tanh function. This candidate value vector contains possible new information. Then, the input gate controls how much information of this candidate value vector is added to the cell state. Formula:

[0091] it=σ(Wi·[ht-1,xt]+bi)

[0092] c~t=tanh(Wc·[ht-1,xt]+bc)

[0093] The forget gate determines which information is discarded from the cell state. It reads the hidden state of the previous moment (ht-1) and the input of the current moment (xt), and outputs a value between 0 and 1 through the sigmoid function. This value indicates which information in the cell state should be retained and which should be forgotten. The closer the output value is to 0, the more it should be forgotten, and the closer it is to 1, the more it should be retained.

[0094] Formula: ft=σ(Wf·[ht-1,xt]+bf)

[0095] The output gate determines which information in the cell state should be output to the current output value (i.e., hidden state) of the LSTM unit. It reads the hidden state of the previous moment (ht-1) and the input of the current moment (xt), and determines which information should be output through the sigmoid function. Then, it also processes the cell state through the tanh function (to obtain a value between -1 and 1), and multiplies it with the output of the sigmoid function to obtain the final hidden state. Formula:

[0096] ot=σ(Wo·[ht-1,xt]+bo)

[0097] The cell state is the core part of the LSTM unit. It is similar to a conveyor belt that runs directly on the entire chain to store long-term information. The cell state updates its own state based on the output of the forget gate and the output of the input gate. The forget gate determines which old information is retained, and the input gate determines which new information is added.

[0098] Formula: ct = ft*ct-1+it*c~t

[0099] The hidden state is the output of the LSTM unit at each time step. It contains all the useful information of the current time step and will be passed to the LSTM unit of the next time step as input. The hidden state can also be used as the final output of the entire LSTM network for subsequent tasks such as classification and regression. Formula: ht = ot*tanh(ct).

[0100] The forget gate, input gate, and output gate help the LSTM network effectively capture long-term dependencies in sequence data by controlling the flow and forgetting of information. The cell state is used to store this information, while the hidden state is the output of the LSTM network at each time step for information transmission and subsequent processing.

[0101] Through Python data analysis and LSTM neural network algorithm, two scenarios were applied:

[0102] 1) Use Python algorithms to receive and analyze multi-dimensional data from the system in real time, covering key indicators such as CPU usage, memory usage, disk performance, and the duration of each business link. By intelligently comparing these indicators with preset thresholds, the system can quickly identify and report any situation that exceeds the normal range, thereby helping the operation and maintenance team to accurately locate the occurrence and potential causes of system problems. This automated monitoring and diagnosis mechanism greatly improves the efficiency and accuracy of system maintenance.

[0103] 2) By applying the LSTM neural network algorithm, we deeply mined the historical data of the previous five years, and comprehensively considered the impact of historical holidays on the system, as well as external factors such as major security events that occurred in history, as input features to build and train an accurate data model. This model aims to capture long-term dependencies and complex patterns in time series data, thereby improving the ability to predict future states. After rigorous model training and optimization, the LSTM model can effectively use historical laws and current trends to accurately predict the operating status of the system in the next 7 days, providing strong data support and risk warning for operation and maintenance decisions.

[0104] In step S5, the cleaned data, analysis data and data date are displayed in the form of data tables and bar charts through Excel; and an alarm mechanism is set according to preset alarm rules and thresholds.

[0105] Specifically, Excel displays data to users in the form of data tables, bar charts, etc. These visual charts can intuitively reflect the operating status of the system and the execution of business processes, helping users quickly understand the status of the system.

[0106] Set up an alarm mechanism based on preset alarm rules and thresholds. When the system is abnormal or the performance index exceeds the threshold, an alarm is automatically triggered to notify the user for processing. The alarm notification is sent to the user by email.

[0107] After multiple system services, the calling relationship is:

[0108] Pay attention to the performance indicators of each call, such as throughput (TPS), response time, and error records.

[0109] Throughput: The actual throughput of components, platforms, and physical devices can be calculated based on the topology.

[0110] QPS(TPS)=Number of concurrent connections / Average response time

[0111] Response time: includes the response time of the overall call and the response time of each service, etc.

[0112] Error log: Count the number of exceptions per unit time based on service returns.

[0113] In summary, the solution proposed in this invention can break data silos, realize unified alarm, and build a three-dimensional IT monitoring and operation and maintenance management system. From the business perspective, it can perceive the business and system operation status in real time, realize the two-way drive of business and IT, ensure business continuity, and improve business efficiency. It can make full use of cloud technology, build an intelligent operation and maintenance management platform, quickly locate business problems, and continuously improve business health.

[0114] The second aspect of the present invention discloses a full-link intelligent root cause analysis system based on power grid big data. Figure 2 is a structural diagram of a full-link intelligent root cause analysis system based on power grid big data according to an embodiment of the present invention; Figure 2 As shown, the system 100 includes:

[0115] The first processing module 101 is configured to deploy a probe to a target system to capture required data;

[0116] The second processing module 102 is configured to compress and encrypt the data; connect to the Hive database using the pyhive library; and write the compressed and encrypted data into the Hive table;

[0117] The third processing module 103 is configured to clean the data in the Hive database through Spark; apply the Python algorithm to analyze the cleaned data, and return the analyzed data and the date of the data;

[0118] The fourth processing module 104 is configured to display the cleaned data, analysis data and data date in the form of data tables and bar charts through Excel; and set an alarm mechanism according to preset alarm rules and thresholds.

[0119] According to the system of the second aspect of the present invention, the first processing module 101 is specifically configured so that these probes can collect system operation data in real time, including but not limited to user request information, service call chain data, performance indicators, etc.

[0120] According to business needs, obtain information such as psutil (used to obtain system information), requests (if data needs to be obtained through network requests), time (used for timing), and logging (used for logging).

[0121] CPU usage: Use the psutil.cpu_percent() function to get the current CPU usage. This function can receive a time interval as a parameter to smooth the result, but it is usually set to None or 0 to get the instant usage.

[0122] Memory usage: Use the psutil.virtual_memory() function to obtain memory usage, including total memory, available memory, usage rate, etc.

[0123] Disk I / O: The psutil.disk_io_counters() function can obtain information such as the number and time of disk read and write operations.

[0124] Network bandwidth: The psutil.net_io_counters() function provides the number of bytes sent and received by the network interface, which can be used to estimate the network bandwidth usage.

[0125] Error logs: You can capture error logs by reading the system's log files. You can also use a logging library (such as Python's logging module) to log errors directly in the application code.

[0126] Timed collection: Use the time.sleep() function in combination with a loop to collect the above data at a fixed time. Integrate the probe code into the target system, which may include modifying the application code, configuring services, or deploying it as an independent process.

[0127] According to the system of the second aspect of the present invention, the second processing module 102 is specifically configured as follows: compressing the data includes:

[0128] The data is compressed using the zlib compression algorithm.

[0129] Encrypting the data includes:

[0130] The data is encrypted using the AES encryption algorithm.

[0131] The method further comprises:

[0132] In Hive, indexes are created based on data access patterns, and queries are optimized through partitioning, bucketing, and sorting.

[0133] Specifically, prepare the configuration information required for Hive connection, such as host name, port, user name, database, etc. Use the pyhive library to connect to the Hive database. After connecting to the database, you can implement SQL queries and operations through the cursor() method of the connection object. This method returns a cursor object that can be used to execute queries and obtain results.

[0134] Use the zlib compression algorithm to compress data and the AES encryption algorithm to encrypt data.

[0135] Data compression: Use zlib's ˋcompress()ˋ or ˋcompressobj()ˋ method to compress data. ˋcompress()ˋ is suitable for compressing small data blocks that need to be completed at one time, while ˋcompressobj()ˋ is more suitable for scenarios where large amounts of data need to be processed step by step. After compression, a byte string (bytes) will be obtained, which contains the compressed data and possible zlib header information.

[0136] Data encryption: Use a library such as pycryptodome or cryptography to perform AES encryption. AES encryption requires a key and an initialization vector (IV). The key length is usually 128, 192, or 256 bits, and the IV length must be the same as the block size (for AES, the block size is 128 bits). Use an encryption mode such as CBC, ECB, and decide how to handle data padding. Encrypt the data using the selected key, IV, encryption mode, and padding. After encryption, you will get an encrypted byte string that contains the encrypted data, but may also contain some additional information (such as IV, depending on the encryption method and subsequent usage requirements).

[0137] Write the encrypted (and compressed) data to the Hive table. Note that if the data is encrypted and compressed, it needs to be decrypted and decompressed first. Use the hive.Connection class of pyhive to create a connection to the Hive database. Configure the host name, port, user name, and other information. Call SQL statements to create a database and data table, and insert the collected data into the data table.

[0138] In Hive, indexes are created based on data access patterns, and queries are optimized to improve retrieval efficiency and accuracy. Hive does not support a wide range of index types like traditional relational databases, but it can optimize query performance through partitioning, bucketing, and sorting.

[0139] According to the system of the second aspect of the present invention, the third processing module 103 is specifically configured as follows: the cleaning of data in the Hive database by Spark includes deleting invalid data:

[0140] Loop through the current data and record the column field where the data is located. The column fields of the Hive database are I1, I2, I3...I96, where I1 is the beginning and I96 is the end. During the traversal process, null values ​​and non-positive values ​​are recorded as invalid points. First, the data of the invalid points are assigned to null. If the invalid points have exceeded 5 invalid values, no subsequent judgment is required, and the loop is directly exited, and the current data is recorded as invalid data and deleted.

[0141] The data cleaning of the Hive database by Spark also includes data deduplication:

[0142] Concatenate each piece of data in the Hive database into a string and record it in an array to determine whether the data in the array is repeated. If the string data of two records is the same, they are considered to be duplicate records.

[0143] The data cleaning of the Hive database through Spark also includes data completion:

[0144] The column index of the invalid point is obtained, and the average value of the previous non-empty data and the next non-empty data of the index column is taken to complete the data of the invalid point.

[0145] Specifically, Spark uses in-memory computing technology to efficiently process data in memory. Compared with traditional disk-based computing systems, Spark has faster computing speed and higher performance. This feature makes Spark particularly efficient when processing large-scale data sets. Spark is not only a data processing framework, but also has a series of powerful components, such as Spark SQL, Spark Streaming, MLlib, and GraphX. These components can be seamlessly integrated and provide a one-stop solution platform to meet various data processing and analysis needs.

[0146] The Python algorithm analyzes the data and returns the analyzed data and the data date. The data analysis process uses APIs such as Sequential, ModelCheckpoint, Dense, LSTM, Dropout, Activation, and MinMaxScaler.

[0147] Python has significant advantages in implementing neural network algorithms, including:

[0148] 1. Python has a large number of libraries and frameworks designed specifically for neural networks, such as TensorFlow, PyTorch, Keras, Caffe, etc. These libraries and frameworks provide efficient tensor calculations, automatic differentiation, model training and optimization, etc., which greatly simplifies the design and implementation process of neural networks.

[0149] 2. Python is a high-level programming language that provides flexible data structures and control flow statements, allowing developers to freely define the structure and training process of neural networks.

[0150] 3. With the help of Python's object-oriented programming features, developers can easily create and manage complex neural network models.

[0151] 4. Python supports multiple programming paradigms, including procedural, object-oriented, and functional programming.

[0152] This diversity makes Python highly scalable in implementing neural network algorithms.

[0153] 5. Python can be easily integrated with other programming languages ​​and tools, such as C++, Java, etc., to achieve more efficient computing or utilize specific features of other languages.

[0154] LSTM neural network algorithm: In the LSTM neural network algorithm, the forget gate, input gate, output gate, cell state, and hidden state are interrelated components that each play a different role.

[0155] The input gate determines which new information should be added to the cell state. It also reads the hidden state of the previous moment (ht-1) and the input of the current moment (xt), and uses the sigmoid function to determine which information needs to be updated. At the same time, it also generates a new candidate value vector (c~t) through the tanh function. This candidate value vector contains possible new information. Then, the input gate controls how much information of this candidate value vector is added to the cell state. Formula:

[0156] it=σ(Wi·[ht-1,xt]+bi)

[0157] c~t=tanh(Wc·[ht-1,xt]+bc)

[0158] The forget gate determines which information is discarded from the cell state. It reads the hidden state of the previous moment (ht-1) and the input of the current moment (xt), and outputs a value between 0 and 1 through the sigmoid function. This value indicates which information in the cell state should be retained and which should be forgotten. The closer the output value is to 0, the more it should be forgotten, and the closer it is to 1, the more it should be retained.

[0159] Formula: ft=σ(Wf·[ht-1,xt]+bf)

[0160] The output gate determines which information in the cell state should be output to the current output value (i.e., hidden state) of the LSTM unit. It reads the hidden state of the previous moment (ht-1) and the input of the current moment (xt), and determines which information should be output through the sigmoid function. Then, it also processes the cell state through the tanh function (to obtain a value between -1 and 1), and multiplies it with the output of the sigmoid function to obtain the final hidden state. Formula:

[0161] ot=σ(Wo·[ht-1,xt]+bo)

[0162] The cell state is the core part of the LSTM unit. It is similar to a conveyor belt that runs directly on the entire chain to store long-term information. The cell state updates its own state based on the output of the forget gate and the output of the input gate. The forget gate determines which old information is retained, and the input gate determines which new information is added.

[0163] Formula: ct = ft*ct-1+it*c~t

[0164] The hidden state is the output of the LSTM unit at each time step. It contains all the useful information of the current time step and will be passed to the LSTM unit of the next time step as input. The hidden state can also be used as the final output of the entire LSTM network for subsequent tasks such as classification and regression. Formula: ht = ot*tanh(ct).

[0165] The forget gate, input gate, and output gate help the LSTM network effectively capture long-term dependencies in sequence data by controlling the flow and forgetting of information. The cell state is used to store this information, while the hidden state is the output of the LSTM network at each time step for information transmission and subsequent processing.

[0166] Through Python data analysis and LSTM neural network algorithm, two scenarios were applied:

[0167] 1) Use Python algorithms to receive and analyze multi-dimensional data from the system in real time, covering key indicators such as CPU usage, memory usage, disk performance, and the duration of each business link. By intelligently comparing these indicators with preset thresholds, the system can quickly identify and report any situation that exceeds the normal range, thereby helping the operation and maintenance team to accurately locate the occurrence and potential causes of system problems. This automated monitoring and diagnosis mechanism greatly improves the efficiency and accuracy of system maintenance.

[0168] 2) By applying the LSTM neural network algorithm, we deeply mined the historical data of the previous five years, and comprehensively considered the impact of historical holidays on the system, as well as external factors such as major security events that occurred in history, as input features to build and train an accurate data model. This model aims to capture long-term dependencies and complex patterns in time series data, thereby improving the ability to predict future states. After rigorous model training and optimization, the LSTM model can effectively use historical laws and current trends to accurately predict the operating status of the system in the next 7 days, providing strong data support and risk warning for operation and maintenance decisions.

[0169] According to the system of the second aspect of the present invention, the fourth processing module 104 is specifically configured to display the data to the user in the form of a data table, a bar chart, etc. through Excel. These visual charts can intuitively reflect the operating status of the system and the execution of the business process, helping the user to quickly understand the situation of the system.

[0170] Set up an alarm mechanism based on preset alarm rules and thresholds. When the system is abnormal or the performance index exceeds the threshold, an alarm is automatically triggered to notify the user for processing. The alarm notification is sent to the user by email.

[0171] After multiple system services, the calling relationship is:

[0172] Pay attention to the performance indicators of each call, such as throughput (TPS), response time, and error records.

[0173] Throughput: The actual throughput of components, platforms, and physical devices can be calculated based on the topology.

[0174] QPS(TPS)=Number of concurrent connections / Average response time

[0175] Response time: includes the response time of the overall call and the response time of each service, etc.

[0176] Error log: Count the number of exceptions per unit time based on service returns.

[0177] The third aspect of the present invention discloses an electronic device. The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the steps of any one of the full-link intelligent root cause analysis methods based on power grid big data disclosed in the first aspect of the present invention are implemented.

[0178] Figure 3 is a structural diagram of an electronic device according to an embodiment of the present invention, such as Figure 3 As shown, the electronic device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, near field communication (NFC) or other technologies. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covered on the display screen, or a button, a trackball or a touch pad set on the housing of the electronic device, or an external keyboard, touch pad or mouse, etc.

[0179] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a structural diagram of the part related to the technical solution of the present disclosure, and does not constitute a limitation on the electronic device to which the technical solution of the present application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0180] The fourth aspect of the present invention discloses a computer-readable storage medium. A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the steps in the steps of any one of the steps in the full-link intelligent root cause analysis method based on power grid big data disclosed in the first aspect of the present invention are implemented.

[0181] Please note that the technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. The above embodiments only express several implementation methods of the present application, and their descriptions are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, without departing from the concept of the present application, several variations and improvements can be made, which all belong to the scope of protection of the present application. Therefore, the scope of protection of the patent in this application shall be based on the attached claims.

Claims

1. A full-link intelligent root cause analysis method based on power grid big data, characterized in that: The method comprises: Step S1: Determine the requirements and scope of full-link monitoring by analyzing business requirements; Step S2: write Python probe code according to the requirements and scope, and deploy the probe to the target system to capture the required data; Step S3, compressing and encrypting the data; connecting to the Hive database using the pyhive library; writing the compressed and encrypted data into the Hive table; Step S4: Clean the data in the Hive database using Spark; apply the Python algorithm to analyze the cleaned data and return the analyzed data and the date of the data; Step S5: display the cleaned data, analysis data and data date in the form of a data table and a bar chart through Excel; and set an alarm mechanism according to preset alarm rules and thresholds.

2. A full-link intelligent root cause analysis method based on power grid big data according to claim 1, characterized in that: In step S3, compressing the data includes: The data is compressed using the zlib compression algorithm.

3. A full-link intelligent root cause analysis method based on power grid big data according to claim 1, characterized in that: In the step S3, encrypting the data includes: The data is encrypted using the AES encryption algorithm.

4. A full-link intelligent root cause analysis method based on power grid big data according to claim 1, characterized in that: In step S3, the method further includes: In Hive, indexes are created based on data access patterns, and queries are optimized through partitioning, bucketing, and sorting.

5. The full-link intelligent root cause analysis method based on power grid big data according to claim 1 is characterized in that: In step S4, the cleaning of data in the Hive database by Spark includes deleting invalid data: Loop through the current data and record the column field where the data is located. The column fields of the Hive database are I1, I2, I3...I96, where I1 is the beginning and I96 is the end. During the traversal process, empty values ​​and non-positive values ​​are recorded as invalid points, and the data of the invalid points are first assigned to null; if the invalid points have exceeded 5 invalid values, no subsequent judgment is required, and the loop is exited directly, and the current data is recorded as invalid data and deleted.

6. A full-link intelligent root cause analysis method based on power grid big data according to claim 1, characterized in that: In step S4, the cleaning of data in the Hive database by Spark also includes data deduplication: Concatenate each piece of data in the Hive database into a string and record it in an array to determine whether the data in the array is repeated. If the string data of two records is the same, they are considered to be duplicate records.

7. A full-link intelligent root cause analysis method based on power grid big data according to claim 5, characterized in that: In step S4, the cleaning of data in the Hive database by Spark also includes data completion: Get the column index of the invalid point, and take the average of the previous non-empty data and the next non-empty data of the index column to complete the data of the invalid point.

8. A full-link intelligent root cause analysis system based on power grid big data, characterized in that: The system comprises: A first processing module is configured to deploy a probe to a target system to capture required data; The second processing module is configured to compress and encrypt the data; connect to the Hive database using the pyhive library; and write the compressed and encrypted data into the Hive table; The third processing module is configured to clean the data in the Hive database through Spark; apply the Python algorithm to analyze the cleaned data, and return the analyzed data and the date of the data; The fourth processing module is configured to display the cleaned data, analysis data and data date in the form of data tables and bar charts through Excel; and set an alarm mechanism according to preset alarm rules and thresholds.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the steps in a full-link intelligent root cause analysis method based on power grid big data described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored on the computer-readable storage medium. When the computer program is executed by the processor, the steps in the full-link intelligent root cause analysis method based on power grid big data described in any one of claims 1 to 7 are implemented.

Images