A dynamic modeling method and system for enterprise-level massive power data search engine

Abstract

The present invention relates to a method for dynamic modeling of a massive power data search engine at an enterprise level, comprising the following steps: step S1: collecting power data based on sensors or meters, integrating the collected power data into network devices, configuring sFlow agents on the network devices, and regularly generating sFlow data packets; step S2: encrypting and transmitting sFlow data packets to a data storage platform based on TLS / SSL protocols; step S3: the data storage platform uses HBase as a distributed database and uses Elasticsearch to build a full-text index; step S4: dynamically modeling power data using machine learning and data mining algorithms, and using streaming computing and real-time data processing technology to update and optimize the dynamic modeling results in real time; step S5: displaying dynamic modeling results and search engine query results through data visualization technology, so that users can perform data analysis and decision-making. The present invention realizes real-time collection, processing and analysis of power data, and ensures high performance and availability of the system.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to an enterprise-level massive power data search engine dynamic modeling method and system. Background Art

[0002] The massive power data search engine needs to ensure the real-time performance of the data, but some search engine technologies currently face certain challenges in processing real-time data, and need to better support the indexing and retrieval of real-time data. As the scale of power data continues to increase, the existing data management engine cannot meet the storage and processing needs of massive data. Traditional relational databases or file systems cannot carry such large-scale data, resulting in insufficient data storage and processing capabilities. In addition, the existing data management engine has performance bottlenecks and insufficient real-time performance when processing large-scale real-time data, which seriously affects the stability and reliability of power grid operation. Summary of the invention

[0003] In order to solve the above problems, the purpose of the present invention is to provide an enterprise-level massive power data search engine dynamic modeling method to achieve real-time collection, processing and analysis of power data, cope with the ever-increasing data volume and query load, and ensure the high performance and availability of the system.

[0004] To achieve the above object, the present invention adopts the following technical solutions:

[0005] A method for dynamic modeling of an enterprise-level massive power data search engine comprises the following steps:

[0006] Step S1: Collect power data based on sensors or meters, integrate the collected power data into network devices, configure sFlow agents on network devices, and regularly generate sFlow data packets, which contain the collected power data and related network traffic information;

[0007] Step S2: Encrypt the sFlow data packet based on the TLS / SSL protocol and transmit it to the data storage platform;

[0008] Step S3: The data storage platform uses HBase as a distributed database to store metadata information and indexes of power data, and stores the metadata information and index information of power data in HBase; uses Elasticsearch to build a full-text index, supporting fast retrieval by multiple dimensions such as time, sensor ID, and geographic location;

[0009] Step S4: Use machine learning and data mining algorithms to dynamically model the power data, including time series analysis, prediction models, and anomaly detection, and use streaming computing and real-time data processing technology to update and optimize the dynamic modeling results in real time;

[0010] Step S5: Display dynamic modeling results and search engine query results through data visualization technology to facilitate users to conduct data analysis and decision-making.

[0011] Furthermore, the step S1 is specifically as follows:

[0012] Collect power data based on sensors or meters, and transmit the collected power data to network devices through Modbus communication protocol;

[0013] An sFlow agent is configured on the network device so that it can generate sFlow data packets regularly. The sFlow agent integrates the collected power data and related network flow information into the sFlow data packets according to predefined rules. The sFlow data packets should contain the collected power data and related network flow information.

[0014] sFlow packets are transmitted to the specified sFlow collector through the UDP protocol.

[0015] Furthermore, the step S2 is specifically as follows:

[0016] First, establish the certificates required for TLS / SSL communication between the data storage platform and the sFlow agent. In the sFlow agent configuration, enable the TLS / SSL encryption option and configure the certificate, private key, and other related TLS / SSL parameters. In the data storage platform configuration, enable the TLS / SSL option and configure the certificate, private key, and other related TLS / SSL parameters.

[0017] The encrypted sFlow data packets generated by the sFlow agent are transmitted to the data storage platform via the TLS / SSL protocol. The data storage platform uses the corresponding certificate and private key to decrypt and verify the received data packets.

[0018] Furthermore, the sFlow data packet is encrypted and transmitted through the TLS / SSL protocol, and a virtual private network tunnel is established to encrypt and transmit the sFlow data packet, as follows:

[0019] Use Diffie-Hellman to exchange keys. During this process, the two communicating parties negotiate a symmetric key for subsequent data encryption and decryption:

[0020] The communicating parties need to select two large prime numbers p and g, where p is a large prime number and g is a primitive root of p. The two parameters can be used as public parameters for use by the communicating parties.

[0021] Key negotiation process:

[0022] The sFlow agent selects a private random number a and calculates the public key A:

[0023] A=g^amodp

[0024] The data storage platform selects a private random number b and calculates the public key B:

[0025] B=g^b modp

[0026] The sFlow agent sends public key A to the data storage platform, and the data storage platform sends public key B to the sFlow agent.

[0027] Key generation process:

[0028] The sFlow agent uses the public key B sent by the data storage platform and its own private key a to calculate the shared key:

[0029] s=B^a modp

[0030] The data storage platform uses the public key A sent by the sFlow agent and its own private key b to calculate the shared key:

[0031] s=A^b modp

[0032] Use a symmetric encryption algorithm to encrypt communication data. During the encryption process, the negotiated symmetric key is used to encrypt the data:

[0033] The sFlow agent uses the symmetric key s to encrypt the plaintext data to obtain the ciphertext data;

[0034] Bob uses the same symmetric key s to decrypt the ciphertext data and obtain the original plaintext data

[0035] During the encryption process, the formula for encrypting data using the symmetric key s is: Ciphertext = AES_Encrypt (Plaintext, s)

[0036] During the decryption process, the formula for decrypting data using the symmetric key s is: Plaintext = AES_Decrypt (Ciphertext, s)

[0037] AES_Encrypt means using the AES algorithm to encrypt plaintext data, AES_Decrypt means using the AES algorithm to decrypt ciphertext data, s means the negotiated symmetric key, Plaintext means the original plaintext data, and Ciphertext means the encrypted ciphertext data;

[0038] Use message authentication codes to protect the integrity of communication data:

[0039] Let the encrypted data be Ciphertext, the message authentication code be MAC, the symmetric key be s, and the HMAC algorithm use the SHA-256 hash function

[0040] Message authentication code generation process: MAC = HMAC_SHA256 (Ciphertext, s)

[0041] Message authentication code verification process: MAC'=HMAC_SHA256(Ciphertext,s) Compare MAC and MAC', if they are equal, the integrity of the communication data is verified.

[0042] Furthermore, the step S3 is specifically as follows:

[0043] The metadata information of power data is stored in HBase. This information is used as the key field for retrieval. A table structure is created in HBase to store metadata information and index information. The metadata information is stored in the metadata column family, and the index information is stored in the index column family. The row key of each record is the timestamp. The metadata column family stores sensor identification and geographic location metadata information, and the index column family stores timestamp and data type index information.

[0044] Synchronize the metadata and index information stored in HBase to Elasticsearch, build a full-text index, create indexes and mappings in Elasticsearch, and support fast retrieval by time, sensor ID, and geographic location.

[0045] Furthermore, the metadata information and index information stored in HBase are synchronized to Elasticsearch to construct a full-text index, as follows:

[0046] Establish a connection between Elasticsearch and HBase, use HBase's API to read data from the power_metadata table, and assume that hbase.read_power_metadata() is a function that reads data from the power_metadata table from HBase and returns a dictionary in which the key is the row key and the value is a dictionary containing metadata information and index information.

[0047] Create an empty list bulk_data to store the data inserted into Elasticsearch in batches, traverse the data read from HBase, build a corresponding Elasticsearch document for each row of data, and assume that the metadata information includes the timestamp timestamp, sensor identifier sensor_id, geographic location location, and data type data_type fields;

[0048] For each row of data, convert it into Elasticsearch document format and add it to the bulk_data list. Each document includes the index name, row key, and document content.

[0049] Use Elasticsearch's Bulk API to batch insert the data in bulk_data into an index named power_metadata and set refresh to True to ensure that the data is immediately available.

[0050] An index_mapping dictionary is defined, which contains the mapping information of the power_metadata index;

[0051] Use the indices.create method of Elasticsearch to create an index named power_metadata and set its mapping to the mapping defined in index_mapping.

[0052] Furthermore, the step S3 is specifically as follows:

[0053] Modeling of power data based on ARIMA, and analysis of historical data to obtain seasonality and trend patterns of power data, thus predicting future power demand and usage;

[0054] Based on the outlier detection algorithm, anomaly detection is performed on power data to find abnormal points in the data;

[0055] The modeling results are stored in Elasticsearch to achieve real-time query and analysis of data. When the real-time data changes, the updated data is written into the index in Elasticsearch in real time through the streaming computing platform.

[0056] Monitor dynamic modeling results based on Elasticsearch, monitor anomalies and trend changes in dynamic modeling results, and immediately trigger an alarm or take corresponding real-time adjustment measures once an anomaly is found.

[0057] Furthermore, the dynamic modeling results are monitored based on Elasticsearch, as follows:

[0058] Based on the aggregation function and query statements of Elasticsearch, the dynamic modeling results are monitored regularly. By writing scheduled tasks or using the Watcher function of Elasticsearch, query and aggregation operations are performed regularly to monitor abnormal conditions and trend changes in sensor data.

[0059] In Elasticsearch, the aggregation function is used to calculate the statistical indicators of sensor data, including the average, standard deviation, maximum, and minimum values, to monitor data anomalies. At the same time, thresholds are set. When the data exceeds the preset threshold, it is considered an abnormal situation.

[0060] Set Watcher alert rules to trigger alerts and notify relevant personnel when monitoring indicators exceed set thresholds or meet specific conditions.

[0061] Furthermore, the step S5 is specifically as follows:

[0062] Use Kibana to create dynamic dashboards to display dynamic modeling results, real-time trends of sensor data, anomaly detection results, and prediction model output information;

[0063] Use Kibana's search function and Elasticsearch's full-text search capability to display search engine query results. Users can enter keywords or conditions in Kibana's search box to retrieve data related to metadata information, and display query results in charts and tables.

[0064] Use Kibana's reporting function to regularly generate reports on dynamic modeling results and send them to relevant personnel via email or other means. At the same time, set alarm rules to automatically trigger alarms to notify relevant personnel when data exceeds preset thresholds or conditions.

[0065] An enterprise-level massive power data search engine dynamic modeling system includes a processor, a memory, and a computer program stored in the memory. When the processor executes the computer program, it specifically executes the steps in the enterprise-level massive power data search engine dynamic modeling method as described above.

[0066] The present invention has the following beneficial effects:

[0067] 1. The present invention realizes real-time collection, processing and analysis of power data, copes with the ever-increasing data volume and query load, ensures high performance and availability of the system, realizes multi-dimensional analysis and query, improves decision-making efficiency and accuracy, and reduces the cost and error of manual intervention;

[0068] 2. The present invention uses HBase to store metadata information and index information of power data, and uses Elasticsearch to construct full-text indexes, supports multi-dimensional rapid retrieval, realizes efficient data storage and retrieval, and meets the needs of power data analysis and query;

[0069] 3. Monitoring dynamic modeling results based on Elasticsearch can help users discover abnormal situations in a timely manner and take corresponding measures to ensure the stable operation of the system and data quality. At the same time, combined with data visualization tools, users can intuitively understand the monitoring results, so as to make better real-time decisions and adjustments; BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0071] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0072] refer to Figure 1 In this embodiment, a method for dynamic modeling of an enterprise-level massive power data search engine includes the following steps:

[0073] Step S1: Collect power data based on sensors or meters, integrate the collected power data into network devices, configure sFlow agents on network devices, and regularly generate sFlow data packets, which contain the collected power data and related network traffic information;

[0074] Step S2: Encrypt the sFlow data packet based on the TLS / SSL protocol and transmit it to the data storage platform;

[0075] Step S3: The data storage platform uses HBase as a distributed database to store metadata information and indexes of power data, and stores metadata information (such as timestamp, sensor ID, data type, etc.) and index information (such as data location, partition information, etc.) of power data in HBase; HBase provides fast random read and write capabilities, suitable for storing large amounts of metadata information and index data. Metadata information (such as timestamp, sensor ID, data type, etc.) and index information (such as data location, partition information, etc.) of power data are stored in HBase. Use Elasticsearch to build a full-text index, supporting fast retrieval by multiple dimensions such as time, sensor ID, and geographic location;

[0076] Step S4: Use machine learning and data mining algorithms to dynamically model the power data, including time series analysis, prediction models, and anomaly detection, and use streaming computing and real-time data processing technology to update and optimize the dynamic modeling results in real time;

[0077] Step S5: Display dynamic modeling results and search engine query results through data visualization technology to facilitate users to conduct data analysis and decision-making.

[0078] In this embodiment, the step S1 is specifically as follows:

[0079] Collect power data based on sensors or meters, and transmit the collected power data to network devices through Modbus communication protocol;

[0080] An sFlow agent is configured on the network device so that it can generate sFlow data packets regularly. The sFlow agent integrates the collected power data and related network traffic information into the sFlow data packets according to predefined rules; the sFlow data packets should contain the collected power data (such as current, voltage, power) and related network traffic information (such as source IP, target IP, port number, traffic size, etc.);

[0081] sFlow packets are transmitted via UDP protocol to the designated sFlow collector (such as sFlow-RT, Elasticsearch, etc.) for further processing and analysis.

[0082] In this embodiment, the step S2 is specifically as follows:

[0083] First, establish the certificates required for TLS / SSL communication between the data storage platform and the sFlow agent. You can choose a self-signed certificate or obtain a certificate from a trusted certificate authority (CA).

[0084] In the configuration of the sFlow agent, enable the TLS / SSL encryption option and configure the certificate, private key, and other related TLS / SSL parameters. This ensures that the sFlow agent can use the TLS / SSL protocol to encrypt the generated sFlow packets. In the configuration of the data storage platform, enable the TLS / SSL option and configure the certificate, private key, and other related TLS / SSL parameters; this ensures that the data storage platform can receive and decrypt sFlow packets transmitted via the TLS / SSL protocol.

[0085] The encrypted sFlow data packets generated by the sFlow agent are transmitted to the data storage platform via the TLS / SSL protocol. The data storage platform uses the corresponding certificate and private key to decrypt and verify the received data packets.

[0086] In this embodiment, the sFlow data packet is encrypted and transmitted through the TLS / SSL protocol, and a virtual private network tunnel is established to encrypt and transmit the sFlow data packet, as follows:

[0087] Use Diffie-Hellman to exchange keys. During this process, the two communicating parties negotiate a symmetric key for subsequent data encryption and decryption:

[0088] The communicating parties need to select two large prime numbers p and g, where p is a large prime number and g is a primitive root of p. The two parameters can be used as public parameters for use by the communicating parties.

[0089] Key negotiation process:

[0090] The sFlow agent selects a private random number a and calculates the public key A:

[0091] A=g^amodp

[0092] The data storage platform selects a private random number b and calculates the public key B:

[0093] B=g^b modp

[0094] The sFlow agent sends public key A to the data storage platform, and the data storage platform sends public key B to the sFlow agent.

[0095] Key generation process:

[0096] The sFlow agent uses the public key B sent by the data storage platform and its own private key a to calculate the shared key:

[0097] s=B^a modp

[0098] The data storage platform uses the public key A sent by the sFlow agent and its own private key b to calculate the shared key:

[0099] s=A^b modp

[0100] Use a symmetric encryption algorithm to encrypt communication data. During the encryption process, the negotiated symmetric key is used to encrypt the data:

[0101] The sFlow agent uses the symmetric key s to encrypt the plaintext data to obtain the ciphertext data;

[0102] Bob uses the same symmetric key s to decrypt the ciphertext data and obtain the original plaintext data

[0103] During the encryption process, the formula for encrypting data using the symmetric key s is: Ciphertext = AES_Encrypt (Plaintext, s)

[0104] During the decryption process, the formula for decrypting data using the symmetric key s is: Plaintext = AES_Decrypt (Ciphertext, s)

[0105] AES_Encrypt means using the AES algorithm to encrypt plaintext data, AES_Decrypt means using the AES algorithm to decrypt ciphertext data, s means the negotiated symmetric key, Plaintext means the original plaintext data, and Ciphertext means the encrypted ciphertext data;

[0106] Use message authentication codes to protect the integrity of communication data:

[0107] Let the encrypted data be Ciphertext, the message authentication code be MAC, the symmetric key be s, and the HMAC algorithm use the SHA-256 hash function

[0108] Message authentication code generation process: MAC = HMAC_SHA256 (Ciphertext, s)

[0109] Message authentication code verification process: MAC'=HMAC_SHA256(Ciphertext,s) Compare MAC and MAC', if they are equal, the integrity of the communication data is verified.

[0110] In this embodiment, step S3 is specifically as follows:

[0111] The metadata information of power data is stored in HBase. This information is used as the key field for retrieval. A table structure is created in HBase to store metadata information and index information. The metadata information is stored in the metadata column family, and the index information is stored in the index column family. The row key of each record is the timestamp. The metadata column family stores sensor identification and geographic location metadata information, and the index column family stores timestamp and data type index information.

[0112] Synchronize the metadata and index information stored in HBase to Elasticsearch, build a full-text index, create indexes and mappings in Elasticsearch, and support fast retrieval by time, sensor ID, and geographic location.

[0113] In this embodiment, the metadata information and index information stored in HBase are synchronized to Elasticsearch to construct a full-text index, as follows:

[0114] Establish a connection between Elasticsearch and HBase, use HBase's API to read data from the power_metadata table, and assume that hbase.read_power_metadata() is a function that reads data from the power_metadata table from HBase and returns a dictionary in which the key is the row key and the value is a dictionary containing metadata information and index information.

[0115] Create an empty list bulk_data to store the data inserted into Elasticsearch in batches, traverse the data read from HBase, build a corresponding Elasticsearch document for each row of data, and assume that the metadata information includes the timestamp timestamp, sensor identifier sensor_id, geographic location location, and data type data_type fields;

[0116] For each row of data, convert it into Elasticsearch document format and add it to the bulk_data list. Each document includes the index name, row key, and document content.

[0117] Use Elasticsearch's Bulk API to batch insert the data in bulk_data into an index named power_metadata and set refresh to True to ensure that the data is immediately available.

[0118] An index_mapping dictionary is defined, which contains the mapping information of the power_metadata index; in this example, we specify that the timestamp field is of date type, the sensor identification field is of keyword type, the geographic location field is of geographic point type, the data type field is of keyword type, and so on.

[0119] Use the indices.create method of Elasticsearch to create an index named power_metadata and set its mapping to the mapping defined in index_mapping.

[0120] In this embodiment, step S3 is specifically as follows:

[0121] Modeling of power data based on ARIMA, and analysis of historical data to obtain seasonality and trend patterns of power data, thus predicting future power demand and usage;

[0122] Based on the outlier detection algorithm, anomaly detection is performed on power data to find abnormal points in the data, such as sudden increase or decrease in power consumption, so that timely processing and adjustments can be made.

[0123] The modeling results are stored in Elasticsearch to achieve real-time query and analysis of data. When the real-time data changes, the updated data is written into the index in Elasticsearch in real time through the streaming computing platform.

[0124] Monitor dynamic modeling results based on Elasticsearch, monitor anomalies and trend changes in dynamic modeling results, and immediately trigger an alarm or take corresponding real-time adjustment measures once an anomaly is found.

[0125] In this embodiment, the dynamic modeling results are monitored based on Elasticsearch as follows:

[0126] Based on the aggregation function and query statements of Elasticsearch, the dynamic modeling results are monitored regularly. By writing scheduled tasks or using the Watcher function of Elasticsearch, query and aggregation operations are performed regularly to monitor abnormal conditions and trend changes in sensor data.

[0127] In Elasticsearch, the aggregation function is used to calculate the statistical indicators of sensor data, including the average, standard deviation, maximum, and minimum values, to monitor data anomalies. At the same time, thresholds are set. When the data exceeds the preset threshold, it is considered an abnormal situation.

[0128] Set up Watcher alert rules to trigger alerts and notify relevant personnel when monitoring indicators exceed set thresholds or meet specific conditions. You can choose to send alert information via email, Slack, etc., or call external systems to make real-time adjustments. Once an abnormal situation is found, the corresponding real-time adjustment measures can be triggered through Watcher alerts. For example, you can call external systems to perform automated operations, such as shutting down abnormal sensor equipment, adjusting data collection frequency, sending control instructions, etc., and notify relevant personnel to intervene manually.

[0129] In this embodiment, step S5 is specifically as follows:

[0130] Use Kibana to create dynamic dashboards to display dynamic modeling results, real-time trends of sensor data, anomaly detection results, and prediction model output information;

[0131] Use Kibana's search function and Elasticsearch's full-text search capability to display search engine query results. Users can enter keywords or conditions in Kibana's search box to retrieve data related to metadata information, and display query results in charts and tables.

[0132] Use Kibana's reporting function to regularly generate reports on dynamic modeling results and send them to relevant personnel via email or other means. At the same time, set alarm rules to automatically trigger alarms to notify relevant personnel when data exceeds preset thresholds or conditions.

[0133] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0134] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0135] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0136] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0137] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A dynamic modeling method for enterprise-level massive power data search engine, characterized in that: The following steps are involved: Step S1: Collect power data based on sensors or meters, integrate the collected power data into network devices, configure sFlow agents on network devices, and regularly generate sFlow data packets, which contain the collected power data and related network traffic information; Step S2: Encrypt the sFlow data packet based on the TLS / SSL protocol and transmit it to the data storage platform; Step S3: The data storage platform uses HBase as a distributed database to store metadata information and indexes of power data, and stores the metadata information and index information of power data in HBase; uses Elasticsearch to build a full-text index, supporting fast retrieval by multiple dimensions such as time, sensor ID, and geographic location; The step S3 is specifically as follows: The metadata information of power data is stored in HBase. This information is used as the key field for retrieval. A table structure is created in HBase to store metadata information and index information. The metadata information is stored in the metadata column family, and the index information is stored in the index column family. The row key of each record is the timestamp. The metadata column family stores sensor identification and geographic location metadata information, and the index column family stores timestamp and data type index information. Synchronize metadata and index information stored in HBase to Elasticsearch, build a full-text index, create indexes and mappings in Elasticsearch, and support fast retrieval by time, sensor ID, and geographic location. The metadata information and index information stored in HBase are synchronized to Elasticsearch to build a full-text index, as follows: Establish a connection between Elasticsearch and HBase, and use the HBase API to read data from the power_metadata table. Let hbase.read_power_metadata() be a function that reads data from the power_metadata table in HBase and returns a dictionary where the key is the row key and the value is a dictionary containing metadata information and index information. Create an empty list bulk_data to store the data to be inserted into Elasticsearch in batches. Traverse the data read from HBase and build a corresponding Elasticsearch document for each row of data. Set the metadata information to include the timestamp timestamp, sensor ID sensor_id, geographic location location, and data type data_type fields. For each row of data, convert it to Elasticsearch document format and add it to the bulk_data list. Each document includes the index name, row key, and document content. Use Elasticsearch's Bulk API to batch insert the data in bulk_data into an index named power_metadata, and set refresh to True to ensure that the data is immediately available; An index_mapping dictionary is defined, which contains the mapping information of the power_metadata index; Use the indices.create method of Elasticsearch to create an index named power_metadata and set its mapping to the mapping defined in index_mapping; Step S4: Use machine learning and data mining algorithms to dynamically model the power data, including time series analysis, prediction models, and anomaly detection, and use streaming computing and real-time data processing technology to update and optimize the dynamic modeling results in real time; Step S5: Display dynamic modeling results and search engine query results through data visualization technology to facilitate users to perform data analysis and decision-making; The step S4 is specifically as follows: Modeling of power data based on ARIMA, and analysis of historical data to obtain seasonality and trend patterns of power data, thus predicting future power demand and usage; Based on the outlier detection algorithm, anomaly detection is performed on power data to find abnormal points in the data; The modeling results are stored in Elasticsearch to achieve real-time query and analysis of data. When the real-time data changes, the updated data is written into the index in Elasticsearch in real time through the streaming computing platform. Monitor dynamic modeling results based on Elasticsearch, monitor abnormalities and trend changes in dynamic modeling results, and immediately trigger alarms or take corresponding real-time adjustment measures once abnormalities are found; The monitoring of dynamic modeling results based on Elasticsearch is as follows: Based on the aggregation function and query statements of Elasticsearch, the dynamic modeling results are monitored regularly. By writing scheduled tasks or using the Watcher function of Elasticsearch, query and aggregation operations are performed regularly to monitor abnormal conditions and trend changes in sensor data. In Elasticsearch, the aggregation function is used to calculate the statistical indicators of sensor data, including the average, standard deviation, maximum, and minimum values, to monitor data anomalies. At the same time, thresholds are set. When the data exceeds the preset threshold, it is considered an abnormal situation. Set Watcher alert rules to trigger alerts and notify relevant personnel when monitoring indicators exceed set thresholds or meet specific conditions.

2. According to claim 1, a method for dynamic modeling of enterprise-level massive power data search engine is characterized in that: The step S1 is specifically as follows: Collect power data based on sensors or meters, and transmit the collected power data to network devices through Modbus communication protocol; Configure sFlow agents on network devices to enable them to generate sFlow packets regularly. The sFlow agents integrate the collected power data and related network traffic information into sFlow packets according to pre-defined rules. The sFlow data packet should contain the collected power data and related network traffic information; sFlow packets are transmitted to the specified sFlow collector through the UDP protocol.

3. The enterprise-level massive power data search engine dynamic modeling method according to claim 1 is characterized in that: The step S2 specifically comprises: First, establish the certificates required for TLS / SSL communication between the data storage platform and the sFlow agent; In the sFlow agent configuration, enable the TLS / SSL encryption option and configure the certificate, private key, and other related TLS / SSL parameters; In the data storage platform configuration, enable the TLS / SSL option and configure the certificate, private key, and other related TLS / SSL parameters; The encrypted sFlow data packets generated by the sFlow agent are transmitted to the data storage platform via the TLS / SSL protocol. The data storage platform uses the corresponding certificate and private key to decrypt and verify the received data packets.

4. The enterprise-level massive power data search engine dynamic modeling method according to claim 3 is characterized in that: The sFlow data packet is encrypted and transmitted through the TLS / SSL protocol, and a virtual private network tunnel is established to encrypt and transmit the sFlow data packet, as follows: Use Diffie-Hellman to exchange keys. During this process, the two communicating parties negotiate a symmetric key for subsequent data encryption and decryption: The two communicating parties need to select two large prime numbers p and g, where p is a large prime number and g is a primitive root of p. The two parameters are public parameters for use by both communicating parties. Key negotiation process: The sFlow agent selects a private random number a and calculates the public key A: A = g^a mod p The data storage platform selects a private random number b and calculates the public key B: B = g^b mod p The sFlow agent sends public key A to the data storage platform, and the data storage platform sends public key B to the sFlow agent. Key generation process: The sFlow agent uses the public key B sent by the data storage platform and its own private key a to calculate the shared key: s = B^a mod p The data storage platform uses the public key A sent by the sFlow agent and its own private key b to calculate the shared key: s = A^b mod p Use a symmetric encryption algorithm to encrypt communication data. During the encryption process, the negotiated symmetric key is used to encrypt the data: The sFlow agent uses the symmetric key s to encrypt the plaintext data to obtain the ciphertext data; Bob uses the same symmetric key s to decrypt the ciphertext data and obtain the original plaintext data During the encryption process, the formula for encrypting data using the symmetric key s is: Ciphertext = AES_Encrypt(Plaintext, s) During the decryption process, the formula for decrypting data using the symmetric key s is: Plaintext = AES_Decrypt(Ciphertext, s) AES_Encrypt means using the AES algorithm to encrypt plaintext data, AES_Decrypt means using the AES algorithm to decrypt ciphertext data, s means the negotiated symmetric key, Plaintext means the original plaintext data, and Ciphertext means the encrypted ciphertext data; Use message authentication codes to protect the integrity of communication data: Let the encrypted data be Ciphertext, the message authentication code be MAC, the symmetric key be s, and the HMAC algorithm use the SHA-256 hash function Message authentication code generation process: MAC = HMAC_SHA256(Ciphertext, s) Message authentication code verification process: MAC' = HMAC_SHA256(Ciphertext, s) Compare MAC and MAC', if they are equal, the integrity of the communication data is verified.

5. The enterprise-level massive power data search engine dynamic modeling method according to claim 1 is characterized in that: The step S5 is specifically as follows: Use Kibana to create dynamic dashboards to display dynamic modeling results, real-time trends of sensor data, anomaly detection results, and prediction model output information; Use Kibana's search function and Elasticsearch's full-text search capability to display search engine query results. Users can enter keywords or conditions in Kibana's search box to retrieve data related to metadata information, and display query results in charts and tables. Use Kibana's reporting function to regularly generate reports on dynamic modeling results and send them to relevant personnel via email or other means. At the same time, set alarm rules to automatically trigger alarms to notify relevant personnel when data exceeds preset thresholds or conditions.

6. An enterprise-level massive power data search engine dynamic modeling system, characterized in that: It includes a processor, a memory and a computer program stored in the memory. When the processor executes the computer program, it specifically executes the steps in the enterprise-level massive power data search engine dynamic modeling method as described in any one of claims 1-5.

Images