A power grid data integration method and system

Abstract

The application discloses a kind of power grid data integration method and system, method includes constructing basic environment, configuration cluster client and establish cluster storage main body;Build column database and task channel, and according to the cluster storage main body information of task channel acquisition, pre-process measurement data, and the measurement data after pre-processing is imported into the measurement table in column database;According to the data information of measurement table, data mapping is carried out, and according to data mapping result, it is stored and counted.The application can gather large amount and high frequency measurement data, and compare the difference of memory difference, to ensure the uniqueness of measurement data, and through giving unique identification, it is quickly positioned, and the upper limit of setting write cycle and classification capacity, every other write cycle or over classification capacity reaches classification capacity upper limit, measurement table is written into column database, guarantee the timeliness of measurement data writing and improve the writing efficiency of measurement data.

Description

Technical Field

[0001] This invention relates to the field of data processing, and specifically to a method and system for integrating power grid data. Background Technology

[0002] Power grid measurement data mainly refers to real-time / near real-time operating data (such as voltage, current, power, frequency), metering data (electricity consumption, load curves), and equipment status data collected by devices such as sensors, smart meters, SCADA systems, and PMUs (synchronous phasor measurement units).

[0003] Currently, in the process of integrating power grid measurements, ETL tools are used to periodically extract data from various measurement systems, such as metering automation systems and power energy acquisition systems. After cleaning and transformation, the data is stored in a relational database to form a historical measurement data center, which facilitates report generation and offline modeling.

[0004] However, in the traditional ETL batch processing mode, the delay from the acquisition of measurement data to its application is 5-10 minutes, and the data writing speed is slow. Summary of the Invention

[0005] Based on this, the purpose of this invention is to provide a power grid data integration method and system, which aims to solve the problem that the delay from acquisition to application of measurement data is 5-10 minutes and the data writing speed is slow under the current traditional ETL batch processing mode.

[0006] To achieve the above objectives, the present invention proposes a power grid data integration method, the power grid data integration method comprising:

[0007] Build the basic environment, configure the cluster client, and establish the cluster storage entity;

[0008] A columnar database and task channel are established, and cluster storage entity information is obtained according to the task channel. Measurement data is preprocessed and imported into the measurement table in the columnar database. The measurement table is parsed, classified, and enumerated information is defined. The enumeration information definition process includes real-time enumeration mining, predictive enumeration definition, and dynamic adjustment of association weights. Implicit enumeration candidate values ​​are extracted as basic features through real-time enumeration mining, and these implicit enumeration candidate values ​​are synchronized to the input layer of the prediction model in the predictive enumeration definition. The future enumeration probability distribution generated by the predictive enumeration predefined is used as a forward-looking feature to feed back into the real-time mining direction. The cluster storage entity load feature in the dynamic adjustment of association weights is used as a constraint feature to limit the storage priority of enumeration values.

[0009] Data mapping is performed based on the data information in the measurement table, the data mapping results are transferred to the entity table, and consistency statistics are performed on the measurement table and the entity table.

[0010] According to one aspect of the above technical solution, the steps of constructing the basic environment, configuring the cluster client, and establishing the cluster storage entity include:

[0011] Configure a big data infrastructure environment, support compilation of Java and Scala development languages, and support deployment of development environment jobs according to the cluster client;

[0012] Based on the actual data volume of the measurement data and business rules, create the cluster storage entity, data storage duration, and partition size.

[0013] According to one aspect of the above technical solution, the steps of building a columnar database and task channel, obtaining cluster storage entity information based on the task channel, preprocessing measurement data, importing the preprocessed measurement data into the measurement table in the columnar database, parsing and classifying the measurement table, and defining enumeration information, the enumeration information definition process including real-time enumeration mining, predictive enumeration definition, and dynamic adjustment of association weights, extracting implicit enumeration candidate values ​​as basic features through real-time enumeration mining, synchronizing these implicit enumeration candidate values ​​to the input layer of the prediction model in the predictive enumeration definition, using the future enumeration probability distribution generated by the predictive enumeration predefined as a forward-looking feature to feed back into the real-time mining direction, and using the cluster storage entity load characteristics in the dynamic adjustment of association weights as constraint features to limit the storage priority of enumeration values, include:

[0014] Design the table name of the columnar database for the measurement information of any measurement table, set the partition size of the measurement information data parsing process and the table fields of the measurement table, and complete the automatic creation of the measurement table.

[0015] Based on the information collected by the cluster client and the business information of the measurement table, the uniqueness of the current measurement table is analyzed, and a unique identifier for each row of measurement data in the measurement table is generated.

[0016] Based on one aspect of the above technical solution, a series of parsers for the data parsing process are designed for the data information of the measurement table, and a communication path for the cluster storage entity is constructed.

[0017] Based on the data volume of the measurement table, checkpoints are constructed in the processing framework environment, and the cluster storage entity information in the measurement table is subscribed to in real time.

[0018] According to one aspect of the above technical solution, the measurement data in the measurement table is parsed through a processing framework, a domain knowledge graph is introduced, the measurement data in the measurement table is parsed through the domain knowledge graph, data lineage tracing is used to verify the field logic of the related tables of the current measurement table, and the isolated forest algorithm is used to identify the degree of matching between the measurement data in the current measurement table and the columnar database, filter out normal data and abnormal data, classify the time period to which the normal data belongs, and merge the abnormal data into the same time period.

[0019] The enumeration information of the categorized measurement tables is defined, and the information is written into the columnar database of the corresponding table fields based on the association between the enumeration information and the cluster storage entity. The table creation rules of the measurement tables are analyzed, and the measurement tables written into the columnar database and the table fields of the corresponding columnar database are verified. If the content of the measurement table is consistent with the table fields, the writing is successful; otherwise, a columnar database matching the measurement table is automatically created.

[0020] Set the write cycle and the classification capacity limit. Write the measurement table to the columnar database every write cycle or when the classification capacity reaches the classification capacity limit.

[0021] According to one aspect of the above technical solution, the steps of performing data mapping based on the data information of the measurement table, transferring the data mapping result to the entity table, and performing consistency statistics on the measurement table and the entity table include:

[0022] Based on the data information of the measurement table and according to the custom rules, a uniquely identified entity table is generated, which covers the maximum data range that appears in the columnar database.

[0023] The big data processing engine reads the data content of the measurement table from different table fields in the columnar database and uses custom rules to transfer the data content of the measurement table to the entity table with a unique identifier.

[0024] Consistency statistics are performed on the data content of the measurement table and the data content of the uniquely identified entity table.

[0025] According to one aspect of the above technical solution, during the process of transferring and statistically analyzing the data information of the measurement table, the data information of the measurement table is monitored.

[0026] At a minimum, the cluster storage entity information and data service information of the measurement table should be collected. The number of records in the measurement table should be counted in real time through the processing framework for real-time data monitoring. At the same time, the time period for writing the data information of the measurement table into the columnar database should be captured to monitor the timeliness of the data.

[0027] This invention also proposes a power grid data integration system, which is used to implement the above-mentioned power grid data integration method. The system includes:

[0028] The configuration module is used to build the basic environment, configure cluster clients, and establish the cluster storage entity.

[0029] The import module is used to build a columnar database and task channel, and obtain cluster storage entity information according to the task channel. It preprocesses the measurement data and imports the preprocessed measurement data into the measurement table in the columnar database. The measurement table is parsed, classified, and enumerated information is defined. The enumeration information definition process includes real-time enumeration mining, predictive enumeration definition, and dynamic adjustment of association weights. Implicit enumeration candidate values ​​are extracted as basic features through real-time enumeration mining, and these implicit enumeration candidate values ​​are synchronized to the input layer of the prediction model in the predictive enumeration definition. The future enumeration probability distribution generated by the predictive enumeration predefined is used as a forward-looking feature to feed back into the real-time mining direction. The cluster storage entity load feature in the dynamic adjustment of association weights is used as a constraint feature to limit the storage priority of the enumeration values.

[0030] The transfer module is used to perform data mapping based on the data information of the measurement table, transfer the data mapping result to the entity table, and perform consistency statistics on the measurement table and the entity table.

[0031] The present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the power grid data integration method described above.

[0032] The present invention also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the power grid data integration method as described above.

[0033] In summary, the power grid data integration method proposed by this invention configures a big data infrastructure environment to support compilation in multiple programming languages, configures cluster clients and establishes a cluster storage entity, and sets data storage duration and partition size for the cluster storage entity to accommodate large volumes of data. Simultaneously, a columnar database and task channels are built. Data information from the measurement tables of the cluster storage entity is obtained through the task channels. While maintaining the uniqueness of the measurement tables, the measurement data is aggregated. Based on the write cycle and classification capacity limit rules, the aggregated measurement data is written into the measurement tables in the columnar database. The measurement tables are then parsed, classified, and enumerated with information definitions. The information definition process includes real-time enumeration mining, predictive enumeration definition, and dynamic adjustment of association weights. Real-time enumeration mining extracts latent enumeration candidate values ​​as basic features, which are then synchronized to the input layer of the prediction model in the predictive enumeration definition. The future enumeration probability distribution generated by the predictive enumeration predefined is used as a forward-looking feature to feed back into the real-time mining direction. The cluster storage main load characteristics in the dynamic adjustment of association weights are used as constraint features to limit the storage priority of enumeration values. Data mapping is performed based on the data information in the measurement table, and the data mapping results are transferred to the entity table. Consistency statistics are performed on the measurement table and the entity table to ensure the accuracy of the transfer process.

[0034] This invention achieves feature sharing through data interoperability, using current values, future values, and storage rules to generate a complete enumeration information system. Accuracy and efficiency are continuously optimized through a feedback loop. The enumeration information definition of the measurement table not only enables static field labeling but also dynamically responds to business changes and adapts to future needs. Furthermore, it can aggregate large volumes and high-frequency measurement data for post-processing difference comparison to ensure data uniqueness. Unique identifiers are assigned for rapid location. A write cycle and classification capacity limit are set; the measurement table is written to the columnar database every write cycle or when the classification capacity limit is reached, ensuring timely and efficient data writing.

[0035] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0036] Figure 1 This is a flowchart of the power grid data integration method in Embodiment 1 of the present invention;

[0037] Figure 2 This is a schematic diagram of the power grid data integration system in Embodiment 2 of the present invention;

[0038] Figure 3 This is a structural block diagram of the electronic device in Embodiment 4 of the present invention. Detailed Implementation

[0039] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0040] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0041] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0042] Example 1

[0043] like Figure 1 The diagram shows a flowchart of a power grid data integration method according to Embodiment 1 of the present invention. The power grid data integration method includes the following steps S01-S03, wherein:

[0044] S01. Build the basic environment, configure the cluster client and establish the cluster storage entity.

[0045] Building the basic environment includes configuring the big data infrastructure environment, cluster client configuration, and cluster storage entity configuration. The basic big data infrastructure environment configuration includes support for Java and Scala development languages, and necessary dependency packages for accessing services such as MRS-HBASE, MRS-KAFKA, MRS-HDFS, MRS-YARN, MRS-KLINK, and DWS. It also includes an MRS cluster client to support development environment job deployment and application debugging. Finally, it involves creating the cluster storage entity, data storage duration, and partition size according to the actual data volume of the measurement data and business rules.

[0046] S02. Build a columnar database and task channel, and obtain cluster storage entity information according to the task channel. Preprocess the measurement data and import the preprocessed measurement data into the measurement table in the columnar database. Analyze, classify, and define enumeration information in the measurement table. The enumeration information definition process includes real-time enumeration mining, predictive enumeration definition, and dynamic adjustment of association weights. Extract implicit enumeration candidate values ​​as basic features through real-time enumeration mining, and synchronize these implicit enumeration candidate values ​​to the input layer of the prediction model in the predictive enumeration definition. Use the future enumeration probability distribution generated by the predictive enumeration predefined as a forward-looking feature to feed back into the real-time mining direction. Use the cluster storage entity load feature in the dynamic adjustment of association weights as a constraint feature to limit the storage priority of enumeration values.

[0047] Build a columnar database, design the table name for the columnar database for the measurement information on each measurement table, set the partition size for the measurement information data parsing process and the table fields of the measurement table, and complete the automatic creation of measurement tables.

[0048] The system collects information about the corresponding cluster clients from the main storage unit of the cluster, and combines this information with the business information from the measurement tables of the corresponding main storage unit.

[0049] The uniqueness of measurement data in the current measurement table is analyzed to comply with the uniqueness and dispersion of columnar databases, and a unique identifier is generated for each row of measurement data in the measurement table. The generation of the unique identifier for each row of measurement data can employ a "ciphertext + plaintext" approach. The plaintext design involves tracking and analyzing the measurement data for each measurement table, selecting appropriate combinations of business fields for the design. The "ciphertext + plaintext" approach ensures the uniqueness, dispersion, and readability of the columnar database, avoids "hotspot" issues, and improves business usability.

[0050] For the data information in the measurement table, a series of parsers that conform to the actual business situation are designed in the data parsing process, so as to reduce the data deserialization and serialization time and improve the data processing speed when carrying out subsequent data processing.

[0051] The communication path of the cluster storage entity is constructed. Based on the data volume of the measurement table, checkpoints are constructed in the processing framework environment. The status of the measurement data in the measurement table of the columnar database is checked periodically according to the checkpoints, and the cluster storage entity information in the measurement table is subscribed to in real time.

[0052] The processing framework parses the measurement data in the measurement table, filters out abnormal data based on business rules, and performs special processing, such as handling data with a business time of one year ago, incorrect business time format, or empty data. Normal data is categorized according to its time period, while abnormal data is merged into the same time period to ensure all data belongs to a specific day, preparing for subsequent data merging and disk storage, and improving the speed of concurrent data processing. More specifically, during the measurement table parsing process, a domain knowledge graph is introduced. This domain knowledge graph can be the business rules of power systems or intelligent manufacturing. The measurement data in the measurement table is parsed using the domain knowledge graph. Simultaneously, data lineage tracing is used to verify the field logic of the related tables in the current measurement table, and the isolated forest algorithm is used to identify the degree of matching between the measurement data in the current measurement table and the columnar database, avoiding the misclassification of abnormal data as normal data.

[0053] The enumeration information of the categorized measurement tables is defined, and the information is written into the columnar database of the corresponding table fields based on the association between the enumeration information and the cluster storage entity. The table creation rules of the measurement tables are analyzed, and the measurement tables written into the columnar database and the table fields of the corresponding columnar database are verified. If the content of the measurement table is consistent with the table fields, the writing is successful; otherwise, a columnar database matching the measurement table is automatically created.

[0054] In the process of defining enumeration information for the measurement table, the definition of enumeration information can be extended to a dynamic sensing combined with predictive enumeration method, specifically:

[0055] By combining NLP and time series analysis, implicit enumeration values ​​are extracted from historical data of the categorized measurement tables, equipment logs, business documents and other data (such as "unmarked anomaly" of equipment fault codes can be automatically classified as a new enumeration item), and semantic associations of enumeration values ​​are constructed through knowledge graphs (such as "sudden temperature rise" and "sensor failure" are strongly correlated), so as to realize real-time enumeration mining. Specifically, the NLP-based text enumeration extraction process is as follows: After classifying the text data, a named entity recognition model is used to identify "state entities" (such as "voltage overload" and "communication interruption"). Similar entities are merged through word vector clustering (such as Word2Vec+K-Means) (such as "signal loss" and "communication interruption" being grouped into the same enumeration candidate value). The time-series analysis-based anomaly enumeration extraction process is as follows: For the time-series data of the measurement table (such as temperature values ​​recorded every 5 minutes), anomaly fluctuations (such as a sudden temperature rise of 20°C) are detected using isolated forest or DBSCAN. Combined with the time window (such as the anomaly lasting more than 10 minutes), it is determined to be a "persistent anomaly", and a new enumeration value is generated (such as "sudden temperature rise anomaly"). At the same time, high-frequency repeated codes in the device log are automatically marked as "high-frequency undefined enumerations".

[0056] Using predictive enumeration definitions, this method predicts new enumeration values ​​that may appear in the future (such as new states predicted by the fluctuation characteristics of seasonal measurement data) based on time series models such as LSTM. It reserves space for field expansion in advance in the columnar database, reducing the frequency of creating new tables when inconsistencies arise. Specifically, it employs a bidirectional LSTM (Bi-LSTM) + Attention mechanism. The input layer consists of extracted features, including time features, fluctuation features, and correlation features. The output layer is the probability distribution of new enumeration values ​​that may appear in the next 1-3 months (e.g., the probability of "winter low-temperature protection" appearing in December is 0.85). The basic model is trained using full historical data to learn general time series patterns (such as annual seasonal fluctuations). The prediction model is evaluated using accuracy (the proportion of predicted enumeration values ​​that actually appear in the future) and recall (the proportion of actually appearing new enumeration values ​​that are predicted). Model retraining is triggered when the accuracy is lower than a preset accuracy threshold. For high-probability new enumeration values, dynamic fields are set within the reserved field extension space. The field type can be set to a variable-length string to accommodate new enumeration values ​​of different formats. When a new enumeration value is officially entered into the database, the reserved field is automatically renamed, and the field mapping relationship is recorded to avoid historical data query anomalies. When a reserved field conflicts with an existing field, the field renaming rule is automatically triggered.

[0057] The association weights are dynamically adjusted, and reinforcement learning is used to update the association weights of "enumeration information - cluster storage entity" in real time (such as the association degree between a certain type of measurement table and "workshop A storage cluster" increases as the production plan changes), prioritizing the matching of high-weight storage entities and optimizing write efficiency.

[0058] In the process of defining enumeration information, implicit enumeration candidate values ​​are extracted as basic features through real-time enumeration mining, and these implicit enumeration candidate values ​​are synchronized to the input layer of the prediction model. The future enumeration probability distribution generated by the predictive enumeration is used as a forward-looking feature to feed back the focus of real-time mining. Finally, the cluster storage main load feature in the dynamic adjustment of association weights is used as a constraint feature to limit the storage priority of enumeration values. This application achieves feature sharing through data interoperability, and uses the linkage of current values, future values, and storage rule decisions to generate a complete enumeration information system, continuously optimizing accuracy and efficiency through feedback loop. The definition of enumeration information in the measurement table can not only achieve static field labeling, but also dynamically respond to business changes and adapt to future needs.

[0059] A write cycle and a categorization capacity limit are set. Every write cycle or when the categorization capacity limit is reached, the measurement table is written to the columnar database to improve the write speed of measurement data. In this embodiment, the write cycle can be 20 seconds, and the categorization capacity limit can be 3000 records.

[0060] In addition, the program that writes measurement tables to the columnar database in real time can create a connection to the columnar database every 30 minutes to ensure that data writing is running normally.

[0061] It's worth noting that a task data retry mechanism is implemented to address potential issues such as full garbage collection at the basic unit (region) of the columnar database. When the measurement data writing program automatically catches an exception, the task data retry mechanism is executed. If the retry still fails, the data is written to a high-performance key-value database. The Java background program periodically reads the data written to the high-performance key-value database and rewrites it back to the columnar database.

[0062] S03. Perform data mapping based on the data information in the measurement table, transfer the data mapping result to the entity table, and perform consistency statistics on the measurement table and the entity table.

[0063] Based on the data information of the measurement table and according to the custom rules, a uniquely identified entity table is generated, which covers the maximum data range that appears in the columnar database.

[0064] The big data processing engine reads the data content of the measurement table from different table fields in the columnar database and uses custom rules to transfer the data content of the measurement table to the entity table with a unique identifier.

[0065] Consistency statistics are performed on the data content of the measurement table and the data content of the uniquely identified entity table.

[0066] During the process of transferring and statistically analyzing the data information from the measurement table, the data information from the measurement table is monitored.

[0067] At a minimum, the cluster storage entity information and data service information of the measurement table should be collected. The number of records in the measurement table should be counted in real time through the processing framework for real-time data monitoring. At the same time, the time period for writing the data information of the measurement table into the columnar database should be captured to monitor the timeliness of the data.

[0068] In summary, the power grid data integration method proposed by this invention configures a big data infrastructure environment to support compilation in multiple programming languages, configures cluster clients and establishes a cluster storage entity, and sets data storage duration and partition size for the cluster storage entity to accommodate large volumes of data. Simultaneously, a columnar database and task channels are built. Data information from the measurement tables of the cluster storage entity is obtained through the task channels. While maintaining the uniqueness of the measurement tables, the measurement data is aggregated. Based on the write cycle and classification capacity limit rules, the aggregated measurement data is written into the measurement tables in the columnar database. The measurement tables are then parsed, classified, and enumerated with information definitions. The information definition process includes real-time enumeration mining, predictive enumeration definition, and dynamic adjustment of association weights. Real-time enumeration mining extracts latent enumeration candidate values ​​as basic features, which are then synchronized to the input layer of the prediction model in the predictive enumeration definition. The future enumeration probability distribution generated by the predictive enumeration predefined is used as a forward-looking feature to feed back into the real-time mining direction. The cluster storage main load characteristics in the dynamic adjustment of association weights are used as constraint features to limit the storage priority of enumeration values. Data mapping is performed based on the data information in the measurement table, and the data mapping results are transferred to the entity table. Consistency statistics are performed on the measurement table and the entity table to ensure the accuracy of the transfer process.

[0069] This invention achieves feature sharing through data interoperability, using current values, future values, and storage rules to generate a complete enumeration information system. Accuracy and efficiency are continuously optimized through a feedback loop. The enumeration information definition of the measurement table not only enables static field labeling but also dynamically responds to business changes and adapts to future needs. Furthermore, it can aggregate large volumes and high-frequency measurement data for post-processing difference comparison to ensure data uniqueness. Unique identifiers are assigned for rapid location. A write cycle and classification capacity limit are set; the measurement table is written to the columnar database every write cycle or when the classification capacity limit is reached, ensuring timely and efficient data writing.

[0070] Example 2

[0071] Another aspect of this invention provides a power grid data integration system; please refer to [link / reference needed]. Figure 2 The diagram shown is a structural schematic of the power grid data integration system in Embodiment 2 of the present invention. The power grid data integration system includes:

[0072] Configuration module 11 is used to build the basic environment, configure the cluster client, and establish the cluster storage entity;

[0073] Import module 12 is used to build a columnar database and task channel, and obtain cluster storage entity information according to the task channel. It preprocesses the measurement data and imports the preprocessed measurement data into the measurement table in the columnar database. It parses, classifies and defines enumeration information in the measurement table. The enumeration information definition process includes real-time enumeration mining, predictive enumeration definition and dynamic adjustment of association weights. The latent enumeration candidate value is extracted as a basic feature through real-time enumeration mining and synchronized to the input layer of the prediction model in the predictive enumeration definition. The future enumeration probability distribution generated by the predictive enumeration predefined is used as a forward-looking feature to feed back to the real-time mining direction. The cluster storage entity load feature in the dynamic adjustment of association weights is used as a constraint feature to limit the storage priority of the enumeration value.

[0074] The transfer module 13 is used to perform data mapping based on the data information of the measurement table, transfer the data mapping result to the entity table, and perform consistency statistics on the measurement table and the entity table.

[0075] Building the basic environment includes configuring the big data infrastructure environment, cluster client configuration, and cluster storage entity configuration. The basic big data infrastructure environment configuration includes support for Java and Scala development languages, and necessary dependency packages for accessing services such as MRS-HBASE, MRS-KAFKA, MRS-HDFS, MRS-YARN, MRS-KLINK, and DWS. It also includes an MRS cluster client to support development environment job deployment and application debugging. Finally, it involves creating the cluster storage entity, data storage duration, and partition size according to the actual data volume of the measurement data and business rules.

[0076] Build a columnar database, design the table name for the columnar database for the measurement information on each measurement table, set the partition size for the measurement information data parsing process and the table fields of the measurement table, and complete the automatic creation of measurement tables.

[0077] The system collects information about the corresponding cluster clients from the main storage unit of the cluster, and combines this information with the business information from the measurement tables of the corresponding main storage unit.

[0078] The uniqueness of measurement data in the current measurement table is analyzed to comply with the uniqueness and dispersion of columnar databases, and a unique identifier is generated for each row of measurement data in the table. The generation of the unique identifier for each row can employ a "ciphertext + plaintext" approach. The plaintext design involves tracking and analyzing the measurement data in each measurement table, selecting appropriate combinations of business fields for design. The "ciphertext + plaintext" approach ensures the uniqueness, dispersion, and readability of the columnar database, avoids the problem of centralized resource access, and improves business usability.

[0079] For the data information in the measurement table, a series of parsers that conform to the actual business situation are designed in the data parsing process, so as to reduce the data deserialization and serialization time and improve the data processing speed when carrying out subsequent data processing.

[0080] The communication path of the cluster storage entity is constructed. Based on the data volume of the measurement table, checkpoints are constructed in the processing framework environment. The status of the measurement data in the measurement table of the columnar database is checked periodically according to the checkpoints, and the cluster storage entity information in the measurement table is subscribed to in real time.

[0081] The processing framework parses the measurement data in the measurement table, filters out abnormal data based on business rules, and performs special processing, such as handling data with a business time of one year ago, incorrect business time format, or empty data. Normal data is categorized according to its time period, while abnormal data is merged into the same time period. This ensures all data belongs to a specific day and time, preparing for subsequent data merging and disk storage, and improving the speed of concurrent data processing. More specifically, during the measurement table parsing process, a domain knowledge graph is introduced. This domain knowledge graph can be the business rules of power systems or intelligent manufacturing. The measurement data in the measurement table is parsed using the domain knowledge graph. Simultaneously, data lineage tracing is used to verify the field logic of the related tables in the current measurement table, and the isolated forest algorithm is used to identify the degree of matching between the measurement data in the current measurement table and the columnar database, avoiding the misclassification of abnormal data as normal data.

[0082] The enumeration information of the categorized measurement tables is defined, and the information is written into the columnar database of the corresponding table fields based on the association between the enumeration information and the cluster storage entity. The table creation rules of the measurement tables are analyzed, and the measurement tables written into the columnar database and the table fields of the corresponding columnar database are verified. If the content of the measurement table is consistent with the table fields, the writing is successful; otherwise, a columnar database matching the measurement table is automatically created.

[0083] In the process of defining enumeration information for the measurement table, the definition of enumeration information can be extended to a dynamic sensing combined with predictive enumeration method, specifically:

[0084] By combining NLP and time series analysis, implicit enumeration values ​​are extracted from historical data of the categorized measurement tables, equipment logs, business documents and other data (such as "unmarked anomaly" of equipment fault codes can be automatically classified as a new enumeration item), and semantic associations of enumeration values ​​are constructed through knowledge graphs (such as "sudden temperature rise" and "sensor failure" are strongly correlated), so as to realize real-time enumeration mining. Specifically, the NLP-based text enumeration extraction process is as follows: After classifying the text data, a named entity recognition model is used to identify "state entities" (such as "voltage overload" and "communication interruption"). Similar entities are merged through word vector clustering (such as Word2Vec+K-Means) (such as "signal loss" and "communication interruption" being grouped into the same enumeration candidate value). The time-series analysis-based anomaly enumeration extraction process is as follows: For the time-series data of the measurement table (such as temperature values ​​recorded every 5 minutes), anomaly fluctuations (such as a sudden temperature rise of 20°C) are detected using isolated forest or DBSCAN. Combined with the time window (such as the anomaly lasting more than 10 minutes), it is determined to be a "persistent anomaly", and a new enumeration value is generated (such as "sudden temperature rise anomaly"). At the same time, high-frequency repeated codes in the device log are automatically marked as "high-frequency undefined enumerations".

[0085] Using predictive enumeration definitions, this method predicts new enumeration values ​​that may appear in the future (such as new states predicted by the fluctuation characteristics of seasonal measurement data) based on time series models such as LSTM. It reserves space for field expansion in advance in the columnar database, reducing the frequency of creating new tables when inconsistencies arise. Specifically, it employs a bidirectional LSTM (Bi-LSTM) + Attention mechanism. The input layer consists of extracted features, including time features, fluctuation features, and correlation features. The output layer is the probability distribution of new enumeration values ​​that may appear in the next 1-3 months (e.g., the probability of "winter low-temperature protection" appearing in December is 0.85). The basic model is trained using full historical data to learn general time series patterns (such as annual seasonal fluctuations). The prediction model is evaluated using accuracy (the proportion of predicted enumeration values ​​that actually appear in the future) and recall (the proportion of actually appearing new enumeration values ​​that are predicted). Model retraining is triggered when the accuracy is lower than a preset accuracy threshold. For high-probability new enumeration values, dynamic fields are set within the reserved field extension space. The field type can be set to a variable-length string to accommodate new enumeration values ​​of different formats. When a new enumeration value is officially entered into the database, the reserved field is automatically renamed, and the field mapping relationship is recorded to avoid historical data query anomalies. When a reserved field conflicts with an existing field, the field renaming rule is automatically triggered.

[0086] The association weights are dynamically adjusted, and reinforcement learning is used to update the association weights of "enumeration information - cluster storage entity" in real time (such as the association degree between a certain type of measurement table and "workshop A storage cluster" increases as the production plan changes), prioritizing the matching of high-weight storage entities and optimizing write efficiency.

[0087] In the process of defining enumeration information, implicit enumeration candidate values ​​are extracted as basic features through real-time enumeration mining, and these implicit enumeration candidate values ​​are synchronized to the input layer of the prediction model. The future enumeration probability distribution generated by the predictive enumeration is used as a forward-looking feature to feed back the focus of real-time mining. Finally, the cluster storage main load feature in the dynamic adjustment of association weights is used as a constraint feature to limit the storage priority of enumeration values. This application achieves feature sharing through data interoperability, and uses the linkage of current values, future values, and storage rule decisions to generate a complete enumeration information system, continuously optimizing accuracy and efficiency through feedback loop. The definition of enumeration information in the measurement table can not only achieve static field labeling, but also dynamically respond to business changes and adapt to future needs.

[0088] A write cycle and a categorization capacity limit are set. Every write cycle or when the categorization capacity limit is reached, the measurement table is written to the columnar database to improve the write speed of measurement data. In this embodiment, the write cycle can be 20 seconds, and the categorization capacity limit can be 3000 records.

[0089] In addition, the program that writes measurement tables to the columnar database in real time can create a connection to the columnar database every 30 minutes to ensure that data writing is running normally.

[0090] It's worth noting that a task data retry mechanism is implemented to address potential issues such as full garbage collection at the basic unit (region) of the columnar database. When the measurement data writing program automatically catches an exception, the task data retry mechanism is executed. If the retry still fails, the data is written to a high-performance key-value database. The Java background program periodically reads the data written to the high-performance key-value database and rewrites it back to the columnar database.

[0091] Based on the data information of the measurement table and according to the custom rules, a uniquely identified entity table is generated, which covers the maximum data range that appears in the columnar database.

[0092] The big data processing engine reads the data content of the measurement table from different table fields in the columnar database and uses custom rules to transfer the data content of the measurement table to the entity table with a unique identifier.

[0093] Consistency statistics are performed on the data content of the measurement table and the data content of the uniquely identified entity table.

[0094] During the process of transferring and statistically analyzing the data information from the measurement table, the data information from the measurement table is monitored.

[0095] At a minimum, the cluster storage entity information and data service information of the measurement table should be collected. The number of records in the measurement table should be counted in real time through the processing framework for real-time data monitoring. At the same time, the time period for writing the data information of the measurement table into the columnar database should be captured to monitor the timeliness of the data.

[0096] In summary, the power grid data integration system proposed by this invention configures a big data infrastructure environment to support compilation in multiple programming languages, configures cluster clients and establishes a cluster storage entity, and sets data storage duration and partition size for the cluster storage entity to accommodate large volumes of data. Simultaneously, it establishes a columnar database and task channels, acquires data information from the measurement tables of the cluster storage entity through the task channels, and aggregates the measurement data while maintaining the uniqueness of the measurement tables. Based on the write cycle and classification capacity limit rules, the aggregated measurement data is written into the measurement tables in the columnar database. The measurement tables are then parsed, classified, and enumerated with information definitions. The information definition process includes real-time enumeration mining, predictive enumeration definition, and dynamic adjustment of association weights. Real-time enumeration mining extracts latent enumeration candidate values ​​as basic features, which are then synchronized to the input layer of the prediction model in the predictive enumeration definition. The future enumeration probability distribution generated by the predictive enumeration predefined is used as a forward-looking feature to feed back into the real-time mining direction. The cluster storage main load characteristics in the dynamic adjustment of association weights are used as constraint features to limit the storage priority of enumeration values. Data mapping is performed based on the data information in the measurement table, and the data mapping results are transferred to the entity table. Consistency statistics are performed on the measurement table and the entity table to ensure the accuracy of the transfer process.

[0097] This invention achieves feature sharing through data interoperability, using current values, future values, and storage rules to generate a complete enumeration information system. Accuracy and efficiency are continuously optimized through a feedback loop. The enumeration information definition of the measurement table not only enables static field labeling but also dynamically responds to business changes and adapts to future needs. Furthermore, it can aggregate large volumes and high-frequency measurement data for post-processing difference comparison to ensure data uniqueness. Unique identifiers are assigned for rapid location. A write cycle and classification capacity limit are set; the measurement table is written to the columnar database every write cycle or when the classification capacity limit is reached, ensuring timely and efficient data writing.

[0098] Example 3

[0099] In another aspect, the present invention provides a computer-readable storage medium having stored thereon one or more computer programs that, when executed by a processor, implement the above-described power grid data integration method.

[0100] Those skilled in the art will understand that the logic or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable storage medium" can mean any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0101] More specific examples (a non-exhaustive list) of computer-readable storage media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable storage media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0102] Example 4

[0103] Figure 3 This is a structural block diagram of an electronic device provided in Embodiment 4. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the power grid data integration method in the above embodiments. Figure 3 The electronic device 30 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0104] like Figure 3 As shown, the electronic device 30 can be manifested as a general-purpose computing device, such as a server device. The components of the electronic device 30 may include, but are not limited to: at least one processor 31, at least one memory 32, and a bus 33 connecting different system components (including memory 32 and processor 31).

[0105] Bus 33 includes a data bus, an address bus, and a control bus.

[0106] The memory 32 may include volatile memory, such as RAM 321 (random access memory), and / or cache memory 322, and may further include ROM 323 (read-only memory).

[0107] The memory 32 may also include a program tool 325 having a set (at least one) of program modules 324, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0108] The processor 31 executes various functional applications and data processing by running computer programs stored in the memory 32, such as the power grid data integration method of the present invention as described above.

[0109] Electronic device 30 can also communicate with one or more external devices 34 (e.g., keyboard, pointing device, etc.). This communication can be performed via I / O interface 35 (input / output interface). Furthermore, electronic device 30 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public network, such as the Internet) via network adapter 36. Figure 3 As shown, network adapter 36 communicates with other modules of the model-generated electronic device 30 via bus 33. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the model-generated electronic device 30, including but not limited to: microcode, device drivers, redundant processors, disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.

[0110] It should be noted that although several units / modules or sub-units / modules of the electronic device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0111] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0112] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A method of integrating grid data, the method comprising: The power grid data integration method comprises: Build the basic environment, configure the cluster client and establish the cluster storage subject; Build a columnar database and a task channel, obtain cluster storage subject information according to the task channel, preprocess measurement data, import the preprocessed measurement data into a measurement table in the columnar database, analyze and classify the measurement table, and define enumeration information, the enumeration information definition process includes real-time enumeration mining, predictive enumeration definition and dynamic adjustment of correlation weight, combined with NLP and time series analysis, extract implicit enumeration candidate values with enumeration attributes from the measurement table as basic features, and build semantic association of enumeration values through knowledge graph to realize real-time enumeration mining, and synchronize the implicit enumeration candidate values to the input layer of the prediction model to extract time features, fluctuation features and correlation features of the implicit enumeration candidate values, perform predictive enumeration definition, and output future enumeration probability distribution to display the probability distribution of new enumeration values in the future period, use the future enumeration probability distribution as a forward-looking feature to indicate the new enumeration direction, and return the real-time enumeration mining direction, use the cluster storage subject load feature in the dynamic adjustment of correlation weight as a constraint feature, preferentially match high-weight storage subjects, optimize write efficiency, and limit the storage priority of enumeration values; Map data according to data information of the measurement table, store data mapping results to an entity table, and perform consistency statistics on the measurement table and the entity table.

2. The grid data integration method of claim 1, wherein, The step of building the basic environment, configuring the cluster client and establishing the cluster storage subject comprises: Configure a big data basic environment, support Java and scala development language compilation, support development environment job deployment according to the cluster client; Create a cluster storage subject, data storage duration and partition size based on the actual data size and business rules of measurement data.

3. The grid data integration method of claim 1, wherein, The step of building a columnar database and a task channel, obtaining cluster storage subject information according to the task channel, preprocessing measurement data, importing the preprocessed measurement data into a measurement table in the columnar database, analyzing and classifying the measurement table, and defining enumeration information, the enumeration information definition process includes real-time enumeration mining, predictive enumeration definition and dynamic adjustment of correlation weight, combined with NLP and time series analysis, extract implicit enumeration candidate values with enumeration attributes from the measurement table as basic features, and build semantic association of enumeration values through knowledge graph to realize real-time enumeration mining, and synchronize the implicit enumeration candidate values to the input layer of the prediction model to extract time features, fluctuation features and correlation features of the implicit enumeration candidate values, perform predictive enumeration definition, and output future enumeration probability distribution to display the probability distribution of new enumeration values in the future period, use the future enumeration probability distribution as a forward-looking feature to indicate the new enumeration direction, and return the real-time enumeration mining direction, use the cluster storage subject load feature in the dynamic adjustment of correlation weight as a constraint feature, preferentially match high-weight storage subjects, optimize write efficiency, and limit the storage priority of enumeration values comprises: The table name of the column database is designed according to the measurement information of any tensor measurement table, the size of the partition of the measurement information data analysis process and the table field of the measurement table are set, and the automatic creation of the measurement table is completed; Based on the acquisition information of the cluster client and the business information of the measurement table, the uniqueness of the current measurement table is analyzed, and the unique identifier of each row of measurement data in the measurement table is generated.

4. The grid data integration method of claim 3, wherein, A series of parsers of the data analysis process are designed for the data information of the measurement table, and a communication path of the cluster storage subject is constructed. According to the data magnitude of the measurement table, the checkpoint in the processing framework environment is constructed, and the cluster storage subject information in the measurement table is subscribed in real time.

5. The grid data integration method of claim 4, wherein, The measurement data in the measurement table is parsed by the processing framework, the domain knowledge graph is introduced, the measurement data of the measurement table is parsed by the domain knowledge graph, the data blood relationship tracking is used to check the field logic of the associated table of the current measurement table, and the isolated forest algorithm is used to identify the matching degree of the measurement data in the current measurement table and the column database, so as to screen out normal data and abnormal data, classify the time period to which the normal data belongs, and merge the abnormal data in the same time period; The enumeration information of the classified measurement table is defined, and the enumeration information and the association of the cluster storage subject are written into the table field of the column database according to the enumeration information, the table building rules of the measurement table are analyzed, the measurement table and the table field of the column database are checked, if the measurement table content is consistent with the table field, the writing is successful, if not, the column database matching the measurement table is automatically created; The writing period and the upper limit of the classification capacity are set, and every other writing period or when the classification capacity reaches the upper limit of the classification capacity, the measurement table is written into the column database.

6. The grid data integration method of claim 1, wherein, The step of performing data mapping according to the data information of the measurement table, storing the data mapping result into an entity table, and performing consistency statistics on the measurement table and the entity table comprises: Based on the data information of the measurement table, and according to the self-defined rules, an entity table with a unique identifier is generated, which covers the maximum data range of the column database; The data content of the measurement table in different table fields in the column database is read by the big data processing engine, and the self-defined rules are used to store the data content of the measurement table into the entity table with a unique identifier; The consistency statistics is performed on the data content of the measurement table and the data content in the entity table with a unique identifier.

7. The grid data integration method of claim 6, wherein, In the process of storing and statistics of the data information of the measurement table, the data information of the measurement table is monitored; At least the cluster storage subject information and the data business information of the measurement table are collected, the number of records of the measurement table is counted in real time by the processing framework to perform real-time data monitoring, and the time period of writing the data information of the measurement table into the column database is grabbed to monitor the timeliness of the data.

8. A power grid data integration system characterized by, The power grid data integration system is used to realize the power grid data integration method in any one of claims 1-7, and the system comprises: A configuration module is configured to construct a basic environment, configure a cluster client, and establish a cluster storage subject; A configuration module is configured to construct a basic environment, configure a cluster client, and establish a cluster storage subject; The import module is used to build a columnar database and a task channel, obtain cluster storage subject information according to the task channel, pre-process measurement data, import the pre-processed measurement data into a measurement table in the columnar database, analyze and classify the measurement table, and define enumeration information. The enumeration information definition process includes real-time enumeration mining, predictive enumeration definition, and dynamic adjustment of correlation weight. In combination with NLP and time series analysis, implicit enumeration candidate values with enumeration attributes that are not labeled as abnormal are extracted from the measurement table as basic features. The semantic correlation of the enumeration values is constructed through a knowledge graph to realize real-time enumeration mining. The implicit enumeration candidate values are synchronized to the input layer of a prediction model to extract time features, fluctuation features, and correlation features of the implicit enumeration candidate values, perform predictive enumeration definition, and output future enumeration probability distribution to display the probability distribution of new enumeration values in the future time period. The future enumeration probability distribution is used as a forward-looking feature to indicate a new enumeration direction, which in turn benefits the real-time enumeration mining direction. The cluster storage subject load features in the dynamic adjustment of correlation weight are used as constraint features to preferentially match high-weight storage subjects, optimize write efficiency, and limit the storage priority of enumeration values. The conversion module is used to perform data mapping according to the data information of the measurement table, convert the data mapping result to an entity table, and perform consistency statistics on the measurement table and the entity table.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the power grid data integration method of any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor executes the computer program to implement the power grid data integration method of any one of claims 1-7.

Images