Multi-source multi-dimensional data acquisition and convergence management method and device for high-proportion new energy hydropower area

Abstract

The invention discloses a collection and convergence management method and device for a high-proportion new energy hydropower region, and the method comprises the steps: collecting various types of data through a data collection and monitoring control system SCADA in a target region; integrating wind power plant data, photovoltaic power station data, hydropower station data and access power grid side data according to a unified standard and specification through a data preprocessing method based on a neural network; secondly, a real-time data cloud platform is established, metadata is indexed through a data cube, a unified architecture storage model of'phenomenon-data-data source 'is formed, and corresponding equipment is formed; and finally, converting into an IEC 61850 protocol through a CIM model so as to provide interoperability and communication capability between equipment in an electric power automation system. The method is applied to a hydropower station in a new energy access scene and a regional power grid of a near region of the hydropower station, fusion, cleaning and access optimization are carried out on complex high-dimensional heterogeneous data of multi-source information interaction, a power information physical system is facilitated to realize data standard integration and safe sharing, and safe and stable operation of the power grid is supported.

Description

Technical Field

[0001] The present invention relates to the technical field of electric power digital management, and in particular to a collection and aggregation management method and equipment for a high-proportion new energy hydropower area. Background Art

[0002] The integration and coordinated operation of renewable energy, hydropower, energy storage and other flexible resources is an important new power generation system that has emerged in recent years. Through coordinated operation, it can greatly improve the consumption of renewable energy and provide controllable, adjustable and stable power supply for the power grid. At the same time, with the rapid development of power measurement and communication technology, the access of a large amount of information such as wide-area measurement and external information (environment, meteorology, society, etc.), the power system has developed into a complex high-dimensional heterogeneous power information-physical system with multi-source information interaction. Under the above background, the hydropower regional power system with a large amount of new energy access shows the characteristics of huge data volume and complex variables, which significantly affect the analysis and dispatch model of the power system. Therefore, it is urgent to define the standard format of data according to the characteristics of multi-source heterogeneous data of power grid and new energy, unify the conversion format of heterogeneous data, realize seamless management of data, and solve key problems such as data fusion and cleaning, access optimization, etc. in the process of new energy consumption. Summary of the invention

[0003] The present invention provides a collection and aggregation management method and equipment for areas with a high proportion of new energy hydropower, aiming to solve the above technical problems.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is: The multi-source and multi-dimensional data collection method for areas with a high proportion of renewable energy and hydropower includes: Step 1, data collection: Taking the target hydropower station and its nearby regional power grid as the target, data collection is based on the internal data of the data acquisition and monitoring control system SCADA, energy management system EMS, wide area monitoring system WAMS, distribution management system DMS, power consumption information collection system, marketing management system, distribution network micro-synchronous phasor measurement device D-PMU, advanced measurement system AMI, etc. within the target range, as well as external data such as geographic information system GIS, meteorological information system, Internet data, public service department data, social and economic data, etc. Step 2: Data preprocessing based on neural network: After obtaining the original data, the data format, integrity and synchronization are considered to perform data preprocessing.

[0005] The specific contents of data preprocessing in the above Step 2 include: 1) Data format verification: Use data validation rules to check whether the data conforms to the expected format and range, including verification of data length, data type, and data range; 2) Data missing processing: predict the missing data part through neural network and reconstruct it; 3) Data anomaly processing: Based on the identification of bad data by mutual verification of multi-source data, the rules between data are constructed into correlation coefficients between neurons through rule-based neural network classifiers. The neural network trained by multi-source data can identify data that violates the rules. 4) Data anomaly processing: Data duplication processing: Use Siamese twin neural network to remove duplicate data, input the data into two neural networks, the two neural networks map the input to the new space respectively, and judge the similarity of the data by calculating the loss function, so as to distinguish the duplicate data. The loss function is as follows:

[0006] Where Y represents whether the input data X belongs to the same category; if the input belongs to the same category, it is 0, and if it does not belong to the same category, it is 1.

[0007] The above two neural networks can be LSTM and CNN respectively.

[0008] The above-mentioned multi-source and multi-dimensional data collection method for high-proportion new energy hydropower areas is used as a convergence management method, and the convergence management method includes: Step 1, data aggregation; the real-time data cloud platform has a built-in real-time data service, and the real-time data service in the real-time data cloud platform acts as an application proxy to request data from the real-time data service in each region, and provides the data returned by each region to each application after comprehensive analysis; the application uses the id as a globally unique index to access data, and the proxy analyzes the maintenance area to which the id belongs, and uses the real-time data of the maintenance area as the data source; when the data required by the application belongs to different maintenance areas, the proxy is responsible for requesting data from different areas and returning it to the application after merging. The actual storage method and location of the real-time data are transparent to the application; Step 2: Verification of information model based on RDF; Step 3: Multi-source and multi-dimensional data management.

[0009] The RDF-based information model verification in Step 2 above specifically includes: Step 2.1. Based on the definition of the information model, write SHACL constraint rules, including data type constraint sh:datatype, property constraint sh:PropertyShape, and value constraints sh:in, sh:minCount, sh:maxCount; Step 2.2, use the rdflib tool to extract the RDF information of the sample data; Step 2.3. Use the SHACL verification engine to perform verification operations, find error information that does not comply with the constraint rules and make corrections.

[0010] The multi-source and multi-dimensional data management in the above Step 3 includes: 1) By designing a distributed storage architecture with storage and computing separation, the data to be cleaned is deployed in an independent cluster through HDFS, and the data cleaning algorithm is deployed in another cluster through Hadoop computing components, so as to achieve on-demand elastic expansion of computing and storage and meet the requirements of efficient data sharing; 2) Use data cube to store aggregated data; design fact tables and dimension tables of the data model according to the determined dimensions and metrics; load the processed power data into the designed fact tables and dimension tables through data extraction, conversion and loading; and establish data indexes, including the name and attributes of the database, the tables in the database, the name of the table, the columns and partitions of the table and their attributes, the attributes of the table, and the directory where the table data is located.

[0011] An electronic device using the above-mentioned data aggregation management method includes a processor, a communication interface, a memory and a communication bus. The processor, the communication interface and the memory communicate with each other through the communication bus. The processor can call the logic instructions in the memory to execute the real-time data aggregation and processing method.

[0012] The logic instructions in the above-mentioned memory are implemented in the form of software functional units and sold or used as independent products.

[0013] The present invention provides a collection, aggregation and management method and equipment for areas with a high proportion of new energy and hydropower. With the trend of a sharp increase in new energy installed capacity and a large number of power measurement and communication equipment connected, a multi-source and multi-dimensional data collection and management method is proposed to address the problem of multi-source heterogeneity of massive data and the lack of a unified data management tool, and related equipment and devices are completed on a theoretical basis. This method can efficiently collect multi-source electrical / non-electrical information in the region, and use data cubes to perform distributed storage on a cloud platform with a unified structure, which is conducive to the reading and calling of massive information. It provides technical support for the business of digital twin modeling and intelligent scheduling of regional power grids. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The present invention will be further described below in conjunction with the accompanying drawings and embodiments: Figure 1 It is a schematic diagram of a bad data screening method based on multi-source data mutual verification of the present invention; Figure 2 This is a schematic diagram of the practical application of the present invention based on neural network data preprocessing; Figure 3 This is a schematic diagram of the overall architecture of real-time data services; Figure 4 This is a schematic diagram of the overall architecture of the distributed cluster storage solution; Figure 5 This is a schematic diagram of data cube storage; Figure 6 It is a schematic diagram of the structure of part of the electronic equipment. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solutions and advantages of the present invention clearer, the following content will systematically and completely describe the specific technical solutions of the present invention in combination with the drawings provided according to the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0016] The multi-source and multi-dimensional data collection method for areas with a high proportion of renewable energy and hydropower includes: Step 1, data collection; taking the target hydropower station and the regional power grid in its vicinity as the target, data collection is carried out based on the internal data of the data acquisition and monitoring control system SCADA, energy management system EMS, wide area monitoring system WAMS, distribution management system DMS, power consumption information collection system, marketing management system, distribution network micro-synchronous phasor measurement device D-PMU, advanced measurement system AMI, etc. within the target range, and external data such as geographic information system GIS, meteorological information system, Internet data, public service department data, social and economic data. Some of the collected data are shown in Table 1.

[0017] Table 1 Data collection table

[0018] At the same time, the data collection performance quality should meet the following requirements: (1) The real-time data change update delay is less than 2 seconds; (2) The system real-time data scanning time period is adjustable from 1 to 10 seconds; (3) Within any 5 minutes, the average CPU load rate of the server and workstation is ≤30%; (4) Within any 5 minutes, taking into account a certain margin, the average load rate of the network is ≤20%; (5) When the dual networks of the main LAN are operated in a split mode, the load rate of each network is ≤12%; (6) Under single network operation, the average network load rate is ≤24% within any 30 seconds; (7) Spare space ≥ 20%.

[0019] Step 2: Data preprocessing based on neural network: After obtaining the original data, the data format, integrity and synchronization are considered to perform data preprocessing.

[0020] The specific contents of data preprocessing in the above Step 2 include: 1) Data format verification: Use data validation rules to check whether the data conforms to the expected format and range, including verification of data length, data type, and data range; 2) Data missing processing: predict the missing data part through neural network and reconstruct it; The missing processing code is: import tensorflow as tf from tensorflow.keras.models import Model from tensorflow.keras.layers import Input, Dense input_dim = X_train.shape[1] input_layer = Input(shape=(input_dim,)) encoded = Dense(64, activation='relu')(input_layer) encoded = Dense(32, activation='relu')(encoded) decoded = Dense(64, activation='relu')(encoded) decoded = Dense(input_dim, activation='sigmoid')(decoded) autoencoder = Model(input_layer, decoded) autoencoder.compile(optimizer='adam', loss='mean_squared_error') autoencoder.fit(X_train, X_train, epochs=50, batch_size=256,validation_split=0.2) X_train_pred = autoencoder.predict(X_train) 3) Data anomaly processing: Based on the identification of bad data by mutual verification of multi-source data, the rules between data are constructed into the correlation coefficient between neurons through the rule-based neural network classifier. The neural network trained by multi-source data can identify data that violates the rules; the principle is as follows Figure 1 shown.

[0021] The code is as follows: model = Sequential() model.add(Dense(64, input_dim=input_dim, activation='relu')) model.add(Dense(32, activation='relu')) model.add(Dense(1, activation='sigmoid')) model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy']) model.fit(X_train, y_train, epochs=50, batch_size=32, validation_split=0.2) predictions = model.predict(X_test) anomalies = X_test[predictions>0.5] # Assume the threshold is 0.5 4) Data anomaly processing: Data duplication processing: Use Siamese twin neural network to remove duplicate data, input the data into two neural networks, the two neural networks map the input to the new space respectively, and judge the similarity of the data by calculating the loss function, so as to distinguish the duplicate data. The loss function is as follows:

[0022] Where Y represents whether the input data X belongs to the same category; if the input belongs to the same category, it is 0, and if it does not belong to the same category, it is 1.

[0023] The above two neural networks can be LSTM and CNN respectively.

[0024] Figure 2The application of the above data preprocessing process in the 2019 wind power data of Zhangbei County, Zhangjiakou City was demonstrated, and the data of 4 stations were detected and repaired, with the accuracy rate exceeding 98%.

[0025] The above-mentioned multi-source and multi-dimensional data collection method for high-proportion new energy hydropower areas is used as a convergence management method, and the convergence management method includes: Step 1, data aggregation; the real-time data cloud platform has a built-in real-time data service, and the real-time data service in the real-time data cloud platform acts as an application proxy to request data from the real-time data service in each region, and provides the data returned by each region to each application after comprehensive analysis; the application uses the id as a globally unique index to access data, and the proxy analyzes the maintenance area to which the id belongs, and uses the real-time data of the maintenance area as the data source; when the data required by the application belongs to different maintenance areas, the proxy is responsible for requesting data from different areas and returning it to the application after merging. The actual storage method and location of the real-time data are transparent to the application; Step 2: RDF-based information model verification: When application systems interact with each other, model verification is required to ensure the semantic consistency between data and information models.

[0026] Step 3: Multi-source and multi-dimensional data management.

[0027] The RDF-based information model verification in Step 2 above specifically includes: Step 2.1. Based on the definition of the information model, write SHACL constraint rules, including data type constraint sh:datatype, property constraint sh:PropertyShape, and value constraints sh:in, sh:minCount, sh:maxCount; Step 2.2, use the rdflib tool to extract the RDF information of the sample data; Step 2.3. Use the SHACL verification engine to perform verification operations, find error information that does not comply with the constraint rules and make corrections.

[0028] The multi-source and multi-dimensional data management in the above Step 3 includes: 1) By designing a distributed storage architecture with storage and computing separation, the data to be cleaned is deployed in an independent cluster through HDFS, and the data cleaning algorithm is deployed in another cluster through Hadoop computing components, so as to achieve elastic expansion of computing and storage on demand and meet the efficient sharing of data. The overall architecture of the distributed cluster storage solution is as follows: Figure 4 shown.

[0029] 2) Use data cube to store the aggregated data; design the fact table and dimension table of the data model according to the determined dimensions and measures; load the processed power data into the designed fact table and dimension table through data extraction, conversion and loading; and establish data indexes, including the name and attributes of the database, the table in the database, the name of the table, the columns and partitions of the table and their attributes, the attributes of the table, and the directory where the table data is located. The data cube storage diagram is as follows Figure 5 shown.

[0030] An electronic device using the above data aggregation management method, such as Figure 6 As shown, the electronic device includes but is not limited to: a processor 810, a communication interface 820, a memory 830 and a communication bus 840. The processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call the logic instructions in the memory 830 to execute the above-mentioned real-time data collection and processing method.

[0031] In addition, the logic instructions in the memory 830 can be implemented in the form of software functional units and can be sold or used as independent products. These instructions can be stored in a computer-readable storage medium. It is understood that the substantial part of the technical solution of the present invention or the contribution to the prior art can be embodied as a software product, and the computer software product is stored in various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0032] In addition, the present invention also provides a computer program product, including a computer program stored on a non-transitory computer-readable storage medium. When a processor executes the computer program, the computer can execute the method for processing and analyzing multi-source heterogeneous data of a regional power grid provided by the present invention.

[0033] In addition, the present invention also provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program implements a method for processing and analyzing multi-source heterogeneous data of a regional power grid and its characteristics when executed by a processor.

[0034] The device embodiments described above are for illustration only. Components described as separate units may or may not be physically separated, and components shown as units may or may not be physical units, and may be located in one place or distributed over multiple network units. As needed, some or all of the modules may be selected to achieve the purpose of the present embodiment. Those skilled in the art may understand and implement these contents without creative work.

[0035] IEC 61850 protocol conversion IEC 61850 is a communication protocol and data model standard. Converting the CIM model of data storage to the IEC61850 protocol is intended to provide interoperability and communication capabilities between devices in the power system. The detailed steps are as follows: 1) Parse the CIM model file, use the this.model.listSubjects method to get all the subjects, and create an ArrayList object to save the class name; 2) Get a StmtIterator object, traverse all attribute values ​​in the StmtIterator object through the inner while loop, and create a HashMap object to save; 3) Use the DocumentBuilder object to create a new empty Document object and use the createElement method to create the SCL root element; 4) Use the createElement method to create substation elements and logical node elements; 5) Create IEDName, Address and GOOSESecurity elements and save them in the communication element, and add the communication element to the substation element; 6) Use TransformerFactory to save the generated document to a file.

[0036] According to the above steps, the CIM model can be converted into the IEC 61850 protocol, providing a basis for reading and calling the database.

[0037] The present invention proposes a multi-source and multi-dimensional data collection and management method for high-proportion renewable energy hydropower areas. The method collects multiple types of data through the data acquisition and monitoring control system (SCADA) in the target area, and integrates wind farm data, photovoltaic power station data, hydropower station data and grid access data with unified standards and specifications through a data preprocessing method based on a neural network; secondly, a real-time data cloud platform is established, and metadata is indexed through a data cube to form a unified architecture storage model of "phenomenon-data-data source" and form corresponding equipment; finally, the CIM model is converted into the IEC 61850 protocol, aiming to provide interoperability and communication capabilities between devices in the power automation system. This method is applied to hydropower stations and their nearby regional power grids in the scenario of renewable energy access, and integrates, cleans and optimizes the access of complex high-dimensional heterogeneous data of multi-source information interaction, which is conducive to the standard integration and secure sharing of data in the power information-physical system, and supports the safe and stable operation of the power grid.

Claims

1. A multi-source and multi-dimensional data collection method for high-proportion new energy hydropower areas, characterized in that: include: Step 1, data collection; The target hydropower station and its nearby regional power grid are taken as the target, and data collection is based on internal data such as data acquisition and monitoring control system SCADA, energy management system EMS, wide area monitoring system WAMS, distribution management system DMS, power consumption information collection system, marketing management system, distribution network micro-synchronous phasor measurement device D-PMU, advanced measurement system AMI, and external data such as geographic information system GIS, meteorological information system, Internet data, public service department data, and social and economic data within the target range; Step 2: Data preprocessing based on neural network: After obtaining the original data, the data format, integrity and synchronization are considered to perform data preprocessing.

2. The multi-source and multi-dimensional data collection method for high-proportion new energy hydropower areas according to claim 1 is characterized in that: The specific contents of the data preprocessing in Step 2 include: 1) Data format verification: Use data validation rules to check whether the data conforms to the expected format and range, including verification of data length, data type, and data range; 2) Data missing processing: predict the missing data part through neural network and reconstruct it; 3) Data anomaly processing: Based on the identification of bad data by mutual verification of multi-source data, the rules between data are constructed into correlation coefficients between neurons through rule-based neural network classifiers. The neural network trained by multi-source data can identify data that violates the rules. 4) Data anomaly processing: Data duplication processing: Use Siamese twin neural network to remove duplicate data, input the data into two neural networks, the two neural networks map the input to the new space respectively, and judge the similarity of the data by calculating the loss function, so as to distinguish the duplicate data. The loss function is as follows: Where Y represents whether the input data X belongs to the same category; if the input belongs to the same category, it is 0, and if it does not belong to the same category, it is 1.

3. A method for managing the multi-source and multi-dimensional data collection method for high-proportion new energy hydropower areas according to claim 2, characterized in that: Convergence management methods include: Step 1, data aggregation; the real-time data cloud platform has a built-in real-time data service, and the real-time data service in the real-time data cloud platform acts as an application proxy to request data from the real-time data service in each region, and provides the data returned by each region to each application after comprehensive analysis; the application uses the id as a globally unique index to access data, and the proxy analyzes the maintenance area to which the id belongs, and uses the real-time data of the maintenance area as the data source; when the data required by the application belongs to different maintenance areas, the proxy is responsible for requesting data from different areas and returning it to the application after merging. The actual storage method and location of the real-time data are transparent to the application; Step 2: Verification of information model based on RDF; Step 3: Multi-source and multi-dimensional data management.

4. The multi-source multi-dimensional data aggregation management method for high-proportion new energy hydropower areas according to claim 3 is characterized in that: The RDF-based information model verification in Step 2 specifically includes: Step 2.

1. Based on the definition of the information model, write SHACL constraint rules, including data type constraint sh:datatype, property constraint sh:PropertyShape, and value constraints sh:in, sh:minCount, sh:maxCount; Step 2.2, use the rdflib tool to extract the RDF information of the sample data; Step 2.

3. Use the SHACL verification engine to perform verification operations, find error information that does not comply with the constraint rules and make corrections.

5. The multi-source multi-dimensional data aggregation management method for high-proportion new energy hydropower areas according to claim 4 is characterized in that: The multi-source multi-dimensional data management in Step 3 includes: 1) By designing a distributed storage architecture with storage and computing separation, the data to be cleaned is deployed in an independent cluster through HDFS, and the data cleaning algorithm is deployed in another cluster through Hadoop computing components, so as to achieve on-demand elastic expansion of computing and storage and meet the requirements of efficient data sharing; 2) Use data cube to store aggregated data; design fact tables and dimension tables of the data model according to the determined dimensions and metrics; load the processed power data into the designed fact tables and dimension tables through data extraction, conversion and loading; and establish data indexes, including the name and attributes of the database, the tables in the database, the name of the table, the columns and partitions of the table and their attributes, the attributes of the table, and the directory where the table data is located.

6. An electronic device using the data aggregation management method described in claim 5, characterized in that the electronic device includes a processor, a communication interface, a memory and a communication bus, the processor, the communication interface, and the memory communicate with each other through the communication bus, and the processor can call the logical instructions in the memory to execute the real-time data aggregation and processing method.

7. The electronic device for multi-source and multi-dimensional data for areas with a high proportion of new energy and hydropower as described in claim 6 is characterized in that the logical instructions in the memory are implemented in the form of software functional units and are sold or used as independent products.

Images