Big data platform for realizing energy information collection method

Abstract

The invention discloses a big data platform for realizing an energy information collection method, which relates to the technical field of energy information management and comprises a data acquisition unit, a data storage unit, a data calculation unit, a data analysis unit, a visualization unit, a standard management unit, a data service unit and a data security unit. The data acquisition unit completes acquisition, cleaning and standardization of multiple types of energy data, the storage unit provides rapid retrieval, the calculation unit supports real-time and batch calculation, the analysis unit realizes trend prediction and association mining, and a data result is displayed in the visualization unit. The data service unit provides an interface, and the data security unit ensures the security of data transmission and access. The big data platform for realizing the energy information collection method provided by the invention has efficient multi-source data collection and standardized processing capability, can quickly complete data storage and retrieval, supports real-time calculation and batch analysis, and realizes deep association mining and trend prediction of energy data.

Description

Technical Field

[0001] The present invention relates to the technical field of energy information management, and in particular to a big data platform for realizing an energy information collection method. Background Art

[0002] At present, the energy big data platform needs to achieve comprehensive monitoring of multiple types of energy. The existing technology has the following problems:

[0003] (1) The collection of multiple types of energy data, such as natural gas, crude oil, coal, and electricity, is operated by relevant entities in different industries. The degree of informatization, data collection, and storage methods of each entity are different. The main enterprises in each energy industry are under the unified management of different energy regulatory agencies, making data monitoring difficult.

[0004] (2) Data types and formats are diverse. It is unrealistic to establish data connections between existing systems. The cost of data interoperability is high and there are risks in data security management.

[0005] (III) The collection of multiple types of energy data lacks a unified management tool and relies on manual statistics and reporting. Data updates are slow and the statistical process is time-consuming and labor-intensive.

[0006] (iv) Different industries have different energy data management models, data sharing methods, and technical support capabilities, which limit the sharing and use of energy data. Summary of the invention

[0007] In view of the above-mentioned problems, the present invention is proposed.

[0008] Therefore, the technical problem solved by the present invention is: it solves the problems of high difficulty in collecting multi-type energy data, inconsistent data formats, low data storage and retrieval efficiency, and complex multi-source data correlation analysis. By building a multi-module big data platform, it provides full-process support for data collection, cleaning, storage, calculation and analysis, and realizes comprehensive monitoring and efficient management of multi-type energy data.

[0009] In order to solve the above technical problems, the present invention provides the following technical solutions: a big data platform for realizing an energy information collection method, comprising:

[0010] Data acquisition unit, data storage unit, data calculation unit, data analysis unit, visualization unit, standard management unit, data service unit and data security unit;

[0011] The data acquisition unit is used to collect multiple types of energy data from energy equipment and sensors, the data being data of different energy types, and to perform format conversion, field standardization and data cleaning on the collected data to generate a structured data set;

[0012] The data storage unit is connected to the data acquisition unit and is used to store standardized multi-type energy data. The data storage unit supports fast retrieval operations at the index and field levels.

[0013] The data computing unit is connected to the data storage unit and is used to perform distributed computing operations, including real-time computing, batch computing and graph computing, so as to process and calculate different energy data;

[0014] The data analysis unit is connected to the data calculation unit and is used to analyze the calculated energy data, including time series analysis, cluster analysis and association rule mining, to obtain energy trend prediction data;

[0015] The visualization unit is connected to the data analysis unit and is used to display the analyzed data in the form of charts and maps to provide a visualized energy data monitoring interface;

[0016] The standard management unit is connected to the data acquisition unit and the data storage unit, and is used to perform format standardization management on multiple types of energy data, and unify various data formats through a predefined standard model;

[0017] The data service unit is connected to the data analysis unit and the visualization unit, and is used to provide a data access interface to the external system and execute the distribution and call of energy data;

[0018] The data security unit is connected to the data acquisition unit, the data storage unit and the data service unit, and is used for data access permission control, data encryption operation and recording data access logs, providing security management functions.

[0019] As a preferred solution of a big data platform for realizing a method for collecting energy information described in the present invention, wherein: the data acquisition unit includes an interface adaptation module, a cleaning module and a standardization module;

[0020] The interface adapter module is connected to different types of energy devices and sensors, and calls the built-in communication protocol to convert the data into an internal format and transmit it to the cleaning module;

[0021] The cleaning module performs integrity check on the data output by the interface adapter module, detects and corrects abnormal data through an algorithm, and transmits the corrected data to the standardization module;

[0022] The standardization module performs field standardization operations on the cleaned data and converts it into a unified structured format for use by the data storage unit.

[0023] As a preferred solution of a big data platform for realizing a method for collecting energy information described in the present invention, wherein: the data storage unit includes a wide table construction module, an index optimization module and a data backup module;

[0024] The wide table construction module associates the collected standardized data with the business subject data table and constructs the wide table by setting the data type and field structure;

[0025] The index optimization module generates indexes for commonly used fields in wide tables and dynamically adjusts the indexes based on the frequency of field access;

[0026] The data backup module replicates the wide table in real time and performs periodic backup, and synchronizes the backup data to off-site storage.

[0027] As a preferred solution of a big data platform for realizing a method for collecting energy information described in the present invention, wherein: the data calculation unit includes a real-time processing module, a batch processing module and a graph calculation module;

[0028] The real-time processing module extracts real-time energy data from the data storage unit and executes a continuous processing algorithm to generate a dynamic data stream;

[0029] The batch processing module periodically extracts data from the storage unit and performs aggregate calculations, and stores the processing results for subsequent use by the data analysis unit;

[0030] The graph computing module establishes a relationship graph model between energy devices and analyzes the interaction relationship between energy devices through weighted operations on nodes and edges.

[0031] As a preferred solution of a big data platform for realizing a method for collecting energy information described in the present invention, wherein: the data analysis unit includes a time series analysis module, a cluster analysis module and an association mining module;

[0032] The time series analysis module predicts and analyzes energy data through a time series model algorithm, and outputs prediction results for subsequent visualization unit display;

[0033] The cluster analysis module uses a clustering algorithm to classify patterns in the energy consumption data and generates usage group data for use by the association mining module;

[0034] The association mining module mines the association relationship between devices from the data output by the cluster analysis module, and generates rules for the data service unit to call.

[0035] As a preferred solution of a big data platform for realizing a method for collecting energy information described in the present invention, wherein: the visualization unit includes a custom view module and a multi-dimensional data display module;

[0036] The custom view module presents the data and chart type selected by the user as a visual view;

[0037] The multi-dimensional data display module displays energy data in multiple dimensions.

[0038] As a preferred solution of a big data platform for realizing a method for collecting energy information described in the present invention, wherein: the standard management unit includes an API interface management module, a real-time push module and a load balancing module;

[0039] The API interface management module provides a data access interface for external systems and supports retrieval operations by field and page;

[0040] The real-time push module pushes energy data updates to external subscribers via the WebSocket protocol, synchronizing energy data changes in real time;

[0041] The load balancing module dynamically allocates system resources according to the number of visits and gives priority to processing API interfaces that meet preset standards.

[0042] As a preferred solution of a big data platform for realizing a method for collecting energy information described in the present invention, wherein: the data service unit includes a format conversion module, a field specification module and a data model management module;

[0043] The format conversion module converts the energy data formats from different sources into the standard format required by the platform;

[0044] The field specification module performs mapping processing on the data field name and data type to ensure field consistency;

[0045] The data model management module maintains the standard data model and regularly updates the field structure to adapt to changes in data sources.

[0046] As a preferred solution of a big data platform for realizing a method for collecting energy information described in the present invention, wherein: the data security unit includes an authority control module, an encryption module and a log recording module;

[0047] The permission control module limits the data access scope based on the permission configuration of the preset role;

[0048] The encryption module performs AES-256 encryption on sensitive data to prevent data leakage during transmission and storage;

[0049] The logging module records access behaviors and operation types, and stores them in encrypted form on a regular basis.

[0050] As a preferred solution of a big data platform for realizing the energy information collection method described in the present invention, the energy information collection method includes acquiring multiple types of energy data through the data acquisition unit, calling the interface adapter module to complete the data format conversion, using the cleaning module to remove invalid data, and using the standardization module to unify the field format, and outputting the result to the data storage unit;

[0051] The wide table construction module of the data storage unit stores the standardized data in a wide table, and creates indexes for frequently accessed fields with the help of the index optimization module;

[0052] The data computing unit receives the stored data, the real-time processing module performs dynamic data stream processing, the batch processing module performs periodic aggregation calculations, and the graph computing module generates an interactive relationship model between energy devices;

[0053] The data analysis unit performs time series forecasting analysis, cluster classification and association mining on the processed data, extracts energy trends and equipment relationship rules, and transmits the analysis results to the visualization unit and data service unit;

[0054] The visualization unit generates user-defined views and multi-dimensional displays of the analyzed data, and the data service unit calls the format conversion module and the field specification module to provide a unified energy data access interface to the external system;

[0055] The data security unit is used to perform permission control, encryption processing and log recording to ensure the security of data transmission and access.

[0056] Beneficial effects of the present invention: The big data platform for implementing the energy information collection method provided by the present invention has efficient multi-source data collection and standardized processing capabilities, can quickly complete data storage and retrieval, support real-time calculation and batch analysis, and realize deep correlation mining and trend prediction of energy data. At the same time, the platform provides flexible multi-dimensional data visualization and convenient data sharing interface, and has security measures such as permission control, encryption and logging, which effectively improves data security and compliance. On the whole, the platform enhances the management, analysis and application capabilities of energy data, provides accurate and comprehensive technical support for energy monitoring and decision-making, and promotes the intelligent development of the energy industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0058] Figure 1 An overall structural diagram of a big data platform for implementing an energy information collection method provided in one embodiment of the present invention.

[0059] Figure 2 A structural diagram of a data acquisition unit of a big data platform for implementing an energy information collection method provided in one embodiment of the present invention.

[0060] Figure 3 A structural diagram of a data storage unit of a big data platform for implementing an energy information collection method provided in one embodiment of the present invention.

[0061] Figure 4 A structural diagram of a data computing unit of a big data platform for implementing an energy information aggregation method provided in one embodiment of the present invention.

[0062] Figure 5 A structural diagram of a data analysis unit of a big data platform for implementing an energy information collection method provided in one embodiment of the present invention.

[0063] Figure 6 A visualization unit structure diagram of a big data platform for implementing an energy information collection method provided in one embodiment of the present invention.

[0064] Figure 7 A standard management unit structure diagram of a big data platform for implementing an energy information collection method provided in one embodiment of the present invention.

[0065] Figure 8 A structural diagram of a data service unit of a big data platform for implementing an energy information aggregation method provided in one embodiment of the present invention.

[0066] Fig. 9 A data security unit structure diagram of a big data platform for implementing an energy information collection method provided in one embodiment of the present invention.

[0067] Fig.10 A module function diagram of a big data platform for implementing an energy information collection method provided in accordance with an embodiment of the present invention. DETAILED DESCRIPTION

[0068] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0069] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0070] Example 1

[0071] Reference Figure 1-Figure 9 , which is an embodiment of the present invention, provides a big data platform for implementing an energy information collection method, including:

[0072] Data collection unit 100, data storage unit 200, data calculation unit 300, data analysis unit 400, visualization unit 500, standard management unit 600, data service unit 700 and data security unit 800;

[0073] The data collection unit 100 is used to collect multiple types of energy data from energy equipment and sensors, the data being data of different energy types, and to perform format conversion, field standardization and data cleaning on the collected data to generate a structured data set;

[0074] The data storage unit 200 is connected to the data acquisition unit 100 and is used to store standardized multi-type energy data. The data storage unit 200 supports fast retrieval operations at the index and field levels.

[0075] The data computing unit 300 is connected to the data storage unit 200 and is used to perform distributed computing operations, including real-time computing, batch computing and graph computing, so as to process and calculate different energy data;

[0076] The data analysis unit 400 is connected to the data calculation unit 300 and is used to analyze the calculated energy data, including time series analysis, cluster analysis and association rule mining, to obtain energy trend prediction data;

[0077] The visualization unit 500 is connected to the data analysis unit 400 and is used to display the analyzed data in the form of charts and maps, providing a visualized energy data monitoring interface;

[0078] The standard management unit 600 is connected to the data acquisition unit 100 and the data storage unit 200, and is used to perform format standardization management on multiple types of energy data, and unify various data formats through a predefined standard model;

[0079] The data service unit 700 is connected to the data analysis unit 400 and the visualization unit 500, and is used to provide a data access interface to an external system and execute the distribution and call of energy data;

[0080] The data security unit 800 is connected to the data acquisition unit 100, the data storage unit 200 and the data service unit 700, and is used for data access permission control, data encryption operation and recording data access logs, and provides security management functions.

[0081] The data acquisition unit 100 includes an interface adaptation module 101, a cleaning module 102 and a standardization module 103;

[0082] The interface adapter module 101 is connected to different types of energy devices and sensors, calls the built-in communication protocol to convert the data into an internal format and transmits it to the cleaning module 102;

[0083] The cleaning module 102 performs integrity check on the data output by the interface adapter module 101, detects and corrects abnormal data through an algorithm, and transmits the corrected data to the standardization module 103;

[0084] The standardization module 103 performs field standardization operations on the cleaned data and converts it into a unified structured format for use by the data storage unit 200.

[0085] The data storage unit 200 includes a wide table construction module 201, an index optimization module 202 and a data backup module 203;

[0086] The wide table construction module 201 associates the collected standardized data with the business subject data table and constructs a wide table by setting the data type and field structure;

[0087] The index optimization module 202 generates indexes for commonly used fields in the wide table and dynamically adjusts the indexes according to the frequency of field access to reduce query time consumption;

[0088] The data backup module 203 performs real-time replication of the wide table and performs periodic backup, and synchronizes the backup data to a remote storage location to increase data security.

[0089] The data computing unit 300 includes a real-time processing module 301, a batch processing module 302 and a graph computing module 303;

[0090] The real-time processing module 301 extracts real-time energy data from the data storage unit 200 and executes a continuous processing algorithm to generate a dynamic data stream;

[0091] The batch processing module 302 periodically extracts data from the storage unit and performs aggregate calculations, and stores the processing results for subsequent use by the data analysis unit 400;

[0092] The graph calculation module 303 establishes a relationship graph model between energy devices and analyzes the interaction relationship between energy devices through weighted operations of nodes and edges.

[0093] It should be noted that the real-time processing module 301 extracts real-time energy data from the data storage unit 200. The extracted data will be pre-processed first, specifically including operations such as removing noise data and filling missing data, so as to ensure the integrity and accuracy of the data.

[0094] To achieve real-time performance, this module uses a continuous processing algorithm, which mainly includes the following steps:

[0095] The data is divided into multiple small segments at fixed time intervals (for example, every 5 seconds or every 1 minute), so that the processing data is updated in each time interval. The sliding window mechanism enables the system to update real-time data in a short period of time and generate a dynamic data stream.

[0096] In each sliding window, real-time data is aggregated, such as calculating the average, maximum, minimum, or sum, to help quickly identify short-term trends in energy consumption. These aggregated data reflect real-time energy usage.

[0097] Set a predetermined threshold range to monitor whether there are abnormal conditions in real-time data that exceed the threshold, such as excessive energy consumption. If data exceeding the threshold is found, the system will automatically mark it as abnormal data and generate an alarm message for subsequent analysis and processing.

[0098] The real-time processing module eventually generates a dynamic data stream, including real-time aggregation results and abnormal alarm information, and transmits these results to the data analysis unit 400 for further analysis and processing.

[0099] The batch processing module 302 extracts large batches of historical data from the data storage unit 200 regularly by setting a scheduling time interval (eg, every hour or every day), thereby ensuring that the data is automatically updated on time.

[0100] Perform periodic aggregation and statistical analysis on the extracted batch data. Specifically, it includes the following:

[0101] Energy consumption data is collected at set time periods (such as daily or weekly) to generate total energy usage, facilitating long-term trend analysis.

[0102] Calculate the energy usage frequency of different time periods or equipment to identify high-frequency usage time periods or equipment, and provide support for equipment optimization and management.

[0103] The results of aggregation and analysis are directly stored in the data storage unit 200 for subsequent use by the data analysis unit 400. This processing method can reduce the pressure on the real-time processing module and improve the efficiency of the system.

[0104] The graph calculation module 303 extracts standardized data from the data storage unit 200 and constructs a relationship graph model according to the interaction relationship between energy devices. In the model, each node represents an energy device, and the connection between nodes represents the interaction relationship or data flow between devices.

[0105] In the graph model, each connection line (i.e. each pair of interacting devices) is assigned a weight to represent the strength of the interaction. The weight calculation takes into account multiple factors, such as the frequency of data transmission between two devices and the stability of the interaction. The specific description is as follows:

[0106] The system calculates the actual data transmission frequency between two devices, that is, the number of interactions between the devices. The higher the frequency, the closer the interaction between the devices.

[0107] The stability score of the interaction is calculated by counting the success rate or failure rate of the interaction between devices. The higher the score, the more stable the interaction.

[0108] The frequency of data transmission and interaction stability are combined to assign weights to the connections between devices. The higher the weight, the closer the interaction between devices and the higher the priority.

[0109] Using the weighted relationship graph, the module further analyzes the key interactions in the energy network, such as identifying important equipment nodes or critical paths. This provides support for subsequent energy management and fault prediction.

[0110] The data analysis unit 400 includes a time series analysis module 401, a cluster analysis module 402 and an association mining module 403;

[0111] The time series analysis module 401 performs prediction analysis on energy data through a time series model algorithm and outputs prediction results for subsequent display by the visualization unit 500;

[0112] The cluster analysis module 402 uses a clustering algorithm to classify patterns of energy consumption data and generates usage group data for use by the association mining module 403;

[0113] The association mining module 403 mines association relationships between devices from the data output by the cluster analysis module 402 and generates rules for the data service unit 700 to call.

[0114] It should be noted that the time series analysis module 401 first performs time series sorting on the input energy data, including sorting the data in chronological order and filling in missing data to ensure the continuity of the time series data.

[0115] Time series prediction model: Based on the sorted time series data, the module uses different prediction models to analyze future energy usage trends. Commonly used models include the following:

[0116] The ARIMA model is suitable for time series data with relatively stable trends. The ARIMA model captures the periodic fluctuations of data by setting the difference order and parameters, and is suitable for predicting stable trends in energy consumption.

[0117] The LSTM neural network model is suitable for time series data with volatility or sudden changes. By simulating short-term and long-term dependencies, LSTM can effectively predict energy usage trends with irregular fluctuations.

[0118] Prediction result output: The time series analysis module outputs the prediction result to the visualization unit 500, which displays the future energy usage trend in the form of a time series chart to provide data support for energy management.

[0119] The cluster analysis module 402 normalizes the energy consumption data to eliminate differences in units and magnitudes of different energy types, ensuring that the data is suitable for cluster analysis.

[0120] The module uses a clustering algorithm (such as K-means or DBSCAN) to classify patterns in energy consumption data. The implementation steps of the K-means algorithm include:

[0121] Randomly select several initial cluster center points as the initial state of the clustering algorithm.

[0122] Using Euclidean distance or cosine similarity, each data point is assigned to the cluster corresponding to the closest center point.

[0123] The new center point of each cluster is calculated, and the process of assignment and updating is repeated until the algorithm converges and finally a stable cluster is obtained.

[0124] The clustering algorithm groups similar energy consumption patterns to generate usage group data, which provides a data source for the subsequent association mining module 403 to further analyze the specific usage patterns of equipment or areas.

[0125] The association mining module 403 uses association rule mining algorithms (such as Apriori or FP-Growth) based on the usage group data to find out the potential relationships and common combinations between devices. The specific steps of association rule mining include:

[0126] The device combinations in the usage group are analyzed to extract the device combinations with high usage frequency, namely frequent itemsets.

[0127] Association rules are generated by calculating support and confidence. For example, a rule may show that "device A and device B often operate together during a specific time period."

[0128] The generated association rules will be transmitted to the data service unit 700 for automatic scheduling and optimization of energy management. Through these association rules, potential patterns of energy use can be identified, providing a basis for energy management and prediction.

[0129] The visualization unit 500 includes a custom view module 501 and a multi-dimensional data display module 502;

[0130] The custom view module 501 presents the data and chart types selected by the user as visual views, including line charts and bar charts;

[0131] The multi-dimensional data display module 502 displays energy data in multiple dimensions and generates a chart view including multiple attributes such as time, geography, and energy type.

[0132] The standard management unit 600 includes an API interface management module 601, a real-time push module 602 and a load balancing module 603;

[0133] The API interface management module 601 provides a data access interface for external systems and supports retrieval operations by field and page;

[0134] The real-time push module 602 pushes energy data updates to external subscribers via the WebSocket protocol, synchronizing energy data changes in real time;

[0135] The load balancing module 603 dynamically allocates system resources according to the access volume and gives priority to processing API interfaces that meet preset standards.

[0136] The data service unit 700 includes a format conversion module 701, a field specification module 702 and a data model management module 703;

[0137] The format conversion module 701 converts the energy data formats from different sources into the standard format required by the platform;

[0138] The field specification module 702 performs mapping processing on the data field name and data type to ensure field consistency;

[0139] The data model management module 703 maintains the standard data model and regularly updates the field structure to adapt to changes in data sources.

[0140] The data security unit 800 includes an authority control module 801, an encryption module 802 and a log recording module 803;

[0141] The permission control module 801 limits the data access scope based on the permission configuration of the preset role to ensure clear access levels;

[0142] The encryption module 802 performs AES-256 encryption on sensitive data to prevent data leakage during transmission and storage;

[0143] The log recording module 803 records access behaviors and operation types, and stores them in encrypted form on a regular basis.

[0144] The method for realizing energy information collection includes acquiring multiple types of energy data through the data acquisition unit 100, calling the interface adaptation module 101 to complete the format conversion of the data, using the cleaning module 102 to remove invalid data, and using the standardization module 103 to unify the field format, and outputting the result to the data storage unit 200;

[0145] The wide table construction module 201 of the data storage unit 200 stores the standardized data in a wide table, and creates indexes for frequently accessed fields with the help of the index optimization module 202;

[0146] The data calculation unit 300 receives the stored data, the real-time processing module 301 performs dynamic data stream processing, the batch processing module 302 performs periodic aggregation calculations, and the graph calculation module 303 generates an interactive relationship model between energy devices;

[0147] The data analysis unit 400 performs time series prediction analysis, cluster classification and association mining on the processed data, extracts energy trends and equipment relationship rules, and transmits the analysis results to the visualization unit 500 and the data service unit 700;

[0148] The visualization unit 500 generates user-defined views and multi-dimensional displays for the analysis data, and the data service unit 700 calls the format conversion module 701 and the field specification module 702 to provide a unified energy data access interface to the external system;

[0149] The data security unit 800 performs permission control, encryption processing and log recording to ensure the security of data transmission and access.

[0150] It should be noted that the interactive relationship model refers to the relationship network established between energy devices through the graph computing module, which aims to analyze the association and interaction behavior between devices. Specifically, this model regards energy devices and their attributes as nodes, and the interactive relationships between devices, such as data transmission, power sharing, or control signal exchange, are represented as edges between nodes. By weighting nodes and edges, the interactive relationship model can quantify and display the connection strength or influence between devices, thereby analyzing the dependencies, propagation paths, and other relationships between devices. The model is used to discover and analyze the network topology of each energy device in the system, which helps to identify key equipment or potential failure points, and can provide a basis for optimizing energy scheduling, failure prediction, and resource allocation.

[0151] Example 2

[0152] Reference Fig.10 , is an embodiment of the present invention, which provides a big data platform for realizing the energy information collection method, including: the present invention establishes a collection mechanism for multiple types of energy data, which can more intuitively realize the unified monitoring of multiple types of energy production and consumption, facilitate the analysis of energy demand and trends, and have a positive impact on the planning and development of the energy industry, enable regional energy monitoring "holographic panorama", promote energy government affairs, enterprises, and users to communicate, benefit and win-win, and at the same time provide convenient service applications for energy regulatory agencies.

[0153] By establishing multiple types of energy data templates to provide basic support for energy data collection, energy data template design can improve the efficiency and accuracy of data collection, and ensure data uniformity and availability. First, carry out energy data collection and analyze data fields. For each energy type, determine which field information needs to be collected, such as output, consumption, timestamp, location, etc. And formulate a unified data format standard based on different energy type data to ensure data consistency and accuracy. Then establish data templates for different energy types, and the templates support data reporting and import.

[0154] Establish a wide table for energy big data, and assemble multiple types of energy data or business theme-related data tables into a wide table through associated fields. Determine the appropriate data type for each field in the wide table, such as integer, floating point number, string, etc. The purpose of wide table design is to improve the efficiency of data query. The wide table handles the associations between multiple tables in advance to avoid a large number of associated concatenations. The fields are indexed. Creating an index can locate the target data, thereby achieving the purpose of fast retrieval. For arbitrary field combination queries that are frequently used in wide tables, an independent index for each column is used to ensure a balance of read and write performance, thereby improving query efficiency and facilitating data maintenance and call.

[0155] Establish appropriate multi-type energy data collection methods and authority control methods to realize the collection and integration of multi-type energy data, while ensuring the accuracy and security of the data, and improving the efficiency of multi-type energy updates. The data collection mechanism design includes data reporting and importing, template recognition, data preprocessing and correction. Authority control includes authority configuration, data access supervision, and data operation records.

[0156] The data collection mechanism combines the multi-energy energy data template designed in the early stage, establishes energy type selection and associated matching templates, provides corresponding template content filling tools, and provides easy-to-use and convenient filling instructions to improve data filling efficiency. In addition, due to the different levels of informatization of energy industry data subjects, offline data maintenance extensions are provided for multi-type energy data, and offline maintenance is performed according to the defined data fields and values. After the maintenance is completed, template import is supported, thereby realizing the collection of multi-type energy data.

[0157] Establish data preprocessing and correction scripts. On the one hand, design data cleaning steps in accordance with energy data to remove or correct inaccurate, incomplete or inconsistent data. On the other hand, automatically verify the correctness and compliance of reported and imported data, identify data errors in a timely manner, and ensure the correctness and consistency of data.

[0158] Permission control is used to implement permission management in data maintenance, access, and calling. Permission control includes account permission authorization, permission verification, operation records, etc., to ensure the operational security of data.

[0159] Build a data update mechanism to automatically collect reported or imported data from various data template sources and integrate them into wide tables. Set a reasonable update cycle based on the frequency of data changes to ensure that the data in the wide table remains up-to-date.

[0160] A service-oriented architecture is adopted to build a multi-type energy integrated monitoring platform. The platform's architecture design adopts a microservice architecture to support high modularity and scalability. Under this architecture, the platform is decomposed into multiple independent services, each of which is responsible for handling a specific function of the platform, such as data collection, processing, analysis, and visualization. These services interact through communication protocols to ensure efficient data flow and processing. In order to support real-time processing and analysis of large-scale data, a distributed computing framework is adopted. In addition, the entire platform is deployed on cloud-based basic resources, using the elastic computing, storage, and network resources provided by cloud services to achieve on-demand allocation and automatic expansion of resources. To ensure the high availability and fault tolerance of the platform, strategies such as load balancing, automatic failover, and data backup are adopted. In terms of security, multi-layer security measures, including network isolation, data encryption, access control, and security auditing, are implemented to protect the platform and data from unauthorized access and various security threats. Not only does it ensure the flexibility and maintainability of the platform, but it can also meet the requirements of high performance, high reliability, and high security, providing users with stable and reliable energy monitoring services.

[0161] A large amount of data and statistical indicators will be accumulated during the monitoring of multi-type energy data. By using standardized design methods and establishing authentication and authorization mechanisms, data security can be ensured while enabling data exchange and communication between different systems and applications. Data management and sharing establish a centralized data storage environment that allows the storage of structured and unstructured data, manages data access and sharing through a data directory, and configures regular synchronization strategies to ensure data consistency. Unified data management and interaction can be achieved, and data sharing can be carried out for diverse needs.

[0162] It aims to achieve the collection of multi-type energy data by building a multi-type energy comprehensive monitoring platform, provide basic capabilities such as multi-type energy data collection, storage, analysis, and services, realize multi-type energy comprehensive monitoring, promote the empowerment and value-added of energy data, and improve the management level of energy regulatory agencies.

[0163] Driven by the monitoring needs of energy regulatory agencies for multiple types of energy data, multiple types of energy data are integrated and convenient data maintenance, reporting and unified call are realized. The relatively scattered energy monitoring and statistical methods of energy regulatory agencies are unified to form basic capabilities such as data collection, data storage, data analysis and data services, and provide basic energy monitoring and data sharing services for energy regulatory agencies at all levels to carry out energy planning. Then, by building a comprehensive monitoring platform for multiple types of energy, integrated monitoring of multiple types of energy data is realized, channels supporting cross-border integration of big data in the energy field are established, a data collection mechanism with authority control is formed, an energy data resource sharing mechanism is established, and energy upstream and downstream data sharing and exchange are realized. Energy data is promoted to empower and add value in economic development, industry development, service application, etc., and cross-domain and cross-industry energy data sharing tools are provided. Based on the platform, personalized energy data analysis functions can be continuously built to provide "practical, useful and easy-to-use" energy data for energy regulatory departments at all levels. Provide data basis for energy regulatory agencies at all levels to carry out macroeconomic decision-making and energy planning and construction, assist in improving the level of energy management, and help the high-quality development of the energy industry and optimize the energy business environment.

[0164] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A big data platform for realizing an energy information collection method, characterized in that: include: A data acquisition unit (100), a data storage unit (200), a data calculation unit (300), a data analysis unit (400), a visualization unit (500), a standard management unit (600), a data service unit (700), and a data security unit (800); The data collection unit (100) is used to collect multiple types of energy data from energy equipment and sensors, the data being data of different energy types, and to perform format conversion, field standardization and data cleaning on the collected data to generate a structured data set; The data storage unit (200) is connected to the data acquisition unit (100) and is used to store standardized multi-type energy data, and the data storage unit (200) supports fast retrieval operations at the index and field levels; The data computing unit (300) is connected to the data storage unit (200) and is used to perform distributed computing operations, including real-time computing, batch computing and graph computing, so as to process and calculate different energy data; The data analysis unit (400) is connected to the data calculation unit (300) and is used to analyze the calculated energy data, including time series analysis, cluster analysis and association rule mining, to obtain energy trend prediction data; The visualization unit (500) is connected to the data analysis unit (400) and is used to display the analyzed data in the form of charts and maps, providing a visualized energy data monitoring interface; The standard management unit (600) is connected to the data acquisition unit (100) and the data storage unit (200) and is used to perform format standardization management on multiple types of energy data and unify various data formats through a predefined standard model; The data service unit (700) is connected to the data analysis unit (400) and the visualization unit (500) and is used to provide a data access interface to an external system and execute the distribution and call of energy data; The data security unit (800) is connected to the data acquisition unit (100), the data storage unit (200) and the data service unit (700) and is used for data access permission control, data encryption operations and recording data access logs, thereby providing a security management function.

2. The big data platform for realizing the energy information collection method according to claim 1, characterized in that: The data acquisition unit (100) comprises an interface adaptation module (101), a cleaning module (102) and a standardization module (103); The interface adapter module (101) is connected to different types of energy devices and sensors, and calls a built-in communication protocol to convert data into an internal format and transmit it to the cleaning module (102); The cleaning module (102) performs integrity check on the data output by the interface adapter module (101), detects and corrects abnormal data through an algorithm, and transmits the corrected data to the standardization module (103); The standardization module (103) performs field standardization operations on the cleaned data and converts it into a unified structured format for use by the data storage unit (200).

3. The big data platform for realizing the energy information collection method according to claim 2, characterized in that: The data storage unit (200) comprises a wide table construction module (201), an index optimization module (202) and a data backup module (203); The wide table construction module (201) associates the collected standardized data with the business subject data table, and constructs a wide table by setting the data type and field structure; The index optimization module (202) generates indexes for commonly used fields in the wide table, and dynamically adjusts the indexes according to the field access frequency; The data backup module (203) performs real-time replication of the wide table and performs periodic backup, and synchronizes the backup data to a remote storage location.

4. The big data platform for realizing the energy information collection method according to claim 3, characterized in that: The data computing unit (300) includes a real-time processing module (301), a batch processing module (302) and a graph computing module (303); The real-time processing module (301) extracts real-time energy data from the data storage unit (200) and executes a continuous processing algorithm to generate a dynamic data stream; The batch processing module (302) periodically extracts data from the storage unit and performs aggregate calculations, and stores the processing results for subsequent use by the data analysis unit (400); The graph calculation module (303) establishes a relationship graph model between energy devices and analyzes the interaction relationship between energy devices through weighted operations of nodes and edges.

5. The big data platform for realizing the energy information collection method according to claim 4, characterized in that: The data analysis unit (400) includes a time series analysis module (401), a cluster analysis module (402) and an association mining module (403); The time series analysis module (401) performs prediction analysis on energy data through a time series model algorithm, and outputs prediction results for subsequent display by a visualization unit (500); The cluster analysis module (402) uses a clustering algorithm to classify patterns of energy consumption data and generates usage group data for use by the association mining module (403); The association mining module (403) mines the association relationship between devices from the data output by the cluster analysis module (402), and generates rules for the data service unit (700) to call.

6. The big data platform for realizing the energy information collection method according to claim 5, characterized in that: The visualization unit (500) comprises a custom view module (501) and a multi-dimensional data display module (502); The custom view module (501) presents the data and chart type selected by the user as a visual view; The multi-dimensional data display module (502) displays energy data in multiple dimensions.

7. The big data platform for realizing the energy information collection method according to claim 6, characterized in that: The standard management unit (600) includes an API interface management module (601), a real-time push module (602) and a load balancing module (603); The API interface management module (601) provides a data access interface for the external system and supports retrieval operations by field and page; The real-time push module (602) pushes energy data updates to external subscribers via the WebSocket protocol, synchronizing energy data changes in real time; The load balancing module (603) dynamically allocates system resources according to the access volume and gives priority to processing API interfaces that meet preset standards.

8. The big data platform for realizing the energy information collection method according to claim 7, characterized in that: The data service unit (700) comprises a format conversion module (701), a field specification module (702) and a data model management module (703); The format conversion module (701) converts the energy data formats from different sources into the standard format required by the platform; The field specification module (702) performs mapping processing on the data field name and data type to ensure field consistency; The data model management module (703) maintains the standard data model and regularly updates the field structure to adapt to changes in data sources.

9. The big data platform for realizing the energy information collection method according to claim 8, characterized in that: The data security unit (800) comprises an authority control module (801), an encryption module (802) and a log recording module (803); The authority control module (801) limits the data access scope based on the authority configuration of the preset role; The encryption module (802) performs AES-256 encryption processing on sensitive data to prevent data leakage during transmission and storage; The log recording module (803) records access behaviors and operation types, and stores them in encrypted form on a regular basis.

10. The big data platform for realizing the energy information collection method according to claim 9, characterized in that: The method for realizing energy information collection comprises acquiring multiple types of energy data through the data acquisition unit (100), calling the interface adaptation module (101) to complete the format conversion of the data, using the cleaning module (102) to remove invalid data, and using the standardization module (103) to unify the field format, and outputting the result to the data storage unit (200); The wide table construction module (201) of the data storage unit (200) stores the standardized data in a wide table, and creates indexes for frequently accessed fields with the help of the index optimization module (202); The data calculation unit (300) receives the stored data, the real-time processing module (301) performs dynamic data stream processing, the batch processing module (302) performs periodic aggregation calculations, and the graph calculation module (303) generates an interactive relationship model between energy devices; The data analysis unit (400) performs time series prediction analysis, cluster classification and association mining on the processed data, extracts energy trends and equipment relationship rules, and transmits the analysis results to the visualization unit (500) and the data service unit (700); The visualization unit (500) generates user-defined views and multi-dimensional displays for the analysis data, and the data service unit (700) calls the format conversion module (701) and the field specification module (702) to provide a unified energy data access interface to the external system; The data security unit (800) performs permission control, encryption processing and log recording to ensure the security of data transmission and access.

Images