Energy data management system based on cloud platform

Abstract

The invention discloses an energy data management system based on a cloud platform, and relates to the technical field of energy data management, and the system comprises an energy data collection module which comprises equipment data, environment data and third-party demand data; according to the data transmission module, a communication network uses 4G / 5G, LoRaWAN, NB-IoT and optical fibers, and a message queue uses Kafka, RabbitMQ and the like to manage asynchronous data transmission. In the invention, the edge-cloud collaborative architecture module has the functions of task dynamic allocation, data consistency, resource optimization and system integrity, a traditional algorithm and a deep learning model are fused, the analysis precision and scene adaptability are improved, unnecessary data transmission is reduced and bandwidth resources are saved through data preprocessing of edge computing nodes; the dynamic data partitioning module reasonably distributes storage resources according to the data access frequency and importance, the storage efficiency is improved, hot area data is stored in a high-speed storage cluster, and it is ensured that data with the high real-time requirement can be rapidly accessed and processed.

Description

Technical Field

[0001] The present invention relates to the technical field of energy data management, and in particular to an energy data management system based on a cloud platform. Background Art

[0002] Energy refers to the material or energy form that exists in nature and can be transformed and used in various activities such as production, transportation, lighting, heating, driving machinery, etc. Energy is an important material basis for supporting the development and survival of human society. It exists in various forms, including but not limited to fossil energy, nuclear energy, hydropower, wind energy, solar energy, bioenergy, etc.; in the process of human social development, energy is widely used in various fields of production and life, such as industrial production, transportation, home heating, electricity production, etc.; the use of energy directly affects the development of social economy and environmental sustainability; therefore, the efficient use of energy and the development and utilization of renewable energy have become one of the important issues of today's society and environmental protection.

[0003] However, traditional management systems rely on centralized cloud platforms to process data, and edge devices only serve as data collection terminals, resulting in high transmission delays, unable to meet real-time monitoring needs, and data analysis relies on preset rules. It lacks adaptive machine learning models and is difficult to explore deep-level energy consumption patterns. Therefore, there is an urgent need for an energy data management system based on a cloud platform. Summary of the invention

[0004] The purpose of the present invention is to provide an energy data management system based on a cloud platform in order to solve the problem that the current traditional management system relies on a centralized cloud platform to process data, and the edge device only serves as a data acquisition terminal, resulting in high transmission delay, inability to meet real-time monitoring needs and data analysis dependence on preset rules, lacks an adaptive machine learning model, and is difficult to explore deep-level energy consumption patterns.

[0005] To achieve the above object, the present invention provides the following technical solution: an energy data management system based on a cloud platform, comprising:

[0006] Energy data collection module: including equipment data, environmental data, and third-party demand data;

[0007] Data transmission module: The communication network uses 4G / 5G, LoRaWAN, NB-IoT, and optical fiber, and the message queue uses Kafka, RabbitMQ, etc. to manage asynchronous data transmission;

[0008] Data storage module: including real-time database, relational database, big data storage and cold data archiving, and converting data into a unified format;

[0009] Edge-cloud collaborative architecture module: including edge computing node unit and dynamic data partitioning module;

[0010] Data processing module: performs data cleaning, data standardization operations, and data filtering;

[0011] Adaptive analysis module: uses a multi-dimensional analysis model combined with time series analysis (ARIMA), clustering algorithm (K-means), and deep learning (CNN) to dynamically generate an energy consumption pattern portrait. According to historical data and external environment parameters, the prediction and optimization module predicts future energy consumption trends and generates energy-saving strategies through a genetic algorithm.

[0012] Preferably, the device data in the energy data acquisition module includes the power generation end and the power transmission and distribution end, the environmental data includes weather, temperature and humidity, wind speed, and solar irradiance intensity, and the third-party demand data includes electricity consumption, time-of-use electricity price data, smart meter data, and user behavior analysis.

[0013] Preferably, the power generation end includes power generation volume, unit efficiency, voltage frequency, current frequency, and equipment status, and the power transmission and distribution end is the line, grid load, transformer operation status, and fault alarm.

[0014] Preferably, the data transmission module transmits the collected device data, environmental data, and third-party demand data to the cloud platform. At the same time, it compresses the data to reduce bandwidth occupancy and ensures transmission security through encryption. The data transmission module is compatible with multiple communication protocols (such as MQTT, HTTP, Modbus, etc.) and adapts to different devices and systems.

[0015] Preferably, in the data storage module, real-time databases (InfluxDB, TimescaleDB) store high-frequency data, relational databases (MySQL / PostgreSQL) store device metadata and user information, big data storage is processed by Hadoop HDFS or cloud storage (AWS S3) for PB-level data, and cold data is archived regularly to low-cost storage.

[0016] Preferably, for the edge-cloud collaborative architecture module, edge computing nodes: are deployed at the energy device end, with lightweight AI models built-in to preprocess the original data and only upload key data to the cloud to reduce bandwidth occupancy. Dynamic data partitioning module: divides data into hot zones, warm zones, and cold zones according to data type and priority and allocates them to different storage clusters in the cloud.

[0017] Preferably, the edge-cloud collaborative architecture module combines edge computing and cloud computing, dynamically allocates tasks to the edge or the cloud according to requirements, the edge processes real-time tasks, and the cloud processes complex analysis, maintaining data consistency between the edge and the cloud and ensuring system integrity.

[0018] Preferably, the data processing module performs data cleaning: removing duplicate data, handling missing values, and correcting incorrect data (such as outliers, noise data, etc.); data standardization: converting data with different units or formats into a unified standardized format for subsequent analysis; and data filtering: filtering out unnecessary data according to business requirements to reduce storage and computing burdens.

[0019] Preferably, the adaptive analysis module: Time series analysis is used to analyze the time series data of energy consumption to capture its trends, seasonality, and periodic changes; the clustering algorithm performs clustering analysis on the energy consumption data to identify different energy consumption patterns or user behavior patterns; deep learning uses convolutional neural networks (CNNs) to extract features and recognize patterns from complex energy consumption data, especially for processing multi-dimensional spatio-temporal data. The dynamic generation of energy consumption pattern portraits combines the analysis results of time series analysis, clustering algorithms, and deep learning, enabling the system to dynamically generate energy consumption pattern portraits for each user or device.

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0021] In the present invention, the edge-cloud collaborative architecture module has the functions of task dynamic allocation, data consistency, resource optimization, and system integrity. It integrates traditional algorithms and deep learning models to improve analysis accuracy and scenario adaptability. Through data preprocessing at the edge computing nodes, unnecessary data transmission is reduced, saving bandwidth resources. The dynamic data partitioning module reasonably allocates storage resources according to data access frequency and importance, improving storage efficiency. Hot zone data is stored in a high-speed storage cluster to ensure that data with high real-time requirements can be quickly accessed and processed.

[0022] In the present invention, time series analysis uses the ARIMA model to predict the energy consumption trend in the short term in the future, helping to identify the peaks and troughs of energy consumption. The clustering algorithm can classify similar energy consumption patterns through clustering, helping to identify the energy consumption characteristics of different users or devices, and thus providing a basis for personalized energy-saving strategies. Deep learning can capture the non-linear relationships in energy consumption data and identify complex energy consumption patterns, especially the energy consumption changes affected by multiple variables (such as temperature, humidity, device status, etc.). The energy consumption pattern portrait includes the time distribution characteristics of energy consumption (such as peak hours, trough hours), the seasonal changes of energy consumption, the energy consumption changes under different external environmental conditions, and the energy consumption behavior patterns of users or devices (such as the usage habits of high-energy-consuming devices). Then, based on historical energy consumption data and external environmental parameters (such as weather, temperature, etc.), the ARIMA and CNN models are used to predict the future energy consumption trend, and the genetic algorithm (GA) is used to optimize the energy-saving strategy to generate the optimal energy-saving plan. This adaptive analysis module can dynamically adjust the analysis model and prediction results according to real-time data to adapt to different energy consumption scenarios. Combining time series, clustering, and deep learning, the system can comprehensively analyze energy consumption data from multiple perspectives, provide more accurate predictions and optimization strategies. The energy-saving strategy generated by the genetic algorithm can maximize the energy use efficiency while ensuring user comfort. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic flowchart of the method of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0024] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0025] Please refer to Figure 1 , an energy data management system based on a cloud platform, including:

[0026] Energy data collection module: including device data, environmental data, and third-party demand data;

[0027] Data transmission module: The communication network uses 4G / 5G, LoRaWAN, NB-IoT, optical fiber, and message queues use Kafka, RabbitMQ, etc. to manage asynchronous data transmission;

[0028] Data storage module: including real-time database, relational database, big data storage, and cold data archiving, and converting the data into a unified format;

[0029] Edge-cloud collaborative architecture module: including edge computing node unit, dynamic data partitioning module;

[0030] Data processing module: perform data cleaning, data standardization operations, and data filtering;

[0031] Adaptive analysis module: use a multi-dimensional analysis model combined with time series analysis (ARIMA), clustering algorithm (K-means), and deep learning (CNN) to dynamically generate an energy consumption pattern portrait. Through the prediction and optimization module, according to historical data and external environment parameters, predict future energy consumption trends, and generate energy-saving strategies through genetic algorithms.

[0032] Embodiment 1

[0033] As a preferred embodiment of the present invention: The device data in the energy data acquisition module includes the power generation end and the power transmission and distribution end, the environmental data includes weather, temperature and humidity, wind speed, solar irradiance intensity, and the third-party demand data includes electricity consumption, time-of-use electricity price data, smart meter data, and user behavior analysis;

[0034] The power generation end includes equipment such as power station boilers, steam turbines, and generators, and the power transmission and distribution end includes transformers, circuit breakers, relays, etc. Obtain energy consumption data such as electricity, gas, and water, as well as external environmental data, weather, temperature and humidity, wind speed, solar irradiance intensity, and real-time collect electricity consumption, time-of-use electricity price data, and smart meter data from various sensors, smart meters, environmental monitoring equipment, etc., so as to conduct user behavior analysis.

[0035] Embodiment 2

[0036] As a preferred embodiment of the present invention: The power generation end includes power generation capacity, unit efficiency, voltage frequency, current frequency, and equipment status, and the power transmission and distribution end is the line, grid load, transformer operation status, and fault alarm;

[0037] Power generation capacity refers to the amount of electric energy output by the generator during energy conversion. Unit efficiency refers to the energy conversion ability of the power unit during power generation, that is, the ratio between the input energy and the output energy. Voltage frequency refers to the periodic change frequency of the voltage waveform in the AC power system. The frequency of the current refers to the frequency of the electric field. Electric power load, also known as "electrical load", is the total electric power consumed by the electrical equipment of the electric energy user from the power system at a certain moment, which is called the electrical load. The transformer operation status includes no-load status, load status, short-circuit status, operation status, hot standby status, cold standby status, and maintenance status. The fault alarm of the power transmission and distribution end means that when various equipment or components in the power transmission and distribution system show abnormal conditions, an alarm is issued through the alarm system to remind the operation and maintenance personnel to handle it in time, prevent the fault from expanding, and ensure the stable operation of the power system.

[0038] Example 3

[0039] As a preferred embodiment of the present invention: The data transmission module transmits the collected device data, environmental data, and third-party requirement data to the cloud platform. At the same time, it compresses the data to reduce the bandwidth occupancy and ensures transmission security through encryption. The data transmission module is compatible with multiple communication protocols (such as MQTT, HTTP, Modbus, etc.) and adapts to different devices and systems;

[0040] Among them, the collected data is compressed through data compression algorithms (such as GZIP, LZ4, etc.) to reduce the bandwidth occupancy during the transmission process and improve the transmission efficiency. The transmitted data is encrypted through encryption technologies (such as TLS / SSL, AES, etc.) to ensure that the data is not stolen or tampered with during the transmission process. The MQTT protocol is lightweight and suitable for low-bandwidth and unstable networks, and is commonly used for data transmission of Internet of Things devices. The HTTP protocol is suitable for interacting with web services and is widely used for data transmission in cloud platforms. This data transmission module is a key component to ensure the efficient, safe, and reliable transmission of energy data. It is responsible for transmitting energy data between different systems, devices, or platforms and supports real-time monitoring, analysis, and decision-making.

[0041] Example 4

[0042] As a preferred embodiment of the present invention: In the data storage module, the real-time database (InfluxDB, TimescaleDB) stores high-frequency data, the relational database (MySQL / PostgreSQL) stores device metadata and user information, the big data storage is Hadoop HDFS or cloud storage (AWS S3) for processing PB-level data, and the cold data is archived regularly to low-cost storage;

[0043] The real-time database stores high-frequency data such as sensor data, monitoring data, and log data. High performance: Optimized for time-series data, it supports high-throughput data writing and querying. Time-series optimization: Built-in functions such as time windows, downsampling, and aggregation are suitable for processing time-related data. Scalability: Supports distributed deployment and can handle large-scale time-series data; The relational database stores structured data such as device metadata, user information, and configuration information. Structured data: Supports complex relational models and is suitable for storing data that needs to be frequently queried and updated. Transaction support: Provides ACID transaction guarantees to ensure data consistency and integrity. Mature ecosystem: Supported by rich tools and frameworks, it is easy to integrate and develop; The big data storage stores PB-level large-scale data, usually used in scenarios such as data analysis and machine learning. High capacity: Can store massive amounts of data and is suitable for processing unstructured or semi-structured data. Distributed storage: Supports horizontal expansion and can handle the storage and computing of large-scale data sets. Cost-effectiveness: Cloud storage (such as AWS S3) provides a pay-as-you-go model, suitable for long-term storage of large-scale data; The cold data archive regularly archives infrequently accessed historical data (cold data) to low-cost storage. Low cost: Uses tapes, low-performance hard drives, or low-cost options of cloud storage (such as AWS Glacier) to store cold data. Long-term preservation: Suitable for storing data that needs to be retained for a long time but is rarely accessed, such as historical logs and backup data. Access latency: Cold data usually requires a long recovery time and is not suitable for scenarios with frequent access.

[0044] Example 5

[0045] As a preferred embodiment of the present invention: Edge-cloud collaboration architecture module, Edge computing node: Deployed at the energy device end, with a lightweight AI model built in, it preprocesses the raw data and only uploads the key data to the cloud, reducing bandwidth occupancy. Dynamic data partitioning module: According to the data type and priority, divides the data into hot zones, warm zones, and cold zones and allocates them to different storage clusters in the cloud;

[0046] Functions of the edge computing node: Lightweight AI model: A lightweight AI model is built in to preprocess the raw data. Data preprocessing: Preliminary processing of the data is performed at the edge node to extract key information. Data upload: Only uploads the key data to the cloud, reducing bandwidth occupancy and improving transmission efficiency; Functions of the dynamic data partitioning module: Data classification: According to the data type and priority, divides the data into hot zones, warm zones, and cold zones. Storage allocation: Allocates the data in different partitions to different storage clusters in the cloud to optimize the utilization of storage resources.

[0047] Example 6

[0048] As a preferred embodiment of the present invention: The edge-cloud collaborative architecture module combines edge computing and cloud computing, dynamically allocates tasks to the edge or the cloud according to requirements, processes real-time tasks at the edge, and processes complex analysis in the cloud, maintaining data consistency between the edge and the cloud to ensure system integrity;

[0049] Functions of the edge-cloud collaborative architecture module: Dynamic task allocation, data consistency, resource optimization, and system integrity. It integrates traditional algorithms and deep learning models to improve analysis accuracy and scenario adaptability. Through data preprocessing by edge computing nodes, unnecessary data transmission is reduced, saving bandwidth resources. The dynamic data partitioning module reasonably allocates storage resources according to data access frequency and importance, improving storage efficiency. Hot zone data is stored in a high-speed storage cluster to ensure that data with high real-time requirements can be quickly accessed and processed.

[0050] Embodiment 7

[0051] As a preferred embodiment of the present invention: The data processing module, data cleaning: removing duplicate data, handling missing values, correcting incorrect data (such as outliers, noisy data, etc.), data standardization: converting data of different units or formats into a unified standardized format for subsequent analysis, data filtering: filtering out unnecessary data according to business requirements to reduce storage and computing burdens;

[0052] Data cleaning is the first step in data processing, aiming to remove noise, errors, and inconsistencies in the data to ensure data accuracy and integrity, including removing duplicate data, handling missing values, and correcting incorrect data. Duplicate data is removed using a deduplication algorithm (such as deduplication based on a primary key or unique identifier) to identify and delete duplicate data. Missing values are processed by deleting missing values and filling in missing values. Incorrect data is corrected using outlier detection methods and noisy data processing methods. Data standardization methods include Z-score standardization: converting data into a distribution with a mean of 0 and a standard deviation of 1, Min-Max standardization: scaling data to a specified range (such as 0 to 1), and decimal scaling standardization: scaling data to between [-1, 1].

[0053] Embodiment 8

[0054] As a preferred embodiment of the present invention: Adaptive Analysis Module: Time series analysis is used to analyze the time series data of energy consumption, capture its trends, seasonality, and periodic changes. The clustering algorithm performs clustering analysis on the energy consumption data to identify different energy consumption patterns or user behavior patterns. Deep learning uses convolutional neural networks (CNNs) to extract features and identify patterns from complex energy consumption data, especially for processing multi-dimensional spatio-temporal data. The dynamically generated energy consumption pattern portrait combines the analysis results of time series analysis, clustering algorithms, and deep learning, enabling the system to dynamically generate an energy consumption pattern portrait for each user or device;

[0055] Time series analysis, through the ARIMA model, predicts the energy consumption trend in the short term future, helping to identify the peaks and valleys of energy consumption. The clustering algorithm, through clustering, can classify similar energy consumption patterns, helping to identify the energy consumption characteristics of different users or devices, and thus providing a basis for personalized energy-saving strategies. Deep learning can capture the non-linear relationships in energy consumption data and identify complex energy consumption patterns, especially the energy consumption changes under the influence of multiple variables (such as temperature, humidity, device status, etc.); The energy consumption pattern portrait includes the time distribution characteristics of energy consumption (such as peak hours, valley hours), the seasonal changes in energy consumption, the energy consumption changes under different external environmental conditions, and the energy consumption behavior patterns of users or devices (such as the usage habits of high-energy-consuming devices). Then, based on historical energy consumption data and external environmental parameters (such as weather, temperature, etc.), the ARIMA and CNN models are used to predict the future energy consumption trend, and the energy-saving strategy is optimized through the Genetic Algorithm (GA) to generate the optimal energy-saving plan. This adaptive analysis module can dynamically adjust the analysis model and prediction results according to real-time data to adapt to different energy consumption scenarios. By combining time series, clustering, and deep learning, the system can comprehensively analyze energy consumption data from multiple perspectives, provide more accurate predictions and optimization strategies. The energy-saving strategy generated by the genetic algorithm can maximize the energy use efficiency while ensuring user comfort.

[0056] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and without departing from the spirit or basic characteristics of the present invention, the present invention can be implemented in other specific forms. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present invention. Any reference signs in the claims should not be regarded as limiting the claimed rights.

Claims

1. An energy data management system based on a cloud platform, characterized by: include: Energy data collection module: including equipment data, environmental data, and third-party demand data; Data transmission module: The communication network uses 4G / 5G, LoRaWAN, NB-IoT, and optical fiber, and the message queue uses Kafka, RabbitMQ, etc. to manage asynchronous data transmission; Data storage module: including real-time database, relational database, big data storage and cold data archiving, and converting data into a unified format; Edge-cloud collaborative architecture module: including edge computing node unit and dynamic data partitioning module; Data processing module: perform data cleaning, data standardization and data filtering; Adaptive analysis module: Uses a multi-dimensional analysis model combined with time series analysis (ARIMA), clustering algorithm (K-means) and deep learning (CNN) to dynamically generate energy consumption pattern portraits. Through the prediction and optimization module, it predicts future energy consumption trends based on historical data and external environmental parameters, and generates energy-saving strategies through genetic algorithms.

2. The cloud platform-based energy data management system according to claim 1, characterized in that: The equipment data in the energy data acquisition module includes the power generation end and the transmission and distribution end; the environmental data includes weather, temperature and humidity, wind speed, and solar radiation intensity; the third-party demand data includes electricity consumption, time-of-use electricity price data, smart meter data, and user behavior analysis.

3. The cloud platform-based energy data management system according to claim 2, characterized in that: The power generation end includes power generation, unit efficiency, voltage frequency, current frequency, and equipment status, and the transmission and distribution end includes lines, grid load, transformer operating status, and fault alarm.

4. The cloud platform-based energy data management system according to claim 1, characterized in that: The data transmission module transmits the collected device data, environmental data, and third-party demand data to the cloud platform, while compressing the data to reduce bandwidth occupancy and ensuring transmission security through encryption. The data transmission module is compatible with multiple communication protocols (such as MQTT, HTTP, Modbus, etc.) and adapts to different devices and systems.

5. The cloud platform-based energy data management system according to claim 1, characterized in that: In the data storage module, the real-time database (InfluxDB, TimescaleDB) stores high-frequency data, the relational database (MySQL / PostgreSQL) stores device metadata and user information, the big data storage is Hadoop HDFS or cloud storage (AWS S3) to process PB-level data, and the cold data is archived regularly to low-cost storage.

6. The cloud platform-based energy data management system according to claim 1, characterized in that: The edge-cloud collaborative architecture module, edge computing node: deployed at the energy device end, with a built-in lightweight AI model, pre-processes the original data, and only uploads key data to the cloud to reduce bandwidth usage, dynamic data partitioning module: divides the data into hot zones, warm zones, and cold zones according to the data type and priority, and distributes them to different storage clusters in the cloud.

7. The cloud platform-based energy data management system according to claim 1, characterized in that: The edge-cloud collaborative architecture module combines edge computing and cloud computing, dynamically allocates tasks to the edge or cloud according to needs, processes real-time tasks at the edge, and processes complex analysis at the cloud, maintaining data consistency between the edge and the cloud to ensure system integrity.

8. The cloud platform-based energy data management system according to claim 1, characterized in that: The data processing module includes data cleaning: removing duplicate data, processing missing values, and correcting erroneous data (such as outliers, noise data, etc.); data standardization: converting data in different units or formats into a unified standardized format to facilitate subsequent analysis; and data filtering: filtering out unnecessary data according to business needs to reduce storage and computing burdens.

9. The cloud platform-based energy data management system according to claim 1, characterized in that: The adaptive analysis module: time series analysis is used to analyze the time series data of energy consumption to capture its trends, seasonality and cyclical changes. The clustering algorithm performs cluster analysis on the energy consumption data to identify different energy consumption patterns or user behavior patterns. Deep learning uses convolutional neural networks (CNNs) to perform feature extraction and pattern recognition on complex energy consumption data, especially processing multi-dimensional spatiotemporal data. The dynamic generation of energy consumption pattern portraits combines the analysis results of time series analysis, clustering algorithms and deep learning. The system can dynamically generate energy consumption pattern portraits for each user or device.

Images