Intelligent electric meter data acquisition system
Through the multi-level transmission architecture and Hadoop MapReduce framework, combined with distributed file system and NoSQL database, the problem of latency and inefficiency in smart meter data processing is solved, and the efficient real-time transmission and processing of smart meter data is realized, and the real-time system is improved. Sex and extensibility.
Patent Information
- Application Number
- CN202411361868.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-27
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-09-27
AI Technical Summary
Traditional smart meter data processing methods are difficult to effectively deal with the high frequency acquisition, transmission, storage and management of massive data, resulting in increased delays, data loss, and inefficient processing efficiency during data acquisition, affecting the real-time, reliability and scalability of the system.
Adopting a multi-level transmission architecture and Hadoop MapReduce framework, through the coordinated work of the data acquisition layer, data transmission layer, data processing layer, data storage layer and data application layer, real-time transmission and parallel processing of data from smart meters to central data centers. Efficient storage and analysis using distributed file systems and NoSQL databases.
It improves the real-time and reliability of data transmission, enhances the data processing efficiency and system scalability, ensures the efficiency and quality of the data acquisition process, and improves the real-time, reliability and scalability of the entire system.
Smart Images

Figure CN119094582A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a smart meter data acquisition system. Background Art
[0002] With the continuous growth of electricity demand and the increasing complexity of the power grid, the traditional power grid system faces many challenges, so it is necessary to introduce advanced information technology and communication technology to improve the efficiency, reliability and sustainability of the power grid. Among them, smart meters are the key components of smart grids, which can collect users' electricity consumption data in real time and transmit it to the central system for processing and analysis.
[0003] At present, the large-scale deployment of smart meters has generated a massive amount of electricity consumption data, which is characterized by high frequency, massiveness, diversity and timeliness. Faced with many challenges such as high-frequency data collection and transmission, storage and management of massive data, real-time data processing and analysis, traditional data processing methods are difficult to effectively cope with, resulting in increased delays, data loss, and low processing efficiency during data collection, affecting the real-time, reliability and scalability of the entire system. Summary of the invention
[0004] The purpose of this application is to solve at least one of the above-mentioned technical defects, especially the technical defects in the prior art such as increased delay, data loss, and low processing efficiency during data acquisition, which affect the real-time, reliability and scalability of the entire system.
[0005] The present application provides a smart meter data acquisition system, the system comprising a data acquisition layer, a data transmission layer, a data processing layer, a data storage layer and a data application layer;
[0006] The data collection layer is used to read the user's original data from the smart meter;
[0007] The data transmission layer is used to sequentially transmit the original data to the local gateway, the regional concentrator and the central data center using a multi-level transmission architecture;
[0008] The data processing layer is constructed using the Hadoop MapReduce framework and is used to pre-process the raw data in the local gateway to obtain intermediate data, and to perform parallel aggregation processing on the intermediate data in the regional concentrator to obtain power consumption data;
[0009] The data storage layer is used to store the original data in a distributed file system, and to store the intermediate data and the power consumption data in a NoSQL database;
[0010] The data application layer is used to perform data analysis on the electricity consumption data using data analysis tools and machine learning languages.
[0011] Optionally, the data transmission layer is used to sequentially transmit the original data to a local gateway, a regional concentrator and a central data center using a multi-stage transmission architecture, including:
[0012] The data transmission layer uses a short-distance transmission method and the MQTT transmission protocol to transmit the original data to a local gateway for preliminary processing, and uses a medium-distance transmission method and the MQTT transmission protocol to transmit the original data after preliminary processing to a regional concentrator for secondary processing, and uses a long-distance transmission method and the AMQP transmission protocol to transmit the original data after secondary processing to a central data center.
[0013] Optionally, the data processing layer is used to pre-process the original data in the local gateway to obtain the intermediate data, including:
[0014] The data processing layer uses a data filtering algorithm in the local gateway to filter the original data, and uses a data aggregation algorithm and a data compression algorithm to aggregate and compress the filtered original data to obtain intermediate data.
[0015] Optionally, the data processing layer is used to perform parallel aggregation processing on the intermediate data in the regional concentrator to obtain power consumption data, including:
[0016] The data processing layer in the regional concentrator determines the processing task of the intermediate data, and uses the MapReduce programming model to divide the intermediate data into multiple data blocks based on the processing task, and processes each data block in parallel to generate power consumption data.
[0017] Optionally, the data processing layer processes each data block in parallel to generate power consumption data, including:
[0018] The data processing layer uses the Map function to process each data block in parallel to generate an intermediate key-value pair for each data block, and uses the Reduce function to aggregate data blocks with the same intermediate key to generate electricity consumption data.
[0019] Optionally, the data processing layer uses a Map function to process each data block in parallel to generate an intermediate key-value pair for each data block, including:
[0020] After extracting the attribute ID and power consumption in each data block, the data processing layer uses a Map function to convert the attribute ID of the data block into a data key and the power consumption of the data block into a data value for each data block, and generates an intermediate key-value pair corresponding to the data block based on the data key and the data value.
[0021] Optionally, the data processing layer uses a Reduce function to aggregate data blocks with the same intermediate key to generate power consumption data, including:
[0022] The data processing layer determines the data key in each intermediate key-value pair, and adds the intermediate key-value pair of the data key to the same data set to form multiple data sets;
[0023] For each data set, the data processing layer uses a Reduce function to aggregate the data values in each intermediate key-value pair in the data set to generate the electricity consumption data corresponding to the data set.
[0024] Optionally, the data storage layer further includes:
[0025] The data storage layer regularly performs data backup and recovery tests on the distributed file system and the NoSQL database, and monitors and manages the distributed file system and the NoSQL database in real time through monitoring and management tools.
[0026] Optionally, the data application layer is used to use data analysis tools and machine learning languages to perform data analysis on the power consumption data, including:
[0027] The data application layer uses Hive to query the electricity consumption data in the NoSQL database and generate an electricity consumption statistical report, and uses Pig to convert the format of the electricity consumption data, so as to use the Mahout learning library and the SparkMllib learning library to perform abnormal analysis and electricity consumption prediction on the converted electricity consumption data.
[0028] Optionally, the data application layer further includes:
[0029] The data application layer uses a data visualization tool to visualize the analysis results of the power consumption data.
[0030] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:
[0031] The application provides a smart meter data acquisition system, including a data acquisition layer, a data transmission layer, a data processing layer, a data storage layer and a data application layer. The data acquisition layer can read the original data of the user from the smart meter, and then the data transmission layer can use a multi-level transmission architecture to transmit the original data to the local gateway, the regional concentrator and the central data center in sequence, so as to improve the real-time and reliability of data transmission; during the transmission process, the data processing layer constructed by using the Hadoop MapReduce framework can pre-process the original data in the local gateway to obtain intermediate data, and parallelly summarize the intermediate data in the regional concentrator to obtain power consumption data, so that the distributed parallel processing of MapReduce can further improve the data processing efficiency and capacity; finally, the data storage layer can store the original data in a distributed file system, and store the intermediate data and power consumption data in a NoSQL database. Here, based on the high scalability and fault tolerance of the distributed file system, the NoSQL database can be used in combination to improve the fast reading and writing operations of the data, so the data application layer can obtain the power consumption data from the NoSQL database in real time for data analysis to ensure the rapid response capability of the system. Based on this, the present application utilizes the mutual cooperation and joint action of various data layers to improve the efficiency and quality of data transmission during the data collection process, thereby improving the real-time, reliability and scalability of the entire system. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0033] Figure 1 A schematic diagram of the structure of a smart meter data acquisition system provided in an embodiment of the present application;
[0034] Figure 2 An application logic diagram of a smart meter data acquisition system provided in an embodiment of the present application. DETAILED DESCRIPTION
[0035] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0036] At present, the large-scale deployment of smart meters has generated a massive amount of electricity consumption data, which is characterized by high frequency, massiveness, diversity and timeliness. Faced with many challenges such as high-frequency data collection and transmission, storage and management of massive data, real-time data processing and analysis, traditional data processing methods are difficult to effectively cope with, resulting in increased delays, data loss, and low processing efficiency during data collection, affecting the real-time, reliability and scalability of the entire system.
[0037] Based on this, this application proposes the following technical solutions, see below for details:
[0038] In one embodiment, Figure 1 As shown, Figure 1 A structural schematic diagram of a smart meter data acquisition system provided in an embodiment of the present application; the present application provides a smart meter data acquisition system, including a data acquisition layer, a data transmission layer, a data processing layer, a data storage layer and a data application layer.
[0039] The data collection layer is used to read the user's raw data from the smart meter.
[0040] The data transmission layer is used to transmit the original data to the local gateway, regional concentrator and central data center in sequence using a multi-level transmission architecture.
[0041] The data processing layer is constructed using the Hadoop MapReduce framework, which is used to pre-process the original data in the local gateway to obtain intermediate data, and to perform parallel aggregation processing on the intermediate data in the regional concentrator to obtain power consumption data.
[0042] The data storage layer is used to store raw data in a distributed file system, and to store intermediate data and power consumption data in a NoSQL database.
[0043] The data application layer is used to analyze electricity consumption data using data analysis tools and machine learning languages.
[0044] In this embodiment, the smart meter data acquisition system of the present application can be functionally divided into five data layers, namely, data acquisition layer, data transmission layer, data processing layer, data storage layer and data application layer. Among them, the data acquisition layer is mainly responsible for collecting data from smart meters, the data transmission layer is mainly responsible for transmitting data in each node, the data processing layer is mainly responsible for processing the collected data, and the data storage layer is mainly responsible for storing the data before and after processing. Finally, the data application layer is responsible for analyzing and applying the processed data, so as to realize a closed loop from data collection to application in the system.
[0045] It is understandable that smart meters can monitor and record relevant data on user electricity consumption in real time, usually collecting data at a frequency of seconds or minutes. These data include various information such as electricity consumption, voltage, current, power factor, etc., which can provide basic data support for subsequent processing and analysis. Based on this, firstly, the present application can read the original data of user electricity consumption from the smart meter in real time through the data collection layer to ensure the real-time nature of the data; then the data transmission layer can adopt a multi-level transmission architecture and an efficient transmission protocol to transmit the original data collected by the data collection layer through the local gateway and concentrator in turn to the central data center, ensuring the reliability, real-time nature and efficiency of data transmission.
[0046] Secondly, in order to reduce the pressure and delay of data transmission, the data processing layer can be equipped with local computing power in the local gateway, so that it can perform simple computing power, so that the original data can be pre-processed in the local gateway to reduce the dependence on the central data center. For example, the original data can be pre-processed by filtering, compressing and other pre-processing operations to extract effective intermediate data, thereby reducing the pressure of data transmission by reducing the amount of data.
[0047] Furthermore, the data processing layer of the present application can be constructed using the Hadoop MapReduce framework. The Hadoop MapReduce framework here is an open source distributed programming model that allows developers to write applications that process large data sets. Its core idea is to decompose the task of big data into multiple small data tasks, which are handed over to Map for distributed processing, and finally combined by Reduce. Therefore, using this data processing layer to process raw data can improve data processing efficiency; for the real-time processing requirements of raw data, the data processing layer can also be combined with memory computing frameworks such as Apache Spark to further improve the real-time and efficiency of data processing.
[0048] Finally, the present application can also store and manage the massive smart meter electricity consumption data transmitted from the data transmission layer through the data storage layer. In order to meet the needs of large-scale data storage and efficient data reading and writing, the smart meter data acquisition system can use the Hadoop distributed file system and the NoSQL database for data storage and management; in the present application, the Hadoop distributed file system can use HDFS, and the NoSQL database can use HBase.
[0049] Among them, HDFS is a highly scalable distributed file system that can be expanded horizontally and increase storage nodes to cope with the growth of data volume. Through the distributed architecture, HDFS can also easily process PB-level data and support parallel read and write operations, thereby improving the efficiency of data storage and reading; in the smart meter data acquisition system, HDFS is mainly responsible for storing the original data of the smart meter. Each data block of the original data can be distributed and stored on different nodes and will be copied to multiple storage nodes. When a node fails, the data can be restored from other replica nodes to ensure that the data is not lost. Therefore, it has a high fault tolerance rate, thereby ensuring reliable storage and high availability of data. As a high-performance distributed NoSQL database, HBase can implement fast random read and write operations on large-scale data sets. It belongs to the column storage model and can process structured and semi-structured data; in the smart meter data acquisition system, HBase can be used to efficiently store and query intermediate data and electricity consumption data to meet the needs of real-time query and analysis of the data application layer. The data application layer here is the top layer of the smart meter data acquisition system, which can implement power consumption pattern analysis, load prediction, fault detection and other functions based on the read intermediate data or electricity consumption data.
[0050] To expand on this, HDFS can split massive amounts of raw data into fixed-size blocks, usually 128MB or 256MB, and then distribute these blocks across multiple nodes. The block size is calculated as follows:
[0051]
[0052] In the formula, Indicates the number of data blocks; Indicates the total size of the original data; Indicates the size of each data block.
[0053] HDFS can also improve data fault tolerance through the replication mechanism. Each data block stores three copies by default, distributed on different nodes. The expression of the replica storage space is as follows:
[0054]
[0055] In the formula, The default value is 3.
[0056] In this application, HDFS mainly uses a master-slave architecture for data management; NameNode is responsible for managing file system metadata, while DataNode is responsible for actual data storage and reading. The smart meter data collection system can obtain data block location information through NameNode, and then directly interact with DataNode to read and write data.
[0057] In addition, HBase uses a column family storage model, where data can be organized and stored according to row keys and column families. The structure of the data model can be shown as follows:
[0058]
[0059] In the formula, Represents the row key; Represents a column family; Represents a column qualifier; Represents the stored value.
[0060] Hbase can divide the table into multiple partitions (Regions) according to the row key range. Each partition is stored on a different RegionServer to achieve horizontal expansion and load balancing of data. The partition division process is as follows:
[0061]
[0062] In the formula, Indicates the row key range, which defines the row key interval contained in each partition.
[0063] It should be noted that when the data storage layer writes data, it can first write it to the MemStore in the memory, then asynchronously write it to the HFile in HDFS, and record it to the WAL (Write-Ahead Log) at the same time to ensure the persistence of the data; the expression of this process can be as follows:
[0064]
[0065] When reading data, the data storage layer can first search in MemStore. If it misses, it searches in HFile. To improve reading performance, HBase can use Bloom Filter and Block Cache for optimization. The expression of this process can be as follows:
[0066]
[0067] Furthermore, encryption technology, such as SSL / TLS, can be used to protect data security during the transmission process of the data transmission layer and the storage process of the data storage layer. At the same time, a complete data access control mechanism is also established in the smart meter data acquisition system, such as identity authentication, authority management, and audit mechanism to ensure that only authorized users can access the data, and use aggregation, generalization or perturbation technology to anonymize data, and remove or replace direct identifiers to de-identify data to protect user privacy.
[0068] In an example, the calculation expression for data anonymization is as follows:
[0069]
[0070] In the formula, Indicates the anonymized result; Represents an anonymization function.
[0071] Specifically, the anonymization function may include techniques such as aggregation, generalization, or perturbation. Aggregation means aggregating detailed data into statistical data, and the specific expression is as follows:
[0072]
[0073] In the formula, Represents the aggregation result; Represents the electricity consumption of the i-th user among n users.
[0074] Generalization means generalizing a specific value into a range. In this application, the range can be: Age∈[20,30].
[0075] In another example, the identifier of data de-identification may include attributes such as ID, name, address, etc., and its calculation expression is as follows:
[0076]
[0077] In the formula, represents the de-identification result, Represents the de-identification function.
[0078] Specifically, the de-identification function may include removing the identification or replacing the identification. Wherein, replacing the identification means replacing the user ID with a pseudo-anonymous ID, and the specific expression is as follows:
[0079]
[0080] In the formula, Indicates the replacement result. Represents a hash function.
[0081] Furthermore, the smart meter data acquisition system can adopt standardized interfaces and protocols, such as RESTful API and OPC UA, to ensure the interoperability and compatibility of the system, and realize data exchange and integration between different systems through message middleware and data bus technology. Among them, ESTful API is an interface design style based on HTTP protocol, which can use standard HTTP methods (such as GET, POST, PUT, DELETE) for data operations; while OPC UA (OLE for Process Control Unified Architecture) is a communication protocol used in industrial automation systems, supporting cross-platform and cross-language data exchange. In addition, the message middleware used by the smart meter data acquisition system can be Apache Kafka, and the data bus technology can be Apache Camel.
[0082] In the above embodiment, the smart meter data collection system includes a data collection layer, a data transmission layer, a data processing layer, a data storage layer and a data application layer. The data collection layer can read the original data of the user from the smart meter, and then the data transmission layer can use a multi-level transmission architecture to transmit the original data to the local gateway, the regional concentrator and the central data center in sequence, so as to improve the real-time and reliability of data transmission; during the transmission process, the data processing layer constructed by using the Hadoop MapReduce framework can pre-process the original data in the local gateway to obtain intermediate data, and perform parallel aggregation processing on the intermediate data in the regional concentrator to obtain power consumption data, so that the distributed parallel processing of MapReduce can further improve the data processing efficiency and capacity; finally, the data storage layer can store the original data in a distributed file system, and store the intermediate data and power consumption data in a NoSQL database. Here, based on the high scalability and fault tolerance of the distributed file system, the NoSQL database can be used in combination to improve the fast reading and writing operations of the data, so the data application layer can obtain the power consumption data from the NoSQL database in real time for data analysis to ensure the rapid response capability of the system. Based on this, the present application utilizes the mutual cooperation and joint action of various data layers to improve the efficiency and quality of data transmission during the data collection process, thereby improving the real-time, reliability and scalability of the entire system.
[0083] In one embodiment, the data transmission layer is used to sequentially transmit the original data to the local gateway, the regional concentrator and the central data center using a multi-stage transmission architecture, which may include:
[0084] The data transmission layer uses short-distance transmission and the MQTT transmission protocol to transmit the original data to the local gateway for preliminary processing, and uses medium-distance transmission and the MQTT transmission protocol to transmit the original data after preliminary processing to the regional concentrator for secondary processing, and uses long-distance transmission and the AMQP transmission protocol to transmit the original data after secondary processing to the central data center.
[0085] In this embodiment, the local gateway can receive the original data of multiple smart meters for preliminary aggregation and processing to ensure the integrity of the data; the regional concentrator can receive data from multiple local gateways, further process and store the data, and ensure that the data is not lost during transmission; the central data center can receive and store data from each regional concentrator, perform unified management and processing, and provide a basis for data analysis and decision-making.
[0086] It can be understood that, according to the distribution characteristics of the local gateway, regional concentrator and central data center, the data transmission layer can use a short-distance transmission method to transmit the original data to the local gateway for preliminary processing, and use a medium-distance transmission method to transmit the original data after preliminary processing to the regional concentrator for secondary processing, and use a long-distance transmission method to transmit the original data after secondary processing to the central data center; in addition, the data transmission layer can also use the MQTT transmission protocol and the AMQP transmission protocol as data transmission protocols to further improve data transmission efficiency.
[0087] Among them, the MQTT transmission protocol refers to a lightweight message transmission protocol designed for low-bandwidth, unreliable network environments. It has the characteristics of low bandwidth occupancy, low latency and high reliability, and is suitable for data transmission of IoT devices; therefore, in the smart meter data acquisition system, the MQTT transmission protocol can be used for data transmission between smart meters and local gateways, and between local gateways and regional concentrators to ensure the real-time and reliability of data. The AMQP transmission protocol is an open source standardized message transmission protocol that supports reliable message transmission and queue management. It has the characteristics of high reliability, high flexibility and security, and is suitable for enterprise-level applications; therefore, in the smart meter data acquisition system, the AMQP transmission protocol can be used for data transmission between regional concentrators and central data centers to ensure that data is not lost during the data transmission process, and supports complex message routing and processing mechanisms.
[0088] Specifically, the short-distance transmission method can be wireless communication, such as ZigBee and LoRa, or wired communication, such as PLC and power line communication. There is no restriction here. These communication technologies have the characteristics of low power consumption, wide coverage and high reliability, and are suitable for short-distance data transmission within the cell; medium-distance transmission methods refer to more efficient transmission methods, such as Ethernet, cellular networks, etc. These transmission methods have higher bandwidth and transmission rate, and are suitable for medium-distance data transmission; long-distance transmission methods refer to high-speed networks, such as fiber optic networks, the Internet, etc. These transmission methods have extremely high bandwidth and reliability, and are suitable for long-distance data transmission.
[0089] In one embodiment, the data processing layer is used to pre-process the original data in the local gateway to obtain the intermediate data, which may include:
[0090] The data processing layer uses a data filtering algorithm in the local gateway to filter the original data, and uses a data aggregation algorithm and a data compression algorithm to aggregate and compress the filtered original data to obtain intermediate data.
[0091] In this embodiment, the data processing layer can perform pre-processing operations on the original data in the local gateway, including filtering the original data using a data filtering algorithm, and aggregating and compressing the filtered original data using a data aggregation algorithm and a data compression algorithm to obtain intermediate data, thereby reducing the frequency and amount of data transmission and reducing the burden on the communication network.
[0092] Specifically, data compression algorithms can be used to reduce the volume of original data, thereby reducing the network bandwidth usage. They usually include Huffman coding and Delta coding. Among them, Huffman coding can achieve lossless data compression by constructing an optimal binary tree; while Delta coding is suitable for time series data, which reduces the amount of data by storing the difference between adjacent data points. In short, the specific expression of the data compression algorithm can be shown as follows:
[0093]
[0094] In the formula, Indicates the size of the original data; Indicates the size of the compressed data; C indicates the compression ratio.
[0095] Data aggregation algorithms can be used to aggregate and process data to reduce the amount of data transmitted; common aggregation operations include summation, average, maximum / minimum. Sum means calculating the total power consumption within a certain period of time, and its expression is as follows:
[0096]
[0097] In the formula, Indicates the sum result; Represents the power consumption of the i-th data point among n data points.
[0098] The average value means calculating the average power consumption within a certain period of time. The specific expression is as follows:
[0099]
[0100] To the middle, represents the mean value; n represents the number of data points.
[0101] The maximum / minimum value indicates the maximum and minimum power consumption within a period of time. The specific expression is as follows:
[0102]
[0103] Data filtering algorithms can be used to filter and extract important data, remove noise and unnecessary data, and can usually include low-pass filtering and threshold filtering. Among them, low-pass filtering is used to smooth time series data and filter out high-frequency noise. Its expression is as follows:
[0104]
[0105] In the formula, Represents the output of filtered data; Represents input data; Represents the filter coefficient.
[0106] Threshold filtering means that only data points exceeding a certain threshold are transmitted. The specific expression is as follows:
[0107]
[0108] Where T represents the preset filtering threshold.
[0109] In one embodiment, the data processing layer is used to perform parallel aggregation processing on the intermediate data in the regional concentrator to obtain the power consumption data, which may include:
[0110] The data processing layer determines the processing tasks of the intermediate data in the regional concentrator, and uses the MapReduce programming model to divide the intermediate data into multiple data blocks based on the processing tasks, and processes each data block in parallel to generate power consumption data.
[0111] In this embodiment, the MapReduce programming model is a distributed computing model, which can perform batch processing and analysis on massive data based on processing tasks. The processing tasks here can be set according to the actual needs of the smart meter data acquisition system, such as daily electricity consumption statistics, abnormal electricity consumption behavior monitoring, load forecasting, regional electricity consumption statistics and comparison, etc., without any restrictions here.
[0112] It is understandable that the MapReduce programming model improves data processing efficiency through parallel processing and can process massive data sets. It can execute tasks in parallel on different computing nodes, utilize the computing resources of the cluster, and improve the throughput of data processing.
[0113] In addition, the MapReduce programming model can also be used in conjunction with the Apache Spark memory computing framework. Apache Spark here refers to an efficient memory computing framework that can perform distributed data processing tasks in memory, avoiding frequent disk I / O operations. Spark's memory computing model greatly improves the speed of data processing and can perform real-time data processing and iterative calculations. In the smart meter data collection system, Spark can be used to process and analyze the transmitted power consumption data in real time. For example, it can monitor the user's power consumption in real time and promptly discover and respond to abnormal power consumption behavior. In addition, Spark provides a resilient distributed dataset (RDD) as its core data abstraction, supporting parallel operations on large-scale data sets; here, RDD can process data through a series of transformations and actions, and provide a fault recovery mechanism, so that complex analysis tasks can be efficiently performed, such as power consumption pattern recognition and load forecasting.
[0114] In summary, the data processing layer can use the MapReduce programming model to perform batch processing and analysis on intermediate data, and regularly execute MapReduce jobs to aggregate, count and analyze large-scale data, and generate reports and analysis results. At the same time, Apache Spark can be used to process and analyze intermediate data transmitted in real time, and through components such as Spark Streaming, it can realize the processing of real-time data streams, detect abnormal situations in a timely manner and take corresponding measures.
[0115] In one embodiment, the data processing layer processes each data block in parallel to generate power consumption data, which may include:
[0116] The data processing layer uses the Map function to process each data block in parallel, generates an intermediate key-value pair for each data block, and uses the Reduce function to aggregate data blocks with the same intermediate key to generate electricity consumption data.
[0117] In this embodiment, the data processing layer can divide the processing task into two stages through the MapReduce programming model: the Map stage and the Reduce stage. In the Map stage, the data processing layer can use the Map function to process each data block in parallel to generate an intermediate key-value pair; in the Reduce stage, the data processing layer can use the Reduce function to aggregate all data blocks with the same intermediate key to generate the final power consumption data.
[0118] Specifically, in the Map phase, the Map function can receive a pair of key values as input and generate a set of intermediate key-value pairs: ;in, and Represents the input key-value pair. and Represents an intermediate key-value pair; the data processing layer can then use the intermediate key-value pair as Group by values with the same key and pass them to the Reduce function: ; Among them, with the same key Value In the Reduce phase, the Reduce function receives a key and an associated set of values and generates one or more output results: ;in, and It is the key-value pair of the final output.
[0119] In one embodiment, the data processing layer uses a Map function to process each data block in parallel, and the process of generating an intermediate key-value pair for each data block may include:
[0120] After extracting the attribute ID and power consumption in each data block, the data processing layer uses the Map function to convert the attribute ID of the data block into a data key and the power consumption of the data block into a data value for each data block, and generates an intermediate key-value pair corresponding to the data block based on the data key and data value.
[0121] In this embodiment, the data processing layer can first extract the attribute ID and power consumption in each data block, where the attribute ID can be one or a combination of multiple attributes such as user ID, address, and timestamp, and is not limited here. Then, for each data block, the data processing layer can use the Map function to convert the attribute ID of the data block into a data key, and convert the power consumption of the data block into a data value, and generate an intermediate key-value pair corresponding to the data block according to the data key and the data value.
[0122] For example, when the processing task of intermediate data is daily electricity consumption statistics, the data processing layer can extract the user ID, timestamp and electricity consumption of each data block in the intermediate data. The data format generated after extraction is as follows:
[0123]
[0124] Based on the above table, the goal of the Map function is to convert each record in the above table into an intermediate key-value pair; where the data key is "user ID_timestamp" and the data value is "electricity consumption". For example, for the data of user_001 on 2023-07-24, the intermediate key-value pairs output by the Map function are ("user_001_2023-07-24", 1.5), ("user_001_2023-07-24", 1.2), ("user_001_2023-07-24", 1.0).
[0125] In one embodiment, the data processing layer uses the Reduce function to aggregate data blocks with the same intermediate key to generate power consumption data, which may include:
[0126] The data processing layer determines the data key in each intermediate key-value pair, and adds the intermediate key-value pairs with consistent data keys to the same data set to form multiple data sets.
[0127] For each data set, the data processing layer uses the Reduce function to aggregate the data values in each intermediate key-value pair in the data set to generate the electricity consumption data corresponding to the data set.
[0128] In this embodiment, the data processing layer can first determine the data key in each intermediate key-value pair, and add the intermediate key-value pairs with consistent data keys to the same data set to form multiple data sets, and then transmit each data set to each Reduce task node, so that the data values in each intermediate key-value pair in the data set can be aggregated in parallel to generate electricity consumption data corresponding to each data set.
[0129] For example, for the data set corresponding to each attribute ID, the Reduce function can sort the intermediate key-value pairs in the data set in chronological order in its corresponding Reduce task node to ensure that the data corresponding to the same data key can be aggregated in order. For example, in a data set with a data key of "user_001_2023-07-24", which contains intermediate key-value pairs ("user_001_2023-07-24", 1.5), ("user_001_2023-07-24", 1.2), and ("user_001_2023-07-24", 1.0), the data processing layer can first aggregate the data values of the data set to obtain an aggregate key-value pair ("user_001_2023-07-24", [1.5, 1.2, 1.0]); then the data processing layer can use the Reduce function to aggregate the aggregate key-value pair, that is, to obtain a data value list [1.5, 1.2, 1.0] for the data key "user_001_2023-07-24". 1.0] are added to obtain the electricity consumption data of the data set ("user_001_2023-07-24", 3.7).
[0130] In one embodiment, the data storage layer may further include:
[0131] The data storage layer regularly performs data backup and recovery tests on the distributed file system and NoSQL database, and monitors and manages the distributed file system and NoSQL database in real time through monitoring and management tools.
[0132] In this embodiment, the data storage layer can regularly perform data backup and recovery tests on the distributed file system and NoSQL database to ensure the security and integrity of the data, and through monitoring and management tools such as Ambari and Cloudera Manager, the distributed file system and NoSQL database can be monitored and managed in real time to ensure the stable operation of the smart meter data collection system.
[0133] In one embodiment, the process of using the data application layer to analyze the power consumption data using data analysis tools and machine learning languages may include:
[0134] The data application layer uses Hive to query the electricity consumption data in the NoSQL database and generate electricity consumption statistical reports, and uses Pig to convert the format of electricity consumption data, so as to use the Mahout learning library and Spark Mllib learning library to perform anomaly analysis and electricity consumption prediction on the converted electricity consumption data.
[0135] In this embodiment, the data application layer serves as the top layer of the smart meter data collection system. It is mainly responsible for analyzing the power consumption data, helping to make data-driven decisions, and realizing functions such as power consumption pattern analysis, load forecasting, and fault detection through various data analyses.
[0136] Among them, Hive is a data warehouse tool based on Hadoop, which supports querying and analyzing large-scale data sets using a query language similar to SQL. It converts queries into MapReduce jobs and can process and analyze historical data in batches. Therefore, in this application, the data application layer can use Hive to query and preliminarily analyze historical electricity consumption data stored in the distributed file system; for example, Hive can be used to query electricity consumption in different time periods and analyze users' electricity consumption habits. Pig is a high-level data flow scripting language that can simplify the processing of large-scale data sets. It can be converted into a series of MapReduce jobs for execution, which is suitable for data pipelines and ETL (extraction, transformation, loading) operations; therefore, the data application layer can use Pig for data cleaning and preprocessing; for example, through the Pig script, electricity consumption data can be converted into a data format suitable for analysis.
[0137] In addition, the Mahout learning library refers to a distributed machine learning library suitable for machine learning tasks of large-scale data sets. It provides a variety of commonly used algorithms, such as clustering, classification, and collaborative filtering. In the data analysis of this application, the data application layer can use the Mahout learning library to classify and cluster users' electricity consumption behaviors to identify the electricity consumption patterns of different user groups. The Spark Mllib learning library is a Spark machine learning library that provides a wealth of machine learning algorithms and tools and supports fast execution of machine learning tasks in memory. In the data analysis of this application, the data application layer can use the Spark Mllib learning library for load forecasting, anomaly detection, and user behavior analysis. For example, the time series prediction algorithm in the SparkMLlib learning library can be used to predict future electricity demand and help optimize resource allocation.
[0138] In one embodiment, the data application layer may further include:
[0139] The data application layer uses data visualization tools to visualize the analysis results of electricity consumption data.
[0140] In this embodiment, the data application layer can also use data visualization tools to visualize the analysis results of electricity consumption data, such as using Tableau and Power BI to create interactive charts and dashboards, and intuitively display the analysis results to management and users, such as displaying electricity consumption trend charts, regional electricity consumption distribution charts, abnormal electricity consumption alerts, etc., without limitation here.
[0141] Among them, Tableau refers to a data visualization tool that supports access to multiple data sources and rich visualization functions. Users can create interactive charts and dashboards by dragging and dropping. Therefore, in this application, the data application layer can use Tableau to intuitively display the analysis results to users, such as creating a real-time monitoring dashboard to display the real-time power consumption of each area, power consumption trend charts, etc.; and Power BI can support data integration, data analysis and visualization, and provide rich chart types and powerful data analysis functions. In this application, the data application layer can use Power BI to create various data reports and visualization charts to help management make data-driven decisions, such as generating monthly power consumption reports, charts to predict future power demand, etc.
[0142] In order to better explain the text matching method of this application, the following will be Figure 2 To further illustrate, schematically, Figure 2 As shown, Figure 2 An application logic diagram of a smart meter data acquisition system provided in an embodiment of the present application.
[0143] Figure 2 In the smart meter data collection system, after collecting data from the smart meter, it can use SSL / TLS encryption to transmit the data, and perform preliminary processing and aggregation of the collected data during the transmission process, and then synchronously store the data before and after processing in databases such as HDFS to reflect its high scalability and high availability; when it is necessary to further summarize and analyze these data, the smart meter data collection system can also perform in-depth processing and analysis on the data after preliminary processing. In this process, MapReduce and Spark can be used for efficient parallel processing and storage in the database; then, when applied, the smart meter data collection system can use Hive and Pig to query and preliminarily process the data in the database, and use Mahout and MLlib for machine learning and data mining to obtain the final results and visualize them. In the process of data application, the smart meter data collection system can use RESTful and OPC UA for standardized interfaces, and use Kafka and Camel for data exchange and integration to improve system integration and compatibility; at the same time, the smart meter data collection system can also use SSL / TLS for data encryption, use RBAC for access control, and protect privacy through data anonymization, thereby improving the data security and privacy protection of the system.
[0144] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0145] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.
[0146] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A smart meter data acquisition system, It is characterized in that The system includes a data acquisition layer, a data transmission layer, a data processing layer, a data storage layer and a data application layer; The data collection layer is used to read the user's original data from the smart meter; The data transmission layer is used to sequentially transmit the original data to the local gateway, the regional concentrator and the central data center using a multi-level transmission architecture; The data processing layer is constructed using the Hadoop MapReduce framework and is used to pre-process the raw data in the local gateway to obtain intermediate data, and to perform parallel aggregation processing on the intermediate data in the regional concentrator to obtain power consumption data; The data storage layer is used to store the original data in a distributed file system, and to store the intermediate data and the power consumption data in a NoSQL database; The data application layer is used to perform data analysis on the electricity consumption data using data analysis tools and machine learning languages.
2. The smart meter data acquisition system according to claim 1, It is characterized in that The data transmission layer is used to sequentially transmit the original data to the local gateway, the regional concentrator and the central data center using a multi-level transmission architecture, including: The data transmission layer uses a short-distance transmission method and the MQTT transmission protocol to transmit the original data to a local gateway for preliminary processing, and uses a medium-distance transmission method and the MQTT transmission protocol to transmit the original data after preliminary processing to a regional concentrator for secondary processing, and uses a long-distance transmission method and the AMQP transmission protocol to transmit the original data after secondary processing to a central data center.
3. The smart meter data acquisition system according to claim 1, It is characterized in that The data processing layer is used to pre-process the original data in the local gateway to obtain the intermediate data, including: The data processing layer uses a data filtering algorithm in the local gateway to filter the original data, and uses a data aggregation algorithm and a data compression algorithm to aggregate and compress the filtered original data to obtain intermediate data.
4. The smart meter data acquisition system according to claim 1, It is characterized in that The data processing layer is used to perform parallel aggregation processing on the intermediate data in the regional concentrator to obtain the power consumption data, including: The data processing layer in the regional concentrator determines the processing task of the intermediate data, and uses the MapReduce programming model to divide the intermediate data into multiple data blocks based on the processing task, and processes each data block in parallel to generate power consumption data.
5. The smart meter data acquisition system according to claim 4, It is characterized in that The data processing layer processes each data block in parallel to generate power consumption data, including: The data processing layer uses the Map function to process each data block in parallel to generate an intermediate key-value pair for each data block, and uses the Reduce function to aggregate data blocks with the same intermediate key to generate electricity consumption data.
6. The smart meter data acquisition system according to claim 5, It is characterized in that The data processing layer uses the Map function to process each data block in parallel to generate an intermediate key-value pair for each data block, including: After extracting the attribute ID and power consumption in each data block, the data processing layer uses a Map function to convert the attribute ID of the data block into a data key and the power consumption of the data block into a data value for each data block, and generates an intermediate key-value pair corresponding to the data block based on the data key and the data value.
7. The smart meter data acquisition system according to claim 5, It is characterized in that The data processing layer uses the Reduce function to aggregate data blocks with the same intermediate key to generate power consumption data, including: The data processing layer determines the data key in each intermediate key-value pair, and adds the intermediate key-value pairs with the same data key to the same data set to form multiple data sets; For each data set, the data processing layer uses a Reduce function to aggregate the data values in each intermediate key-value pair in the data set to generate the electricity consumption data corresponding to the data set.
8. The smart meter data acquisition system according to claim 1, It is characterized in that The data storage layer also includes: The data storage layer regularly performs data backup and recovery tests on the distributed file system and the NoSQL database, and monitors and manages the distributed file system and the NoSQL database in real time through monitoring and management tools.
9. The smart meter data acquisition system according to claim 1, It is characterized in that The data application layer is used to use data analysis tools and machine learning languages to analyze the power consumption data, including: The data application layer uses Hive to query the electricity consumption data in the NoSQL database and generate an electricity consumption statistical report, and uses Pig to convert the format of the electricity consumption data, so as to use the Mahout learning library and the Spark Mllib learning library to perform abnormal analysis and electricity consumption prediction on the converted electricity consumption data.
10. The smart meter data acquisition system according to claim 1 or 9, It is characterized in that The data application layer also includes: The data application layer uses a data visualization tool to visualize the analysis results of the power consumption data.
Citation Information
Patent Citations
Method for directly accessing hardware data into a big data platform
CN109657003A
Control method of industrial digital cloud twin system based on 5G and IPv6
CN117527864A
Enterprise multi-level data real-time acquisition and processing system based on concrete industry
CN117762102A
Data processing system and method in industrial Internet of Things gateway
CN117872934A
Collection gateway multi-monitoring-point data aggregation reporting method and collection gateway multi-monitoring-point data aggregation reporting system
CN118368345A
Cited By
Electric energy meter data acquisition system suitable for multiple scenes
CN122349070A
