A smart meter data acquisition system

Abstract

This application provides a smart meter data acquisition system comprising five data layers. The data acquisition layer reads raw data from the smart meter and utilizes a multi-level transmission architecture in the data transmission layer to improve real-time data transmission. During transmission, the data processing layer employs MapReduce to preprocess the raw data into intermediate data at the local gateway and then aggregates the intermediate data rows into electricity consumption data at the regional concentrator, thereby improving data processing efficiency. Simultaneously, the data storage layer stores the obtained data in a distributed file system and a NoSQL database to enhance fast read and write operations. Therefore, the data application layer can acquire electricity consumption data in real time for data analysis, ensuring the system's rapid response capability. Based on this, this application improves data transmission efficiency and quality during data acquisition through the collaboration of each data layer, thereby enhancing the real-time performance and reliability of the entire system.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a smart meter data acquisition system. Background Technology

[0002] With the continuous growth of electricity demand and the increasing complexity of power grids, traditional power grid systems face many challenges. Therefore, it is necessary to introduce advanced information and communication technologies to improve the efficiency, reliability, and sustainability of the power grid. Among these technologies, smart meters are a key component of smart grids, capable of collecting users' electricity consumption data in real time and transmitting it to a central system for processing and analysis.

[0003] Currently, the large-scale deployment of smart meters has generated massive amounts of electricity consumption data, characterized by high frequency, massive volume, diversity, and timeliness. Traditional data processing methods struggle to effectively address the numerous challenges posed by high-frequency data acquisition and transmission, massive data storage and management, and real-time data processing and analysis. This results in increased latency, data loss, and low processing efficiency during data acquisition, impacting the real-time performance, reliability, and scalability of the entire system. Summary of the Invention

[0004] The purpose of this application is to at least solve one of the aforementioned technical defects, particularly the technical defects in the prior art that cause increased latency, data loss, and low processing efficiency during data acquisition, affecting the real-time performance, reliability, and scalability of the entire system.

[0005] This application provides a smart meter data acquisition system, which includes a data acquisition layer, a data transmission layer, a data processing layer, a data storage layer, and a data application layer;

[0006] The data acquisition layer is used to read the user's raw data from the smart meter;

[0007] The data transmission layer is used to transmit the raw data sequentially to the local gateway, the regional concentrator, and the central data center using a multi-level transmission architecture.

[0008] The data processing layer is built using the Hadoop MapReduce framework. It is used to preprocess the raw data in the local gateway to obtain intermediate data, and to perform parallel aggregation processing on the intermediate data in the regional concentrator to obtain electricity consumption data.

[0009] The data storage layer is used to store the raw data in a distributed file system, and to store the intermediate data and the electricity consumption data in a NoSQL database;

[0010] The data application layer is used to perform data analysis on the electricity consumption data using data analysis tools and machine learning languages.

[0011] Optionally, the data transmission layer is used to sequentially transmit the raw data to the local gateway, regional concentrator, and central data center using a multi-level transmission architecture, including:

[0012] The data transmission layer uses short-distance transmission and the MQTT transmission protocol to transmit the raw data to the local gateway for preliminary processing, uses medium-distance transmission and the MQTT transmission protocol to transmit the pre-processed raw data to the regional concentrator for secondary processing, and uses long-distance transmission and the AMQP transmission protocol to transmit the secondary-processed raw data to the central data center.

[0013] Optionally, the data processing layer is used to preprocess the raw data in the local gateway to obtain intermediate data, including:

[0014] The data processing layer uses a data filtering algorithm to filter the original data in the local gateway, and uses a data aggregation algorithm and a data compression algorithm to aggregate and compress the filtered original data to obtain intermediate data.

[0015] Optionally, the data processing layer is used to perform parallel aggregation processing on the intermediate data in the regional concentrator to obtain electricity consumption data, including:

[0016] The data processing layer, within the regional concentrator, determines the processing task for the intermediate data, and uses the MapReduce programming model to divide the intermediate data into multiple data blocks based on the processing task, and processes each data block in parallel to generate electricity consumption data.

[0017] Optionally, the data processing layer performs parallel processing on each data block to generate electricity consumption data, including:

[0018] The data processing layer uses a Map function to process each data block in parallel, generating intermediate key-value pairs for each data block, and uses a Reduce function to aggregate data blocks with the same intermediate key to generate electricity consumption data.

[0019] Optionally, the data processing layer uses a Map function to process each data block in parallel, generating intermediate key-value pairs for each data block, including:

[0020] After extracting the attribute ID and power consumption from each data block, the data processing layer uses a Map function to convert the attribute ID of the data block into a data key and the power consumption of the data block into a data value, and generates an intermediate key-value pair corresponding to the data block based on the data key and the data value.

[0021] Optionally, the data processing layer uses the Reduce function to aggregate data blocks with the same intermediate key to generate electricity consumption data, including:

[0022] The data processing layer determines the data key in each intermediate key-value pair and adds the intermediate key-value pair of the data key to the same data set to form multiple data sets;

[0023] For each dataset, the data processing layer uses the Reduce function to aggregate the data values ​​in each intermediate key-value pair within the dataset to generate the electricity consumption data corresponding to that dataset.

[0024] Optionally, the data storage layer further includes:

[0025] The data storage layer periodically performs data backup and recovery tests on the distributed file system and the NoSQL database, and monitors and manages the distributed file system and the NoSQL database in real time through monitoring and management tools.

[0026] Optionally, the data application layer is used to perform data analysis on the electricity consumption data using data analysis tools and machine learning languages, including:

[0027] The data application layer uses Hive to query electricity consumption data in the NoSQL database and generate electricity consumption statistics reports. It also uses Pig to convert the electricity consumption data into a format, and uses the Mahout learning library and SparkMLlib learning library to perform anomaly analysis and electricity consumption prediction on the converted electricity consumption data.

[0028] Optionally, the data application layer further includes:

[0029] The data application layer uses data visualization tools to visualize the analysis results of the electricity consumption data.

[0030] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0031] This application provides a smart meter data acquisition system, comprising a data acquisition layer, a data transmission layer, a data processing layer, a data storage layer, and a data application layer. The data acquisition layer reads raw data from the smart meter. The data transmission layer employs a multi-level transmission architecture to sequentially transmit the raw data to a local gateway, a regional concentrator, and a central data center, thereby improving the real-time performance and reliability of data transmission. During transmission, the data processing layer, built using the Hadoop MapReduce framework, preprocesses the raw data at the local gateway to obtain intermediate data. The regional concentrator then performs parallel aggregation processing on the intermediate data to obtain electricity consumption data. Through the distributed parallel processing of MapReduce, the data processing efficiency and capacity are further improved. Finally, the data storage layer stores the raw data in a distributed file system and stores the intermediate data and electricity consumption data in a NoSQL database. The combination of the high scalability and fault tolerance of the distributed file system with the use of a NoSQL database improves the speed of data read and write operations. Therefore, the data application layer can retrieve electricity consumption data from the NoSQL database in real time for data analysis, ensuring the system's rapid response capability. Based on this, this application utilizes the mutual cooperation and joint action of various data layers to improve the efficiency and quality of data transmission during the data acquisition process, thereby improving the real-time performance, reliability, and scalability of the entire system. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a schematic diagram of the structure of a smart meter data acquisition system provided in an embodiment of this application;

[0034] Figure 2 This is an application logic diagram of a smart meter data acquisition system provided in an embodiment of this application. Detailed Implementation

[0035] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0036] Currently, the large-scale deployment of smart meters has generated massive amounts of electricity consumption data, characterized by high frequency, massive volume, diversity, and timeliness. Traditional data processing methods struggle to effectively address the numerous challenges posed by high-frequency data acquisition and transmission, massive data storage and management, and real-time data processing and analysis. This results in increased latency, data loss, and low processing efficiency during data acquisition, impacting the real-time performance, reliability, and scalability of the entire system.

[0037] Based on this, this application proposes the following technical solution, as detailed below:

[0038] In one embodiment, such as Figure 1 As shown, Figure 1 This is a schematic diagram of the structure of a smart meter data acquisition system provided in an embodiment of this application. This application provides a smart meter data acquisition system, including a data acquisition layer, a data transmission layer, a data processing layer, a data storage layer, and a data application layer.

[0039] The data acquisition layer is used to read raw user data from smart meters.

[0040] The data transmission layer is used to transmit raw data sequentially to the local gateway, regional concentrator, and central data center using a multi-level transmission architecture.

[0041] The data processing layer is built using the Hadoop MapReduce framework. It is used to preprocess the raw data in the local gateway to obtain intermediate data, and to perform parallel aggregation processing on the intermediate data in the regional concentrator to obtain electricity consumption data.

[0042] The data storage layer is used to store raw data in a distributed file system, and intermediate data and electricity consumption data in a NoSQL database.

[0043] The data application layer is used to perform data analysis on electricity consumption data using data analysis tools and machine learning languages.

[0044] In this embodiment, the smart meter data acquisition system of this application can be functionally divided into five data layers: a data acquisition layer, a data transmission layer, a data processing layer, a data storage layer, and a data application layer. The data acquisition layer is primarily responsible for acquiring data from the smart meter; the data transmission layer is primarily responsible for transmitting data between various nodes; the data processing layer is primarily responsible for processing the acquired data; the data storage layer is primarily responsible for storing the data before and after processing; and finally, the data application layer is responsible for analyzing and applying the processed data. This system achieves a closed loop in the data acquisition and application process.

[0045] Understandably, smart meters can monitor and record users' electricity consumption data in real time, typically collecting data at a frequency of seconds or minutes. This data includes various information such as electricity consumption, voltage, current, and power factor, providing basic data support for subsequent processing and analysis. Based on this, this application firstly reads the raw data of users' electricity consumption from the smart meter in real time through the data acquisition layer, ensuring the real-time nature of the data; then, the data transmission layer can adopt a multi-level transmission architecture and an efficient transmission protocol to transmit the raw data collected by the data acquisition layer sequentially through the local gateway and concentrator before finally transmitting it to the central data center, ensuring the reliability, real-time nature, and efficiency of data transmission.

[0046] Secondly, to reduce data transmission pressure and latency, the data processing layer can be equipped with local computing capabilities in the local gateway, enabling it to perform simple calculations. This allows for preliminary preprocessing of raw data at the local gateway, reducing reliance on a central data center. For example, filtering and compression can be performed on the raw data to extract effective intermediate data, thereby reducing the data volume and lowering the data transmission pressure.

[0047] Furthermore, the data processing layer of this application can be built using the Hadoop MapReduce framework. The Hadoop MapReduce framework is an open-source distributed programming model that allows developers to write applications for processing large datasets. Its core idea is to decompose large-scale data tasks into multiple smaller data tasks, which are then processed in a distributed manner by the Map class, and finally merged by the Reduce class. Therefore, using this data processing layer to process raw data can improve data processing efficiency. For real-time processing requirements of raw data, the data processing layer can also be combined with in-memory computing frameworks such as Apache Spark to further improve the real-time performance and efficiency of data processing.

[0048] Finally, this application can also use a data storage layer to store and manage the massive amounts of electricity consumption data from smart meters transmitted from the data transmission layer. To meet the needs of large-scale data storage and efficient data reading and writing, the smart meter data acquisition system can use a Hadoop Distributed File System and a NoSQL database for data storage and management; in this application, the Hadoop Distributed File System can be HDFS, and the NoSQL database can be HBase.

[0049] HDFS is a highly scalable distributed file system capable of horizontal scaling, adding storage nodes to handle increasing data volume. Through its distributed architecture, HDFS can easily handle petabyte-scale data and supports parallel read / write operations, thereby improving data storage and retrieval efficiency. In the smart meter data acquisition system, HDFS primarily stores the raw data from the smart meters. Each data block of this raw data can be distributed across different nodes and replicated to multiple storage nodes. When a node fails, data can be recovered from other replica nodes, ensuring no data loss and thus providing high fault tolerance, ensuring reliable data storage and high availability. HBase, as a high-performance distributed NoSQL database, enables fast random read / write operations on large-scale datasets. It uses a column-oriented storage model and can handle structured and semi-structured data. In the smart meter data acquisition system, HBase can be used to efficiently store and query intermediate data and electricity consumption data, meeting the real-time query and analysis needs of the data application layer. This data application layer is the top layer of the smart meter data acquisition system, enabling functions such as electricity consumption pattern analysis, load forecasting, and fault detection based on the read intermediate or electricity consumption data.

[0050] In more detail, HDFS can divide massive amounts of raw data into fixed-size data blocks, typically 128MB or 256MB. These data blocks can then be distributed across multiple nodes. The calculation method for the data block size is as follows:

[0051]

[0052] In the formula, Indicates the number of data blocks; Indicates the total size of the original data; This indicates the size of each data block.

[0053] HDFS can also improve data fault tolerance through a replication mechanism. Each data block is stored with three replicas by default, distributed across different nodes. The expression for the replica storage space is as follows:

[0054]

[0055] In the formula, The time-replication factor has a default value of 3.

[0056] In this application, HDFS primarily uses a master-slave architecture for data management; the NameNode is responsible for managing file system metadata, while the DataNodes are responsible for the actual data storage and retrieval. The smart meter data acquisition system can obtain data block location information through the NameNode and then directly interact with the DataNodes to read and write data.

[0057] In addition, HBase employs a column family storage model, where data can be organized and stored according to row keys and column families. The structure of this data model can be seen as follows:

[0058]

[0059] In the formula, Indicates row key; Indicates a family or clan; Indicates column qualifiers; This represents the stored value.

[0060] HBase can divide a table into multiple partitions (Regions) based on row key ranges, with each partition stored on a different RegionServer, achieving horizontal scaling and load balancing. The partitioning process is as follows:

[0061]

[0062] In the formula, This indicates the row key range, defining the range of row keys contained in each partition.

[0063] It should be noted that when the data storage layer writes data, it can first write to the MemStore in memory, and then asynchronously write to the HFile in HDFS, while simultaneously recording it in the WAL (Write-Ahead Log) to ensure data persistence; the expression for this process can be shown below:

[0064]

[0065] When reading data, the data storage layer can first search in MemStore; if the data is not found there, it can then search in HFile. To improve read performance, HBase can use Bloom Filters and Block Caches for optimization. The expression for this process can be shown below:

[0066]

[0067] Furthermore, encryption technologies, such as SSL / TLS, can be used to protect data security during the transmission process at the data transmission layer and the storage process at the data storage layer. Simultaneously, the smart meter data acquisition system also establishes a comprehensive data access control mechanism, such as authentication, access control, and auditing mechanisms, to ensure that only authorized users can access the data. It also employs aggregation, generalization, or perturbation techniques for data anonymization and removes or replaces direct identifiers for data de-identification to protect user privacy.

[0068] In one example, the computational expression for data anonymization is shown below:

[0069]

[0070] In the formula, This indicates the anonymization result; This represents an anonymous function.

[0071] Specifically, anonymization functions can include techniques such as aggregation, generalization, or perturbation. Aggregation, in particular, represents combining detailed data into statistical data, and its specific expression is as follows:

[0072]

[0073] In the formula, Indicates the aggregation result; This represents the electricity consumption of the i-th user out of n users.

[0074] Generalization means generalizing a specific value to a range, and in this application, the range can be: Age∈[20,30].

[0075] In another example, the identifier for data de-identification can include attributes such as ID, name, and address, and its calculation expression is shown below:

[0076]

[0077] In the formula, This represents the de-identification result. This represents the deidentification function.

[0078] Specifically, the de-identification function can include removing or replacing the identifier. Replacing the identifier means replacing the user ID with a pseudo-anonymous ID, as shown in the following expression:

[0079]

[0080] In the formula, Indicates the replacement result. This represents a hash function.

[0081] Furthermore, smart meter data acquisition systems can employ standardized interfaces and protocols, such as RESTful APIs and OPC UA, to ensure system interoperability and compatibility. Through message middleware and data bus technologies, they can achieve data exchange and integration between different systems. RESTful APIs are an interface design style based on the HTTP protocol, allowing data manipulation using standard HTTP methods (such as GET, POST, PUT, DELETE). OPC UA (OLE for Process Control Unified Architecture) is a communication protocol used in industrial automation systems, supporting cross-platform and cross-language data exchange. Additionally, the message middleware used by the smart meter data acquisition system can be Apache Kafka, and the data bus technology can be Apache Camel.

[0082] In the above embodiments, the smart meter data acquisition system includes a data acquisition layer, a data transmission layer, a data processing layer, a data storage layer, and a data application layer. The data acquisition layer reads raw data from the smart meter. The data transmission layer uses a multi-level transmission architecture to sequentially transmit the raw data to a local gateway, a regional concentrator, and a central data center, thereby improving the real-time performance and reliability of data transmission. During transmission, the data processing layer, built using the Hadoop MapReduce framework, preprocesses the raw data at the local gateway to obtain intermediate data. Then, the regional concentrator performs parallel aggregation processing on the intermediate data to obtain electricity consumption data. Through the distributed parallel processing of MapReduce, the data processing efficiency and capacity are further improved. Finally, the data storage layer stores the raw data in a distributed file system and stores the intermediate data and electricity consumption data in a NoSQL database. The combination of the high scalability and fault tolerance of the distributed file system with the use of a NoSQL database improves the speed of data read and write operations. Therefore, the data application layer can obtain electricity consumption data from the NoSQL database in real time for data analysis, ensuring the system's rapid response capability. Based on this, this application utilizes the mutual cooperation and joint action of various data layers to improve the efficiency and quality of data transmission during the data acquisition process, thereby improving the real-time performance, reliability, and scalability of the entire system.

[0083] In one embodiment, the data transmission layer, used to sequentially transmit raw data to a local gateway, a regional concentrator, and a central data center using a multi-level transmission architecture, may include:

[0084] The data transmission layer uses short-distance transmission and the MQTT transmission protocol to transmit the raw data to the local gateway for initial processing, and uses medium-distance transmission and the MQTT transmission protocol to transmit the initially processed raw data to the regional concentrator for secondary processing. Finally, it uses long-distance transmission and the AMQP transmission protocol to transmit the secondary processed raw data to the central data center.

[0085] In this embodiment, the local gateway can receive raw data from multiple smart meters for preliminary aggregation and processing to ensure data integrity; while the regional concentrator can receive data from multiple local gateways for further processing and storage to ensure that data is not lost during transmission; and the central data center can receive and store data from various regional concentrators for unified management and processing, providing a basis for data analysis and decision-making.

[0086] Understandably, based on the distribution characteristics of local gateways, regional concentrators, and central data centers, the data transmission layer can use short-distance transmission to transmit raw data to the local gateway for initial processing, medium-distance transmission to transmit the initially processed raw data to the regional concentrator for secondary processing, and long-distance transmission to transmit the secondary processed raw data to the central data center. In addition, the data transmission layer can also use MQTT and AMQP transmission protocols as data transmission protocols to further improve data transmission efficiency.

[0087] MQTT is a lightweight message transmission protocol designed for low-bandwidth, unreliable network environments. It features low bandwidth consumption, low latency, and high reliability, making it suitable for data transmission in IoT devices. Therefore, in smart meter data acquisition systems, MQTT can be used for data transmission between smart meters and local gateways, and between local gateways and regional concentrators, ensuring data real-time performance and reliability. AMQP, on the other hand, is an open-source, standardized message transmission protocol that supports reliable message transmission and queue management. It features high reliability, high flexibility, and security, making it suitable for enterprise-level applications. Therefore, in smart meter data acquisition systems, AMQP can be used for data transmission between regional concentrators and the central data center, ensuring no data loss during transmission and supporting complex message routing and processing mechanisms.

[0088] Specifically, short-range transmission methods can be wireless communication, such as ZigBee and LoRa, or wired communication, such as PLC and power line communication; there are no restrictions here. These communication technologies are characterized by low power consumption, wide coverage, and high reliability, making them suitable for short-range data transmission within a residential area. Medium-range transmission methods refer to more efficient transmission methods, such as Ethernet and cellular networks. These transmission methods have higher bandwidth and transmission rates, making them suitable for medium-range data transmission. Long-range transmission methods refer to high-speed networks, such as fiber optic networks and the Internet. These transmission methods have extremely high bandwidth and reliability, making them suitable for long-range data transmission.

[0089] In one embodiment, the process by which the data processing layer preprocesses raw data in a local gateway to obtain intermediate data may include:

[0090] The data processing layer uses a data filtering algorithm in the local gateway to filter the raw data, and then uses a data aggregation algorithm and a data compression algorithm to aggregate and compress the filtered raw data to obtain intermediate data.

[0091] In this embodiment, the data processing layer can perform preprocessing operations on the raw data in the local gateway, including filtering the raw data using a data filtering algorithm, and aggregating and compressing the filtered raw data using a data aggregation algorithm and a data compression algorithm to obtain intermediate data, thereby reducing the frequency and amount of data transmission and reducing the burden on the communication network.

[0092] Specifically, data compression algorithms can be used to reduce the size of the original data, thereby reducing network bandwidth consumption. These typically include Huffman coding and Delta coding. Huffman coding achieves lossless data compression by constructing an optimal binary tree, while Delta coding is suitable for time-series data, reducing data volume by storing the differences between adjacent data points. In summary, the specific expression for a data compression algorithm can be shown below:

[0093]

[0094] In the formula, Indicates the size of the original data; C represents the size of the compressed data; C represents the compression ratio.

[0095] Data aggregation algorithms can be used to summarize and process data, reducing the amount of data transmitted. Common aggregation operations include summation, averaging, and maximum / minimum values. Summation, for example, calculates the total electricity consumption over a specific period, and its expression is as follows:

[0096]

[0097] In the formula, Indicates the summation result; This represents the electricity consumption of the i-th data point out of n data points.

[0098] The average value represents the average electricity consumption over a certain period of time, and its expression is as follows:

[0099]

[0100] In the middle, represents the average value; n represents the number of data points.

[0101] The maximum / minimum value represents the maximum and minimum electricity consumption within a certain period of time, and its expression is as follows:

[0102]

[0103] Data filtering algorithms can be used to filter and extract important data, and remove noise and unnecessary data. They typically include low-pass filtering and threshold filtering. Low-pass filtering is used to smooth time series data and can filter out high-frequency noise; its specific expression is as follows:

[0104]

[0105] In the formula, This represents the output of the filtered data; This represents the input data; This represents the filter coefficients.

[0106] Threshold filtering means only transmitting data points that exceed a certain threshold. Its expression is as follows:

[0107]

[0108] In the formula, T represents the preset filtering threshold.

[0109] In one embodiment, the data processing layer is used to perform parallel aggregation processing on intermediate data in a regional concentrator to obtain electricity consumption data, which may include:

[0110] In the regional concentrator, the data processing layer determines the processing tasks for intermediate data, and uses the MapReduce programming model to divide the intermediate data into multiple data blocks based on the processing tasks, and processes each data block in parallel to generate electricity consumption data.

[0111] In this embodiment, the MapReduce programming model is a distributed computing model that can perform batch processing and analysis of massive amounts of data based on processing tasks. The processing tasks can be set according to the actual needs of the smart meter data acquisition system, such as daily electricity consumption statistics, abnormal electricity consumption behavior monitoring, load forecasting, regional electricity consumption statistics and comparison, etc., and there are no restrictions here.

[0112] Understandably, the MapReduce programming model improves data processing efficiency through parallel processing, enabling it to handle massive datasets. It can execute tasks in parallel on different computing nodes, utilizing the computing resources of the cluster to increase data processing throughput.

[0113] In addition, the MapReduce programming model can be used in conjunction with the Apache Spark in-memory computing framework. Here, Apache Spark refers to a high-efficiency in-memory computing framework capable of executing distributed data processing tasks in memory, avoiding frequent disk I / O operations. Spark's in-memory computing model greatly improves data processing speed, enabling real-time data processing and iterative computation. In smart meter data acquisition systems, Spark can be used to process and analyze transmitted electricity consumption data in real time. For example, it can monitor user electricity consumption in real time, promptly detecting and responding to abnormal electricity consumption behavior. Furthermore, Spark provides Resilient Distributed Datasets (RDDs) as its core data abstraction, supporting parallel operations on large-scale datasets. These RDDs can be processed through a series of transformations and actions, and provide fault recovery mechanisms, thus efficiently executing complex analytical tasks such as electricity consumption pattern recognition and load forecasting.

[0114] In summary, the data processing layer can use the MapReduce programming model to batch process and analyze intermediate data, and periodically execute MapReduce jobs to aggregate, statistically analyze, and generate reports and analysis results on large-scale data. At the same time, it can use Apache Spark to process and analyze intermediate data transmitted in real time, and use components such as Spark Streaming to process real-time data streams, promptly detect anomalies, and take corresponding measures.

[0115] In one embodiment, the process of the data processing layer processing various data blocks in parallel to generate electricity consumption data may include:

[0116] The data processing layer uses the Map function to process each data block in parallel, generating intermediate key-value pairs for each data block, and uses the Reduce function to aggregate data blocks with the same intermediate key to generate electricity consumption data.

[0117] In this embodiment, the data processing layer can divide the processing task into two stages using the MapReduce programming model: the Map stage and the Reduce stage. In the Map stage, the data processing layer can use Map functions to process each data block in parallel, generating intermediate key-value pairs; in the Reduce stage, the data processing layer can use Reduce functions to aggregate all data blocks with the same intermediate key, generating the final electricity consumption data.

[0118] Specifically, in the Map phase, the Map function can accept a key-value pair as input and generate a set of intermediate key-value pairs: ;in, and This indicates that the input is a key-value pair. and This represents an intermediate key-value pair; the data processing layer can then use the keys on the intermediate key-value pair. Group the data and pass values ​​with the same key to the Reduce function: Among them, those with the same key value They are combined into a list. In the Reduce phase, the Reduce function takes a key and a set of associated values, producing one or more output results: ;in, and These are the key-value pairs in the final output.

[0119] In one embodiment, the process by which the data processing layer uses a Map function to process each data block in parallel and generate intermediate key-value pairs for each data block may include:

[0120] After extracting the attribute ID and power consumption from each data block, the data processing layer uses a Map function to convert the attribute ID of the data block into a data key and the power consumption of the data block into a data value. Then, it generates an intermediate key-value pair corresponding to the data block based on the data key and the data value.

[0121] In this embodiment, the data processing layer can first extract the attribute ID and electricity consumption from each data block. The attribute ID can be one or a combination of attributes such as user ID, address, and timestamp, and there are no restrictions on this. Next, for each data block, the data processing layer can use a Map function to convert the attribute ID of the data block into a data key and the electricity consumption of the data block into a data value, and generate an intermediate key-value pair corresponding to the data block based on the data key and the data value.

[0122] For example, when the intermediate data processing task is daily electricity consumption statistics, the data processing layer can extract the user ID, timestamp, and electricity consumption of each data block in the intermediate data. The data format generated after extraction is shown in the table below:

[0123]

[0124] Based on the table above, the goal of the Map function is to convert each record in the table into an intermediate key-value pair; where the data key is "User ID_Timestamp" and the data value is "Electricity Consumption". For example, for the data of user_001 on 2023-07-24, the intermediate key-value pairs output by the Map function are ("user_001_2023-07-24", 1.5), ("user_001_2023-07-24", 1.2), and ("user_001_2023-07-24", 1.0).

[0125] In one embodiment, the data processing layer uses a Reduce function to aggregate data blocks with the same intermediate key to generate electricity consumption data. This process may include:

[0126] The data processing layer determines the data key in each intermediate key-value pair and adds intermediate key-value pairs with consistent data keys to the same data set, forming multiple data sets.

[0127] For each dataset, the data processing layer uses the Reduce function to aggregate the data values ​​in each intermediate key-value pair within the dataset to generate the electricity consumption data corresponding to that dataset.

[0128] In this embodiment, the data processing layer can first determine the data key in each intermediate key-value pair, and add intermediate key-value pairs with the same data key to the same data set to form multiple data sets. Then, each data set is sent to each Reduce task node, so that the data values ​​in each intermediate key-value pair in the data set can be aggregated in parallel to generate the electricity consumption data corresponding to each data set.

[0129] For example, for each dataset corresponding to an attribute ID, the Reduce function can sort the intermediate key-value pairs in the dataset in chronological order within its corresponding Reduce task node, ensuring that data corresponding to the same data key can be aggregated in sequence. For example, in a dataset with the key "user_001_2023-07-24", it contains intermediate key-value pairs ("user_001_2023-07-24", 1.5), ("user_001_2023-07-24", 1.2), and ("user_001_2023-07-24", 1.0). The data processing layer can first aggregate the data values ​​of this dataset to obtain aggregated key-value pairs ("user_001_2023-07-24", [1.5, 1.2, 1.0]). Then, the data processing layer can use the Reduce function to aggregate these aggregated key-value pairs, i.e., to aggregate the list of data values ​​[1.5, 1.2, 1.0] for the key "user_001_2023-07-24". Add the values ​​in 1.0 to obtain the electricity consumption data for this dataset ("user_001_2023-07-24", 3.7).

[0130] In one embodiment, the data storage layer may further include:

[0131] The data storage layer regularly performs data backup and recovery tests on the distributed file system and NoSQL database, and monitors and manages the distributed file system and NoSQL database in real time through monitoring and management tools.

[0132] In this embodiment, the data storage layer can periodically perform data backup and recovery tests on the distributed file system and NoSQL database to ensure data security and integrity. In addition, monitoring and management tools such as Ambari and Cloudera Manager can be used to monitor and manage the distributed file system and NoSQL database in real time to ensure the stable operation of the smart meter data acquisition system.

[0133] In one embodiment, the process of using data analysis tools and machine learning languages ​​to perform data analysis on the electricity consumption data in the data application layer may include:

[0134] The data application layer uses Hive to query electricity consumption data in the NoSQL database and generate electricity consumption statistics reports. It also uses Pig to convert the electricity consumption data into a format, and then uses the Mahout learning library and the Spark Mllib learning library to perform anomaly analysis and electricity consumption prediction on the converted electricity consumption data.

[0135] In this embodiment, the data application layer, as the top layer of the smart meter data acquisition system, is mainly responsible for analyzing electricity consumption data, helping to make data-driven decisions, and realizing functions such as electricity consumption pattern analysis, load forecasting, and fault detection through various data analyses.

[0136] Hive is a Hadoop-based data warehouse tool that supports querying and analyzing large-scale datasets using a SQL-like query language. It transforms queries into MapReduce jobs, enabling batch processing and analysis of historical data. Therefore, in this application, the data application layer can utilize Hive to query and perform preliminary analysis of historical electricity consumption data stored in a distributed file system; for example, Hive can be used to query electricity consumption for different time periods and analyze users' electricity consumption habits. Pig, on the other hand, is a high-level data flow scripting language that simplifies the processing of large-scale datasets. It can be converted into a series of MapReduce jobs for execution, making it suitable for data pipelines and ETL (extract, transform, load) operations. Therefore, the data application layer can utilize Pig for data cleaning and preprocessing; for example, Pig scripts can be used to convert electricity consumption data into a data format suitable for analysis.

[0137] Furthermore, the Mahout learning library is a distributed machine learning library suitable for machine learning tasks on large-scale datasets. It provides various commonly used algorithms, such as clustering, classification, and collaborative filtering. In the data analysis of this application, the data application layer can utilize the Mahout learning library to perform user electricity consumption behavior classification and clustering analysis to identify the electricity consumption patterns of different user groups. The Spark MLlib learning library is a Spark machine learning library that provides a rich set of machine learning algorithms and tools, supporting fast execution of machine learning tasks in memory. In the data analysis of this application, the data application layer can utilize the Spark MLlib learning library for load forecasting, anomaly detection, and user behavior analysis. For example, the time series forecasting algorithm in the Spark MLlib learning library can be used to predict future electricity demand, helping to optimize resource allocation.

[0138] In one embodiment, the data application layer may further include:

[0139] The data application layer uses data visualization tools to visualize the analysis results of electricity consumption data.

[0140] In this embodiment, the data application layer can also use data visualization tools to visualize the analysis results of electricity consumption data, such as using Tableau and Power BI to create interactive charts and dashboards to intuitively display the analysis results to management and users, such as displaying electricity consumption trend charts, regional electricity consumption distribution maps, abnormal electricity consumption alarms, etc., without any restrictions.

[0141] Tableau refers to a data visualization tool that supports access to multiple data sources and offers rich visualization features. Users can create interactive charts and dashboards through drag-and-drop operations. Therefore, in this application, the data application layer can use Tableau to intuitively display analysis results to users, such as creating real-time monitoring dashboards to display real-time electricity consumption in various regions and electricity consumption trend charts. Power BI supports data integration, data analysis, and visualization, and provides rich chart types and powerful data analysis functions. In this application, the data application layer can use Power BI to create various data reports and visualization charts to help management make data-driven decisions, such as generating monthly electricity consumption reports and charts predicting future electricity demand.

[0142] To better explain the text matching method of this application, the following will be conducted through... Figure 2 To further illustrate, illustratively, such as Figure 2 As shown, Figure 2 This is an application logic diagram of a smart meter data acquisition system provided in an embodiment of this application.

[0143] Figure 2 In this system, after collecting data from smart meters, the smart meter data acquisition system can transmit the data using SSL / TLS encryption. During transmission, the collected data undergoes preliminary processing and aggregation, and the data before and after processing is simultaneously stored in databases such as HDFS to demonstrate its high scalability and high availability. When further aggregation and analysis of this data is required, the smart meter data acquisition system can also perform in-depth processing and analysis on the pre-processed data. In this process, MapReduce and Spark can be used for efficient parallel processing and storage in the database. Then, in application, the smart meter data acquisition system can use Hive and Pig to query and perform preliminary processing on the data in the database, and Mahout and MLlib for machine learning and data mining to obtain the final results and visualize them. During data application, the smart meter data acquisition system can use RESTful and OPC UA for standardized interfaces, and Kafka and Camel for data exchange and integration to improve system integration and compatibility. Simultaneously, the smart meter data acquisition system can also use SSL / TLS for data encryption, RBAC for access control, and data anonymization for privacy protection, thereby improving the system's data security and privacy protection.

[0144] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0145] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0146] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A smart meter data acquisition system, characterized in that, The system includes a data acquisition layer, a data transmission layer, a data processing layer, a data storage layer, and a data application layer; The data acquisition layer is used to read the user's raw data from the smart meter; The data transmission layer is used to transmit the raw data sequentially to the local gateway, regional concentrator, and central data center using a multi-level transmission architecture, including: The data transmission layer uses a short-distance transmission method and the MQTT transmission protocol to transmit the raw data to the local gateway for preliminary processing, and uses a medium-distance transmission method and the MQTT transmission protocol to transmit the raw data after preliminary processing to the regional concentrator for secondary processing. Finally, it uses a long-distance transmission method and the AMQP transmission protocol to transmit the raw data after secondary processing to the central data center. The data processing layer is built using the Hadoop MapReduce framework. It is used to preprocess the raw data in the local gateway to obtain intermediate data, and to perform parallel aggregation processing on the intermediate data in the regional concentrator to obtain electricity consumption data, including: The data processing layer uses a data filtering algorithm to filter the original data in the local gateway, and uses a data aggregation algorithm and a data compression algorithm to aggregate and compress the filtered original data to obtain intermediate data. The data processing layer, within the regional concentrator, determines the processing task for the intermediate data and, using the MapReduce programming model, divides the intermediate data into multiple data blocks based on the processing task. The data processing layer uses the Map function to process each data block in parallel, generating intermediate key-value pairs for each data block, and uses the Reduce function to aggregate data blocks with the same intermediate key to generate electricity consumption data. The data storage layer is used to store the raw data in a distributed file system, and to store the intermediate data and the electricity consumption data in a NoSQL database; The data application layer is used to perform data analysis on the electricity consumption data using data analysis tools and machine learning languages.

2. The smart meter data acquisition system according to claim 1, characterized in that, The data processing layer uses a Map function to process each data block in parallel, generating intermediate key-value pairs for each data block. This process includes: After extracting the attribute ID and power consumption from each data block, the data processing layer uses a Map function to convert the attribute ID of the data block into a data key and the power consumption of the data block into a data value, and generates an intermediate key-value pair corresponding to the data block based on the data key and the data value.

3. The smart meter data acquisition system according to claim 1, characterized in that, The data processing layer uses the Reduce function to aggregate data blocks with the same intermediate key to generate electricity consumption data. This process includes: The data processing layer determines the data key in each intermediate key-value pair and adds intermediate key-value pairs with the same data key to the same data set, forming multiple data sets; For each dataset, the data processing layer uses the Reduce function to aggregate the data values ​​in each intermediate key-value pair within the dataset to generate the electricity consumption data corresponding to that dataset.

4. The smart meter data acquisition system according to claim 1, characterized in that, The data storage layer further includes: The data storage layer periodically performs data backup and recovery tests on the distributed file system and the NoSQL database, and monitors and manages the distributed file system and the NoSQL database in real time through monitoring and management tools.

5. The smart meter data acquisition system according to claim 1, characterized in that, The data application layer is used for the process of performing data analysis on the electricity consumption data using data analysis tools and machine learning languages, including: The data application layer uses Hive to query electricity consumption data in the NoSQL database and generate electricity consumption statistics reports. It also uses Pig to convert the electricity consumption data into a format, and uses the Mahout learning library and Spark MLlib learning library to perform anomaly analysis and electricity consumption prediction on the converted electricity consumption data.

6. The smart meter data acquisition system according to claim 1 or 5, characterized in that, The data application layer also includes: The data application layer uses data visualization tools to visualize the analysis results of the electricity consumption data.

Images