A distributed intelligent metering data synchronization management method and system

Abstract

The application discloses a kind of distributed wisdom metering data synchronization management method and system, it is related to the field of metering data management, this method includes: using edge node to the metering data obtained by connecting wisdom metering system is cached;Metering data is differentially synchronized in batches according to time window, and transmission data is obtained;Synchronous transmission of transmission data is carried out through transmission channel, and message queue is generated;Based on center node, equipment data analysis is carried out on message queue, and distributed database is constructed, for distributed query.The technical problems that the existing metering data management exists, such as low synchronization efficiency and accuracy, and insufficient security and reliability, are solved, the technical effects of improving the synchronization efficiency and accuracy of metering data, providing safe and reliable data support are achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of metering data management, in particular to a distributed intelligent metering data synchronization management method and system. BACKGROUND

[0002] The accuracy and real-time performance of metering data are crucial for enterprise operation decision-making. With the rapid increase in the number of metering devices and the explosive growth of data volume, how to efficiently and accurately synchronize and manage these metering data has become a problem to be solved. At present, the main method to solve this problem is to use centralized data management, that is, all metering data are first sent to a central server for storage and processing. However, when facing massive data, on the one hand, due to the huge amount of data, the processing capacity and storage space of the central server are easily reached to the limit, resulting in data synchronization delay and data loss; on the other hand, the centralized data management method also has great challenges in data security and reliability.

[0003] In the related art, the metering data management has the technical problems of low synchronization efficiency and accuracy, and insufficient security and reliability. SUMMARY

[0004] The present application provides a distributed intelligent metering data synchronization management method and system, which uses edge nodes to cache metering data, and performs batch differential synchronization according to time windows, synchronously transmits transmission data to a central node through a transmission channel, and analyzes device data based on the central node to construct a distributed database, thereby improving the synchronization efficiency and accuracy of metering data, and providing safe and reliable data support.

[0005] The present application provides a distributed intelligent metering data synchronization management method, which comprises: using edge nodes to cache metering data obtained by connecting an intelligent metering system; performing batch differential synchronization on the metering data according to time windows to obtain transmission data; synchronously transmitting the transmission data through a transmission channel to generate a message queue; and analyzing device data based on the central node to construct a distributed database for distributed query.

[0006] In a possible implementation, the batch differential synchronization of the metering data according to the time windows to obtain the transmission data performs the following processing: based on business requirements, transmission delay requirements, system load and data change frequency, the time windows of distributed devices are respectively set; the current data in the time window is compared with the synchronization record to determine the change data with a change threshold; the synchronization is triggered by the change data to obtain the transmission data.

[0007] In a possible implementation, the synchronization is triggered by the change data, the transmission data is obtained, and the following processing is performed: a unique identification code is generated based on the change data; a latest difference value is generated according to the unique identification code and the difference value of the change data in the latest record in the synchronization record; the synchronization is triggered by the latest difference value, and the transmission data is obtained.

[0008] In a possible implementation, the synchronization transmission of the transmission data is performed through the transmission channel, a message queue is generated, and the following processing is performed: a communication protocol is selected to determine the transmission channel of the transmission data; a time synchronization mechanism is introduced to perform transmission of the transmission data, transmission confirmation is performed based on the transmission channel, and the message queue is obtained.

[0009] In a possible implementation, the time synchronization mechanism is introduced, and the following processing is performed: a synchronization fault tolerance time of the distributed device is set; the clock drift of the distributed device is checked based on the synchronization fault tolerance time, and a time source is started based on the clock drift degree; and the distributed device is configured by selecting a healthy time source in the time source.

[0010] In a possible implementation, the transmission confirmation is performed based on the transmission channel, and the following processing is performed: the distributed device is used as a message producer, the edge node is used as a message consumer, and partitioned messages obtained by the message producer are transmitted to the message consumer through the transmission channel; the partitioned messages are read by the message consumer and a processing reply is sent, and if it is determined that the processing fails according to the processing reply, message retry of the partitioned messages is performed until it is determined that the processing is successful in the processing reply; and a dead letter queue is extracted based on the message retry for error processing.

[0011] In a possible implementation, device data analysis is performed on the message queue by the center node, and a distributed database is constructed, and the following processing is performed: device data analysis of the distributed device is performed based on the message queue, and device data is obtained; and the device data is stored in the distributed database.

[0012] The application also provides a distributed intelligent metering data synchronization management system, comprising: a metering data caching module configured to cache metering data obtained by connecting an intelligent metering system by using an edge node; a batch differential synchronization module configured to perform batch differential synchronization on the metering data according to a time window, and obtain transmission data; a message queue generation module configured to perform synchronization transmission of the transmission data through a transmission channel, and generate a message queue; and a distributed database construction module configured to perform device data analysis on the message queue based on a center node, and construct a distributed database for distributed query.

[0013] The application provides a distributed intelligent metering data synchronization management method and system. The metering data obtained by connecting an intelligent metering system is cached by an edge node, and then batch differential synchronization is performed on the metering data according to a time window to obtain transmission data. Then, the transmission data is synchronously transmitted through a transmission channel to generate a message queue. Finally, device data analysis is performed on the message queue based on a center node, a distributed database is constructed, and distributed query is performed. The synchronization efficiency and accuracy of metering data are improved, and safe and reliable data support is provided. BRIEF DESCRIPTION OF DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings of the embodiments of the application will be briefly introduced below. In the present application, a flowchart is used to illustrate the operations performed by the system according to the embodiments of the application. It should be understood that the foregoing or the following operations are not necessarily performed in sequence. On the contrary, various steps can be processed in reverse order or simultaneously according to needs. Meanwhile, other operations can be added to these processes, or one or more steps of operations can be removed from these processes.

[0015] Figure 1 A flowchart of a distributed intelligent metering data synchronization management method provided by the embodiments of the application.

[0016] Figure 2 A structure diagram of a distributed intelligent metering data synchronization management system provided by the embodiments of the application.

[0017] Legend: metering data caching module 10, batch differential synchronization module 20, message queue generation module 30, and distributed database construction module 40. DETAILED DESCRIPTION

[0018] The above description is only a summary of the technical solutions of the application. In order to more clearly understand the technical means of the application, the application can be implemented according to the content of the specification, and in order to make the above and other purposes, characteristics and advantages of the application more obvious and easy to understand, the following specific embodiments of the application are described.

[0019] In order to make the purposes, technical solutions and advantages of the application more clear, the application will be further described in detail below with reference to the drawings. The described embodiments should not be regarded as limiting the application. All other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application.

[0020] In the following description, references to "some embodiments" describe a subset of all possible embodiments; however, it is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. The terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only.

[0021] This application provides a distributed smart metering data synchronization management method, such as... Figure 1 As shown, the method includes:

[0022] Step S100: Cache the metering data obtained from connecting to the smart metering system using edge nodes.

[0023] Specifically, edge nodes are deployed at key locations within the smart metering system. These edge nodes are microcomputers or embedded devices with data processing and storage capabilities, located near data sources (such as sensors and instruments in a thermal power plant). The smart metering system is an integrated system combining sensors, communication technologies, and data analysis capabilities for real-time monitoring and recording of energy consumption data. Edge nodes connect to the smart metering system through pre-defined interfaces or protocols, collecting metering data from various metering devices in real time, including power generation, coal consumption, water consumption, and steam flow. The collected metering data is temporarily stored on the edge nodes' local storage media (such as memory or hard drives). Caching strategies can include timestamps and data priority sorting to ensure that important data is not overwritten; for example, FIFO (First-In, First-Out) or Least Recently Used (LRU) caching strategies can be used to manage cached data.

[0024] Step S200: Perform batch differential synchronization of the measurement data according to the time window to obtain the transmission data.

[0025] Specifically, according to the system requirements and the frequency of data synchronization, the time is divided into fixed time windows (such as every minute, every hour, etc.). At the end of each time window, the edge node differentiates the cached metering data, that is, compares the data in the current time window with the data in the last time window through data hashing, timestamp comparison, etc. to identify the newly added or updated data. The differentiated data is packaged into transmission data, including the data itself, timestamp, device ID, etc. information, ready for synchronization transmission through the transmission channel. The packaging process can include data compression, encryption and formatting operations.

[0026] In one possible implementation, the metering data is batch-differentiated and synchronized according to time windows to obtain transmission data, and step S200 further includes step S210 of setting the time windows of the distributed devices based on business requirements, transmission delay requirements, system load and data change frequency. Specifically, through historical data analysis, the change frequency and transmission delay of different metering data are determined, and the time window settings are adjusted according to the operation requirements and system load. Specifically, according to the operation requirements, such as real-time monitoring of temperature, pressure, flow and other key parameters, and the requirement of generating operation reports regularly, the time windows of different metering data are set. For example, for real-time monitoring data, it needs to be synchronized once every second or every minute; for the cumulative data required for generating reports, it can be set to synchronize once every hour or every day. Considering the network delay and communication bandwidth in the data transmission process, a reasonable time window is set to avoid transmission delay caused by data accumulation. For example, if the network condition is poor, the time window can be appropriately extended to reduce the number of data transmissions, thereby reducing the demand for network resources. According to the processing capacity of the edge node and the center node, the time window is reasonably allocated to avoid system overload. If the system load is high, the system pressure can be reduced by increasing the time window or reducing the synchronization frequency. Analyze the change pattern of the metering data, set a shorter time window for frequently changing data to ensure the real-time performance of the data, and set a longer time window for slowly changing data to reduce the amount of data transmission.

[0027] Step S220 compares the current data in the time window with the synchronization record to determine the changed data with a change threshold. Specifically, at the end of each time window, the current metering data is compared with the data synchronized last time. A change threshold is set to determine whether the data has changed significantly. If the data change exceeds the threshold, the data is considered as changed data.

[0028] Step S230, triggering synchronization with the change data to obtain the transmission data. Specifically, when detecting the change data, triggering the synchronization mechanism, and packaging the change data into the transmission data. Through the preset transmission channel, the transmission data is sent to the center node for further processing. This implementation can significantly reduce unnecessary data transmission and improve data transmission efficiency by setting the time window and the change threshold. By reasonably allocating the time window and reducing the synchronization frequency, the processing pressure of the edge node and the center node can be reduced, and system overload can be avoided. For important data changes, real-time synchronization can be triggered to ensure real-time and accuracy of the data.

[0029] In a possible implementation, step S230 further includes step S231 of generating a unique identification code based on the change data, to obtain the transmission data. Specifically, applying a hash function to the change data to generate a fixed-length hash value as a unique identification code (UID). This hash value can uniquely represent the change data, and even if the data content changes slightly, the hash value will also be different. The generated UID is stored in association with the change data for subsequent use.

[0030] Step S232, generating a latest difference value according to the unique identification code and a difference value of the change data identified by a latest record in the synchronization record. Specifically, searching for a latest record matching the UID of the current change data in the synchronization record. If a matching record is found, calculating the difference value between the current change data and the latest record. This difference value can be a binary difference, a text difference, or a difference representation under specific business logic. The calculated difference value is stored in association with the UID as part of the transmission data.

[0031] Step S233, triggering synchronization with the latest difference value to obtain the transmission data. Specifically, encapsulating the calculated difference value, UID, and necessary metadata (such as timestamp, data source identification, etc.) into a transmission data packet. Through the preset transmission channel, the encapsulated transmission data packet is sent to the center node for further processing. This implementation significantly reduces the amount of data transmission by calculating the difference value and only transmitting these difference values, reducing network bandwidth consumption and transmission delay. Since only the difference value is transmitted, the center node can update the database more quickly after receiving the data, improving synchronization efficiency. Reducing data transmission and synchronization time can reduce the processing pressure of the edge node and the center node, optimizing resource utilization.

[0032] Step S300, synchronously transmitting the transmission data through the transmission channel to generate a message queue.

[0033] Specifically, a reliable transmission channel is established between the edge node and the center node, such as using a wired network, a wireless network, or a dedicated data communication link, etc. After the end of each time window, the edge node sends the packaged transmission data to the center node through the transmission channel. After receiving the transmission data, the center node puts it into the message queue for further processing. The message queue is a first-in, first-out data structure used to manage data packets to be processed, which can be implemented using memory data structures (such as linked lists, queues, etc.) or based on message queue systems (such as RabbitMQ, Kafka, etc.).

[0034] In one possible implementation, the synchronous transmission of the transmission data is performed through the transmission channel, and the message queue is generated. Step S300 further includes step S310 of selecting a communication protocol to determine the transmission channel of the transmission data. Specifically, the characteristics of the transmission data are evaluated, including data type, size, real-time requirement, etc. According to the evaluation result, the most suitable protocol is selected from the available communication protocols. For example, for data requiring high real-time performance, TCP / IP protocol can be used because it has reliable connection mechanism and error checking function; for batch data with low real-time requirement, more efficient UDP protocol or HTTP protocol can be selected for transmission. The corresponding communication channel is configured between the edge node and the center node to ensure smooth data transmission.

[0035] Step S320, introducing a time synchronization mechanism to perform the transmission of the transmission data, based on the transmission channel for transmission confirmation, obtaining the message queue. Specifically, the time synchronization mechanism is the key to ensure the time consistency of each node in the distributed system. In the data transmission process, the time synchronization mechanism is introduced to ensure the accuracy of the sending and receiving time stamp of the data, so as to avoid data confusion or loss. Before data transmission, through NTP (Network Time Protocol) or other time synchronization mechanism, the time consistency of the edge node and the center node is ensured. According to the selected communication protocol, the metering data is batch differentiated and synchronized according to the time window, and is sent to the center node through the transmission channel. After receiving the data, the center node sends a feedback confirmation message to the edge node to confirm the integrity and successful reception of the data. In the center node, the received data is organized into a message queue, waiting for subsequent device data analysis. This implementation ensures efficient and accurate synchronization of a large amount of metering data between multiple systems or nodes through the time synchronization mechanism and transmission confirmation.

[0036] In one possible implementation, a time synchronization mechanism is introduced, and step S320 further includes step S321 of setting a synchronization fault tolerance time of the distributed devices. Specifically, in a computer system, each distributed device (such as a sensor, a data collector, etc.) has its own internal clock. Due to hardware differences, network delays, and other factors, these clocks will gradually drift, that is, there will be a small difference in time. In order to handle such drift, a reasonable synchronization fault tolerance time needs to be set for each device first. This fault tolerance time defines the maximum deviation range within which the device clock is considered to be “synchronized”. For example, each distributed device in a certain thermal power plant generates metering data once per second, and considering the time requirements for data transmission and processing, setting the synchronization fault tolerance time to ±5 milliseconds can ensure that the timestamps of the data remain consistent within an acceptable range.

[0037] Step S322, checking the clock drift of the distributed devices with the synchronization fault tolerance time, and starting a time source based on the degree of clock drift. Specifically, the system regularly (such as every minute) checks the difference between the clock of each device and the system standard time (such as the time provided by an NTP server). If the clock drift of a certain device exceeds the synchronization fault tolerance time, it is determined that the device clock needs to be calibrated. Wherein, the clock drift refers to the change of the deviation of the device clock relative to the system standard time over time. The time source refers to a device or network service that provides time synchronization services, such as an NTP (Network Time Protocol) server.

[0038] Step S323, selecting a healthy time source in the time source to configure the distributed devices. Specifically, the system maintains one or more time sources, and selects the best time source according to its health status (such as response time, stability, etc.). Once a healthy time source is determined, the system uses the time source to calibrate the drifting device clock. For example, the system has two NTP servers as time sources, and the system will select the best time source for clock calibration according to their response time in the past period of time (for example, a server with a response time less than 10 milliseconds is considered healthy). This implementation periodically calibrates the device clock through the time synchronization mechanism, which can greatly reduce the data errors caused by the unsynchronized clocks, ensures that all device data is recorded and analyzed under a unified time framework, and ensures the time sequence consistency and reliability of the transmitted data.

[0039] In a possible implementation, the step S320 of transmitting the confirmation of transmission based on the transmission channel further includes a step S324 of taking the distributed device as a message producer, taking the edge node as a message consumer, and transmitting the partitioned message obtained by the message producer to the message consumer through the transmission channel. Specifically, the distributed device (such as a sensor, a data collector, etc.) is taken as the message producer, and is responsible for generating and sending metering data. The data is divided into different partitioned messages, and each partition represents a different data type or time window. The edge node is taken as the message consumer, and is responsible for receiving and processing the partitioned messages. For example, a certain thermal power plant has 100 distributed devices, and each device generates 1 metering data per second. In order to optimize data transmission, the data can be divided into multiple partitions according to the device ID or data type, and each partition contains messages from a specific device or data type. The message producer (distributed device) sends the partitioned message to the transmission channel using a selected communication protocol (such as MQTT, HTTP, etc.). The edge node is taken as the message consumer, and subscribes to these partitioned messages and is ready to receive.

[0040] In step S325, the partitioned message is read by the message consumer and a processing reply is sent. If it is determined according to the processing reply that the processing fails, message retry of the partitioned message is performed until it is determined that the processing in the processing reply is successful. Specifically, after receiving the partitioned message, the edge node attempts to process the messages (such as storing in a local cache, etc.). After processing is completed, the edge node sends a processing reply to the message producer, indicating that the message has been successfully processed. If the processing fails (such as due to network problems, device failure, etc.), the edge node performs message retry, that is, reattempts to process the partitioned message. The edge node maintains a processing status table when processing the partitioned message, recording the processing result of each message. If the processing fails, the edge node reattempts to process the message according to the configured number of retries, and updates the processing status table after each retry. If the number of retries reaches the upper limit and still fails, the message is transferred to the dead letter queue for error processing. For example, the success rate of the edge node when processing the partitioned message is 99%, and then for the remaining 1% of the failed messages, the edge node performs message retry. The number of retries can be configured according to actual conditions, such as 3 times, 5 times, etc.

[0041] Step S326, error handling based on the message retry extracting the dead letter queue. Specifically, for the partition messages that cannot be processed after multiple retries, the edge node will transfer these messages to the dead letter queue. The dead letter queue is a queue specially used to store messages that cannot be processed or have failed processing. When the edge node transfers the message to the dead letter queue, it will record the relevant error information (such as error code, error description, etc.). System administrators can view this information through monitoring tools or log systems, analyze the failure cause, and take appropriate corrective measures (such as repairing device faults, optimizing processing logic, etc.). This implementation ensures the integrity and accuracy of metering data during transmission through the mechanisms of message producers, message consumers, and partition messages. Through the mechanisms of processing acknowledgments and message retries, errors during transmission can be detected and processed in a timely manner, improving system reliability and stability. Through the mechanism of the dead letter queue, messages that cannot be processed can be centrally managed, making it easier for system administrators to troubleshoot and handle errors, thereby improving system maintainability and scalability.

[0042] Step S400, device data analysis based on the center node to the message queue, building a distributed database for distributed query.

[0043] Specifically, the center node (the core computing device responsible for data processing, storage, and query) takes data packets from the message queue, parses and analyzes them, including data cleaning (removing outliers, filling missing values, etc.), format conversion (such as unit conversion, format conversion, etc.), and statistical analysis (such as calculating the average, sum, etc.) operations. The analyzed data is integrated according to device ID, timestamp, and other key fields to form a complete device data set. The integrated device data set is stored in a distributed database. A distributed database is a database system that can store and query data across multiple physical nodes, with high availability and scalability. The distributed database system can be built based on NoSQL databases (such as MongoDB, Cassandra, etc.) or relational databases (such as MySQL Cluster, PostgreSQLXL, etc.). Index and query mechanisms are established in the distributed database to support efficient distributed query operations, allowing users to concurrently access and query data from different nodes, improving query efficiency and response speed. The present application embodiment improves the synchronization efficiency and accuracy of metering data by using edge nodes to cache metering data and batch differentiating synchronization according to time windows, synchronously transmitting transmission data to the center node through the transmission channel, and performing device data analysis on the message queue based on the center node to build a distributed database, etc. technical means, achieving the technical effect of providing safe and reliable data support.

[0044] In a possible implementation, the center node performs device data analysis on the message queue to construct the distributed database, and step S400 further includes step S410, device data analysis on the distributed device using the message queue to obtain device data. Specifically, using a specific parsing library or tool, the device data in each message in the message queue is parsed using a predefined message format or protocol, such as power generation, device temperature, pressure, flow, and various parameters. The parsed device data is cleaned to remove invalid or abnormal data. For example, values that are obviously outside the reasonable range are considered abnormal data and are removed. The cleaned device data is integrated according to the device or time dimension, so as to facilitate subsequent analysis.

[0045] Step S420, store the device data to the distributed database. Specifically, the center node uses a specific database connection library or driver to establish a connection with the distributed database, uses a SQL statement or a specific database operation API, and inserts the cleaned and integrated device data into the distributed database according to a predetermined data model or table structure. In order to improve the query efficiency, an index is created for the device data in the distributed database. The index is a data structure used to speed up the data retrieval speed. The distributed database in this implementation can provide high availability and data consistency guarantee, so that the data integrity and availability can be guaranteed even when some nodes fail. Moreover, the distributed database system is easy to expand and maintain, and can increase or decrease nodes according to actual needs, and adjust the performance and capacity of the system.

[0046] In the foregoing, with reference to Figure 1 A distributed intelligent metering data synchronization management method according to an embodiment of the application is described in detail. Next, with reference to Figure 2 A distributed intelligent metering data synchronization management system according to an embodiment of the application is described.

[0047] The distributed intelligent metering data synchronization management system according to the embodiment of the application is used to solve the technical problems of low synchronization efficiency and accuracy, and insufficient security and reliability of the existing metering data management, and achieves the technical effects of improving the synchronization efficiency and accuracy of the metering data, and providing safe and reliable data support. The distributed intelligent metering data synchronization management system includes a metering data caching module 10, a batch differentiated synchronization module 20, a message queue generation module 30, and a distributed database construction module 40.

[0048] The metering data caching module 10 is configured to cache metering data obtained by connecting a smart metering system through an edge node; the batch differential synchronization module 20 is configured to batch differential synchronization of the metering data according to a time window to obtain transmission data; the message queue generation module 30 is configured to synchronize transmission of the transmission data through a transmission channel to generate a message queue; and the distributed database construction module 40 is configured to analyze device data of the message queue based on a center node to construct a distributed database for distributed query.

[0049] Next, the specific configuration of the batch differential synchronization module 20 will be described in detail. As described above, the batch differential synchronization module 20 is configured to batch differential synchronization of the metering data according to a time window to obtain transmission data, and can further include: a time window setting unit configured to set a time window of a distributed device based on business requirements, transmission delay requirements, system load and data change frequency; a change data determination unit configured to compare current data in the time window with a synchronization record to determine change data according to a change threshold; and a synchronization triggering unit configured to trigger synchronization with the change data to obtain the transmission data.

[0050] The synchronization triggering unit configured to trigger synchronization with the change data to obtain the transmission data can further include: a unique identification code generation subunit configured to generate a unique identification code based on the change data; a difference value identification subunit configured to identify a difference value of the change data according to the unique identification code and a latest record in the synchronization record to generate a latest difference value; and a transmission data acquisition subunit configured to trigger synchronization with the latest difference value to obtain the transmission data.

[0051] Next, the specific configuration of the message queue generation module 30 will be described in detail. As described above, the message queue generation module 30 is configured to synchronize transmission of the transmission data through a transmission channel to generate a message queue, and can further include: a transmission channel determination unit configured to select a communication protocol to determine a transmission channel of the transmission data; and a data transmission unit configured to introduce a time synchronization mechanism to perform transmission of the transmission data, perform transmission confirmation based on the transmission channel, and obtain the message queue.

[0052] The data transmission unit configured to introduce a time synchronization mechanism can further include: a synchronization fault tolerance time setting subunit configured to set a synchronization fault tolerance time of a distributed device; a time source starting subunit configured to check clock drift of the distributed device with the synchronization fault tolerance time, start a time source based on a clock drift degree; and a distributed device configuration subunit configured to select a healthy time source in the time source to configure a distributed device.

[0053] The data transmission unit can further include a transmission subunit configured to take the distributed device as a message producer, take the edge node as a message consumer, and transmit a partition message obtained by the message producer to the message consumer through the transmission channel; a processing reply sending subunit configured to read the partition message by the message consumer and send a processing reply, perform message retry of the partition message until it is determined that processing is successful in the processing reply if it is determined that processing fails according to the processing reply; and an error processing subunit configured to perform error processing based on the message retry to extract a dead letter queue.

[0054] The specific configuration of the distributed database construction module 40 will be described in detail below. As described above, the message queue is analyzed by the center node to analyze device data of the distributed device, and the distributed database is constructed, and the distributed database construction module 40 can further include a device data acquisition unit configured to analyze device data of the distributed device by the message queue to acquire device data; and a data storage unit configured to store the device data to the distributed database.

[0055] The distributed intelligent metering data synchronization management system provided by the embodiments of the present application can execute the distributed intelligent metering data synchronization management method provided by any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0056] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, however, any number of different modules can be used and run on the user terminal and / or server, and each unit and module is only divided according to the functional logic, but is not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy mutual differentiation, and do not limit the protection scope of the present application.

[0057] The specific embodiments described above do not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application. In some cases, the actions or steps described in the present application can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are possible or can be advantageous.

Claims

1. A distributed intelligent metering data synchronization management method, characterized in that, The system comprises: Caching metering data obtained by connecting a smart metering system by an edge node; Batch differential synchronization of the metering data according to a time window to obtain transmission data; Synchronous transmission of the transmission data through a transmission channel to generate a message queue; Device data analysis of the message queue based on a center node to construct a distributed database for distributed query; Wherein, batch differential synchronization of the metering data according to a time window to obtain transmission data comprises: Setting a time window of a distributed device based on business requirements, transmission delay requirements, system load and data change frequency; Comparing the current data in the time window with the synchronization record to determine the change data with a change threshold; Triggering synchronization with the change data to obtain the transmission data; Wherein, triggering synchronization with the change data to obtain the transmission data comprises: Generating a unique identification code based on the change data; Generating a recent difference value according to the unique identification code and the difference value of the change data in the latest record in the synchronization record; Triggering synchronization with the recent difference value to obtain the transmission data, and encapsulating the calculated difference value, UID and necessary metadata into a transmission data packet; Wherein, synchronous transmission of the transmission data through a transmission channel to generate a message queue comprises: Selecting a communication protocol to determine the transmission channel of the transmission data; Introducing a time synchronization mechanism to perform transmission of the transmission data, and performing transmission confirmation based on the transmission channel to obtain the message queue, wherein after the end of each time window, the edge node sends the packaged transmission data to the center node through the transmission channel, and the center node receives the transmission data and puts it into the message queue for processing, and before data transmission, the time synchronization mechanism is used to ensure that the time of the edge node and the center node is consistent; Introducing a time synchronization mechanism comprises: Setting a synchronization fault tolerance time of the distributed device; Checking the clock drift of the distributed device based on the synchronization fault tolerance time, and starting a time source based on the clock drift degree; Selecting a healthy time source in the time source to configure the distributed device.

2. The distributed smart metering data synchronization management method of claim 1, wherein, Performing transmission confirmation based on the transmission channel comprises: Taking a distributed device as a message producer, taking the edge node as a message consumer, and transmitting partitioned messages obtained by the message producer to the message consumer through the transmission channel; Reading the partitioned messages by the message consumer and sending a processing reply, if it is determined that the processing fails according to the processing reply, performing message retry of the partitioned messages until it is determined that the processing is successful in the processing reply; Extracting a dead letter queue based on the message retry for error handling.

3. The distributed smart metering data synchronization management method of claim 1, wherein, Device data analysis of the message queue based on the center node to construct a distributed database comprises: Performing device data analysis of the distributed device with the message queue to obtain device data; Storing the device data to the distributed database.

4. A distributed intelligent metering data synchronization management system, characterized in that, The system is used to implement a distributed smart metering data synchronization management method according to any one of claims 1-3, and the system comprises: The metering data caching module is configured to cache metering data obtained by connecting a smart metering system by using an edge node. The batch differential synchronization module is configured to perform batch differential synchronization on the metering data according to a time window to obtain transmission data. The message queue generation module is configured to perform synchronous transmission of the transmission data through a transmission channel to generate a message queue. The distributed database construction module is configured to perform device data analysis on the message queue based on a center node to construct a distributed database for distributed query.

Images