A multi-cluster mode IoT management platform and data processing method thereof

Abstract

The present invention discloses an Internet of Things management platform in a multi-cluster mode and a data processing method thereof. The Internet of Things management platform comprises a plurality of access unit clusters, a message queue cluster and a data forwarding cluster; the access unit cluster is used to route business data to a message queue cluster of a specified slice according to a device ID; the message queue cluster is used to cache business data, and consumers consume business data released by producers in real time according to rules; the data forwarding cluster is used to distribute business data to a business system according to rules; and the platform also comprises a time series database for storing dynamic data, the time series database slices business data based on business types, and when business data is uploaded through the message queue cluster, it is parsed and stored through a corresponding time series database proxy cluster; the present invention realizes the multi-cluster performance expansion of the Internet of Things management platform and realizes the accurate, efficient and reliable transmission and distribution of data of the Internet of Things system.

Description

Technical Field

[0001] The present invention relates to an Internet of Things management platform and a data processing method thereof, in particular to a multi-cluster mode Internet of Things management platform and a data processing method thereof. Background Art

[0002] The "dual carbon" strategic goal and the new power system require the creation of a global power Internet of Things with "full state perception, full equipment connection, full data integration, and full business intelligence", which puts forward higher requirements for the Internet of Things collection and management system. The Internet of Things Management Platform (hereinafter referred to as the Internet of Things Platform) is the foundation of the new power system. It realizes the standardized access of unified edge Internet of Things agents (hereinafter referred to as edge devices) and Internet of Things smart terminals (hereinafter referred to as end devices), unifies the collection source, aggregates data, and realizes cloud-edge collaboration; it provides real-time and stable data services for the enterprise middle platform and business systems, and supports the construction of an overall smart power business ecosystem.

[0003] With the advancement of the construction of the new power system, the number of devices connected to the IoT platform has increased rapidly. In some provincial IoT platforms, the number of devices has reached millions, and the data transmission volume has reached tens of billions per day. The scale of device access will soon exceed 10 million. The power Internet of Things faces technical challenges such as unified access of massive terminal devices, unified processing and storage of massive data, unified operation and maintenance management of global equipment, and data sharing and use in different business fields.

[0004] In summary, in the production environment of the industrial Internet of Things, it is urgent to propose a multi-cluster mode data processing method and Internet of Things management platform. Summary of the invention

[0005] Purpose of the invention: The purpose of the present invention is to provide a multi-cluster mode IoT management platform and a data processing method thereof for efficient storage, transmission and distribution.

[0006] Technical solution: The multi-cluster mode IoT management platform described in the present invention includes multiple access unit clusters, message queue clusters and data forwarding clusters; the access unit cluster is used to route business data to the message queue cluster of the specified shard according to the device ID, and is also used to transmit management data to the management cluster; the message queue cluster is used to cache business data, and consumers consume business data published by producers in real time according to rules; the data forwarding cluster is used to distribute business data to the business system according to rules;

[0007] It also includes a time series database for storing dynamic data. The time series database shards business data based on business types. When the business data is sent up through the message queue cluster, it is parsed and stored in the database through the corresponding time series database proxy cluster.

[0008] Furthermore, the access unit cluster is used to route business data to a message queue cluster of a specified shard according to a device ID, including: when a device with a device ID of BnRnTnDn reports data, the access unit cluster routes the data to a message queue cluster of a specified shard _Bn according to the device's professional code Bn.

[0009] Furthermore, the management cluster synchronizes the device and service data by subscribing to the service data in the message queue cluster.

[0010] Furthermore, the data forwarding cluster configures data distribution rules according to device region, specialty, type or grouping.

[0011] Furthermore, when the business data is uploaded through the message queue cluster, it is parsed and stored in the database through the corresponding timing library proxy cluster, including: the timing library proxy cluster establishes an index relationship between the device ID and the timing library instance, and shares the index relationship between multiple timing library proxy clusters through the cache index module; when the application service requests business data, the data storage address of the corresponding device is searched through the cache index module to obtain the business data.

[0012] Furthermore, it also includes a relational database for storing static data.

[0013] Furthermore, it includes an HTTP protocol access cluster and an MQTT protocol access cluster, and the HTTP protocol access cluster and the MQTT protocol access cluster respectively include multiple access unit clusters.

[0014] The data processing method of the multi-cluster mode IoT management platform of the present invention includes a data transmission method and a data storage method;

[0015] The data transmission method comprises the following steps: management data is transmitted to the management cluster through the access unit cluster, service data is routed to the message queue cluster of the designated shard according to the device ID through the access unit cluster, the message queue cluster is used to cache service data, consumers consume the service data published by producers in real time according to the rules, and the service data is distributed to the service system according to the rules through the data forwarding cluster;

[0016] The data storage method comprises the following steps: using a time series database to store dynamic data, the time series database shards business data based on business types, and when the business data is sent up through the message queue cluster, it is parsed and stored in the database through the corresponding time series database proxy cluster.

[0017] The electronic device described in the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, the data processing method of the multi-cluster mode Internet of Things management platform is implemented.

[0018] The computer-readable storage medium of the present invention stores a computer program, and when the computer program is executed by a processor, the data processing method of the multi-cluster mode Internet of Things management platform is implemented.

[0019] Beneficial effects: Compared with the prior art, the advantages of the present invention are: (1) the data transmission multi-cluster link of "access unit-message queue-data forwarding service" diverts data according to management and business type at the access unit, and large-scale business data is transmitted to the business system through the core link, and a small amount of management data is processed by the management cluster, thereby realizing the multi-cluster performance expansion of the Internet of Things platform as a whole and realizing large-scale access of devices; (2) the multi-cluster data transmission method based on device ID segmentation, through the device ID based on professional and regional segmentation mechanism, the data is transmitted through independent channels from the access unit, message queue, data forwarding to the business system, so as to achieve fault isolation between different businesses, and realize the accurate, efficient and reliable transmission and distribution of data in the Internet of Things system on the basis of ensuring unified management and unified operation and maintenance of equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is an architecture diagram of the multi-cluster mode IoT management platform of the present invention.

[0021] Figure 2 Schematic diagram of the data transmission link of the present invention.

[0022] Figure 3 This is a diagram of the multi-cluster data storage architecture of the present invention. DETAILED DESCRIPTION

[0023] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0024] The present invention designs a multi-cluster link for data transmission of "access unit-message queue-data forwarding service", and divides data according to management and business type at the access unit. Large-scale business data is transmitted to the business system through the core link, and a small amount of management data is processed by the management cluster, thus realizing the multi-cluster performance expansion of the IoT management platform as a whole. A multi-cluster data transmission method based on device ID segmentation is designed. Through the device ID based on professional and regional segmentation mechanism, data is transmitted through independent channels from access unit, message queue, data forwarding to business system, so as to achieve fault isolation between different businesses and realize accurate, efficient and reliable transmission and distribution of IoT system data.

[0025] like Figure 1As shown, in order to solve the problems of poor scalability and low access performance of existing IoT management platforms, the IoT management platform in multi-cluster mode described in the present invention has the ability to continuously expand performance and realize large-scale access of equipment. Power IoT data is divided into management data and business data, where management data includes static information such as object models, equipment ledgers, IoT APP information, etc.; business data refers to the real-time collection of telesignaling, telemetry, active and reactive power data by equipment. The IoT management platform in multi-cluster mode described in the present invention transmits massive business data in clusters, and realizes efficient and flexible classification and diversion transmission of data on the basis of ensuring unified management and unified operation and maintenance of equipment.

[0026] The multi-cluster mode IoT management platform of the present invention includes HTTP protocol access cluster, MQTT protocol access cluster, message queue cluster, API gateway cluster, management service cluster, message forwarding service cluster, time series library proxy cluster, etc., which realizes MQTT and HTTP protocol access of edge devices, and provides open API interface and data sharing capability for business systems. According to the type and transmission scale of IoT data, the multi-cluster IoT management platform is divided into management cluster and business data cluster. Among them, the management cluster is responsible for the maintenance of the object model of edge devices and end devices, information registration, online access, APP installation and upgrade, etc., and the business cluster is responsible for the real-time transmission and distribution of massive business data such as equipment telesignaling, telemetry, active and reactive power, temperature, humidity, etc. In order to solve the performance problems encountered by the gradual increase in the scale of data, the data transmission multi-cluster link of "access unit-message queue-data forwarding service" of the present invention divides the data according to management and business type at the access unit, and large-scale business data is transmitted to the business system through the core link, and a small amount of management data is processed by the management cluster, which realizes the multi-cluster performance expansion of the IoT management platform as a whole.

[0027] The present invention proposes a data classification and diversion routing method based on device ID number segments. Through the device ID based on professional and regional number segmentation mechanism, and designs data segmentation independent channel transmission from access units, message queues, data forwarding to business systems, fault isolation between different businesses is achieved, realizing accurate, efficient and reliable transmission and distribution of data in the Internet of Things system.

[0028] (1) Device ID segment

[0029] When the equipment data is reported, the IoT management platform needs to transmit, store and distribute the data according to business needs. In the multi-cluster mode, the massive data needs to be routed and diverted according to the equipment ID. This paper designs an identification management method that can be recognized by various professions such as power generation, transmission, transformation, distribution and use in the power Internet of Things. The equipment ID is compiled based on basic attributes such as power profession, region, equipment type, manufacturer, production date and factory serial number to form a unique identification of the equipment in the IoT management platform.

[0030] As shown in Table 1, the device ID uses a decimal integer encoding rule with a total length of 22 bits. The ID can be divided into four sections according to its specific meaning: the first three bits are the first section, representing the business application number (Business, represented by B), with a value range from 100 to 999, indicating that the IoT management platform supports a maximum of 899 business applications for northbound docking; the next four bits are the second section, representing the region number (Region, represented by R) to which the device belongs, corresponding to the administrative region code, of which the first two bits represent the city, such as Shaanxi. Xi'an, Weinan, etc. The last two digits represent the district or county, such as Yanta District and Chang'an District of Xi'an, with a value range of 1000 to 9999; the next four digits are the third segment, representing the device type number (Type, represented by T), with a value range of 1000 to 9999, indicating that the same business system can access up to 8999 types of devices; the last 12 digits are the fourth segment, representing the unified device code (Device, represented by D), which is generated and allocated by the IoT management platform for devices in a certain region according to the factory code of the device. When the edge device is registered, the IoT management platform generates an ID with a BRTD structure for device management and data transmission.

[0031] Table 1 Rules for generating device IDs on the IoT management platform

[0032]

[0033] (2) Multi-cluster data transmission and distribution

[0034] With the rapid development of digital networks, data transmission is divided into synchronous transmission and asynchronous transmission according to the transmission mode, and can be divided into parallel transmission and serial transmission according to the transmission order. Common data transmission technologies exist in multiple ways at the same time. Big data transmission components realize data classification and diversion through the combination of subscription and publishing mode and hierarchical sharding mode.

[0035] When data of different professions, regions, and types are accessed on a large scale, the IoT management platform transmits and distributes data in a hierarchical and fragmented manner in a multi-cluster mode based on the ID structure of the device BRTD. To meet the needs of large-scale business expansion, the IoT management platform divides the data transmission link into multiple access unit clusters, message queue clusters, and data forwarding clusters. Each cluster is further divided into multiple transmission nodes. Data is transmitted between clusters through subscription, publishing, synchronous push, and other methods, such as Figure 2 The end-to-end independent channel design based on device ID can meet the concurrent transmission of massive business data and achieve fault isolation between different businesses.

[0036] Access unit cluster: responsible for massive MQTT long connection management and data access. When a device with device ID BnRnTnDn reports data, the access unit cluster node routes the data to the message queue cluster of the specified shard _Bn according to the device's professional code Bn.

[0037] Message queue cluster: It is a container for transmitting and caching messages in a multi-cluster architecture, which enables the IoT management platform to have the capabilities of peak shaving, application decoupling, and asynchronous communication. It consists of message servers, producers, consumers, topics, message bodies, etc. According to the business professional coding, the message queue topics are divided into shard_B1, shard_B2, shard_B3...shard_Bn to improve the architecture concurrency and data sharing capabilities. The data published by the access unit producer is consumed in real time by the data forwarding unit consumer according to the rules, and the data is cached in shard_Bn for a period of time to facilitate later problem location and data backtracking. In addition, according to the business function requirements of the IoT management platform, the management service cluster will also subscribe to the message queue cluster data to achieve synchronous management of devices and data.

[0038] Data forwarding cluster: responsible for platform data distribution and sharing. Design rule configuration capabilities, dynamically configure different rules based on device region, profession, type or group, etc., for flexible data distribution to meet the construction needs of different business applications. For example, rule _BnTn subscribes to the data of message queue shard _Bn and forwards it to the business system of a certain provincial business application n type of equipment.

[0039] (3) Multi-cluster mode data distributed storage

[0040] While massive amounts of data bring challenges to big data transmission, they also bring problems with data storage. The scale of data storage in the new power system is much larger than that of the traditional system, and relational databases are difficult to meet the needs of real-time message storage and access on such a scale. Therefore, the IoT management platform divides device data into two categories: static and dynamic. According to data characteristics and generation levels, static data is stored in a relational database, and dynamic data is stored in a time series database. In order to ensure the performance and reliability of data access, a time series data access mechanism based on device ID sharding is designed to construct a data access solution with a multi-cluster architecture. Its architecture is as follows: Figure 3 shown.

[0041] The data storage multi-cluster architecture consists of three parts: the timing library instance, the timing library proxy cluster, and the cache index module. When the data is uploaded through different message queue clusters, it is parsed and stored by the corresponding timing library proxy cluster. At the same time, an index relationship between the device ID and the timing library instance is established, and it is shared with multiple timing library proxy clusters through the distributed cache index module. When the application service requests data, the data storage address of the corresponding device can be retrieved in the cache index module to obtain the data. To ensure data reading and writing efficiency, the DB in each timing library instance is sharded based on business type B (db-1, db-2...db-n) to achieve accurate data access and fault isolation for different businesses.

[0042] The data processing method of the multi-cluster mode IoT management platform of the present invention includes a data transmission method and a data storage method;

[0043] The data transmission method comprises the following steps: management data is transmitted to the management cluster through the access unit cluster, service data is routed to the message queue cluster of the designated shard according to the device ID through the access unit cluster, the message queue cluster is used to cache service data, consumers consume the service data published by producers in real time according to the rules, and the service data is distributed to the service system according to the rules through the data forwarding cluster;

[0044] The data storage method comprises the following steps: using a time series database to store dynamic data, the time series database shards business data based on business types, and when the business data is sent up through the message queue cluster, it is parsed and stored in the database through the corresponding time series database proxy cluster.

[0045] The electronic device described in the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, the data processing method of the multi-cluster mode Internet of Things management platform is implemented.

[0046] The computer-readable storage medium of the present invention stores a computer program, and when the computer program is executed by a processor, the data processing method of the multi-cluster mode Internet of Things management platform is implemented.

[0047] The computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store the desired program code in the form of instructions or data structures and that can be accessed by a computer.

[0048] The processor is used to execute the computer program stored in the memory to implement each step of the method involved in the above embodiment.

Claims

1. A multi-cluster mode IoT management platform, characterized in that: It includes multiple access unit clusters, message queue clusters and data forwarding clusters; the access unit cluster is responsible for massive MQTT long connection management and data access, and is used to route business data to the message queue cluster of the specified shard according to the device ID. When a device with a device ID of BnRnTnDn reports data, the access unit cluster routes the data to the message queue cluster of the specified shard _Bn according to the professional code Bn to which the device belongs. The access unit cluster is also used to transmit management data to the management cluster; the message queue cluster is used to cache business data, and consumers consume the business data released by producers in real time according to the rules; the data forwarding cluster is responsible for data distribution and sharing, and is used to distribute business data to the business system according to the rules; It also includes a time series database for storing dynamic data, which shards the business data according to the business type. When the business data is sent up through the message queue cluster, it is parsed and stored in the database through the corresponding time series library agent cluster; the time series library agent cluster establishes an index relationship between the device ID and the time series library instance, and shares the index relationship between multiple time series library agent clusters through a cache index module; when an application service requests business data, the data storage address of the corresponding device is searched through the cache index module to obtain the business data; the data in each time series library instance is sharded according to the business type, including db-1 to db-n.

2. The multi-cluster mode IoT management platform according to claim 1, characterized in that: The management cluster synchronizes the device and service data by subscribing to the service data in the message queue cluster.

3. The multi-cluster mode IoT management platform according to claim 1, characterized in that: The data forwarding cluster configures data distribution rules according to device region, specialty, type or grouping.

4. The multi-cluster mode IoT management platform according to claim 1, characterized in that: Also included is a relational database for storing static data.

5. The multi-cluster mode IoT management platform according to claim 1, characterized in that: It includes an HTTP protocol access cluster and an MQTT protocol access cluster, and the HTTP protocol access cluster and the MQTT protocol access cluster respectively include multiple access unit clusters.

6. A data processing method for a multi-cluster mode IoT management platform, characterized in that: including data transmission methods and data storage methods; The data transmission method comprises the following steps: management data is transmitted to the management cluster through the access unit cluster, business data is responsible for massive MQTT long connection management and data access, the access unit cluster routes the business data to the message queue cluster of the specified shard according to the device ID, when the device with the device ID of BnRnTnDn reports data, the access unit cluster routes the data to the message queue cluster of the specified shard _Bn according to the professional code Bn to which the device belongs, the access unit cluster message queue cluster is used to cache business data, consumers consume the business data released by producers in real time according to the rules, the data forwarding cluster is responsible for data distribution and sharing, and the business data is distributed to the business system according to the rules; The data storage method includes the following steps: using a time series database to store dynamic data, the time series database shards business data according to business types, and when the business data is sent up through the message queue cluster, it is parsed and stored in the database through the corresponding time series library proxy cluster; the time series library proxy cluster establishes an index relationship between a device ID and a time series library instance, and shares the index relationship between multiple time series library proxy clusters through a cache index module; when an application service requests business data, the data storage address of the corresponding device is searched through the cache index module to obtain the business data; the data in each time series library instance is sharded according to the business type, including db-1 to db-n.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is loaded into the processor, the data processing method of the multi-cluster mode Internet of Things management platform according to claim 6 is implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, the data processing method of the multi-cluster mode Internet of Things management platform according to claim 6 is implemented.

Images