An internet of things data real-time computing service system and a construction method

Abstract

The present disclosure provides an Internet of Things data real-time computing service system and a construction method. A collection service obtains on-site telemetry, remote signaling and electric energy data from an Internet of Things device, adopts a Kafka message queue buffer to complete data cleaning, storage, real-time computing, fault diagnosis and push alarm based on Flink, uses a distributed and in-memory database Redis as a cache database to provide support for data analysis, prediction and visual display, monitors servers, application clusters and data by using Prometheus+Grafana, and pushes and displays logs.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of Internet of Things and real-time data processing, and specifically designs an Internet of Things data real-time computing service processing method based on Flink+Kafka in the aspects of Internet of Things data collection, cleaning, storage, computing and monitoring. BACKGROUND

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] In today's society, mankind is stepping into the information age with the progress of the information industry, and the Internet of Things technology is widely used in all aspects of people's production and life. The number of various Internet of Things collection terminal devices has exploded, 5G networks are gradually replacing 4G networks, and the transmission speed is higher. The existing system centered on the mysql relational model cannot meet the needs of supporting industrial communication protocols and real-time forwarding, cleaning, storage, real-time computing and monitoring of data.

[0004] At present, in actual production, the mainstream stream computing framework has Apache Storm, Spark Streaming and Flink. Among them, Apache Storm is one order of magnitude lower than Flink and Spark Streaming in terms of throughput, and Storm lacks automatic management of state inside the application, does not support high-level program programming interface (API), does not support SQL syntax, and does not support machine learning and graph computing scenarios. Spark Streaming processes real-time data in a micro-batch manner, and the division of time window can only be accurate to seconds. In terms of data processing delay, it does not really achieve real-time processing and the size of the divided micro-batch data is difficult to control, which is extremely easy to cause memory overflow. SUMMARY

[0005] In order to solve the above problems, the present disclosure proposes an Internet of Things data real-time computing service system and a construction method, a method for constructing an Internet of Things real-time computing service based on Flink+Kafka, which adopts Internet of Things and real-time data processing technology to meet the needs of high concurrency, high availability, high performance, high stability and high expansion of the system in the case of access of millions of Internet of Things devices.

[0006] According to some embodiments, the present disclosure adopts the following technical solutions:

[0007] An Internet of Things data real-time computing service system comprises:

[0008] A data collection and pushing module is configured to collect measurement data information of Internet of Things devices in real time and push the data information;

[0009] The data cleaning and storage module is based on a data processing framework, constructs a basic table and a data cleaning table, cleans the data, and stores the data;

[0010] The data calculation and batch storage module is based on a data processing framework, performs real-time calculation on data from different data sources, and performs batch storage on the data;

[0011] The decision diagnosis and alarm module is based on decision logic to perform decision tree visualization configuration and data diagnosis, and issue alarm information;

[0012] The data monitoring and visualization module realizes reading and monitoring of nodes and data, and uses a visualization tool to realize self-defined visualization display of the data.

[0013] According to some other embodiments, the present disclosure adopts the technical scheme as follows:

[0014] A construction method of an Internet of Things data actual calculation service system is provided, and the specific construction method is as follows:

[0015] Configure Redis;

[0016] Build a Kafka cluster;

[0017] Start each module of the project, configure jdk, maven, and mysql environments, configure settings of Kafka and Redis in the project, connect a database, and run each module in the project.

[0018] Compared with the prior art, the present disclosure has the beneficial effects that:

[0019] 1. A high-concurrency communication server is constructed based on a Netty communication framework, data acquisition is realized by communicating with field devices of an access platform through NIO asynchronous network communication technology, dynamic loading of multiple protocols is realized through a self-defined protocol template, and the communication state of a terminal device is determined through setting multiple communication flags, so as to ensure the orderliness and accuracy of data acquisition.

[0020] 2. Developed based on Flink+Kafka, supporting multiple client languages, meeting the data access and processing of various monitoring platforms and the all-time operation requirements of the access platform, and having high adaptability and stability.

[0021] 3. The unique service architecture developed based on Java, the Internet of Things platform constructed by Kafka message middleware technology, real-time database / time series database technology, Redis in-memory database technology, Flink stream processing technology, etc., allows processing of million-level data volume and concurrency per minute, and fully meets the requirements of massive data access, storage, real-time calculation, and real-time early warning. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings, which form a part of this disclosure, are intended to provide further understanding of the disclosure and are incorporated herein for illustrative purposes. The description and illustration are intended to explain aspects of the present disclosure and do not limit the present disclosure.

[0023] Figure 1 A real-time data computing service system architecture diagram of the present disclosure;

[0024] Figure 2 A real-time data computing service result visualization flowchart of the present disclosure; DETAILED DESCRIPTION

[0025] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0026] It should be noted that the following detailed description is illustrative only and is intended to provide further description of the present disclosure. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs.

[0027] It should be noted that the terms used herein are only intended to describe specific embodiments and are not intended to limit exemplary embodiments according to the present disclosure. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should also be understood that when the terms "comprise" and / or "include" are used in the specification, they indicate the presence of the features, steps, operations, devices, components, and / or combinations thereof.

[0028] Embodiment 1

[0029] The present disclosure provides an Internet of Things data real-time computing service system, comprising:

[0030] 1) A data acquisition and pushing module for real-time acquisition of measurement data information of Internet of Things devices, and pushing of the data information.

[0031] Specifically, when data acquisition is performed, communication with Internet of Things devices is realized based on a Netty communication framework, device information of a pre-stored database is read to determine a communication protocol used by the devices, a data frame is sent to the Internet of Things devices in a pre-defined mechanism, and data message acquisition is performed.

[0032] A TCP connection is established between the Netty communication framework and the field devices, communication protocols such as Modbus protocol and IEC104 are supported, current, voltage, power and other measurement data of the Internet of Things devices are acquired in real time, and data messages are pushed to a Kafka cluster. This module is the source of data messages and the entry part of the system, and is the basis for subsequent cleaning, storage, real-time computing, and fault diagnosis.

[0033] A data push thread is constructed to push the collected Internet of Things device data information to a Kafka cluster to generate a Kafka data message.

[0034] 2) A data cleaning and storage module, based on a data processing framework, constructs a basic table and a data cleaning table to clean the data and store the data.

[0035] The data storage module is constructed by using a single instance design pattern to reduce the memory overhead, the template method pattern is used to ensure dynamic loading and calling of different specifications, the multi-thread mechanism is used to ensure the independence of the operation of the functional modules, and the lock mechanism is used to ensure the consistency of the data.

[0036] The data processing framework is a Flink framework, based on the Flink framework, the Kafka data message is persistently stored in a Mysql basic table, and the data in the basic table is used as a third data source. The basic table is similar to a log and stores raw data, has the characteristics of large data volume and comprehensiveness, and is used as a backup basic data resource for Redis or cleaning library crash.

[0037] Based on the Flink framework, the data is cleaned and stored, the basic table and the cleaning table are constructed to realize persistence, and the related data is stored in Redis for subsequent use. In the Flink environment, the Kafka data message is stored in Redis, and the data stored in Redis is the first data source. The Redis is a database based on memory. The data read-write speed is tens to hundreds of times faster than Mysql, and has higher performance and efficiency. It is convenient for visual display of data.

[0038] Different cleaning tables are constructed, which are respectively a telemetry table, an electric meter table and an alarm table. Based on Flink, the Kafka data message is persistently stored in different cleaning tables, and the data in the cleaning table is used as a second data source. Among them, the telemetry table refers to single-precision measurement values such as current, voltage and power; the electric meter table refers to double-precision cumulative values such as window values or high-precision electric meter values. Based on Flink, the different Kafka data messages are persistently stored in different cleaning tables to realize better storage, display and utilization of data.

[0039] 3) A data calculation and batch storage module, based on a data processing framework, performs real-time calculation on data from different data sources and batch stores the data.

[0040] Specifically, Flink can generate different time interval timing messages and signals, including 5-minute, 1-hour and 24-hour signals. The signals are configured on demand to realize the required functions.

[0041] In the Flink environment, the Flink timer is set to generate a timing message, which is a timing signal to be executed, and the Flink timer is stored in the Kafka cluster timing message list, providing timing messages for the execution of service instructions.

[0042] A cleaning history library is established, and the results of the calculation services based on the Flink timing message of the data of the first data source, the second data source and the third data source are batch stored in the cleaning history library.

[0043] Specifically, in the data calculation and batch storage module, the timing service includes:

[0044] 3.1) Based on the Flink timer to generate timing messages, such as 5-minute storage signals, 5-minute calculation signals, decision diagnosis signals (1 hour, 24 hours).

[0045] 3.2) The Flink timing message is stored in the Kafka cluster timing message topic, providing timing messages for table building, storage, calculation, and decision diagnosis services.

[0046] 3.3) Table building service based on Flink timing message.

[0047] Wherein, whether to build a table is judged: based on 24-hour message, at 0 o'clock every day, whether the basic table and the cleaning table exist is judged, if not, the basic table, the kilowatt-hour table and the telemetry table of the next month are created.

[0048] 3.4) Calculation service based on Flink timing message. Based on Flink, storage is performed according to the timing message (5-minute timing signal) in Kafka.

[0049] 3.5) Data source: the data source is described above: the first data source is to read Redis data, the second data source is a cleaning data table, and the third data source is a basic data table. Finally, the calculated results are batch stored in the cleaning history library.

[0050] 4) Decision diagnosis and alarm module, based on decision logic to configure decision tree visualization and diagnose data, and issue alarm information.

[0051] 4.1) The above decision diagnosis and alarm module includes:

[0052] The decision tree diagnosis alarm of the visual self-defined configuration is visualized. The platform uses Gooflow to realize the visualization construction and display of the decision tree, and the serial number, state, business method, flow transfer rule and overall structure of the decision tree node and other attributes can be interacted with the front-end visualization interface. Among them, the business method of the node corresponds to the complete fault diagnosis algorithm in the background. The user can customize the configuration of each item of the decision tree through the visualization page. Finally, through the constructed decision tree, a complete alarm service is formed.

[0053] 4.2) Decision service based on Flink timing message. The decision diagnosis is based on the timing signal of Flink, the decision tree information configured by the decision tree configuration platform is used for initialization, the basic table and the cleaning table are used as the data source of the decision diagnosis, and finally the diagnosis result is stored in the diagnosis table and an alarm is given.

[0054] 5) Data monitoring and visualization module, realizing reading and monitoring of nodes and data, using a visualization tool to realize self-defined visualization display of data.

[0055] 5.1) Prometheus uses Exporter to realize reading and monitoring of nodes and data flow, monitors performance information and indicators, and uses Grafana as a visualization tool to realize self-defined visualization display of Prometheus data.

[0056] 5.2) AlertManager or third-party alarm platform (such as CloudAlert) is used for alarm, which can realize WeChat alarm, email alarm and other modes.

[0057] 5.3) Log collection, filtering, analysis and display are completed based on Logstash+Elasticsearch+Kibana. Among them, Logstash collects the logs of different modules of the project in the Kafka message queue, and distinguishes and filters the useless logs of different modules of the project; and the logs are displayed on the Kibana visualization page through different indexes; meanwhile, the log retrieval, analysis and storage functions based on Elasticsearch are provided. After the Elk accesses the project, it greatly facilitates the managers to monitor the project exceptions and errors in real time, and improves the overall stability of the project.

[0058] The present disclosure provides an Internet of Things data real-time computing service system, which constructs a high-concurrency communication server based on a Netty communication framework, realizes data collection through communication with field devices of an access platform through NIO asynchronous network communication technology, realizes dynamic loading of multiple protocols through a self-defined protocol template, determines the communication state of terminal devices through setting of multiple communication flags, and ensures the orderliness and accuracy of data collection.

[0059] Based on Flink+Kafka development, support multiple client languages, meet the data access and processing of various monitoring platforms and the all-time operation requirements of access platforms, have high adaptability and stability.

[0060] Based on the unique service architecture developed by Java, the Internet of Things platform is built by using Kafka message middleware technology, real-time database / time series database technology, Redis in-memory database technology, Flink stream processing technology, etc. It can process millions of data and concurrent volume per minute, fully meet the demand of mass data access, storage, real-time calculation and real-time warning.

[0061] System construction

[0062] The construction method of the Internet of Things data real-time computing service system comprises the following steps:

[0063] 1) Configure Redis

[0064] Download and decompress Redis, configure the redis.windows.conf file, configure the maxmemory, requirepass, port parameters, and finally specify the configuration file to start the Redis service.

[0065] 2) Build Kafka cluster

[0066] First, install the Jdk environment, then download and decompress Kafka, and configure the server.properties file, modify the log.dirs, advertised.listeners, broker.id and other parameters, start zookeeper, then start Kafka to run on port 9092, and connect to the default port 2181 of zookeeper.

[0067] 3) Start each module of the project

[0068] First, configure the jdk, maven, mysql and other environments, configure the settings of Kafka and redis in the project (corresponding to the already started redis and Kafka), connect the database, and run each module of the project.

[0069] A real-time computing service of Internet of Things based on Flink+Kafka is constructed, through the data acquisition service based on the Netty communication framework to obtain the field telemetry, remote signaling and electric data from the Internet of Things device, the Kafka message queue is used for buffering and shunting, the Flink-based stream computing service subscribes to the Kafka Topic message, the data cleaning, storage, real-time calculation, fault diagnosis and push alarm are completed, the distributed and in-memory database Redis is used as the cache database to provide support for data analysis, prediction and visual display, the Prometheus+Grafana is used for monitoring the server, application cluster and data, and the Slf4j+Log4j2+Kafka+Elk is used as the sending, filtering, processing, analysis and display tool of the log. In addition, the service can meet the uninterrupted operation of 724 hours, and can support the high concurrency of million-level data per minute, and ensure the consistency of data in the collection and processing process.

[0070] Embodiment 2

[0071] The present disclosure provides a service method of an Internet of Things data real-time computing service system, specifically comprising:

[0072] Through the data acquisition service based on the Netty communication framework, field telemetry, remote signaling and electric data are obtained from the Internet of Things device;

[0073] The Kafka message queue is used for buffering and shunting, and the Flink-based stream computing service subscribes to the Kafka Topic message to complete the data cleaning and storage, and serve as different data sources respectively;

[0074] The Flink-based stream computing service subscribes to the Kafka Topic message to complete the data cleaning, storage, real-time calculation, fault diagnosis and push alarm, and the distributed and in-memory database Redis is used as the cache database to provide support for data analysis, prediction and visual display;

[0075] The Prometheus+Grafana is used for monitoring the server, application cluster and data, and the Slf4j+Log4j2+Kafka+Elk is used as the sending, filtering, processing, analysis and display tool of the log.

[0076] In addition, the service can meet the uninterrupted operation of 724 hours, and can support the high concurrency of million-level data per minute, and ensure the consistency of data in the collection and processing process.

[0077] As Figure 1As shown, the network communication connection is established to realize data transmission, including the transmission layer, the data collection of the Internet of Things device based on the Netty communication framework, the formation of the queue of Kafka data messages, the batch data storage of the basic table, the cleaning table and Redis in the Flink framework scene, and the data in the basic table as the third data source, the data in the cleaning table as the second data source, and the data in the Redis cluster as the first data source. The Flink timer generates timing messages, such as 5-minute storage signals, 5-minute calculation signals, and decision diagnosis signals (1 hour, 24 hours). The Flink timing messages are stored in the Kafka cluster timing message topic to provide timing messages for table building, storage, calculation, and decision diagnosis services. The table building service based on the Flink timing message. The corresponding calculation service based on Flink is performed on the corresponding timing message, and the decision tree visualization configuration is performed according to the decision logic. Data diagnosis and related alarm services are realized based on the decision logic. After diagnosis, the alarm information is returned to the cleaning history warehouse, and a diagnosis table is formed and returned to the cleaning history warehouse. Alarm messages are sent through SMS, telephone or WeChat to generate reminders.

[0078] The data collection service based on the Netty communication framework obtains on-site telemetry, remote signaling and electric energy data from the Internet of Things device. Kafka message queue caching is used for shunting. The Flink-based stream computing service subscribes to Kafka Topic messages to complete data cleaning, storage, real-time calculation, fault diagnosis and push alarm. The distributed and in-memory database Redis is used as a cache database to support data analysis, prediction and visualization. Prometheus+Grafana is used to monitor servers, application clusters and data. Slf4j+Log4j2+Kafka+Elk is used as a log sending, filtering, processing, analysis and display tool. In addition, the service can meet the 724-hour uninterrupted operation, and can support high concurrency of millions of data per minute to ensure the consistency of data in the collection and processing process.

[0079] Figure 2 As described in the foregoing, the collection, storage, calculation, diagnosis and abnormal log of each module of the Flink service are uniformly sent to the Kafka log Topic by Log4j2 in Flink. Logstash is used as a consumer to continuously format the data in the Kafka log Topic and output it to Elasticsearch. Elasticsearch establishes an inverted index according to the log data, and finally Kibana is used to visualize the log data to clearly display the flow of data in collection, storage, calculation and diagnosis, facilitating troubleshooting by operation and maintenance personnel.

[0080] Those skilled in the art will appreciate that embodiments of the disclosure can be supplied as a method, a system, or a computer program product. Accordingly, the disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the disclosure can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, and the like) embodying computer readable program code.

[0081] The disclosure is described in reference to the flow diagrams and / or block diagrams of the method, apparatus (system) and computer program product according to embodiments of the disclosure. It should be understood that each flow and / or block in the flow diagrams and / or block diagrams, and a combination of flows and / or blocks in the flow diagrams and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, a special purpose computer, an embedded processor or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flow diagrams and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the functions specified in the flow diagram

[0082] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions means which implement the function specified in the flow diagrams and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the functions specified in the flow diagram

[0083] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flow diagrams and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the functions specified in the flow diagram

[0084] The above description is merely illustrative of the disclosure, and not restrictive. Various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the disclosure. Any modification, equivalent replacement, improvement, and the like made within the spirit and principle of the disclosure shall fall within the scope of the disclosure.

[0085] The specific embodiments of the present disclosure are described above with reference to the accompanying drawings, but are not intended to limit the protection scope of the present disclosure, and those skilled in the art should understand that various modifications or changes made on the basis of the technical solutions of the present disclosure without creative labor are still within the protection scope of the present disclosure.

Claims

1. An Internet of Things data real-time computing service system, characterized in that, Comprise: Data acquisition and pushing module, for real-time acquisition of measurement data information of Internet of Things equipment, and pushing data information; Data cleaning and storage module, based on data processing framework, building basic table and data cleaning table, cleaning data; And storing data; Data calculation and batch storage module, based on data processing framework, real-time calculation of data from different data sources and batch storage of data; Decision diagnosis and alarm module, based on decision logic for decision tree visualization configuration and data diagnosis, and issuing alarm information; Data monitoring and visualization module, realize the reading and monitoring of node and data, adopt visualization tool, realize the self-defined visualization display of data; In Flink environment, set Flink timer to generate timing message, the timing message is the timing signal to be executed, and the Flink timer is stored in the Kafka cluster timing message list, which provides timing message for the executed service instruction; Or, establish a cleaning history library, and store the results of the calculation service based on Flink timing message on the first data source, the second data source and the third data source to the cleaning history library.

2. The system of claim 1, wherein, Based on Netty communication framework, realize communication with Internet of Things equipment, determine the communication protocol used by the equipment by reading the device information of the pre-stored database, send data frame to Internet of Things equipment in pre-defined mechanism, and collect data information.

3. The system of claim 1, wherein, Build data pushing thread, push the collected Internet of Things equipment data information to Kafka cluster to generate Kafka data message.

4. The system of claim 3, wherein, The data processing framework is Flink framework, based on the Flink framework, the Kafka data message is persistently stored in the Mysql basic table, and the data in the basic table is used as the third data source.

5. The system of claim 3, wherein, Build different cleaning tables, respectively telemetry table, electric meter table and alarm table, based on Flink, persistently store Kafka data message in different cleaning tables, and the data in the cleaning table is used as the second data source.

6. The system of claim 3, wherein, In Flink environment, store Kafka data message in Redis, the data stored in Redis is the first data source, and the Redis is a database based on memory.

7. The system of claim 1, wherein, Based on the timing signal of the Flink, decision diagnosis is carried out, the decision tree information configured by the decision tree configuration platform is used for initialization, the basic table and the cleaning table are used as the data source of decision diagnosis, and finally the diagnosis result is stored in the diagnosis table and alarmed.

8. The system of claim 1, wherein, Prometheus uses Exporter to realize reading and monitoring of node and data stream, monitors performance information and indicators, and uses Grafana as a visualization tool to realize self-defined visualization display of Prometheus data.

9. A construction method of an Internet of Things data real-time computing service system based on claims 1-8, characterized in that, The specific construction method is: Configure Redis; Build Kafka cluster; Start the various modules of the project, configure the jdk, maven, mysql environment, configure the settings of Kafka, redis in the project, connect the database, and run the module described in claim 1.

Images