Data acquisition and message processing system and method based on Internet of Things equipment

Abstract

The invention particularly relates to a data acquisition and message processing system and method based on Internet of Things equipment. According to the data acquisition and message processing system and method based on the Internet of Things equipment, communication between the Internet of Things equipment and the Mosquitto component is realized by adopting a message queue telemetry transmission (MQTT) protocol, and the real-time requirement of the Internet of Things equipment on data transmission is met; the data acquisition filtering rule is issued through the server side, flexible control over data acquisition of the Internet of Things equipment is achieved, unnecessary data transmission is reduced, and the data acquisition efficiency is improved; a Kafka component is used as a distributed message queue, so that reliable transmission and efficient processing of data are ensured, and mass data generated by large-scale Internet of Things equipment can be dealt with; by means of the ELK component, data storage, query and visual display are achieved, a user can know the equipment state and the data change trend in time, and powerful support is provided for decision and management of Internet of Things application.

Description

Technical Field

[0001] The present invention relates to the technical field of the Internet of Things, and particularly relates to a data acquisition and message processing system and method based on Internet of Things devices. Background Art

[0002] In the applications of the Internet of Things, a large number of Internet of Things devices generate a vast amount of data. How to efficiently collect this data and perform effective processing and analysis on it has become an urgent problem to be solved.

[0003] Traditional data acquisition and processing methods have problems such as poor real-time performance, insufficient data filtering and analysis functions, and fixed parsing rules that cannot adapt to the data formats of multi-source heterogeneous devices. Moreover, traditional solutions are difficult to perceive the online status of devices in real time, cannot dynamically adjust data acquisition strategies, and are also difficult to meet the growing application requirements of the Internet of Things.

[0004] To solve the above problems, the present invention proposes a data acquisition and message processing system and method based on Internet of Things devices. Summary of the Invention

[0005] In order to make up for the defects of the prior art, the present invention provides a simple and efficient data acquisition and message processing system and method based on Internet of Things devices.

[0006] The present invention is realized by the following technical solutions:

[0007] A data acquisition and message processing system based on Internet of Things devices includes Internet of Things devices, a server, a Mosquitto component, a Kafka component, and an ELK component;

[0008] The Internet of Things devices are the sources of data, responsible for establishing a connection with Mosquitto through the Message Queuing Telemetry Transport (MQTT) protocol and sending the collected data to the Mosquitto component;

[0009] The server is connected to the Mosquitto component, responsible for obtaining the online status of the Internet of Things devices and sending data acquisition filtering rules to the Internet of Things devices through the Mosquitto component;

[0010] The Mosquitto component serves as an MQTT message broker server, responsible for receiving the messages sent by the Internet of Things devices, providing lightweight real-time communication, and sending the collected data to the Kafka component;

[0011] The Kafka component serves as a distributed message queue system, responsible for receiving the data sent by the Mosquitto component and storing and delivering the data to subsequent processing components;

[0012] The ELK component includes three parts: Logstash, Elasticsearch, and Kibana. Among them, Logstash is responsible for collecting data from the Kafka component and transmitting it to Elasticsearch for storage, while Kibana is responsible for various displays of the data through the dashboard.

[0013] The server sends the data collection and filtering rules to the Internet of Things devices in the form of regular expressions through the Mosquitto component, and the Internet of Things devices collect data according to the sent data collection and filtering rules.

[0014] A data collection and message processing method based on Internet of Things devices includes the following steps:

[0015] Step S1, data collection of Internet of Things devices

[0016] The Internet of Things devices collect relevant data according to the custom-set data collection period and send the data to the Mosquitto component through the Message Queuing Telemetry Transport (MQTT) protocol. The data content includes, but is not limited to, device status information and sensor data;

[0017] In the said step S1, after receiving the data from the Internet of Things devices, the Mosquitto component performs preliminary processing on the data according to the custom rules, including data format verification and data integrity check, to ensure the accuracy and reliability of the data.

[0018] Step S2, sending down the data collection and filtering rules

[0019] The server sends down the data collection and filtering rules to the Internet of Things devices through the Mosquitto component. After receiving the rules, the devices filter the collected data according to the rules and only send the data that meets the rules to the Mosquitto component, thereby reducing the data transmission volume and improving the pertinence and effectiveness of data collection;

[0020] In the said step S2, the server sends the data collection and filtering rules to the Internet of Things devices in the form of regular expressions through the Mosquitto component, and the Internet of Things devices collect data according to the sent data collection and filtering rules.

[0021] In the said step S2, the data collection and filtering rules include extracting the timestamp, service ID, channel ID, channel name, and description information from the log lines in the custom-specified format.

[0022] Step S3, data transmission and storage

[0023] The Mosquitto component sends the data collected and meeting the data collection filtering rules to the Kafka component. As a distributed message queue, the Kafka component caches and transfers the data to ensure reliable data transmission.

[0024] The ELK component consumes data from the Kafka component, parses, transforms, and stores the data, and provides efficient full-text retrieval and data analysis functions for subsequent data query and analysis.

[0025] Step S4, data display and analysis

[0026] Through the data visualization tool provided by the ELK component, various dashboards and reports are custom-created according to user needs to visually display the data.

[0027] In the said step S3, the ELK component includes three parts: Logstash, Elasticsearch, and Kibana; Logstash consumes data from the Kafka component, parses and transforms the data, and then stores the processed data in Elasticsearch. Elasticsearch provides efficient full-text retrieval and data analysis functions for subsequent data query and analysis.

[0028] In the said step S4, Kibana, as a data visualization tool, obtains the stored data by connecting to Elasticsearch, and custom-creates various dashboards and reports according to user needs to visually display the data;

[0029] Users can view the real-time status of the device, historical data trends, and data statistical analysis results information through the dashboard of Kibana by customizing.

[0030] A data collection and message processing device based on Internet of Things devices includes a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the above method steps when executing the computer program.

[0031] A readable storage medium stores a computer program thereon, and the computer program implements the above method steps when executed by a processor.

[0032] The beneficial effect of the present invention is that: the data collection and message processing system and method based on Internet of Things devices use the Message Queuing Telemetry Transport (MQTT) protocol to realize the communication between Internet of Things devices and the Mosquitto component, which has the characteristics of lightweight and strong real-time performance, and can meet the real-time requirements of Internet of Things devices for data transmission;

[0033] By issuing data collection filtering rules through the server, flexible control over the data collection of Internet of Things (IoT) devices is achieved, reducing unnecessary data transmission and improving the efficiency and pertinence of data collection;

[0034] Using the Kafka component as a distributed message queue ensures reliable data transmission and efficient processing, capable of handling the massive data generated by a large number of IoT devices;

[0035] With the help of the ELK component, data storage, query, and visual display are realized, providing users with an intuitive and convenient IoT data monitoring and analysis platform, which helps users promptly understand the device status and data change trends, and provides strong support for the decision-making and management of IoT applications. Brief Description of the Drawings

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or in the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0037] Appendix Figure 1 It is a schematic diagram of the data collection and message processing method based on IoT devices of the present invention. Detailed Embodiments

[0038] To enable those skilled in the art of the present technology to better understand the technical solutions in the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in combination with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0039] The data collection and message processing system based on IoT devices includes IoT devices, a server, a Mosquitto component, a Kafka component, and an ELK component;

[0040] The IoT devices are the data sources, responsible for establishing a connection with Mosquitto through the Message Queuing Telemetry Transport (MQTT) protocol and sending the collected data to the Mosquitto component;

[0041] The server is connected to the Mosquitto component, responsible for obtaining the online status of IoT devices and issuing data collection filtering rules to IoT devices through the Mosquitto component;

[0042] The server is connected to the Mosquitto component and has two main functions: one is to be able to display in real time whether the device is online, facilitating the management personnel to monitor the device status; the other is to be able to send the data collection regular expression rules to the device through the Mosquitto component, and the device collects the data in the device according to the rules.

[0043] The Mosquitto component, as a Message Queuing Telemetry Transport (MQTT) message broker server, is responsible for receiving the messages sent by the Internet of Things devices, providing lightweight real-time communication, and sending the collected data to the Kafka component;

[0044] The Kafka component, as a distributed message queue system, is responsible for receiving the data sent by the Mosquitto component and storing and delivering the data to the subsequent processing components;

[0045] The ELK component includes three parts: Logstash, Elasticsearch, and Kibana. Among them, Logstash is responsible for collecting data from the Kafka component and transmitting it to Elasticsearch for storage, and Kibana is responsible for various displays of the data through the dashboard.

[0046] The server sends the data collection filtering rules to the Internet of Things devices in the form of regular expressions through the Mosquitto component, and the Internet of Things devices collect data according to the sent data collection filtering rules.

[0047] The data collection and message processing method based on Internet of Things devices includes the following steps:

[0048] Step S1, Internet of Things device data collection

[0049] The Internet of Things device collects relevant data according to the custom-set data collection period and sends the data to the Mosquitto component through the Message Queuing Telemetry Transport (MQTT) protocol. The data content includes but is not limited to device status information and sensor data;

[0050] In the step S1, after receiving the data of the Internet of Things device, the Mosquitto component performs preliminary processing on the data according to the custom rules, including data format verification and data integrity check, to ensure the accuracy and reliability of the data.

[0051] Step S2, data collection filtering rule distribution

[0052] The server sends the data collection and filtering rules to the Internet of Things devices through the Mosquitto component. After receiving the rules, the devices filter the collected data according to the rules and only send the data that meets the rules to the Mosquitto component, thereby reducing the data transmission volume and improving the pertinence and effectiveness of data collection;

[0053] In step S2, the server sends the data collection and filtering rules to the Internet of Things devices in the form of regular expressions through the Mosquitto component, and the Internet of Things devices collect data according to the sent data collection and filtering rules.

[0054] In step S2, the data collection and filtering rules include extracting the timestamp, service ID, channel ID, channel name, and description information from the log lines in a custom specified format.

[0055] An example of the data collection and filtering rules is as follows:

[0056] ^(? <time>[0-9][^]*([0-9]*:)*[^]*).*? D VDCManager:reportVdc data.*? "serviceId":"(? <serviceid>.*?)".*?"channelId":"(? <channelid>.*?)".*?"channelName":"(? <channelname>.*?)".*?"description":"(? <channelgroup>.*? )

[0057] Step S3, data transmission and storage

[0058] The Mosquitto component sends the data that meets the data collection filtering rules collected to the Kafka component. As a distributed message queue, the Kafka component caches and transfers the data to ensure reliable data transmission;

[0059] The ELK component consumes data from the Kafka component, parses, transforms, and stores the data, and provides efficient full-text retrieval and data analysis functions for subsequent data query and analysis.

[0060] Step S4, data display and analysis

[0061] Through the data visualization tool provided by the ELK component, various dashboards and reports are custom-created according to user needs to visually display the data.

[0062] In the said step S3, the ELK component includes three parts: Logstash, Elasticsearch, and Kibana; Logstash consumes data from the Kafka component, parses and transforms the data, and then stores the processed data in Elasticsearch. Elasticsearch provides efficient full-text retrieval and data analysis functions for subsequent data query and analysis.

[0063] In the said step S4, Kibana, as a data visualization tool, obtains the stored data by connecting to Elasticsearch, and custom-creates various dashboards and reports according to user needs to visually display the data;

[0064] Users can customize to view the real-time status of the device, historical data trends, and data statistical analysis result information through the dashboard of Kibana.

[0065] The data collection and message processing device based on Internet of Things devices includes a memory and a processor; the memory is used to store computer programs, and the processor is used to implement the above method steps when executing the computer programs.

[0066] A computer program is stored on the readable storage medium, and when the computer program is executed by the processor, the above method steps are implemented.

[0067] Compared with the prior art, the data collection and message processing system and method based on Internet of Things devices have the following characteristics:

[0068] (1) The communication between Internet of Things devices and the Mosquitto component is realized by using the Message Queuing Telemetry Transport (MQTT) protocol, which is lightweight and has strong real-time performance, and can meet the real-time requirements of data transmission for Internet of Things devices;

[0069] (2) By issuing data collection and filtering rules from the server side, the flexible control of data collection for Internet of Things devices is realized, unnecessary data transmission is reduced, and the efficiency and pertinence of data collection are improved;

[0070] (3) The Kafka component is used as a distributed message queue to ensure the reliable transmission and efficient processing of data, and can handle the massive data generated by a large number of Internet of Things devices;

[0071] (4) With the help of the ELK component, the storage, query and visual display of data are realized, providing users with an intuitive and convenient Internet of Things data monitoring and analysis platform, which helps users to timely understand the device status and data change trends, and provides strong support for the decision-making and management of Internet of Things applications.

[0072] The above-described embodiments are only one of the specific implementation manners of the present invention, and the common changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the protection scope of the present invention.< / channelgroup> < / channelname> < / channelid> < / serviceid> < / time>

Claims

1. A data acquisition and message processing system based on Internet of Things devices, characterized in that: It includes Internet of Things (IoT) devices, a server, a Mosquitto component, a Kafka component, and an ELK component; The IoT devices are the data sources, responsible for establishing a connection with Mosquitto through the Message Queuing Telemetry Transport (MQTT) protocol and sending the collected data to the Mosquitto component; The server is connected to the Mosquitto component, responsible for obtaining the online status of the IoT devices and sending the data collection and filtering rules to the IoT devices through the Mosquitto component; The Mosquitto component serves as an MQTT message broker server, responsible for receiving the messages sent by the IoT devices, providing lightweight real-time communication, and sending the collected data to the Kafka component; The Kafka component, as a distributed message queue system, is responsible for receiving the data sent by the Mosquitto component and storing and delivering the data to subsequent processing components; The ELK component includes three parts: Logstash, Elasticsearch, and Kibana. Among them, Logstash is responsible for collecting data from the Kafka component and transmitting it to Elasticsearch for storage, and Kibana is responsible for various displays of the data through the dashboard; 2. The data acquisition and message processing system based on the Internet of Things device according to claim 1, characterized in that: The server sends the data collection and filtering rules to the IoT devices in the form of regular expressions through the Mosquitto component, and the IoT devices collect data according to the sent data collection and filtering rules.

3. A data acquisition and message processing method based on Internet of Things devices, characterized in that: It includes the following steps: Step S1, IoT device data collection The IoT devices collect relevant data according to the custom-set data collection period and send the data to the Mosquitto component through the MQTT protocol. The data content includes, but is not limited to, device status information and sensor data; Step S2, Data collection and filtering rule distribution The server sends the data collection and filtering rules to the IoT devices through the Mosquitto component. After receiving the rules, the devices filter the collected data according to the rules and only send the data that meets the rules to the Mosquitto component, thereby reducing the data transmission volume and improving the pertinence and effectiveness of data collection; Step S3, Data transmission and storage The Mosquitto component sends the data that meets the data collection and filtering rules to the Kafka component. The Kafka component, as a distributed message queue, caches and delivers the data to ensure reliable data transmission; The ELK component consumes data from the Kafka component, parses, transforms, and stores the data, and provides full-text search and data analysis functions for subsequent data query and analysis. Step S4, Data display and analysis Through the data visualization tool provided by the ELK component, dashboards and reports are custom-created according to user needs to visually display the data.

4. The data acquisition and message processing method based on Internet of Things devices according to claim 3, characterized in that: In the step S1, after the Mosquitto component receives the data from the Internet of Things device, it preliminarily processes the data according to the custom rules, including data format verification and data integrity check, to ensure the accuracy and reliability of the data.

5. The data acquisition and message processing method based on the Internet of Things device according to claim 3, characterized in that: In the step S2, the server sends the data collection filtering rules to the Internet of Things device in the form of regular expressions through the Mosquitto component, and the Internet of Things device collects data according to the sent data collection filtering rules.

6. The data acquisition and message processing method based on the Internet of Things device according to claim 5, characterized in that: In the step S2, the data collection filtering rules include extracting the timestamp, service ID, channel ID, channel name, and description information from the log lines in the custom specified format.

7. The data acquisition and message processing method based on an Internet of Things device according to claim 3, characterized in that: In the step S3, the ELK component includes three parts: Logstash, Elasticsearch, and Kibana; Logstash consumes data from the Kafka component, parses and transforms the data, and then stores the processed data in Elasticsearch. Elasticsearch provides full-text search and data analysis functions for subsequent querying and analysis of the data.

8. The data acquisition and message processing method based on the Internet of Things device according to claim 7, characterized in that: In the step S4, Kibana, as a data visualization tool, obtains the stored data by connecting to Elasticsearch, and customizes the creation of dashboards and reports according to user needs to visually display the data; The user can view the real-time status of the device, the historical data trend, and the data statistical analysis result information through the dashboard of Kibana.

9. A data acquisition and message processing device based on Internet of Things devices, characterized in that: It includes a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the method steps described in any one of claims 3 to 8 when executing the computer program.

10. A readable storage medium, characterized in that: A computer program is stored on the readable storage medium, and when the computer program is executed by the processor, it implements the method steps described in any one of claims 3 to 8.

Images