Internet of Things equipment data acquisition and monitoring system and method

Abstract

The invention relates to an Internet of Things equipment data acquisition and monitoring system and method. A user can efficiently and stably access a network management system to inquire equipment data. The method comprises the following steps: a network device generates monitoring data and sends the monitoring data to an IoTDB database through a network protocol; the IoTDB database packages TAG label data and data types in the monitoring data, and sends the TAG label data and the data types to a RabbitMQ message queue in a specific theme; after monitoring that the RabbitMQ message queue has a specific theme, the network management system extracts TAG label data and a data type, analyzes the TAG label data, and inquires in a preset interface-data class mapping table according to an analysis result and the data type to obtain a corresponding interface list; performing multi-stage interface screening on the corresponding interface list to obtain a final interface list; the network management system queries the IoTDB database through the final interface list to obtain monitoring data and updates the monitoring data to the Redis cache; and the user sends a query request, and the network management system obtains the monitoring data from the Redis cache and returns the monitoring data to the user.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of communication, in particular to an Internet of Things device data acquisition and monitoring system and method. BACKGROUND

[0002] Currently, the mainstream interaction scheme between network devices and network device management system servers (network management systems) is usually based on the MQTT protocol: device data is transmitted via a third-party MQTT Broker, and the network management system acts as a client to interact with the data. When a user queries device information, there are usually two implementation methods: Polling mode: the front end requests the back end system, and the back end system initiates a real-time query to the target device directly through a network protocol (such as SNMP, MODBUS, etc.), and returns the data to the front end after obtaining the data.

[0003] Report storage mode: the device actively reports data to the network management system through a network protocol, and the system persistently stores the data (such as in a database). When a user queries, the front end requests the back end system, and the back end retrieves data from the database and returns it.

[0004] To improve performance, some schemes introduce a caching mechanism: the user request first queries the cache, and if it hits, it is directly returned; if it does not hit, it obtains data from the device or the database according to the above mode and updates the cache.

[0005] However, the existing scheme in the polling mode, high concurrency requests will directly impact the device; in the report storage mode, high concurrency queries will impact the database. Both of them can cause the system to be unstable. The static caching mechanism introduced is difficult to dynamically identify and cache data frequently accessed by users, and is easy to cause waste of memory resources. And it lacks the ability to flexibly configure high-frequency data access strategies, and cannot optimize the cache content according to the actual demand.

[0006] Through the search of the prior art documents, it is found that the patent with the application number 201811011878.X and the patent name of photovoltaic power station monitoring system based on MQTT protocol is discovered. The patent includes a photovoltaic power station intelligent gateway, a photovoltaic power station monitoring cloud platform, a mobile phone App and a computer Web terminal. The photovoltaic power station intelligent gateway communicates with the photovoltaic power station equipment on one hand, and collects photovoltaic power station equipment data and monitors the running state of the equipment in real time. On the other hand, the photovoltaic power station intelligent gateway communicates with the cloud server through the Internet, and publishes the equipment data, state information and fault information to the cloud server according to the self-defined MQTT communication protocol. The photovoltaic power station monitoring system cloud platform is composed of an MQTT message broker server, a background management system, a MySQL database server and a Redis server. The mobile phone App and the computer Web terminal are user and system interaction tools. The photovoltaic power station intelligent gateway and the computer Web terminal or the mobile phone App are used as clients to directly use the MQTT protocol to connect with the message broker, and complete the message subscription and publishing functions. However, the patent lacks the intelligent cache update realized by the RabbitMQ message queue decoupling, the IoTDB trigger mechanism and the multi-level interface screening, resulting in the problems of lacking high efficiency and high stability in querying the equipment data. SUMMARY

[0007] Therefore, it is necessary to provide an Internet of Things equipment data acquisition and monitoring method to realize efficient and stable user access to the network management system to query the equipment data.

[0008] To solve the above technical problems, the present application provides the following technical solutions. On the one hand, the present application provides an Internet of Things equipment data acquisition and monitoring system, which comprises a network device and a network management system. The network management system comprises a plurality of middlewares, which are an IoTDB database, a RabbitMQ message queue and a Redis cache, and the IoTDB database is used as an MQTT server. The IoTDB database is used for receiving the monitoring data generated by the network device, packing part of the data in the monitoring data to obtain packed data, and sending the packed data to the RabbitMQ message queue. The RabbitMQ message queue is used for temporarily blocking the packed data. The network management system is used for monitoring the RabbitMQ message queue, extracting and analyzing the packed data from the RabbitMQ message queue, and the network management system has a preset interface-data class mapping table. The Redis cache is used for storing the monitoring data obtained by querying the IoTDB database by the network management system.

[0009] In another aspect, the present application also provides an Internet of Things device data acquisition and monitoring method, comprising: S1: the network device generates monitoring data and sends it to the IoTDB database through a network protocol; S2: the IoTDB database packs the TAG label data and data type in the monitoring data and sends them to the RabbitMQ message queue in a specific topic; S3: after the network management system detects that the RabbitMQ message queue has a specific topic, it extracts the TAG label data and data type, analyzes the TAG label data to obtain the device domain group and device information, and queries the corresponding interface list in the preset interface-data class mapping table according to the device domain group, device information, and data type; S4: multi-level interface screening is performed on the corresponding interface list to obtain a final interface list; S5: the network management system queries the IoTDB database through the final interface list, obtains the monitoring data, and updates it to the Redis cache; S6: the user issues a query request, and the network management system obtains the monitoring data from the Redis cache and returns it to the user.

[0010] Preferably, the network protocol includes the MQTT protocol.

[0011] Preferably, after step S1, the method further comprises: The IoTDB database identifies and distinguishes the types of the monitoring data of each table.

[0012] Preferably, before step S2, the method further comprises: presetting a trigger in the IoTDB database, the IoTDB database receiving the monitoring data triggering the trigger to execute step S2, and the trigger being an After Insert trigger.

[0013] Preferably, the network management system has a separate thread that cyclically monitors whether the RabbitMQ message queue has a specific topic.

[0014] Preferably, the multi-level interface screening comprises: S401: the network management system obtains the high-frequency interface list in the Redis cache, compares the corresponding interface list with the high-frequency interface list, and obtains a high-frequency interface result list; S402: the network management system compares the high-frequency interface result list with a to-be-optimized interface list to obtain a final interface list, the to-be-optimized interface list being an interface list that needs performance optimization and being preselected by the user in the interface-data class mapping table and stored in the Redis cache.

[0015] Preferably, before step S401, the method further comprises: The network management system records the commonly used interfaces of user requests and stores them in the Redis cache to obtain a high-frequency interface list.

[0016] Preferably, the network management system further comprises a MySQL database, and the step S5 further comprises: The network management system queries the IoTDB database through the final interface list, obtains monitoring data, and updates the MySQL database.

[0017] Preferably, the step S6 further comprises: The user issues a query request, and the network management system obtains monitoring data from the MySQL database and returns it to the user.

[0018] Compared with the prior art, the present application has the following advantages: The present application provides an Internet of Things device data acquisition and monitoring system and method, which realizes low-delay writing and efficient storage of device data through the MQTT protocol and the IoTDB database, and automatically pushes data changes to the RabbitMQ message queue by using the built-in trigger in the IoTDB database, avoiding resource consumption of polling queries, decoupling through the RabbitMQ message queue, improving system throughput, and ensuring high-concurrency stability; at the same time, multi-level interface screening and active cache preheating form an intelligent cache update strategy, which reduces server load pressure and effectively guarantees the high availability and high performance of the system in the case of high-concurrency requests to the backend, reducing the possibility of cache breakdown in a high-concurrency environment; and finally significantly improving the performance, flexibility and user experience of the Internet of Things management system. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 It is a flowchart of the Internet of Things device data acquisition and monitoring method in one embodiment; Figure 2 It is a specific flowchart of the Internet of Things device data acquisition and monitoring method in one embodiment. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0021] Embodiment 1 The present embodiment proposes an Internet of Things device data acquisition and monitoring system, which comprises a network device and a network management system. The network management system comprises a plurality of middlewares, which are an IoTDB database, a RabbitMQ message queue, and a Redis cache, and the IoTDB database simultaneously serves as an MQTT server. The IoTDB database is used to receive monitoring data generated by the network device, package a portion of the monitoring data to obtain packaged data, and send it to the RabbitMQ message queue. The RabbitMQ message queue is used to temporarily block packaged data. The network management system is used to monitor the RabbitMQ message queue, extract and analyze packaged data from the RabbitMQ message queue, and has a preset interface-data class mapping table. The Redis cache is used to store monitoring data obtained by the network management system after querying the IoTDB database.

[0022] Specifically, this embodiment mainly consists of two parts: network devices and a network device management system server (network management system). The network management system is a carrier that can interact with devices via network protocols. Within this system, network devices are grouped and divided into domain groups. These domain groups are the core groups for assigning different user permissions, meaning that users can obtain different permissions by being assigned to different domain groups, achieving the effect of isolated management of device domain groups. The network management system also contains several middleware components, including: Redis caching middleware, RabbitMQ message queue middleware, and IoTDB database (which also acts as an MQTT server).

[0023] Example 2 like Figure 1 As shown in the figure, this embodiment proposes a method for data acquisition and monitoring of Internet of Things (IoT) devices, the method comprising: S1: Network devices generate monitoring data and send it to the IoTDB database via network protocols; The specific implementation of this step is as follows: After the network device generates monitoring data, it sends the data to the IoTDB database via message subscription through the MQTT network protocol to the MQTT Broker built into the IoTDB database, and inserts and saves the data in the database.

[0024] S2: The IoTDB database packages the TAG tag data and data type from the monitoring data and sends them to the RabbitMQ message queue with a specific topic; The specific implementation of this step is as follows: After detecting data changes in the IoTDB database, the Tag column of the data is packaged as a whole and the device data type and sent to the RabbitMQ message queue with a specific topic; the RabbitMQ message queue is run with IoTDB as the producer and the network management system as the consumer. The packaged data will be blocked in the RabbitMQ message queue, waiting for the network management system to consume it.

[0025] S3: After the network management system detects that there is a specific topic in the RabbitMQ message queue, it extracts the TAG tag data and data type, analyzes the TAG tag data to obtain the device domain group and device information, and queries the preset interface-data class mapping table to obtain the corresponding interface list based on the device domain group, device information and data type. The specific implementation of this step is as follows: When the network management system detects that a specific queue contains data, it will retrieve the data and analyze the overall data based on the TAG label of the current data. The overall TAG data can be parsed into the corresponding device domain group and device information, and the device data type will also be retrieved. The network management system integrates an interface-data class mapping table, that is, a list of interfaces corresponding to a certain function. When device data is retrieved from the RabbitMQ message queue, it is analyzed to determine which type of device data the data is, and the corresponding interface list is obtained through the interface-data class mapping integrated by the system.

[0026] S4: Perform multi-level interface filtering on the corresponding interface list to obtain the final interface list; S5: The network management system queries the IoTDB database through the final interface list to obtain monitoring data and updates it to the Redis cache; The specific implementation of this step is as follows: After obtaining the final list of interfaces, the system will perform an up-to-date query on the interface data involved in these lists and update the Redis cache and the cache time. At this time, when the user sends a query request again on the front end, the data result will be retrieved from the cache first and returned for display, instead of continuously sending it to the database or device for retrieval.

[0027] S6: When a user sends a query request, the network management system retrieves monitoring data from the Redis cache and returns it to the user.

[0028] Example 3 This embodiment further supplements the description of the IoT device data acquisition and monitoring method proposed in Embodiment 2.

[0029] The network protocol includes the MQTT protocol.

[0030] After step S1, the following also includes: The IoTDB database identifies and distinguishes the types of monitoring data in each table.

[0031] The specific implementation of this step is as follows: Each table in the IoTDB database will be identified to distinguish the data type, such as the external attribute type of the device, the hardware resource usage type, etc.

[0032] Before step S2, the method further includes: setting a trigger in the IoTDB database, receiving monitoring data, and triggering the trigger to execute step S2, wherein the trigger is an After Insert trigger.

[0033] The specific implementation of this step is as follows: A trigger is pre-registered in the IoTDB database and is implanted into the database as a plugin. The trigger is triggered after Insert, meaning that it will execute an operation after data insertion changes: After the trigger detects data changes in the IoTDB database, it will package the Tag column of the data as a whole and the device data type, and send it to the RabbitMQ message queue with a specific topic.

[0034] The network management system has a separate thread that runs in a loop to monitor whether a specific topic exists in the RabbitMQ message queue.

[0035] The multi-level interface filtering includes: S401: The network management system obtains the list of high-frequency interfaces in the Redis cache, compares the corresponding interface list with the list of high-frequency interfaces, and obtains the list of high-frequency interface results. S402: The network management system compares the list of high-frequency interface results with the list of interfaces to be optimized to obtain the final list of interfaces. The list of interfaces to be optimized is a list of interfaces that need performance optimization, which are pre-selected by the user in the interface-data class mapping table and stored in the Redis cache.

[0036] The specific implementation of this step is as follows: the high-frequency interface list and the corresponding interface list are compared to obtain the high-frequency interface result list.

[0037] The interface-data class mapping table returns data that has been defined by the network management system as interface data of a certain device data class. Users can also select which interfaces are frequently used and require performance optimization, creating a list of interfaces to be optimized. After users select and configure the interfaces requiring performance optimization, the network management system will, under the user's permissions, create mappings for certain data types of interfaces within certain domain groups and save them to the Redis cache, setting them to be permanently stored.

[0038] At this point, after the user has made their interface selection, the list of interfaces that the user selected and that require performance optimization is obtained, and compared with the list of high-frequency interfaces to obtain the final interface list.

[0039] Before step S401, the following are also included: The network management system records frequently requested interfaces by users and stores them in a Redis cache, thus obtaining a list of high-frequency interfaces.

[0040] The specific implementation of this step is as follows: When the network management system receives a user's front-end request, it records the user's frequently used interfaces and saves them to a Redis cache. This data is assigned a corresponding TTL (Time-To-Live), which is the data expiration time. If the interface is called multiple times within a certain period, the TTL will be refreshed. Simultaneously, the system records the number of times the interface is called; that is, each interface access increments the counter and stores it in the Redis cache. Due to the TTL expiration time, the final list of interfaces obtainable from the Redis cache can be considered a high-frequency interface list. This system function is called high-frequency interface recording.

[0041] The network management system also includes a MySQL database, and step S5 also includes: The network management system queries the IoTDB database through the final interface list to obtain monitoring data and updates it to the MySQL database.

[0042] Step S6 also includes: When a user sends a query request, the network management system retrieves monitoring data from the MySQL database and returns it to the user.

[0043] The specific implementation of this step is as follows: The network management system can either send instruction network packets to the device through the user access interface via a specific API to obtain the corresponding data from the device, or it can store the data in a MySQL database after the above steps and query the required information through the database interface.

[0044] The process includes specific multi-level interface filtering, such as... Figure 2 As shown, this embodiment uses a smart water pumping station monitoring system as an example to illustrate... Figure 2 The equipment type is a distributed pump unit. The pump provides data from temperature, pressure, and flow sensors. It is divided into two groups of pump units: Group A and Group B. First, the water pump temperature in domain group B is abnormal. The temperature data is reported to the IoTDB database via the MQTT protocol for storage. After the data is inserted into the IoTDB database, the After Insert trigger is triggered, which packages the relevant TAG information of the record and the device data type into a message and sends it to the RabbitMQ message queue. The water system's listening thread retrieves messages from RabbitMQ, parses the TAGs to determine that the message comes from water pump B in domain group B and that the data type is physical sensor type. The network management system queries the interface-data class mapping table to obtain the corresponding interface list. Next, by recording the high-frequency interface list of previous users' operations and storing it in the Redis cache, a list of high-frequency interface results related to temperature is obtained by comparison. Next, the final list of interfaces is obtained by comparing the user-pre-set list of interfaces to be optimized with the list of high-frequency interfaces. The water system queries the IoTDB database for the latest physical sensor type data based on the final interface list and returns it to the backend to update the cache of multiple Redis interfaces; When a user requests the temperature report for domain group B again from the front end, the water system first checks whether the Redis cache is hit. If it is hit, the cached data is returned directly to the front end, and finally the user's page successfully displays the water system's temperature data.

Claims

1. A data acquisition and monitoring system for Internet of Things (IoT) devices, characterized in that, include: Network equipment and network management systems; The network management system includes multiple middleware components, namely IoTDB database, RabbitMQ message queue, and Redis cache. The IoTDB database also serves as an MQTT server. The IoTDB database is used to receive monitoring data generated by the network device, package a portion of the monitoring data to obtain packaged data, and send it to the RabbitMQ message queue. The RabbitMQ message queue is used to temporarily block packaged data. The network management system is used to monitor the RabbitMQ message queue, extract and analyze packaged data from the RabbitMQ message queue, and has a preset interface-data class mapping table. The Redis cache is used to store monitoring data obtained by the network management system after querying the IoTDB database.

2. A method for data acquisition and monitoring of Internet of Things (IoT) devices, applied to the IoT device data acquisition and monitoring system as described in claim 1, characterized in that, include: S1: Network devices generate monitoring data and send it to the IoTDB database via network protocols; S2: The IoTDB database packages the TAG tag data and data type from the monitoring data and sends them to the RabbitMQ message queue with a specific topic; S3: After the network management system detects that there is a specific topic in the RabbitMQ message queue, it extracts the TAG tag data and data type, analyzes the TAG tag data to obtain the device domain group and device information, and queries the preset interface-data class mapping table to obtain the corresponding interface list based on the device domain group, device information and data type. S4: Perform multi-level interface filtering on the corresponding interface list to obtain the final interface list; S5: The network management system queries the IoTDB database through the final interface list to obtain monitoring data and updates it to the Redis cache; S6: When a user sends a query request, the network management system retrieves monitoring data from the Redis cache and returns it to the user.

3. The method for data acquisition and monitoring of IoT devices according to claim 2, characterized in that, The network protocol includes the MQTT protocol.

4. The method for data acquisition and monitoring of IoT devices according to claim 2, characterized in that, After step S1, the following also includes: The IoTDB database identifies and distinguishes the types of monitoring data in each table.

5. The method for data acquisition and monitoring of IoT devices according to claim 2, characterized in that, Before step S2, the method further includes: presetting a trigger in the IoTDB database; the IoTDB database receives monitoring data and triggers the trigger to execute step S2; the trigger is an After Insert trigger.

6. The method for data acquisition and monitoring of IoT devices according to claim 2, characterized in that, The network management system has a separate thread that runs in a loop to monitor whether a specific topic exists in the RabbitMQ message queue.

7. The method for data acquisition and monitoring of IoT devices according to claim 2, characterized in that, The multi-level interface filtering includes: S401: The network management system obtains the list of high-frequency interfaces in the Redis cache, compares the corresponding interface list with the list of high-frequency interfaces, and obtains the list of high-frequency interface results. S402: The network management system compares the list of high-frequency interface results with the list of interfaces to be optimized to obtain the final list of interfaces. The list of interfaces to be optimized is a list of interfaces that need performance optimization, which are pre-selected by the user in the interface-data class mapping table and stored in the Redis cache.

8. The method for data acquisition and monitoring of IoT devices according to claim 7, characterized in that, Before step S401, the following are also included: The network management system records frequently requested interfaces by users and stores them in a Redis cache, thus obtaining a list of high-frequency interfaces.

9. The method for data acquisition and monitoring of IoT devices according to claim 2, characterized in that, The network management system also includes a MySQL database, and step S5 also includes: The network management system queries the IoTDB database through the final interface list to obtain monitoring data and updates it to the MySQL database.

10. The method for data acquisition and monitoring of IoT devices according to claim 9, characterized in that, Step S6 also includes: When a user sends a query request, the network management system retrieves monitoring data from the MySQL database and returns it to the user.

Images