An urban underground pipe network data monitoring system

Abstract

The application discloses a kind of urban underground pipe network data monitoring systems, belong to urban underground pipe network monitoring technical field, can solve the problem that existing technology cannot be classified and stored to urban underground pipe network data, and it is difficult to deal with high concurrency condition, leading to the problem of low operating efficiency of monitoring system.The system comprises: a plurality of internet of things equipment, for collecting the real-time data of multiple types of underground pipe network;Internet of Things Gateway is connected with a plurality of internet of things equipment, for setting the type label of each real-time data according to the device type of each internet of things equipment;Real-time database is used to store the real-time data of underground pipe network;Control module is connected with internet of things gateway, real-time database and a plurality of internet of things equipment respectively, for controlling a plurality of internet of things equipment and internet of things gateway work, and according to the type label of each real-time data, each real-time data is stored in the field of corresponding type in real-time database.The application is used to monitor urban underground pipe network.

Description

Technical Field

[0001] This invention relates to a data monitoring system for urban underground pipe networks, belonging to the field of urban underground pipe network monitoring technology. Background Technology

[0002] Currently, urban flooding is a frequent occurrence, severely impacting people's lives. Therefore, it is necessary to manage urban underground pipe networks. In the management of urban underground pipe networks, monitoring data such as temperature, humidity, oxygen concentration, and water level is extremely important. Real-time monitoring of these data allows for the timely detection of sudden problems in the underground pipe network, enabling rapid response and preventing escalation. Furthermore, real-time monitoring and proactive protective measures ensure the safety of maintenance personnel during operation.

[0003] However, existing IoT technologies, when collecting data from urban underground pipe networks, cannot categorize and store the collected real-time data according to types such as temperature, humidity, oxygen concentration, and liquid level into real-time and historical databases. This results in overly centralized data storage, leading to data loading delays when the monitoring system accesses specific types of real-time and historical data, severely impacting the system's operational efficiency and user experience. Furthermore, with the rapid development of sensor technology, the types and numbers of sensor devices used for urban underground pipe network environmental management are increasing, frequently resulting in high concurrency during data transmission. Existing IoT technologies are weak in handling high concurrency, easily causing data loss and delays, affecting data transmission stability and efficiency. Summary of the Invention

[0004] This invention provides an urban underground pipe network data monitoring system that can solve the problems of existing technologies being unable to classify and store urban underground pipe network data, and being unable to cope with high concurrency during data transmission, resulting in low operating efficiency of the monitoring system.

[0005] This invention provides a data monitoring system for urban underground pipeline networks, the system comprising:

[0006] Multiple IoT devices are used to collect real-time data from various types of underground pipe networks;

[0007] An IoT gateway, which connects to multiple IoT devices, is used to set the type label for each real-time data based on the device type of each IoT device;

[0008] A real-time database is used to store real-time data of underground pipe networks;

[0009] The control module is connected to the IoT gateway, the real-time database, and multiple IoT devices, respectively, and is used to control the operation of the multiple IoT devices and the IoT gateway, and to store each real-time data into the corresponding type field in the real-time database according to the type label of each real-time data.

[0010] Optionally, the system further includes:

[0011] The MQTT transmission module is connected to both the IoT gateway and the control module, and is used to transmit the real-time data transmitted by the IoT gateway to the control module in the form of messages.

[0012] Optionally, the control module is used to convert the received real-time data in message form into real-time data in JOSN format, and transmit the real-time data in JOSN format to the real-time database.

[0013] Optionally, the connection between the IoT gateway and the IoT device is based on the Modbus-TCP protocol.

[0014] Optionally, the MQTT transmission module uses a multi-threaded approach for data transmission.

[0015] Optionally, the system further includes:

[0016] Multiple historical databases correspond to multiple types of real-time data, used to store different types of real-time data respectively.

[0017] Optionally, the system further includes:

[0018] A building module is used to establish the real-time database and multiple historical databases based on the device types of multiple IoT devices.

[0019] Optionally, the multiple IoT devices include temperature sensors, humidity sensors, oxygen concentration sensors, and liquid level sensors;

[0020] Accordingly, multiple historical databases, including a temperature historical database, a humidity historical database, an oxygen concentration historical database, and a liquid level historical database, are used to store real-time data collected by temperature sensors, humidity sensors, oxygen concentration sensors, and liquid level sensors, respectively.

[0021] Optionally, the system further includes:

[0022] The visualization module, connected to the real-time database and the historical database, is used to visualize the real-time data and the historical data.

[0023] Optionally, the control module is further configured to associate multiple data processing methods with multiple data topics of the MQTT transmission module, determine the corresponding data processing method according to the topic to which the real-time data belongs, and call the corresponding data processing method to process the real-time data.

[0024] The beneficial effects that this invention can produce include:

[0025] This invention utilizes an IoT gateway to set a type label for each real-time data based on the device type of each IoT device. Then, the control module stores each real-time data into the corresponding type field in the real-time database according to the type label. This enables efficient and accurate classification and storage of real-time data, facilitating the monitoring system to quickly access various types of data, improving data transmission efficiency, and ensuring the stable operation of the monitoring system.

[0026] This invention utilizes an IoT gateway to convert real-time data collected by different IoT devices using different protocols into real-time data using a unified protocol. This facilitates unified data management, allows for the access of more types and quantities of sensors in urban underground pipe networks, and enables real-time acquisition of data from multiple types and quantities of sensors in urban underground pipe networks, effectively solving the problem of limited sensor quantity.

[0027] This invention uses MQTT for data transmission, which can efficiently and conveniently achieve stable and reliable message transmission between IoT devices. Its multi-threaded data processing method can effectively alleviate the high concurrency problem when multiple types and numbers of sensors upload data, which is conducive to improving data transmission efficiency, making the entire data transmission and parsing process more fault-tolerant, and ensuring the stable operation of the system. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the structure of an urban underground pipeline network data monitoring system provided in an embodiment of the present invention;

[0029] Figure 2 A flowchart illustrating the configuration and debugging process of an IoT gateway provided in an embodiment of the present invention;

[0030] Figure 3 A schematic diagram of the Modbus-TCP protocol provided in an embodiment of the present invention;

[0031] Figure 4 The working principle of the MQTT transmission module provided in the embodiments of the present invention;

[0032] Figure 5 This is a schematic diagram of the overall architecture of the data transmission method provided in an embodiment of the present invention. Detailed Implementation

[0033] The present invention will now be described in detail with reference to the embodiments, but the present invention is not limited to these embodiments.

[0034] This invention provides a data monitoring system for urban underground pipe networks, such as... Figure 1 As shown, the system includes:

[0035] Multiple IoT devices are used to collect real-time data from various types of underground pipe networks;

[0036] An IoT gateway, which connects to multiple IoT devices, is used to set the type label for each real-time data based on the device type of each IoT device;

[0037] A real-time database is used to store real-time data of underground pipe networks;

[0038] The control module connects to the IoT gateway, the real-time database, and multiple IoT devices. It controls the operation of the multiple IoT devices and the IoT gateway, and stores each real-time data into the corresponding type field in the real-time database according to the type label of each real-time data.

[0039] Specifically, the multiple IoT devices may include multiple temperature sensors, multiple humidity sensors, multiple oxygen concentration sensors, and multiple liquid level sensors; correspondingly, the real-time data may include multiple types of real-time data such as temperature data, humidity data, oxygen concentration data, and liquid level data. These sensors can be installed at different locations within the underground pipe network to collect real-time data from multiple locations and types of underground pipe networks.

[0040] Existing monitoring methods, after collecting real-time data such as temperature, humidity, oxygen concentration, and liquid level from urban underground pipe networks, typically transmit the data directly to a real-time database based on existing data transmission protocols. On one hand, because current technology does not categorize and store different types of data, modules need to determine the data type through searches before they can retrieve specific data, leading to data loading delays and severely impacting the monitoring system's operational efficiency and user experience. On the other hand, with the increasing number and types of sensors, simultaneous data collection and uploading by multiple sensors can easily cause high concurrency issues in the monitoring system, affecting data transmission efficiency.

[0041] To address the aforementioned issues, this embodiment utilizes an IoT gateway to uniformly receive real-time data collected by all IoT devices. By establishing communication connections between the IoT gateway and all IoT devices, various types of real-time data can be retrieved using the IoT gateway. This achieves two main benefits: firstly, it converts real-time data collected by different IoT devices using different protocols into real-time data using a unified protocol, facilitating unified data management; secondly, the IoT gateway can accommodate more types and quantities of sensors in urban underground pipe networks, enabling real-time data collection from multiple types and quantities of sensors, effectively solving the problem of limited sensor quantity; and thirdly, by defining data types for multiple points within the IoT gateway and associating each point with a corresponding type of IoT device, type tags can be set for real-time data during retrieval. This allows for categorized storage of real-time data based on its type tags, improving data retrieval efficiency and the operational efficiency of the monitoring system.

[0042] Specifically, in this embodiment, the connection between the IoT gateway and the IoT device is based on the Modbus-TCP protocol.

[0043] Specifically, this embodiment selects an IoT gateway that can establish communication connections with various types of sensors via the Modbus-TCP protocol. Using this IoT gateway, multiple types and quantities of sensors can be managed in a unified manner, and real-time data from the sensors can be retrieved by periodically sending requests to the sensors.

[0044] Specifically, the configuration and debugging process for the IoT gateway in this embodiment is as follows: Figure 2 As shown, the process includes the following steps:

[0045] (1) Connecting the IoT gateway to the local area network: Set the IP address of the IoT gateway to a fixed local area network IP, set the subnet mask, and set the IoT gateway as the default gateway of the local area network. Then power on the IoT gateway and connect it to the local area network switch via a network cable to connect the IoT gateway to the local area network.

[0046] (2) Loading the sensor connection configuration file: Configure the sensor connection information through the IoT gateway connection tool, and specify the variable names and data types for the data source. After the configuration is completed, the sensor connection configuration file is obtained. Load this file into the IoT gateway. At this time, the IoT gateway has the ability to establish communication connections with various types of sensors in the urban underground pipe network, and can periodically send requests to the sensors to retrieve real-time data from the sensors.

[0047] (3) Testing and Verification: After completing the above two steps, this embodiment performs a series of tests and verifications on the IoT gateway, including functional testing, performance testing, and stability testing, to ensure that the IoT gateway can work properly. For example, the IoT gateway can be connected using an IoT gateway connection tool to view the data transmission between the sensor and the IoT gateway. If problems are found, the aforementioned steps can be modified and adjusted.

[0048] Specifically, Modbus TCP is a protocol that encapsulates the Modbus protocol within the TCP / IP protocol and transmits data over Ethernet. Modbus TCP utilizes the reliability and flow control mechanisms of TCP to ensure reliable data transmission. At the same time, it retains the simplicity and ease of use of the Modbus protocol, making it readily applicable for communication between various industrial automation devices.

[0049] Specifically, a schematic diagram of the Modbus-TCP protocol is shown below. Figure 3 As shown, the Modbus-TCP protocol consists of two parts: the header (MBAP) and the frame structure (PDU).

[0050] The message header is 7 bytes long, and its structure is shown in Table 1.

[0051] Table 1. Structure of the Message Header

[0052] Transaction identifier Protocol Identifier length Unit identifier 2 bytes 2 bytes 2 bytes 1 byte

[0053] The frame structure consists of a function code and data. The function code is 1 byte, and the data length is determined by the specific function. As shown in Table 2, the meaning of the function code varies depending on the object.

[0054] Table 2 Meaning of Frame Structure Function Codes

[0055] Function code meaning 0x01 Read coil 0x02 Read discrete input 0x03 Read holding register 0x04 Read input register 0x05 Write a single coil 0x06 Write a single holding register 0x0F Write multiple coils 0x10 Write multiple holding registers

[0056] To address the high concurrency issue, this embodiment introduces the Message Queuing Telemetry Transport (MQTT) protocol. By configuring connection information in the IoT gateway and control module, MQTT establishes a communication connection with the IoT gateway and control module. In this way, MQTT can transmit real-time data from the IoT gateway to the control module, and the control module can then classify and store the real-time data in a real-time database.

[0057] Specifically, MQTT is a lightweight message transmission protocol based on publish and subscribe, suitable for IoT application scenarios with low bandwidth or unstable networks. Developers can use MQTT to efficiently achieve stable and reliable message transmission between IoT devices, and its multi-threaded data processing method can effectively alleviate the high concurrency problem when multiple types and numbers of sensors upload data, which is conducive to improving data transmission efficiency, making the entire data transmission and parsing process more fault-tolerant, and ensuring the stable operation of the system.

[0058] Specifically, the system may also include:

[0059] The MQTT transmission module connects to both the IoT gateway and the control module, and is used to transmit real-time data transmitted by the IoT gateway to the control module in the form of messages.

[0060] In this embodiment, the MQTT transmission module uses a multi-threaded data processing method for data transmission to alleviate the high concurrency problem of the system.

[0061] Specifically, the working principle of the MQTT transmission module is as follows: Figure 4 As shown in the diagram, this embodiment configures the MQTT transmission module connection information in the IoT gateway. The IoT gateway publishes topics through the MQTT transmission module, and the control module uses Java to develop its connection method with the MQTT transmission module and subscribes to the topics published by the IoT gateway through the MQTT transmission module. In this way, the IoT gateway can periodically pull real-time data from sensors, encapsulate it, and transmit it to the topics it publishes on the MQTT transmission module. Then, the MQTT transmission module pushes the real-time data under that topic in the form of messages to the control module that subscribed to that topic. The control module then classifies and stores the real-time data in the real-time database according to the type tags of the real-time data, realizing the entire process of efficient transmission and classified storage of real-time data.

[0062] Because the real-time data pushed by the MQTT transmission module to the control module is encapsulated message, the control module cannot directly obtain the type tag of each real-time data item, and therefore cannot classify and store the actual data. Therefore, after receiving the message pushed by the MQTT transmission module, the control module needs to parse and process the message information first.

[0063] Specifically, the control module is used to convert the real-time data in the form of received messages into real-time data in JOSN format, and transmit the real-time data in JOSN format to the real-time database.

[0064] The control module in this embodiment uses a message parsing program developed in Java.

[0065] Specifically, in this embodiment, the real-time data includes oxygen concentration data, temperature data, humidity data, high liquid level warning data, and low liquid level warning data. Oxygen concentration data, temperature data, and humidity data are environmental data, defined as HJ type data in this embodiment. This type of data is in string format and records specific temperature, humidity, and oxygen concentration values. High liquid level warning data and low liquid level warning data are remote control data, defined as YC type data in this embodiment. This type of data is in Boolean format and records two states: warning and no warning. For example, for high liquid level warning data, the warning state is 1, and the no warning state is 0. As mentioned above, the IoT gateway has set a type label for each real-time data when receiving the above real-time data. In this embodiment, the type label is the type information in the primary key value of the real-time data. Specifically, this embodiment marks the type of each real-time data by setting different type information in the primary key value of different types of real-time data.

[0066] It is worth noting that, in addition to type information, the primary key value of real-time data can also include information such as the data collection location.

[0067] Specifically, during data transmission, the IoT gateway periodically sends requests to the sensors to retrieve the aforementioned real-time data, encapsulates it, and then transmits it to the MQTT transmission module. The MQTT transmission module pushes the received real-time data to the control module in the form of messages. The control module uses a Java program to convert the message into HJ-type and YC-type datasets in JSON format.

[0068] Then, the control module traverses the HJ class dataset and the YC class dataset, and distinguishes them as temperature data, humidity data, oxygen concentration data, high liquid level warning data or low liquid level warning data by identifying the primary key value of the real-time data.

[0069] After the control module identifies the type of real-time data, it queries the real-time database based on the primary key value of the real-time data. If the real-time database already contains a record of that data, it updates the data information of that record based on the value of the real-time data, including the specific value of the real-time data and the collection time. If the real-time database does not contain a record of that data, it adds a new data record, including the primary key, key value, specific value, collection time, etc.

[0070] Specifically, the system may also include:

[0071] Multiple historical databases correspond to multiple types of real-time data, used to store different types of real-time data respectively.

[0072] It is worth noting that the real-time database in this embodiment only stores the received real-time data briefly. When it receives the real-time data of the most recent time period, it replaces the real-time data of the previous time period with the real-time data of the most recent time period to keep the real-time data in the real-time database as the latest real-time data. This helps to improve the operating efficiency of the real-time database.

[0073] Since real-time databases only retain the latest real-time data and cannot retain historical data, it is inconvenient to retrieve, analyze, and statistically process historical data. Therefore, this embodiment sets up a historical database, which stores all real-time data received from all time periods for a long period of time, thereby achieving the preservation of historical data.

[0074] Specifically, the multiple IoT devices include temperature sensors, humidity sensors, oxygen concentration sensors, and liquid level sensors;

[0075] Accordingly, multiple historical databases, including a temperature historical database, a humidity historical database, an oxygen concentration historical database, and a liquid level historical database, are used to store real-time data collected by temperature sensors, humidity sensors, oxygen concentration sensors, and liquid level sensors, respectively.

[0076] By setting up multiple types of historical databases, different types of real-time data can be classified and stored in different historical databases, which makes it easier for the control system to call, display, and statistically analyze specific types of historical data, thereby improving the efficiency of data transmission and processing.

[0077] Accordingly, after the control module identifies the type of real-time data, it needs to classify and store different types of real-time data into the corresponding historical database.

[0078] Specifically, the historical database stores information including the primary key, key value, specific numerical value, and collection time of the real-time data.

[0079] Specifically, the code for parsing the message using the Java programming language is as follows:

[0080]

[0081]

[0082]

[0083]

[0084] Based on the above method, the overall architecture of the data transmission method in this embodiment is as follows: Figure 5As shown, the architecture comprises a hardware device layer, a data parsing layer, and a data storage layer. The hardware device layer, also known as the physical layer, primarily includes IoT devices such as temperature sensors, humidity sensors, oxygen concentration sensors, and liquid level sensors, as well as IoT gateways. This layer is crucial to the entire architecture and is the first point of contact between the monitoring system and the physical world. The data parsing layer mainly includes an MQTT transmission module, which receives and parses data transmitted from the IoT gateway. The data storage layer includes a real-time database and a historical database. This layer categorizes the parsed data according to types such as temperature, humidity, oxygen concentration, and liquid level, storing it in the corresponding fields of the real-time database and the corresponding historical database.

[0085] Specifically, the system may also include:

[0086] The visualization module connects to the real-time and historical databases to visualize real-time and historical data.

[0087] Specifically, the control module can transmit real-time and historical data to the visualization module, which can then display and analyze the data in the following ways:

[0088] (1) Real-time data display: Real-time data is displayed on the monitoring screen. At the same time, the data in the urban underground pipeline sensor threshold table is combined to provide early warning for the urban underground pipeline sensor equipment. When the data returned by the sensor exceeds the preset warning value, the control system of the monitoring screen will provide early warning prompts to the on-duty personnel and management personnel of the monitoring center through the color change of the icon.

[0089] (2) Historical data display: Display historical data such as temperature, humidity, oxygen concentration, and liquid level in the urban underground pipe network in the form of lists, charts, etc.

[0090] (3) Statistical analysis: Analyze the collected environmental data to determine the items and timing of human intervention, so as to ensure the normal operation of the urban underground pipeline network and the safety of personnel entering and exiting the underground pipeline network.

[0091] Specifically, the control module is also used to associate multiple data processing methods with multiple data topics of the MQTT transmission module, determine the corresponding data processing method according to the topic to which the real-time data belongs, and call the corresponding data processing method to process the real-time data.

[0092] When using the MQTT protocol for data transmission, the handleMessage method is currently used. It iterates through multiple data processing methods for each piece of data under each topic to associate each piece of data with the corresponding data processing method. When there are many subscribed topics, this method has the problems of program complexity, bloat and low efficiency.

[0093] To address the aforementioned issues, this embodiment first defines two interface classes, `MqttService` and `MqttTopic`. Then, it creates a method class `MqttTopicHandle` and adds the annotation `@MqttService` to it based on the name of the first interface. Subsequently, it creates a data processing method within `MqttTopicHandle` and adds the annotation `@MqttTopic` to this method based on the name of the second interface class. A subscription topic corresponding to the data processing method is then added to the `@MqttTopic` annotation, thereby establishing the association between data topics and data processing methods on MQTT.

[0094] During data transmission, after the control module receives data from MQTT, it first iterates through all classes in the `handleMessage` method, selecting the method class `MqttTopicHandle` annotated with `@MqttService`. Then, it selects data processing methods annotated with `@MqttTopic` from `MqttTopicHandle`. Next, it checks if the subscription topic in the `@MqttTopic` annotation of the selected data processing method matches the topic of the received data. If they match, it uses Java reflection to call the data processing method to process the received data. This allows the received data to be associated with the corresponding data processing method based on different topics, enabling the distribution of different data messages according to different topics.

[0095] Specifically, the following code uses the Java programming language to associate data topics with data processing methods:

[0096]

[0097]

[0098]

[0099] The above description is merely a few embodiments of this application and is not intended to limit this application in any way. Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any changes or modifications made by those skilled in the art without departing from the scope of the technical solution of this application using the disclosed technical content are equivalent to equivalent implementation cases and fall within the scope of the technical solution.

Claims

1. A data monitoring system for urban underground pipe networks, characterized in that, The system includes: Multiple IoT devices are used to collect real-time data from various types of underground pipe networks; An IoT gateway, which connects to multiple IoT devices, is used to set the type label for each real-time data based on the device type of each IoT device; A real-time database is used to store real-time data of underground pipe networks; The control module is connected to the IoT gateway, the real-time database, and multiple IoT devices respectively. It is used to control the operation of multiple IoT devices and the IoT gateway, and to store each real-time data into the corresponding type field in the real-time database according to the type label of each real-time data. The MQTT transmission module is connected to both the IoT gateway and the control module, and is used to transmit the real-time data transmitted by the IoT gateway to the control module in the form of messages. Multiple historical databases correspond to multiple types of real-time data, used to store different types of real-time data respectively; The module is used to build the real-time database and multiple historical databases based on the device types of multiple IoT devices; Multiple IoT devices include temperature sensors, humidity sensors, oxygen concentration sensors, and liquid level sensors; Accordingly, multiple historical databases, including a temperature historical database, a humidity historical database, an oxygen concentration historical database, and a liquid level historical database, are used to store real-time data collected by the temperature sensor, humidity sensor, oxygen concentration sensor, and liquid level sensor, respectively. The control module is also used to associate multiple data processing methods with multiple data topics of the MQTT transmission module, determine the corresponding data processing method according to the topic to which the real-time data belongs, and call the corresponding data processing method to process the real-time data.

2. The system according to claim 1, characterized in that, The control module is used to convert the received real-time data in message form into real-time data in JOSN format, and transmit the real-time data in JOSN format to the real-time database.

3. The system according to claim 1, characterized in that, The connection between the IoT gateway and the IoT device is based on protocol.

4. The system according to claim 1, characterized in that, The MQTT transmission module uses a multi-threaded approach for data transmission.

5. The system according to claim 1, characterized in that, The system also includes: The visualization module, connected to the real-time database and the historical database, is used to visualize the real-time data and the historical data.

Images