Solid waste monitoring-oriented efficient data automatic distribution system based on MQTT protocol

Abstract

The invention discloses a high-efficiency automatic data distribution system for solid waste monitoring based on an MQTT protocol, and relates to the technical field of solid waste monitoring, the system uses enterprise codes in data as dynamic subscription topic, data is analyzed in real time and a target topic is matched, automation and intelligence of data distribution are realized, and the distribution efficiency and flexibility are remarkably improved. Meanwhile, a lightweight MQTT protocol is adopted, massive topic and large-scale concurrency are supported, new data sources and monitoring objects can be flexibly accessed, good expandability is achieved, and interconnection and intercommunication with a heterogeneous system are facilitated. The method has remarkable advantages and innovations in the aspects of data distribution automation, routing efficiency, system decoupling, subscription granularity and the like, and the problems of insufficient flexibility, poor real-time performance, low expandability and the like in a massive heterogeneous data distribution scene in the prior art are well solved; the efficiency and the quality of data distribution can be remarkably improved, and the method has a wide application prospect.

Description

Technical Field

[0001] The present invention relates to the technical field of solid waste monitoring, and in particular to an efficient automatic data distribution system for solid waste monitoring based on the MQTT protocol. Background Art

[0002] With the acceleration of industrialization, the amount of solid waste (referred to as "solid waste") is increasing. In order to strengthen the prevention and control of solid waste pollution, various regions have established solid waste monitoring systems, which collect various monitoring indicator data in real time by deploying various sensors and monitoring equipment at emission sources and disposal sites, and timely discover and deal with potential environmental risks.

[0003] However, since solid waste monitoring involves many links, scattered monitoring points, and a wide variety of data, how to achieve efficient distribution and integration of multi-source heterogeneous monitoring data has become a major technical challenge in the construction of solid waste monitoring informatization. Traditional data distribution systems mostly adopt a centralized structure and static configuration method, which is difficult to adapt to the dynamically changing data sources and receivers in solid waste monitoring applications, and the flexibility and scalability of data distribution are insufficient.

[0004] At present, the publish-subscribe model has been widely used in the field of data distribution. MQTT (Message Queuing Telemetry Transport) is a lightweight IoT communication protocol based on the publish-subscribe model. MQTT uses Topic as the transmission medium for messages. The publisher publishes messages to a specific Topic, and the subscriber obtains messages by subscribing to the Topic, thereby realizing the distribution of data between publishers and subscribers.

[0005] MQTT has the advantages of low overhead, low latency, and strong scalability, making it very suitable for data transmission scenarios involving massive devices such as the Internet of Things and mobile Internet. Currently, MQTT has been widely used in many fields such as smart cities, Internet of Vehicles, and power monitoring [3]. Introducing MQTT into the distribution of solid waste monitoring data is expected to solve the problems of poor flexibility and lack of real-time performance of traditional distribution systems.

[0006] In the existing technology, there are limitations of static topic mapping or matching rules, performance bottlenecks based on the topic hierarchy, high coupling between data sources and subscribers, and insufficient semantics of data distribution, as follows:

[0007] 1. Most existing MQTT data distribution solutions use preset static Topic mapping or matching rules to bind data sources to Topics in a one-to-one or one-to-many fixed manner. This static configuration method cannot dynamically adapt to changes in data content. When the data source changes (such as adding, deleting, or modifying monitoring equipment) or the data subscription requirements change, it is necessary to manually modify the Topic mapping or matching rules, which is cumbersome and lacks real-time performance, making it difficult to achieve automated dynamic data distribution;

[0008] 2. Most existing solutions adopt an information distribution model based on the Topic hierarchy structure, and implement refined data routing by designing a multi-level Topic namespace. However, when the Topic hierarchy is too deeply nested or there are too many Topics, the system needs to consume a lot of resources to maintain the Topic tree structure, and the routing efficiency of data distribution will be significantly reduced. In addition, subscribers need to subscribe to multiple levels of Topics to obtain complete data, which has high subscription complexity and limited system scalability;

[0009] 3. Most existing solutions require pre-configuration of the mapping relationship between data sources and topics, or direct publishing to a specified topic on the data source side. This results in a high degree of coupling between the data publishing side and the subscription side, and a lack of decoupling mechanism. The data source needs to perceive the downstream topic information, and when the subscription requirements change, the configuration on the data source side needs to be modified, which has poor flexibility and maintainability;

[0010] 4. Most existing solutions perform data routing based on Topic names, lacking understanding and utilization of the semantics of data content. Topic names can usually only express limited metadata information (such as device number, collection time, etc.), and cannot fully represent the business meaning of the data. This leads to insufficient semantics in data distribution, making it difficult to support intelligent routing and filtering based on data content, and the accuracy of data subscription needs to be improved;

[0011] To this end, this application proposes an efficient automatic data distribution system for solid waste monitoring based on the MQTT protocol to solve the above problems. Summary of the invention

[0012] 1. Technical issues to be resolved

[0013] In view of the deficiencies of the prior art, the present invention aims to solve the above technical problems and proposes an efficient automatic data distribution system for solid waste monitoring based on the MQTT protocol. The system uses the enterprise code in the data as a dynamic subscription topic, and realizes the automation and intelligence of data distribution by real-time data parsing and matching the target topic, which significantly improves the distribution efficiency and flexibility. At the same time, it adopts the lightweight MQTT protocol, supports massive topics and large-scale concurrency, can flexibly access new data sources and monitoring objects, has good scalability, and is easy to interconnect with heterogeneous systems, solving the technical problems raised in the above background.

[0014] (II) Technical solution

[0015] To achieve the above objectives, the technical solution adopted by the present invention is: a method for automatic data distribution based on the MQTT protocol, which uses specific fields in the data (such as enterprise code) as the basis for dynamic subscription, and realizes automation and intelligence of data distribution by real-time parsing of data content and calculating target Topic.

[0016] Preferably, the target Topic uses the following Topic generation rules to calculate data correspondence:

[0017] ${prefix} / ${enterpriseCode} / ${monitoringType}

[0018] Among them, ${prefix} represents the configurable Topic prefix, ${enterpriseCode} represents the enterprise code of the data, and ${monitoringType} represents the monitoring type of the data.

[0019] An automatic data distribution system based on MQTT includes a data acquisition module, a data analysis module, a Topic calculation module, an MQTT service module and a data subscription module; the data acquisition module collects heterogeneous data and converts it into a standard format, the data analysis module extracts key fields in the data, the Topic calculation module calculates the target Topic in real time according to the key fields and preset rules, the MQTT service module publishes data to the target Topic, and the data subscription module obtains data by subscribing to the Topic.

[0020] Preferably, the data acquisition module supports multiple acquisition methods, including but not limited to MQTT, HTTP, FTP, TCPSocket, etc.

[0021] Preferably, the data parsing module extracts key fields from the data in JSON format, and temporarily stores the parsing results in the form of Key-Value in the memory.

[0022] Preferably, the Topic calculation module calculates the target Topic to which each piece of data should be published in real time based on a configurable Topic generation rule.

[0023] Preferably, the MQTT service module provides data publishing and subscription services based on the MQTT protocol, and maintains the tree structure of the Topic.

[0024] A data subscription method where data consumers subscribe to topics based on enterprise code, monitoring type, and other dimensions to achieve fine-grained data subscription. The system automatically pushes data matching topics to subscribers in real time.

[0025] (III) Beneficial effects

[0026] The beneficial effects of the present invention are:

[0027] This efficient automatic data distribution system for solid waste monitoring based on the MQTT protocol uses specific fields in the data (such as enterprise code) as the basis for dynamic subscription. By real-time analysis of data content and calculation of target topics, it realizes the automation and intelligence of data distribution. There is no need to preset static rules. The system can automatically adapt to the dynamic changes of data sources and subscription requirements, which significantly improves the flexibility and real-time performance of data distribution. By adopting a flat topic design and introducing a dynamic routing mechanism based on data content, the limitations of the topic tree structure are overcome. The system can quickly calculate the target topic according to the key fields of the data, realize efficient data routing, and improve the performance and scalability of data distribution. By establishing a mapping relationship between data content and distribution rules, the decoupling of data sources and subscribers is realized, data publishing and subscription can change independently, the coupling degree of the system is greatly reduced, and the flexibility and maintainability of the system are improved.

[0028] This efficient automatic data distribution system for solid waste monitoring based on the MQTT protocol fully mines the semantic information of the data content (such as enterprise code, monitoring type), allows subscribers to flexibly subscribe to specific categories of data according to business needs, and realizes finer-grained data distribution, which improves the pertinence and accuracy of data subscription and reduces unnecessary data transmission and processing overhead. The present invention has significant advantages and innovations in data distribution automation, routing efficiency, system decoupling and subscription granularity, and the present invention solves the problems of insufficient flexibility, poor real-time performance, low scalability and other problems faced by the prior art in the scenario of massive heterogeneous data distribution, which can significantly improve the efficiency and quality of data distribution and has broad application prospects. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] The present invention provides a technical solution: an automatic data distribution method based on the MQTT protocol, which uses specific fields in the data (such as enterprise code) as the basis for dynamic subscription, and realizes automation and intelligence of data distribution by real-time parsing of data content and calculating target Topic.

[0031] The target Topic uses the following Topic generation rules to calculate data correspondence:

[0032] ${prefix} / ${enterpriseCode} / ${monitoringType}

[0033] Among them, ${prefix} represents the configurable Topic prefix, ${enterpriseCode} represents the enterprise code of the data, and ${monitoringType} represents the monitoring type of the data.

[0034] The target topic also uses the following topic generation rules to calculate data correspondence:

[0035] a)"${prefix} / ${enterpriseCode} / ${monitoringType} / ${timestamp}"

[0036] By adding a timestamp to a Topic, you can categorize and subscribe to data based on the time dimension.

[0037] b)"${prefix} / ${enterpriseCode} / ${monitoringType} / ${regionCode}"

[0038] By adding a region code (regionCode) to a Topic, data can be categorized and subscribed to by geographic region.

[0039] c)"${prefix} / ${industryCode} / ${enterpriseCode} / ${monitoringTyp e}"

[0040] By adding industry codes to the Topic, you can classify and subscribe to data by industry category.

[0041] In short, Topic generation rules can be customized and extended according to actual needs, introducing more data dimensions to support more flexible and fine-grained data distribution.

[0042] An automatic data distribution system based on MQTT includes a data acquisition module, a data analysis module, a Topic calculation module, an MQTT service module and a data subscription module; the data acquisition module collects heterogeneous data and converts it into a standard format, the data analysis module extracts key fields in the data, the Topic calculation module calculates the target Topic in real time according to the key fields and preset rules, the MQTT service module publishes data to the target Topic, and the data subscription module obtains data by subscribing to the Topic.

[0043] The data collection module supports multiple collection methods, including but not limited to MQTT, HTTP, FTP, TCP Socket, etc.

[0044] In the present invention, the data collection module supports multiple collection methods, such as MQTT, HTTP, FTP, etc., but the data collection method can be replaced and expanded according to the specific scenario, for example:

[0045] a) Use WebSocket protocol for real-time data collection;

[0046] b) Use AMQP protocol for message queue data collection;

[0047] c) Use file-based data collection methods, such as regularly scanning data files in a specified directory;

[0048] d) Use database triggers or log analysis to collect data.

[0049] The data parsing module extracts key fields from the data in JSON format and temporarily stores the parsing results in the form of Key-Value in the memory.

[0050] In the present invention, the data parsing module uses JSON format to parse data, but the data parsing method can also be replaced according to actual conditions, for example:

[0051] a) Use XML format for data parsing;

[0052] b) Use a custom binary format for data parsing to improve parsing efficiency;

[0053] c) Use schema-based data parsing methods, such as Avro and Protobuf, to obtain stronger data type checking and compatibility;

[0054] In the present invention, the data subscription module supports fine-grained subscription according to dimensions such as enterprise code and monitoring type. However, the subscription granularity can be further refined or expanded, for example:

[0055] a) Support conditional subscription based on data value range (such as temperature > 50 degrees);

[0056] b) Support spatial subscription based on geographic location (such as within a specified coordinate range);

[0057] c) Support event subscription based on complex event rules (such as a combination of multiple conditions).

[0058] The Topic calculation module calculates the target Topic to which each piece of data should be published in real time based on configurable Topic generation rules.

[0059] The MQTT service module provides data publishing and subscription services based on the MQTT protocol and maintains the tree structure of Topic.

[0060] In the present invention, the MQTT service module uses the MQTT protocol as the message middleware, but in fact, other message middleware can be selected as an alternative according to the requirements of performance, function, compatibility, etc., for example:

[0061] a) Use Apache Kafka as the message middleware to provide high-throughput, low-latency data stream processing;

[0062] b) Use RabbitMQ as the message middleware to provide flexible routing and queue support;

[0063] c) Use ZeroMQ as the message middleware to provide lightweight, high-performance distributed messaging.

[0064] A data subscription method where data consumers subscribe to topics based on enterprise code, monitoring type, and other dimensions to achieve fine-grained data subscription. The system automatically pushes data matching topics to subscribers in real time.

[0065] Example

[0066] An efficient automatic data distribution system for solid waste monitoring based on the MQTT protocol. The system uses the enterprise code in the data as the basis for dynamic subscription, and realizes the automation and intelligence of data distribution by real-time analysis of data content and calculation of target Topic. The system mainly includes data acquisition module, data analysis module, Topic calculation module, MQTT service module and data subscription module.

[0067] Data collection module: The data collection module is responsible for collecting data from various solid waste monitoring equipment and systems. The system supports multiple data collection methods, including but not limited to: MQTT, HTTP, FTP, TCP Socket, etc. The data collection module converts the collected heterogeneous data into JSON format and pushes the standardized data to the data analysis module.

[0068] Data parsing module: The data parsing module receives the JSON format monitoring data pushed by the data collection module and parses it in real time. The module extracts key field information by reading specific fields in the JSON data (such as enterpriseCode enterprise code, monitoringType monitoring type, etc.) for subsequent Topic calculation. The parsed data is temporarily stored in the memory in the form of key-value pairs and triggers the Topic calculation module.

[0069] Topic calculation module: The Topic calculation module receives the key field information transmitted by the data analysis module and calculates the target Topic to which each data should be published in real time according to the predefined Topic generation rules. Topic generation rules are configurable, and the default rules are as follows:

[0070] ${prefix} / ${enterpriseCode} / ${monitoringType}

[0071] Among them, ${prefix} represents the Topic prefix, which can be set in the system configuration file; ${enterpriseCode} represents the enterprise code in the data; ${monitoringType} represents the monitoring type of the data.

[0072] For example, if the enterprise code of a piece of data is "E001" and the monitoring type is "airQuality", the target Topic calculated according to the default rule is:

[0073] ${prefix} / E001 / airQuality

[0074] After the topic calculation is completed, the module sends the original data content and the target topic to the MQTT service module.

[0075] MQTT service module, based on the MQTT protocol, provides data publishing and subscription services. The module receives the data content and target Topic transmitted by the Topic calculation module, and publishes the data content to the corresponding Topic in the form of a PUBLISH message. At the same time, the module is also responsible for maintaining the Topic tree structure and processing the client's subscription and unsubscription requests. The MQTT service module can be implemented using an open source MQTT Broker, such as Mosquitto, RabbitMQ, etc.

[0076] Data subscription module, which runs on the data consumer side, obtains relevant data by subscribing to topics of interest. Subscribers can flexibly subscribe to data based on dimensions such as enterprise code and monitoring type, and the system will automatically push data matching topics to subscribers in real time.

[0077] For example, if the environmental protection department of a company (code E002) only cares about the wastewater discharge data of the company, it can subscribe to the following topics:

[0078] ${prefix} / E002 / wasteWater

[0079] After the subscription is successful, all data with enterprise code E002 and monitoring type wasteWater will be pushed to the subscriber in real time.

[0080] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for automatic data distribution based on the MQTT protocol, characterized in that: Specific fields in the data are used as the basis for dynamic subscription. By real-time parsing of data content and calculating target topics, data distribution can be automated and intelligent.

2. The method for automatic data distribution based on the MQTT protocol according to claim 1, characterized in that: The target Topic uses the following Topic generation rules to calculate data correspondence: ${prefix} / ${enterpriseCode} / ${monitoringType} Among them, ${prefix} represents the configurable Topic prefix, ${enterpriseCode} represents the enterprise code of the data, and ${monitoringType} represents the monitoring type of the data.

3. An automatic data distribution system based on MQTT, characterized by: Including data collection module, data analysis module, Topic calculation module, MQTT service module and data subscription module; The data collection module collects heterogeneous data and converts it into a standard format. The data parsing module extracts key fields from the data. The Topic calculation module calculates the target Topic in real time based on the key fields and preset rules. The MQTT service module publishes data to the target Topic. The data subscription module obtains data by subscribing to the Topic.

4. The MQTT-based automatic data distribution system according to feature 3, characterized in that: The data collection module supports multiple collection methods, including but not limited to MQTT, HTTP, FTP, TCP Socket, etc.

5. The MQTT-based automatic data distribution system according to feature 3, characterized in that: The data parsing module extracts key fields from the data in JSON format and temporarily stores the parsing results in the form of Key-Value in the memory.

6. The MQTT-based automatic data distribution system according to feature 3, characterized in that: The Topic calculation module calculates the target Topic to which each piece of data should be published in real time based on configurable Topic generation rules.

7. The MQTT-based automatic data distribution system according to feature 3, characterized in that: The MQTT service module provides data publishing and subscription services based on the MQTT protocol and maintains the tree structure of Topic.

8. A data subscription method, characterized in that: Data consumers can achieve fine-grained data subscription by subscribing to topics divided based on dimensions such as enterprise code and monitoring type. The system automatically pushes data matching the topics to subscribers in real time.