Internet of Things MQTT protocol message shunting method and device, and program product

Abstract

The invention relates to the technical field of Internet of Things communication, in particular to an Internet of Things MQTT protocol message shunting method and device and a program product, and the method comprises the steps: constructing a one-to-many mapping relation between an MQTT theme and a Kafka theme; associating a plurality of Kafka themes in the one-to-many mapping relationship with a plurality of virtual nodes in a one-to-one manner, and constructing a consistent hash ring; and writing the MQTT message into a Kafka cluster based on the consistent hash ring. The invention provides an Internet of Things MQTT protocol message offloading scheme, the message is endowed with an identity label through protocol enhancement, efficient data circulation is realized in combination with theme dynamic mapping, the problem of data skew in the message offloading process is solved based on a load balancing strategy of consistent Hash, and finally the instantaneity of data calculation is improved.

Description

Technical Field

[0001] This disclosure relates to the field of Internet of Things (IoT) communication technology, and in particular to an IoT MQTT protocol message splitting method, apparatus, and program product. Background Technology

[0002] With the surge in IoT devices, platforms need to efficiently handle diverse real-time messages (such as heartbeats, events, and command responses). Current mainstream solutions employ either a single service for unified processing or a distributed service for parsing and distributing messages, but both have significant drawbacks: 1) Resource bottleneck: The distributed service needs to maintain all broker long connections and parse all messages, consuming substantial network and computing resources; 2) Uncontrollable latency: Critical messages lack a priority processing mechanism, leading to response delays when mixed with ordinary messages; 3) Insufficient routing accuracy: Inaccurate mapping between devices and Kafka topics forces the backend to cache all hot data, causing storage pressure and performance degradation. Therefore, there is an urgent need to optimize IoT resource allocation, priority scheduling, and precise routing mechanisms. Summary of the Invention

[0003] This disclosure aims to at least partially solve one of the technical problems in the aforementioned technologies, and to this end proposes an IoT MQTT protocol message splitting method, including: Establish a one-to-many mapping relationship between MQTT topics and Kafka topics; The multiple Kafka topics in the one-to-many mapping relationship are associated one-to-one with multiple virtual nodes to construct a consistent hash ring; The MQTT message is written to the Kafka cluster based on the consistent hashing ring.

[0004] Furthermore, a one-to-many mapping relationship is established between MQTT topics and Kafka topics, including: Determine the Kafka topic group corresponding to the MQTT topic; Get the Kafka topics under the Kafka topic group; Construct a mapping relationship between the MQTT topic and the several Kafka topics.

[0005] Furthermore, the IoT MQTT protocol message splitting method further includes: receiving MQTT messages from the IoT device through an MQTT Broker based on the MQTT protocol with the IoT device ID injected in the variable header.

[0006] Furthermore, based on the aforementioned consistent hashing ring, MQTT messages are written to the Kafka cluster, including: Determine the IoT device ID and MQTT topic corresponding to the MQTT message; Calculate the hash value of the IoT device ID, locate the hash value of the IoT device ID in the consistent hash ring, and based on the location, find the first Kafka topic in the Kafka topic group that corresponds to the MQTT topic; Write the MQTT message to the found Kafka topic.

[0007] Furthermore, the MQTT Broker is directly connected to the Kafka cluster via the TCP protocol.

[0008] This disclosure also proposes an IoT MQTT protocol message routing system, including: IoT devices, MQTT Broker, and Kafka cluster; among them, The MQTT Broker is directly connected to the Kafka cluster via TCP protocol; The MQTT Broker is configured to send MQTT messages received from the IoT device to the Kafka cluster, and write the MQTT messages to the corresponding Kafka topics based on a consistent hashing ring; the consistent hashing ring is constructed based on a one-to-many mapping relationship between MQTT topics and Kafka topics.

[0009] Furthermore, the MQTT Broker is also configured to receive MQTT messages from the IoT device based on the MQTT protocol with the IoT device ID injected in the variable header.

[0010] Furthermore, the MQTT Broker is also configured as follows: Determine the IoT device ID and MQTT topic corresponding to the MQTT message; Calculate the hash value of the IoT device ID, locate the hash value of the IoT device ID in the consistent hash ring, and based on the location, find the first Kafka topic in the Kafka topic group that corresponds to the MQTT topic; Write the MQTT message to the found Kafka topic.

[0011] This disclosure also proposes a computer-readable storage medium storing a computer program or instructions, which, when executed by a processor, are at least used to implement the above-described IoT MQTT protocol message splitting method.

[0012] This disclosure also proposes a computer program product stored in a computer-readable storage medium, characterized in that, when executed by a processor, the computer program product is used to at least implement the above-described IoT MQTT protocol message splitting method.

[0013] Compared with the prior art, the beneficial effects of this disclosure are: This disclosure proposes an IoT MQTT protocol message splitting scheme, which assigns "identity identifiers" to messages through protocol enhancement, achieves efficient data flow by combining dynamic topic mapping, and solves the data skew problem in the message splitting process based on a consistent hash load balancing strategy, ultimately improving the real-time performance of data computation.

[0014] Other features and advantages of this disclosure will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the disclosure. The objects and other advantages of this disclosure may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings. The technical solutions of this disclosure will be further described below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0015] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings: Figure 1 This is a schematic diagram of an IoT MQTT protocol message splitting method provided in an embodiment; Figure 2 This is a schematic diagram illustrating the mapping relationship between MQTT topics and Kafka topics, provided as an example. Figure 3 A schematic diagram of a consistent hash ring structure is provided for this embodiment; Figure 4 A schematic diagram of an IoT MQTT protocol message routing system is provided as an example. Figure 5 A schematic diagram of an electronic device provided for an embodiment; Figure 6 This is a schematic diagram of a computer-readable storage medium provided for an embodiment. Detailed Implementation

[0016] The present disclosure will be described below with reference to the accompanying drawings. The preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present disclosure.

[0017] Figure 1 The message routing method for the IoT MQTT protocol disclosed herein includes: Establish a one-to-many mapping relationship between MQTT topics and Kafka topics; The multiple Kafka topics in the one-to-many mapping relationship are associated one-to-one with multiple virtual nodes to construct a consistent hash ring; The MQTT message is written to the Kafka cluster based on the consistent hashing ring.

[0018] According to some embodiments of this disclosure, the IoT MQTT protocol message splitting method given in this disclosure is based on the one-to-many mapping relationship between MQTT topics and Kafka topics, associating Kafka topics with virtual nodes, constructing a consistent hash ring, and realizing the writing of MQTT messages to the Kafka cluster through the consistent hash ring.

[0019] Furthermore, a one-to-many mapping relationship is established between MQTT topics and Kafka topics, including: Determine the Kafka topic group corresponding to the MQTT topic; Retrieve several Kafka topics under a Kafka topic group; Establish a mapping relationship between MQTT topics and several Kafka topics.

[0020] According to some embodiments of this disclosure, this disclosure achieves intelligent routing from a single MQTT topic to multiple Kafka topic partitions through a namespace isolation mechanism, constructing a horizontally scalable data channel. The corresponding technical solution is as follows: Each MQTT topic is independently bound to a Kafka topic group (e.g., "Business 1 / data" corresponds to "Business 1_topic_p0~Business 1_topic_p3"), and this one-to-many mapping relationship is as follows: Figure 2 As shown in the diagram, during mapping, the business prefix of the MQTT topic (e.g., business 1) is used as the prefix for namespace isolation of the Kafka topics. New businesses automatically create independent topic groups, and the number of partitions for a single business can be dynamically increased (p0→p10). This mapping method allows the system to flexibly distribute data from the same MQTT topic to different Kafka topics according to business needs and data traffic, achieving distributed data storage and processing, thereby improving the overall performance and scalability of the system.

[0021] Furthermore, the IoT MQTT protocol message splitting method disclosed herein also includes: receiving MQTT messages from IoT devices through an MQTT Broker based on the MQTT protocol with the IoT device ID injected in the variable header.

[0022] According to some embodiments of this disclosure, this disclosure achieves decoupled transmission of device identification and message body by injecting a unique IoT device ID into the variable header of the MQTT protocol. The corresponding technical solution is as follows: The message structure of the MQTT protocol consists of a fixed header, a variable header, and a payload. The variable header contains some optional fields related to the message, which can be used to transmit additional control or identification information. A unique IoT device ID is forcibly injected into the Properties field of the variable header. The MQTT data packet structure in this embodiment is shown in Tables 1-3.

[0023] Table 1 MQTT packet structure

[0024] Table 2 Fixed Head Data Structure

[0025] Table 3 Variable Header PUBLISH Message Structure

[0026] In this way, during message transmission, the server or other recipient can accurately extract the IoT device ID from the Properties field of the variable header, thereby achieving precise identification and management of the message source device. Simultaneously, the IoT device ID is independent of the payload, avoiding pollution of the business data structure. The IoT device ID is transmitted completely with the message and can be directly extracted from the protocol header without parsing the JSON payload.

[0027] Furthermore, based on a consistent hashing ring, MQTT messages are written to the Kafka cluster, including: Determine the IoT device ID and MQTT topic corresponding to the MQTT message; Calculate the hash value of the IoT device ID, locate the hash value of the IoT device ID in the consistent hash ring, and find the first Kafka topic in the Kafka topic group that corresponds to the MQTT topic based on the location; Write the MQTT message to the found Kafka topic.

[0028] According to some embodiments of this disclosure, in a Kafka topic group, each topic is associated with n virtual nodes, which are evenly distributed on a consistent hash ring, such as... Figure 3As shown, when an MQTT message is received, the Properties field of the message is parsed to extract the IoT device ID; simultaneously, the message is parsed to obtain the corresponding MQTT topic. Based on the obtained MQTT topic, the corresponding Kafka topic group is found. For example, the topic group may contain multiple topics such as Business 1_topic_p0, Business 1_topic_p1, Business 1_topic_p2, and Business 1_topic_p3. For each topic in the Kafka topic group, since its associated n virtual nodes are evenly distributed on a consistent hash ring, the device data is located at the corresponding position on the hash ring based on the hash value of its connected device ID, and then the search proceeds clockwise to find the first topic that matches the MQTT topic. After finding the matching topic, the MQTT Broker directly writes the received MQTT message to Kafka via the TCP protocol, completing the message transmission and storage.

[0029] Furthermore, the MQTT Broker is directly connected to the Kafka cluster via the TCP protocol.

[0030] According to some embodiments of this disclosure, by removing the intermediate distribution layer service, a direct, zero-intermediate connection channel is established between the MQTT Broker and Kafka, enabling rapid transmission of device data to the message queue. The MQTT Broker has a built-in native Kafka producer client that writes directly to Kafka via TCP protocol, such as... Figure 4 As shown, the MQTT Broker integrates a native Kafka producer client. This allows the MQTT Broker to directly write MQTT messages received from IoT devices to the Kafka cluster via TCP, without any intermediaries. This direct connection significantly simplifies the data transmission process, reducing latency and potential data loss. Furthermore, by eliminating the intermediate distribution layer, the overall system architecture becomes clearer and simpler, facilitating future maintenance and expansion. In addition, it effectively improves system response speed and processing capacity, meeting the stringent real-time and high-concurrency requirements of IoT applications.

[0031] Based on the same technological concept, such as Figure 4 As shown, this disclosure also proposes an IoT MQTT protocol message routing system, including: IoT devices, MQTT Broker, and Kafka cluster; among them, The MQTT Broker is directly connected to the Kafka cluster via TCP protocol; The MQTT Broker is configured to send MQTT messages received from the IoT device to the Kafka cluster, and write the MQTT messages to the corresponding Kafka topics based on a consistent hashing ring; the consistent hashing ring is constructed based on a one-to-many mapping relationship between MQTT topics and Kafka topics.

[0032] like Figure 5 As shown, this disclosure provides an electronic device, which includes a memory and a processor. The memory stores computer programs or instructions, and when the computer programs or instructions are executed by the processor, they are used to implement at least the above-described IoT MQTT protocol message splitting method.

[0033] like Figure 6 As shown, this disclosure provides a computer-readable storage medium storing a computer program or instructions, which, when executed by a processor, are used to implement at least the above-described IoT MQTT protocol message splitting method.

[0034] This disclosure also provides a computer program product stored in a computer-readable storage medium, which, when executed by a processor, is used to at least implement the above-described IoT MQTT protocol message splitting method.

[0035] It is obvious that those skilled in the art can make various modifications and variations to this disclosure without departing from its spirit and scope. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. A message splitting method for the Internet of Things (IoT) MQTT protocol, characterized in that, include: Establish a one-to-many mapping relationship between MQTT topics and Kafka topics; The multiple Kafka topics in the one-to-many mapping relationship are associated one-to-one with multiple virtual nodes to construct a consistent hash ring; The MQTT message is written to the Kafka cluster based on the consistent hashing ring.

2. The method as described in claim 1, characterized in that, Establish a one-to-many mapping between MQTT topics and Kafka topics, including: Determine the Kafka topic group corresponding to the MQTT topic; Get the Kafka topics under the Kafka topic group; Construct a mapping relationship between the MQTT topic and the several Kafka topics.

3. The method as described in claim 2, characterized in that, Also includes: Based on the MQTT protocol with the IoT device ID injected into the variable header, MQTT messages from the IoT device are received through the MQTT Broker.

4. The method as described in claim 3, characterized in that, Based on the aforementioned consistent hashing ring, writing MQTT messages to the Kafka cluster includes: Determine the IoT device ID and MQTT topic corresponding to the MQTT message; Calculate the hash value of the IoT device ID, locate the hash value of the IoT device ID in the consistent hash ring, and based on the location, find the first Kafka topic in the Kafka topic group that corresponds to the MQTT topic; Write the MQTT message to the found Kafka topic.

5. The method as described in claim 3, characterized in that, The MQTT Broker is directly connected to the Kafka cluster via the TCP protocol.

6. An IoT MQTT protocol message routing system, characterized in that, include: IoT devices, MQTT Broker, and Kafka cluster; among them, The MQTT Broker is directly connected to the Kafka cluster via TCP protocol; The MQTT Broker is configured to send MQTT messages received from the IoT device to the Kafka cluster, and write the MQTT messages to the corresponding Kafka topics based on a consistent hashing ring; the consistent hashing ring is constructed based on a one-to-many mapping relationship between MQTT topics and Kafka topics.

7. The system as described in claim 6, characterized in that, The MQTT Broker is also configured to receive MQTT messages from the IoT device based on the MQTT protocol with the IoT device ID injected in the variable header.

8. The system as described in claim 7, characterized in that, The MQTT Broker is also configured as follows: Determine the IoT device ID and MQTT topic corresponding to the MQTT message; Calculate the hash value of the IoT device ID, locate the hash value of the IoT device ID in the consistent hash ring, and based on the location, find the first Kafka topic in the Kafka topic group that corresponds to the MQTT topic; Write the MQTT message to the found Kafka topic.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program or instructions, which, when executed by a processor, are used to implement at least the method described in any one of claims 1-5.

10. A computer program product, said computer program product being stored in a computer-readable storage medium, characterized in that, When the computer program product is executed by a processor, it is used to implement at least the method described in any one of claims 1-5.

Images