A data communication method based on MQTT and Websocket

Abstract

The present invention discloses a data communication method based on MQTT and Websocket: S1, the intelligent terminal device publishes an MQTT message, which carries not only Topic information but also data type information subordinate to the Topic; S2, the MQTT message proxy server receives these MQTT messages, and creates message queues for storage according to the different Topics; S3, the background server subscribes to the corresponding Topic from the MQTT message proxy server according to its own needs; S4, after the background server receives the corresponding MQTT message, according to the client's subscription to the data type, the information corresponding to the corresponding data type under the corresponding Topic is sent to the client with which a long connection is established through the Websocket protocol. The present invention provides a reliable and timely data communication method for the business scenario of "intelligent terminal device-server-client", and can also reduce the number of Topics in the MQTT message proxy server, thereby reducing the occupation of the disk space of the MQTT message proxy server and increasing the rate of message consumption therein.

Description

Technical Field

[0001] The present invention relates to a data transmission method, and in particular to a data communication method based on MQTT and Websocket. Background Art

[0002] With the rapid development of Internet technology, the popularization of smart terminal devices, and the diversification of business needs, the importance of data communication between terminal devices has become increasingly prominent, especially now that there are more and more business scenarios such as "smart terminal device-server-client". How to ensure the real-time data acquisition and reliability of data transmission in actual applications is the key to consider in the data communication process.

[0003] MQTT is a "lightweight" communication protocol based on the publish / subscribe model, which is built on the TCP / IP protocol. The biggest advantage of MQTT is that it can provide real-time and reliable message services for connected remote devices with very little code and limited bandwidth, making it an ideal choice for implementing the Internet of Things architecture.

[0004] The MQTT protocol is based on topics for subscription and publishing. The protocol sets the following message format, including the Header and Payload message carrier, and its Topic information is recorded in its header. Figure 1 This is a typical MQTT message communication process. The publisher and subscriber are decoupled in this way. They are not directly connected, but through an intermediary, which we call a message broker server in MQTT. Figure 1 The Broker in is used to store and forward messages.

[0005] When the MQTT protocol is applied in the business scenario of "smart terminal device-server-client", the smart terminal device is Figure 1 The server is the publisher in the network, and the client connects to the server through the network to obtain information. The HTTP (Hypertext Transfer Protocol) protocol defines how the Web client communicates with the Web server and how the server transmits data back to the client. It is a common communication method between Web clients and servers, which includes two communication mechanisms: short connection and long polling.

[0006] Short connection (traditional polling): Each time the client and server perform an HTTP operation, a connection is established and the connection is terminated when the task is completed.

[0007] Long polling: The client requests data from the server like traditional polling. However, if the server does not have data to return to the client immediately, it will not return an empty result immediately, but will keep the request for a while, waiting for data to arrive. Figure 2 shown.

[0008] As can be seen from the above, when the server and the client communicate using the http protocol, no matter which mechanism is used, HTTP connections are constantly established and then wait for the server to process. The server cannot actively contact the client and can only initiate the connection by the client. This passive method is bound to affect the timeliness of message transmission. In addition, short connections require the server to have a fast processing speed, and long polling requires the server to have a high concurrent processing capability.

[0009] In addition, since smart terminal devices usually collect a lot of data, MQTT may need to set up a lot of topics, and too many topics will make the MQTT proxy server Figure 1 The broker needs to open up more space to store message queues (the broker creates queues to store messages based on the Topic of the MQTT message, and one Topic corresponds to one queue). This also brings about the problem of high time cost for the broker to locate the message queue. Summary of the invention

[0010] The main purpose of the present invention is to provide a reliable and timely data communication method for the business scenario of "smart terminal device-server-client". The data communication method can also reduce the number of Topics in the MQTT message proxy server, thereby reducing the occupancy of the disk space of the MQTT message proxy server. At the same time, it is also beneficial to improve the disk search speed of the MQTT message proxy server, thereby increasing the rate of message consumption therein.

[0011] The object of the present invention is achieved through the following technical solution: a data communication method based on MQTT and Websocket, applied to the business scenario of "intelligent terminal device-server-client", involving objects including intelligent terminal devices, MQTT message proxy servers, background servers and clients, and the data communication method comprises the following steps:

[0012] S1. The smart terminal device publishes an MQTT message with Topic information; the MQTT message complies with the following data transmission protocol, including SOF, Length, Data, CRC16 and EOF; wherein SOF and EOF are used as the frame header and frame tail respectively, Length is used to set the length of Data, and CRC16 is used to verify the integrity of the data frame; the Data part includes msg, time, index, and value, wherein the msg field is used to indicate the data type, the time field is used to indicate the publishing time, the index field is used to indicate the device number, and the value field is used to indicate the data value;

[0013] S2. The MQTT message proxy server receives these MQTT messages and creates message queues for storage according to different topics.

[0014] S3, the backend server subscribes to the corresponding Topic from the MQTT message proxy server according to its own needs;

[0015] S4. After receiving the corresponding MQTT message, the backend server sends the information corresponding to the corresponding data type under the corresponding Topic to the client with which a long connection has been established through the Websocket protocol according to the client's subscription status to the data type.

[0016] The WebSocket protocol is a new network protocol based on TCP. It implements full-duplex communication between the browser and the server, allowing the server to actively send information to the client, so compared with the http protocol, it can better save server resources and bandwidth and achieve the purpose of real-time communication.

[0017] The connection between the intelligent terminal device and the server is established through MQTT, which makes full use of the characteristics of the MQTT lightweight protocol that supports millions of intelligent terminal devices to be connected simultaneously in communication and can provide real-time and reliable message services for connected remote devices. At the same time, the Websocket protocol is introduced to support the server to actively transmit data to the client, which can better save server resources and bandwidth and achieve the purpose of real-time communication.

[0018] In addition, the client of the present invention subscribes to a data type rather than a traditional Topic. In the present invention, Topic is a higher-level concept of the data type subscribed by the client, that is, Topic in the present invention represents a large classification, and Topic / data type in the present invention as a whole corresponds to the topic in the traditional MQTT message. The " / " here is used for hierarchical segmentation. Obviously, the present invention can significantly reduce the number of Topics in MQTT, thereby reducing the occupancy of the disk space of the MQTT message proxy server. The reduction in the number of Topics is conducive to improving the disk search speed of the MQTT message proxy server, thereby increasing the rate of message consumption therein.

[0019] According to the present invention, the Topic in the MQTT message is only used to represent a large classification, and the data type contained therein has the characteristic of great uncertainty. The present invention makes the following improvements:

[0020] The Data part also includes other custom fields, which are used to support users to expand according to the scenarios used.

[0021] In order to further solve the problem of data transmission stability in high-concurrency scenarios, the present invention proposes a data communication method based on MQTT, Kafka and Websocket on the basis of the above scheme. The specific measures are as follows: the server is a Kafka proxy cluster composed of more than two background servers, and step S4 is improved as follows: the background server receives the corresponding MQTT message by using Kafka's partitioning mechanism, and stores the received MQTT message in a Kafka message queue created according to the Topic according to the Topic. The messages in each Kafka message queue are processed by the service corresponding to the Topic, and the client subscribes to the Kafka message queue according to the Topic. The Topic to which the data type belongs is obtained by the corresponding service from the corresponding Kafka message queue, and the message is put into a synchronous blocking queue. The thread pool is used to sequentially extract the MQTT messages in the queue, and then a legitimacy check is performed to determine whether the MQTT message is complete. The MQTT message is then parsed to extract the data therein, and different data types are distinguished according to the msg field in the data. The corresponding Handler function is called to perform data segmentation processing according to the data type, and the processed data is then encapsulated into a message in json format, and the message is pushed to the client that has established a long connection with the background server through the Websocket protocol.

[0022] Kafka is a distributed messaging system that supports partitioned storage and multiple copies. It usually runs in a cluster. Its biggest feature is that it can use the Kafka proxy cluster to achieve load balancing, so that it can process large amounts of data in real time to cope with high-concurrency scenarios. Like MQTT, Kafka also uses the publish / subscribe message processing model, which also has the concept of Topic and creates message queues based on Topic, so it can directly use the Topic information in the MQTT message to connect with MQTT.

[0023] As a preferred embodiment of the present invention:

[0024] The smart terminal device accesses the third-party library org.eclipse.paho.client.mqttv3, calls the connect method of the MqttClient class to establish a connection with the MQTT message proxy server, and monitors the connection status between the smart terminal device and the MQTT message proxy server through the connectionLost callback interface of MqttCallbackExtended. If the connection is disconnected, reconnect.

[0025] The client accesses the third-party library io.crossbar.autobahn, connects to the backend server through the connect method of WebSocketConnnection in the third-party library, and monitors their connection status through the onClose method in the WebSocketConnectionHandler class. When it is detected that their connection is disconnected, the onClose method is called to reconnect.

[0026] Beneficial effects:

[0027] 1) First, the present invention proposes a data communication method based on MQTT and Websocket, in which a connection is established between a smart terminal device and a server through MQTT, which makes full use of the characteristics of the MQTT lightweight protocol that supports millions of smart terminal devices to be connected simultaneously in communication, and can provide real-time and reliable message services for connected remote devices. At the same time, the Websocket protocol is introduced to support the server to actively transmit data to the client, which can better save server resources and bandwidth and achieve the purpose of real-time communication; in addition, the client of the present invention subscribes to a data type rather than a traditional Topic, and the Topic / data type in the present invention corresponds to the topic in the traditional MQTT message as a whole, so the present invention can significantly reduce the number of Topics in MQTT, thereby reducing the occupancy of the disk space of the MQTT message proxy server, and the reduction in the number of Topics is conducive to improving the disk search speed of the MQTT message proxy server, thereby increasing the rate of message consumption therein.

[0028] 2) Secondly, the present invention also provides a data communication method based on MQTT, Kafka and Websocket. On the basis of the data communication method based on MQTT and Websocket, Kafka is introduced to achieve load balancing and solve the stability problem of data transmission in high concurrency scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a typical MQTT message communication process;

[0030] Figure 2 This is a schematic diagram of the HTTP long polling process;

[0031] Figure 3 It is a communication network topology diagram of the data communication method based on MQTT and Websocket according to the first embodiment of the present invention;

[0032] Figure 4 This is the format of the data transmission protocol proposed in Embodiment 1;

[0033] Figure 5 yes Figure 2 The structure of the data part;

[0034] Figure 6 This is a communication network topology diagram of a data communication method based on MQTT and Websocket according to a second embodiment of the present invention. DETAILED DESCRIPTION

[0035] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0036] Embodiment 1

[0037] A data communication method based on MQTT and Websocket, comprising the following steps:

[0038] S1. The smart terminal device publishes an MQTT message with Topic information.

[0039] The MQTT message follows the following data transmission protocol (a customization of the MQTT message payload part), such as Figure 4 , 5 As shown in the figure, it includes SOF, Length, Data, CRC16 and EOF; SOF and EOF are respectively used as the frame header and frame tail, Length is used to set the length of Data, and CRC16 is used to verify the integrity of the data frame; the Data part includes msg, time, index, value and other custom fields, among which the msg field is used to indicate the data type, the time field is used to indicate the release time, the index field is used to indicate the device number, the value field is used to indicate the data value, and the custom field is used to support users to expand according to the used scenario.

[0040] S2. The MQTT message proxy server receives these MQTT messages and creates message queues for storage according to different topics.

[0041] S3. The background server subscribes to the corresponding Topic from the MQTT message proxy server according to its own needs.

[0042] S4. After receiving the corresponding MQTT message, the backend server sends the information corresponding to the corresponding data type under the corresponding Topic to the client with which a long connection has been established through the Websocket protocol according to the client's subscription status to the data type.

[0043] The connection between the intelligent terminal device and the server is established through MQTT, which makes full use of the characteristics of the MQTT lightweight protocol that supports millions of intelligent terminal devices to be connected simultaneously in communication and can provide real-time and reliable message services for connected remote devices. At the same time, the Websocket protocol is introduced to support the server to actively transmit data to the client, which can better save server resources and bandwidth and achieve the purpose of real-time communication.

[0044] In addition, the client of the present invention subscribes to a data type, rather than a traditional Topic. In the present invention, Topic is an upper-level concept of the data type subscribed by the client, that is, Topic in the present invention represents a large classification. For example, there are two experimental tanks, which are large tanks for brewing soy sauce. During the brewing process, it is necessary to collect some data in the tank for real-time observation, mainly temperature data, humidity data, and pH data. At this time, Topics can be set for the two experimental tanks respectively, such as one is Topic_A and the other is Topic_B. Correspondingly, the msg field is Temperature_msg, Humidity_msg, and ph_msg, reflecting the data type included in the Topic. Therefore, the Topic / data type in the present invention corresponds to the topic in the traditional MQTT message as a whole. The " / " here is used for hierarchical segmentation. Obviously, the present invention can significantly reduce the number of Topics of MQTT, thereby reducing the occupancy of the disk space of the MQTT message proxy server. The reduction in the number of Topics is conducive to improving the disk search speed of the MQTT message proxy server, thereby increasing the rate of message consumption therein.

[0045] As the preferred method for establishing an MQTT connection between a smart terminal device and a server:

[0046] The smart terminal device accesses the third-party library org.eclipse.paho.client.mqttv3, and calls the connect method of the MqttClient class to establish a connection with the MQTT message proxy server. At the same time, a reconnection mechanism is implemented, which monitors the connection status between the smart terminal device and the MQTT message proxy server through the connectionLost callback interface of MqttCallbackExtended, and reconnects once the connection is disconnected;

[0047] The backend server accesses the third-party library org.eclipse.paho.client.mqttv3 to establish a connection with the MQTT message broker server and subscribes to the corresponding topics, such as Topic_A and Topic_B. Specifically, the developer writes the topic corresponding to the data type that the client needs to subscribe to into the configuration file, and the server obtains the topic information required to be subscribed by reading the configuration file.

[0048] Preferably, the client in the present invention accesses the third-party library io.crossbar.autobahn, and establishes a connection with the server through the connect method of the WebSocketConnnection class in the third-party library. At the same time, a disconnection and reconnection mechanism is implemented to prevent the problem that the connection is suddenly released and cannot be automatically reconnected. By implementing the onClose method in the WebSocketConnectionHandler class, once the connection is disconnected, this method will be called, so reconnection can be achieved. After establishing the connection, the client sends a message to the server indicating the data types it subscribes to.

[0049] The server uses a Map to store the data types (in the format of Topic_data type, where Topic is the Topic to which the data type belongs) and the connections of the clients that subscribe to the data types, denoted as Map<Topic_data type, List <websocketconnection>>, which is used as the basis for sending data. One data type can correspond to multiple client connections.

[0050] Embodiment 2

[0051] In order to further solve the problem of data transmission stability in high-concurrency scenarios, the server in Example 2 adopts a Kafka proxy cluster, which is usually composed of more than two background servers. On this basis, Example 2 further proposes a data communication method based on MQTT, Kafka and Websocket. The main difference from Example 1 is that it improves step S4 as follows:

[0052] The backend server receives the corresponding MQTT message by using Kafka's partition mechanism and writes it to the disk in a sequential writing mode. Since Kafka is also based on the publish / subscribe mode, it can directly create a Kafka message queue based on the Topic of the received MQTT message and store the MQTT message therein. The message in each Kafka message queue is processed by the service corresponding to the Topic. According to the Topic to which the data type subscribed by the client belongs, the corresponding service obtains the message from the corresponding Kafka message queue and puts it into a synchronous blocking queue. The thread pool is used to extract the MQTT message in the queue in sequence, and then a legality check is performed to determine whether the MQTT message is complete. Then the MQTT message is parsed, the data therein is extracted, and different data types are distinguished according to the msg field in the data, and the corresponding Handler function is called to perform data segmentation processing according to the data type, and then the processed data is encapsulated into a message in json format, and the message is pushed to the client that has established a long connection with the backend server through the Websocket protocol. After the client receives the data, it parses the data according to the data protocol and then displays it.

[0053] Kafka is a distributed messaging system that supports partitioned storage and multiple copies. It usually runs in a cluster. Its biggest feature is that it can use the Kafka proxy cluster to achieve load balancing, so that it can process large amounts of data in real time. Example 2 Based on Example 1, Kafka is introduced to achieve load balancing and solve the stability problem of data transmission in high-concurrency scenarios.

[0054] In general, the first embodiment of the present invention can solve the timeliness and reliability issues of data communication in the "intelligent terminal device-server-client" business scenario, and the second embodiment of the present invention can further solve the problem of data transmission stability in high-concurrency scenarios. At the same time, the first and second embodiments of the present invention can also significantly reduce the number of MQTT topics, thereby reducing the occupancy of the MQTT message proxy server disk space. The reduction in the number of topics is conducive to improving the disk search speed of the MQTT message proxy server, thereby increasing the rate of message consumption therein.< / websocketconnection>

Claims

1. A data communication method based on MQTT and Websocket, applied to the business scenario of "intelligent terminal device-server-client", involving objects including intelligent terminal device, MQTT message proxy server, background server and client, characterized in that: The data communication method comprises the following steps: S1. The smart terminal device publishes an MQTT message with Topic information; the MQTT message complies with the following data transmission protocol, including SOF, Length, Data, CRC16 and EOF; wherein SOF and EOF are used as the frame header and frame tail respectively, Length is used to set the length of Data, and CRC16 is used to verify the integrity of the data frame; the Data part includes msg, time, index, and value, wherein the msg field is used to indicate the data type, the time field is used to indicate the publishing time, the index field is used to indicate the device number, and the value field is used to indicate the data value; S2. The MQTT message proxy server receives these MQTT messages and creates message queues for storage according to different topics. Specifically, Topic represents a large category, which includes the data type defined by the msg field in the MQTT message. The data type represents the different types of data required. S3, the backend server subscribes to the corresponding Topic from the MQTT message proxy server according to its own needs; S4. After receiving the corresponding MQTT message, the backend server sends the information corresponding to the corresponding data type under the corresponding Topic to the client with which a long connection has been established through the Websocket protocol according to the subscription status of the client to the data type; The server is a Kafka proxy cluster composed of more than two background servers, and step S4 is improved as follows: the background server receives the corresponding MQTT message by using the partition mechanism of Kafka, and stores the received MQTT message in a Kafka message queue created according to the Topic of the Topic, and the message in each Kafka message queue is processed by the service corresponding to the Topic, and the corresponding service obtains the message from the corresponding Kafka message queue according to the Topic to which the data type subscribed by the client belongs, and puts it into a synchronous blocking queue, and uses the thread pool to extract the MQTT message in the queue in sequence, and then performs a legitimacy check to determine whether the MQTT message is complete, and then parses the MQTT message, extracts the data therein, and then distinguishes different data types according to the msg field in the data, and calls the corresponding Handler function to perform data segmentation processing according to the data type, and then encapsulates the processed data into a message in json format, and pushes the message to the client that has established a long connection with the background server through the Websocket protocol.

2. The data communication method according to claim 1, characterized in that: The Data part also includes other custom fields, which are used to support users to expand according to the scenarios used.

3. The data communication method according to claim 1, characterized in that: The smart terminal device accesses the third-party library org.eclipse.paho.client.mqttv3, calls the connect method of the MqttClient class to establish a connection with the MQTT message proxy server, and monitors the connection status between the smart terminal device and the MQTT message proxy server through the connectionLost callback interface of MqttCallbackExtended. If the connection is disconnected, reconnect.

4. The data communication method according to claim 1, characterized in that: The client accesses the third-party library io.crossbar.autobahn, connects to the backend server through the connect method of WebSocketConnnection in the third-party library, and monitors their connection status through the onClose method in the WebSocketConnectionHandler class. When it is detected that their connection is disconnected, the onClose method is called to reconnect.

Images