A message publishing method and device based on MQTT

Abstract

The present invention discloses a message publishing method and device based on MQTT. By responding to the publishing operation of the publishing end, a message to be published is obtained, the message to be published is segmented to obtain a plurality of message files, and the segment number of each message file is obtained based on the current thread number and the current timestamp number of each message file. The sub-messages in each message file are numbered to obtain the number of each sub-message, and the sub-message is sent to the subscription end so that the subscription end consumes it. If the subscription end consumes the sub-message, the segment number and number of the consumed sub-message are obtained, and the consumed sub-message is deleted according to the segment number of the consumed sub-message, and the sub-messages in the message files that have not been deleted are continuously sent to the subscription segment for consumption until all the sub-messages in all the message files are published. The method segments and numbers the messages through multi-threading, so that the integrity of the data is ensured when the message is pushed.

Description

Technical Field

[0001] The present invention relates to the technical field of computer data processing, and in particular to a message publishing method and device based on MQTT. Background Art

[0002] MQTT (Message Queue Telemetry Transport) is a message protocol based on the publish / subscribe paradigm under the ISO standard (ISO / IEC PRF 20922). It is a lightweight, simple, open subscription-based data transmission protocol designed specifically for situations where the hardware connection network is unreliable. MQTT is an application layer transmission protocol based on the TCP protocol. When the publisher sends a large amount of data to the MQTT server on the same topic at the same time, some data may be lost or the publisher may disconnect from the MQTT server and reconnect. During the reconnection, if the data is not saved, the MQTT server will not be able to receive the data during the reconnection, resulting in data loss. Summary of the invention

[0003] In order to solve the above technical problems, an embodiment of the present invention provides a message publishing method and device based on an MQTT server to solve the problem of partial data loss when a large amount of message data is sent on the same topic at the same time.

[0004] A first aspect of an embodiment of the present invention provides a message publishing method based on MQTT, the method comprising:

[0005] In response to the publishing operation of the publishing end, a message to be published is obtained, and the message to be published is segmented to obtain a plurality of message files;

[0006] Based on the current thread number and the current timestamp number of each message file, the segment number of each message file is obtained, and the sub-messages in each message file are numbered to obtain the number of each sub-message;

[0007] Send the sub-message to the subscriber so that the subscriber can consume it. If the subscriber consumes the sub-message, the segment number and serial number of the consumed sub-message are obtained.

[0008] According to the segment number of the consumed sub-message, the corresponding message file is found, and the corresponding consumed sub-message is found in the message file according to the number of the consumed sub-message, and the consumed sub-message is deleted. Other sub-messages in the message file continue to be sent to the subscriber until there are no sub-messages in the message file. Then, the message file is deleted, and the sub-messages in the undeleted message files continue to be sent to the subscription segment for consumption until all sub-messages in all message files are published.

[0009] In a possible implementation manner of the first aspect, the message to be published is segmented to obtain several messages, including:

[0010] Get the number of segments based on the maximum number of threads on the current server;

[0011] Based on the number of segments, the message to be published is segmented to obtain several messages.

[0012] In a possible implementation manner of the first aspect, if the subscriber consumes the message, obtaining the segment number and serial number of the consumed message includes:

[0013] If the subscriber consumes the message, the segment number and serial number of the consumed message are recorded, so that the subscriber can send the segment number and serial number of the consumed message to the server at a preset time;

[0014] Receive the segment number and serial number of the message after consumption.

[0015] In a possible implementation manner of the first aspect, after obtaining the serial number and segment number of each message file, the method further includes:

[0016] Each message file is saved in a local database according to the segment number and serial number, and the file name of each message file is named as the segment number of each message file.

[0017] A second aspect of an embodiment of the present invention provides a message publishing device based on an MQTT server, the device comprising:

[0018] A segmentation module is used to respond to the publishing operation of the publishing end, obtain the message to be published, and segment the message to be published to obtain a plurality of message files;

[0019] A numbering module is used to obtain the segment number of each message file based on the current thread number and the current timestamp number of each message file, and to number the sub-messages in each message file to obtain the number of each sub-message;

[0020] The consumption module is used to send the sub-message to the subscriber so that the subscriber can consume it. If the subscriber consumes the sub-message, the segment number and serial number of the consumed sub-message are obtained;

[0021] The deletion module is used to find the corresponding message file according to the segment number of the consumed sub-message, find the corresponding consumed sub-message in the message file according to the number of the consumed sub-message, delete the consumed sub-message, and continue to send other sub-messages in the message file to the subscriber until there are no sub-messages in the message file. Then, the message file is deleted, and the sub-messages in the undeleted message file are continued to be sent to the subscription segment for consumption until all the sub-messages in all message files are published.

[0022] In a possible implementation manner of the second aspect, the message to be published is segmented to obtain several messages, including:

[0023] Get the number of segments based on the maximum number of threads on the current server;

[0024] Based on the number of segments, the message to be published is segmented to obtain several messages.

[0025] In a possible implementation manner of the second aspect, if the subscriber consumes the message, obtaining the segment number and serial number of the consumed message includes:

[0026] If the subscriber consumes the message, the segment number and serial number of the consumed message are recorded, so that the subscriber can send the segment number and serial number of the consumed message to the server at a preset time;

[0027] Receive the segment number and serial number of the message after consumption.

[0028] In a possible implementation manner of the second aspect, after obtaining the serial number and segment number of each of the message files, the method further includes:

[0029] According to the segment number and serial number, each of the message files is saved in a local database, and the file name of each of the message files is named as the segment number of each of the message files.

[0030] A third aspect of an embodiment of the present invention provides a computing device, including:

[0031] Memory for storing computer programs;

[0032] A processor is used to implement the message publishing method based on MQTT as the first aspect when executing the computer program.

[0033] A fourth aspect of an embodiment of the present invention provides a non-temporary computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the MQTT-based message publishing method of the first aspect are implemented.

[0034] The embodiment of the present invention provides a method for obtaining a message to be published in response to a publishing operation of a publishing end, segmenting the message to be published to obtain a plurality of message files, obtaining the segment number of each message file based on the current thread number and the current timestamp number of each message file, numbering the sub-messages in each message file to obtain the number of each sub-message, and sending the sub-message to a subscriber end so that the subscriber end consumes it. If the subscriber end consumes the sub-message, the segment number and number of the consumed sub-message are obtained, and the corresponding message file is found according to the segment number of the consumed sub-message, and the corresponding consumed sub-message is searched in the message file according to the number of the consumed sub-message, and the consumed sub-message is deleted, and other sub-messages in the message file are continuously sent to the subscriber end until there are no sub-messages in the message file, then the message file is deleted, and the sub-messages in the undeleted message file are continuously sent to the subscription segment for consumption until all the sub-messages in all the message files are published. The method segments and numbers the message through multiple threads, so that the integrity of the data is ensured when pushing the message. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 : A flow chart of an embodiment of a message publishing method based on MQTT provided by the present invention;

[0036] Figure 2 : A device structure block diagram of an embodiment of the message publishing method based on MQTT provided by the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] Please refer to Figure 1 , which is a flow chart of an embodiment of a message publishing method based on MQTT provided in an embodiment of the present invention, including steps S101 to S104.

[0039] S101. In response to a publishing operation of a publishing end, a message to be published is obtained, and the message to be published is segmented to obtain a plurality of message files.

[0040] In this embodiment, before the publisher sends a large amount of data to the MQTT server on the same topic at the same time, the server first segments all messages to be sent to obtain multiple message files, each of which includes multiple messages.

[0041] In some embodiments, the step S101 of "segmenting the message to be published to obtain a plurality of message files" includes but is not limited to the following steps:

[0042] Get the number of segments based on the maximum number of threads on the current server;

[0043] Based on the number of segments, the message to be published is segmented to obtain a plurality of message files.

[0044] In this embodiment, when the messages to be published are segmented, 80% of the maximum number of threads of the server is used as the number of segments, that is, the number of files. If all threads are used for segmentation, the server performance will be greatly reduced. All messages are segmented according to the number of segments, and then the segmented messages are saved in corresponding local files to obtain multiple message files.

[0045] S102: Based on the current thread number and the current timestamp number of each message file, the segment number of each message file is obtained, and the sub-messages in each message file are numbered to obtain the number of each sub-message.

[0046] In this embodiment, the current thread ID and the current timestamp corresponding to each message file are used as the segment number, and the segment number is used as the file name of the corresponding message file. Then, the messages in each message file are numbered, and the numbering increases in the form of 01, 02, 03 according to the number of messages.

[0047] It should be noted that the sub-message refers to the message under the message file.

[0048] In some embodiments, "after obtaining the serial number and segment number of each message file" in step S102 includes but is not limited to the following steps:

[0049] Each message file is saved in a local database according to the segment number and serial number, and the file name of each message file is named as the segment number of each message file.

[0050] In this embodiment, all messages are segmented, and then the segmented messages are saved in corresponding local files to obtain multiple message files. Each message file is saved in a local database, and the file name of each message file is named as the segment number of each message file.

[0051] By saving message data locally, local operations can be performed directly without operating the system database, reducing the pressure on the system database.

[0052] S103: Send the sub-message to the subscriber so that the subscriber consumes it. If the subscriber consumes the sub-message, the segment number and serial number of the consumed sub-message are obtained.

[0053] In this embodiment, when the publisher sends a message to the topic, each message is accompanied by the segment number and number of the current message when it is stored in the local file. When the subscriber receives the message, each time the subscriber consumes a message, the subscriber records the segment number and number of the message, and sends the list of segment numbers and numbers of the consumed messages to the server, so that the server obtains the segment number and number of the consumed messages. Segmenting, numbering, and storing messages in files through multi-threading can greatly improve the processing speed and response time of the server. In addition, segment number and number processing of message data can ensure the uniqueness of message data and improve the rate of message retrieval.

[0054] In some embodiments, in step S103, "if the subscriber consumes the message, obtaining the segment number and serial number of the consumed message" includes but is not limited to the following steps:

[0055] If the subscriber consumes the message, the segment number and serial number of the consumed message are recorded, so that the subscriber can send the segment number and serial number of the consumed message to the server at a preset time;

[0056] Receive the segment number and serial number of the message after consumption.

[0057] In this embodiment, each time the subscriber consumes a message, the subscriber records the segment number and number of the message, and then requests the server's http interface at a fixed time to send a list of the segment numbers and numbers of the consumed messages to the server, so that the server obtains the segment numbers and numbers of the consumed messages.

[0058] After receiving the segment number and serial number sent by the subscriber, the server responds to the publisher.

[0059] It should be noted that the preset time is a fixed time, which is generally set to 3-5 seconds.

[0060] S104. Find the corresponding message file according to the segment number of the consumed sub-message, search the corresponding consumed sub-message in the message file according to the number of the consumed sub-message, delete the consumed sub-message, and continue to send other sub-messages in the message file to the subscriber until there are no sub-messages in the message file. Then delete the message file, and continue to send the sub-messages in the undeleted message file to the subscription segment for consumption until all the sub-messages in all the message files are published.

[0061] In this embodiment, the publisher first locates the message file by the segment number, then retrieves the current message according to the number, removes the message that has been consumed by the subscriber, and after removing the message that has been consumed by the subscriber, continues to send other sub-messages in the message file to the subscriber for consumption until there are no sub-messages in the message file, and then deletes the message file. The messages in the remaining files are messages that have not been successfully pushed. The server notifies the publisher to re-push the segment number and number of the message that has not been successfully sent. If there are still messages that have not been successfully pushed, repeat the above operation until all the message files in the local file are cleared, which means that all messages have been successfully consumed and have not been lost. This method can be used to push messages that have not been successfully pushed again when a large number of messages are sent, regardless of whether the message push failure is caused by a large amount of data or an abnormal server connection, until success, thereby ensuring data integrity.

[0062] As an example of this embodiment, if the publishing end needs to publish 1,000 messages, the message is segmented into 10 segments, which is 80% of the maximum number of threads on the server. Then, the message is divided into 10 message files, each of which contains 100 messages and has 100 numbers. When all the 100 messages in the file are consumed and deleted, the current file can be deleted.

[0063] Determine whether there are still messages in the message file. If there are messages, they are messages that have not been successfully pushed. If not, it means that all messages in the message file have been consumed, and the message file is deleted.

[0064] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0065] In one embodiment, Figure 2 As shown, it shows a block diagram of a message publishing device 200 based on MQTT provided in an embodiment of the present application, including a segmentation module 201, a numbering module 202, a consumption module 203 and a deletion module 204, wherein:

[0066] The segmentation module 201 is used to obtain a message to be published in response to a publishing operation of the publishing end, and segment the message to be published to obtain a plurality of message files;

[0067] A numbering module 202 is used to obtain the segment number of each message file based on the current thread number and the current timestamp number of each message file, and to number the sub-messages in each message file to obtain the number of each sub-message;

[0068] The consumption module 203 is used to send the sub-message to the subscriber so that the subscriber consumes it. If the subscriber consumes the sub-message, the segment number and serial number of the consumed sub-message are obtained;

[0069] The deletion module 204 is used to find the corresponding message file according to the segment number of the consumed sub-message, search the corresponding consumed sub-message in the message file according to the number of the consumed sub-message, delete the consumed sub-message, and continue to send other sub-messages in the message file to the subscriber until there are no sub-messages in the message file, then delete the message file, and continue to send the sub-messages in the undeleted message file to the subscription segment for consumption until all the sub-messages in all the message files are published.

[0070] In one embodiment, the message to be published is segmented to obtain several messages, including:

[0071] Get the number of segments based on the maximum number of threads on the current server;

[0072] Based on the number of segments, the message to be published is segmented to obtain several messages.

[0073] In one embodiment, if the subscriber consumes the message, the segment number and serial number of the consumed message are obtained, including:

[0074] If the subscriber consumes the message, the segment number and serial number of the consumed message are recorded, so that the subscriber can send the segment number and serial number of the consumed message to the server at a preset time;

[0075] Receive the segment number and serial number of the message after consumption.

[0076] In one embodiment, after obtaining the serial number and segment number of each message file, the method further includes:

[0077] Each message file is saved in a local database according to the segment number and serial number, and the file name of each message file is named as the segment number of each message file.

[0078] In one embodiment of the present application, a computing device is provided, which includes a memory and a processor, wherein a computer program is stored in the memory, and the above steps are implemented when the processor executes the computer program; the computing device provided in the embodiment of the present application has an implementation principle and technical effects similar to those of the above method embodiment, and will not be repeated here.

[0079] In one embodiment of the present application, a non-temporary computer-readable storage medium is provided, on which a computer program is stored, and the above steps are also implemented when the computer program is executed by a processor; the computer-readable storage medium provided in this embodiment has an implementation principle and technical effects similar to those of the above-mentioned method embodiment, and will not be repeated here.

[0080] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0081] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0082] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A message publishing method based on MQTT, characterized in that: include: In response to a publishing operation of the publishing end, a message to be published is obtained, and the message to be published is segmented to obtain a plurality of message files; Based on the current thread number and the current timestamp number of each message file, the segment number of each message file is obtained, and the sub-messages in each message file are numbered to obtain the number of each sub-message, wherein the current thread number is the current thread ID corresponding to each message file, the current timestamp number is the current timestamp corresponding to each message file, and the segment number is used as the file name of the message file; Sending the sub-message to a subscriber so that the subscriber consumes the sub-message. If the subscriber consumes the sub-message, obtaining the segment number and serial number of the consumed sub-message; According to the segment number of the consumed sub-message, the corresponding message file is found, and according to the number of the consumed sub-message, the corresponding consumed sub-message is searched in the message file, and the consumed sub-message is deleted, and other sub-messages in the message file are continuously sent to the subscriber end until there are no sub-messages in the message file, then the message file is deleted, and the sub-messages in the message file that have not been deleted are continuously sent to the subscriber end for consumption, until all the sub-messages in the message file are completely published; The message to be published is segmented to obtain a plurality of message files, including: Get the number of segments based on the maximum number of threads on the current server; Based on the number of segments, the message to be published is segmented to obtain a plurality of message files.

2. The message publishing method based on MQTT according to claim 1, characterized in that: If the subscriber consumes the message, the segment number and serial number of the consumed message are obtained, including: If the subscriber consumes the message, the segment number and serial number of the consumed message are recorded, so that the subscriber sends the segment number and serial number of the consumed message to the server at a preset time; Receive the segment number and serial number of the consumed message.

3. The message publishing method based on MQTT according to claim 1, characterized in that: After obtaining the segment number of each of the message files and the number of the sub-message in each of the message files, the method further includes: According to the segment number of each message file and the number of the sub-message in each message file, each message file is saved in a local database, and the file name of each message file is named according to the segment number of each message file.

4. A message publishing device based on MQTT, characterized in that: include: A segmentation module, used for obtaining a message to be published in response to a publishing operation of the publishing end, and segmenting the message to be published to obtain a plurality of message files; A numbering module, used to obtain the segment number of each message file based on the current thread number and current timestamp number of each message file, and number the sub-messages in each message file to obtain the number of each sub-message, wherein the current thread number is the current thread ID corresponding to each message file, the current timestamp number is the current timestamp corresponding to each message file, and the segment number is used as the file name of the message file; A consumption module, used for sending the sub-message to a subscriber so that the subscriber consumes the sub-message, and if the subscriber consumes the sub-message, obtaining the segment number and serial number of the consumed sub-message; A deletion module is used to find the corresponding message file according to the segment number of the consumed sub-message, search the corresponding consumed sub-message in the message file according to the number of the consumed sub-message, delete the consumed sub-message, and continue to send other sub-messages in the message file to the subscriber until there are no sub-messages in the message file, then delete the message file, and continue to send the sub-messages in the message file that have not been deleted to the subscriber for consumption until all the sub-messages in the message file are published; The message to be published is segmented to obtain several messages, including: Get the number of segments based on the maximum number of threads on the current server; Based on the number of segments, the message to be published is segmented to obtain a plurality of messages.

5. The message publishing device based on MQTT as claimed in claim 4, characterized in that: If the subscriber consumes the message, the segment number and serial number of the consumed message are obtained, including: If the subscriber consumes the message, the segment number and serial number of the consumed message are recorded, so that the subscriber sends the segment number and serial number of the consumed message to the server at a preset time; Receive the segment number and serial number of the consumed message.

6. The message publishing device based on MQTT as claimed in claim 4, characterized in that: After obtaining the segment number of each of the message files and the number of the sub-messages in each of the message files, the method further includes: According to the segment number of each message file and the number of the sub-message in each message file, each message file is saved in a local database, and the file name of each message file is named according to the segment number of each message file.

7. A computing device, characterized in that include: Memory for storing computer programs; A processor, configured to implement the MQTT-based message publishing method as described in any one of claims 1 to 3 when executing the computer program.

8. A non-transitory computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the message publishing method based on MQTT as described in any one of claims 1 to 3 is implemented.

Images