Distributed system file breakpoint resuming method based on MQTT communication

Abstract

The invention relates to the field of MQTT protocols, and discloses a distributed system file breakpoint resuming method and system based on MQTT communication, and the method comprises the steps: storing a file identifier and a file path of a to-be-sent file in Redis; querying a file path from a distributed shared memory Redis by using the file identifier, and collecting file contents of a to-be-sent file from a distributed file system; calculating the subpackage number of the to-be-sent file, and sending the to-be-sent file to a preset terminal device from the MQTT server; identifying the sent data, and transmitting the unsent data in the to-be-sent file to a preset terminal device by using the sent data so as to complete the breakpoint resume of the first file of the distributed file system; and storing a target file corresponding to the file downloading request into the distributed file system, and after the target file is transmitted from the distributed file system to a service system, finishing second file breakpoint resuming of the distributed file system. According to the invention, file breakpoint resume in a distributed environment can be realized.

Description

Technical Field

[0001] The present invention relates to the field of MQTT protocol, and in particular to a distributed system file breakpoint resume transmission method and system based on MQTT communication. Background Art

[0002] The existing IoT collection system uses the MQTT protocol to communicate with devices, and our terminal devices generally use 3G or 4G signals for communication, which has certain traffic restrictions. In addition to uploading monitoring data, terminal devices need to use network signals and traffic. In business applications, there is often a need to upload or send files, and the terminal device has certain restrictions on the byte size of each transmission (maximum 1KB each time), and it must also cooperate with the device to resume file transmission when traffic is idle or the CPU is idle. Generally, it is relatively simple and easy to implement the above requirements in a single-process system, while it is very difficult to process the distributed architecture system because each file request or file response is not fixed, so the reading, unpacking, sending, and sub-packaging of files are required. A comprehensive solution must not only consider the normal reception and distribution of files when they are sub-packaged, but also ensure that the service does not repeatedly upload file sub-packages and the device does not repeatedly request file sub-packages when resuming the download, which would result in traffic loss. Secondly, IoT platforms are all developed using a distributed architecture. In addition to supporting massive device connections and data collection, there are many business scenarios for uploading and downloading files from devices. For distributed platform applications, the same file processing may be split into multiple sub-packages, and requests and responses are processed on different servers. If there is no complete solution, then the file will be partially stored in different server directories, resulting in incomplete files, or file packet loss when network conditions are poor, and repeated transmission will waste the device's SIM card data. Therefore, a solution is urgently needed that can achieve stable and reliable transmission for both file uploads and downloads, and support the ability to continue the transmission from the last position after an interruption, so as to achieve breakpoint-resume file downloads in a distributed environment. Summary of the Invention

[0003] In order to solve the above problems, the present invention provides a distributed system file breakpoint resumption method and system based on MQTT communication, which can achieve stable and reliable transmission of file upload and download, and supports continuing the transmission from the last position after interruption, realizing file breakpoint resumption in a distributed environment.

[0004] In a first aspect, the present invention provides a method for resuming file transfer in a distributed system based on MQTT communication, comprising:

[0005] Transferring the file to be sent from the business system to the file service component, uploading the file to be sent to the distributed file system through the file service component, and after uploading the file to be sent to the distributed file system, storing the file identifier and file path of the file to be sent in the distributed shared memory Redis;

[0006] Querying the file path from the distributed shared memory Redis using the file identifier, and collecting the file content of the to-be-sent file from the distributed file system based on the file path;

[0007] Calculating the number of sub-packets of the file to be sent according to the file content, and sending the file to be sent from a preset MQTT server to a preset terminal device based on the number of sub-packets, and in the preset terminal device responding to the file to be sent to obtain file response data;

[0008] Identifying, according to the file response data, sent data of the file to be sent, and using the sent data to transmit unsent data in the file to be sent to the preset terminal device, so as to complete the breakpoint resume transfer of the first file of the distributed file system and obtain a first file breakpoint resume transfer result;

[0009] Sending a file download request from the file service component to the preset terminal device, so that the preset terminal device responds to the file download request and obtains request response data;

[0010] Based on the request response data, the target file corresponding to the file download request is stored in the distributed file system, and after the target file is transferred from the distributed file system to the business system, the second file breakpoint resume of the distributed file system is completed to obtain the second file breakpoint resume result.

[0011] In a possible implementation of the first aspect, storing the file identifier and the file path of the to-be-sent file in a distributed shared memory Redis includes:

[0012] Extracting the file identifier and file path of the file to be sent from the distributed file system;

[0013] The file identifier and the file path of the file to be sent are returned to the file service component, so that the file identifier and the file path of the file to be sent are cached in the distributed shared memory Redis through the file service component.

[0014] In a possible implementation of the first aspect, querying the file path from the distributed shared memory Redis using the file identifier includes:

[0015] Sending a file acquisition request from the file service component to a preset MQTT server;

[0016] In the preset MQTT server, based on the file acquisition request, extracting the file identifier of the file to be sent from the distributed file system;

[0017] The file path corresponding to the file identifier is queried from the distributed shared memory Redis.

[0018] In a possible implementation of the first aspect, calculating the number of subpackets of the file to be sent according to the file content includes:

[0019] According to the file content, the number of sub-packets of the file to be sent is calculated using the following formula:

[0020]

[0021] Wherein, n represents the number of subpackets, D represents the file size in the file content, and D0 represents the standard subpacket size.

[0022] In a possible implementation of the first aspect, sending the to-be-sent file from a preset MQTT server to a preset terminal device based on the subpacket quantity includes:

[0023] Splitting the file to be sent into multiple sub-packets based on the number of sub-packets;

[0024] Sending the multiple subpackets from the preset MQTT server to a preset terminal device according to the subpacketization order of the multiple subpackets;

[0025] Within a preset timer time, detecting whether a current sub-packet response of the plurality of sub-packets is received in the preset MQTT server;

[0026] When no current sub-package response of the plurality of sub-packages is received from the preset MQTT server within the preset timer, querying the last sub-package response received from the preset MQTT server, and identifying the last file offset from the last sub-package response;

[0027] Determining a to-be-retransmitted subpacket among the multiple subpackets based on the last file offset;

[0028] Calculating the retransmission delay time of the subpacket to be retransmitted;

[0029] Based on the retransmission delay time, after retransmitting the subpacket to be retransmitted to the preset terminal device, the last file offset is sent to the file service component, so as to instruct the preset MQTT server to continue transmitting the remaining subpackets after the subpacket to be retransmitted in the multiple subpackets to the preset terminal device through the file service component.

[0030] In a possible implementation of the first aspect, the calculating the retransmission delay time of the subpacket to be retransmitted includes:

[0031] The retransmission delay time of the subpacket to be retransmitted is calculated using the following formula:

[0032] t=2rτ

[0033] Wherein, t represents the retransmission delay time, r represents a discrete integer set {0, 1, ..., 2 k -1}, k represents the number of packet retransmissions, and τ represents the one-way propagation delay from the preset MQTT server to the preset terminal device.

[0034] In a possible implementation of the first aspect, storing the target file corresponding to the file download request in the distributed file system based on the request response data includes:

[0035] Using the file identifier in the request response data as a file keyword, constructing a hash storage structure of the target file;

[0036] Using the current file offset in the request response data as the hash address of the target file in the hash storage structure;

[0037] Based on the hash address, the target file is stored in the distributed file system using the hash storage structure.

[0038] In a possible implementation of the first aspect, transferring the target file from the distributed file system to the business system includes:

[0039] After transferring the file path of the target file from the distributed file system to the distributed shared memory Redis, each sub-package of the target file is transferred from the distributed file system to the file service component;

[0040] After collecting the file download progress of each sub-packet transmitted from the distributed file system to the file service component in the business system and transmitting the file download progress from the file service component to the distributed shared memory Redis, determining whether the transmission of the target file from the distributed file system to the file service component is completed;

[0041] When the transfer of the target file from the distributed file system to the file service component is completed, querying the file path of each sub-packet according to the file identifier of each sub-packet;

[0042] Sort each subpackage based on the file path of each subpackage to obtain a subpackage sequence;

[0043] Using a preset file operation function to splice each sub-package in the sub-package sequence to obtain a complete version of the file;

[0044] The complete version of the file is transferred from the file service component to the business system to complete the process of transferring the target file from the distributed file system to the business system.

[0045] In a possible implementation of the first aspect, determining whether the transfer of the target file from the distributed file system to the file service component is completed includes:

[0046] When the sum of the current file offset and the packet size of each sub-packet is greater than the total file size of the target file, it is determined that the transmission of the target file from the distributed file system to the file service component is completed.

[0047] In a second aspect, the present invention provides a distributed system file breakpoint resume transmission system based on MQTT communication, the system comprising:

[0048] A file storage module is used to transfer a file to be sent from a business system to a file service component, so as to upload the file to be sent to a distributed file system through the file service component, and after uploading the file to be sent to the distributed file system, store the file identifier and file path of the file to be sent in a distributed shared memory Redis;

[0049] A content collection module is used to query the file path from the distributed shared memory Redis using the file identifier, and based on the file path, collect the file content of the file to be sent from the distributed file system;

[0050] A sub-packet sending module is configured to calculate the number of sub-packets of the file to be sent according to the file content, and based on the number of sub-packets, send the file to be sent from a preset MQTT server to a preset terminal device, and in the preset terminal device, respond to the file to be sent to obtain file response data;

[0051] a first resuming transmission module, configured to identify, based on the file response data, sent data of the file to be sent, and use the sent data to transmit unsent data in the file to be sent to the preset terminal device, so as to complete the first file breakpoint resuming transmission of the distributed file system and obtain a first file breakpoint resuming transmission result;

[0052] A request response module, configured to send a file download request from the file service component to the preset terminal device, so as to respond to the file download request in the preset terminal device and obtain request response data;

[0053] The second resume transfer module is used to store the target file corresponding to the file download request in the distributed file system based on the request response data, and after transferring the target file from the distributed file system to the business system, complete the second file breakpoint resume transfer of the distributed file system to obtain the second file breakpoint resume transfer result.

[0054] Compared with the existing technology, the technical principle and beneficial effects of this solution are:

[0055] The embodiment of the present invention stores the file identifier and file path of the file to be sent in the distributed shared memory Redis, so as to be used in combination with the distributed file storage middleware FastDFS and the distributed shared memory Redis, and designs the cache structure of Redis, in which the MD5 of the file content is used as the unique identifier of the file to realize content storage. Furthermore, the embodiment of the present invention collects the file content of the file to be sent from the distributed file system based on the file path, so as to prepare for the subsequent setting of the number of sub-packets of the file. Furthermore, the embodiment of the present invention sends the file to be sent from the preset MQTT server to the preset terminal device based on the number of sub-packets, so as to wait for the device network to recover and re-trigger the request to continue transmitting the file when the device network information is not good and the line is disconnected, and send the packOffset of the last transmission to the file service component. The file service component continues to transmit the remaining part of the file according to the packOffset field in the parameter, thereby realizing the file breakpoint resumption of the file to be sent. The embodiment of the present invention responds to the request based on the request Data, the target file corresponding to the file download request is stored in the distributed file system, so that when the server downloads the file from the device, the device splits the file, converts the sub-package file into Base64 characters after serialization, and sends it to the target server in the form of a message through the MQTT protocol. After the target server receives the file sub-package data, it converts the Base64 characters into a Byte array and writes it into the FastDFS distributed file system, and writes the file information into the Redis cache structure to facilitate subsequent file merging, thereby realizing file transmission of the distributed system. Furthermore, the embodiment of the present invention transfers the target file from the distributed file system to the business system for use in a distributed system architecture. In the scenario where the distributed service and the terminal device perform file transmission through the MQTT protocol, due to the uncontrollable network conditions of the terminal device and the limitation of network traffic, a large file needs to be split into multiple sub-packages for transmission, and the distributed service is often deployed on the system side with multi-service redundant deployment. The load balancing method allows the files uploaded by the terminal device to be processed by different servers and finally merged into an original file. Therefore, the embodiment of the present invention proposes a distributed system file breakpoint resumption method and system based on MQTT communication, which can achieve stable and reliable transmission of file uploads and downloads, and supports continuing the transmission from the last position after interruption, thereby realizing file breakpoint resumption in a distributed environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0058] Figure 1 A flowchart of a method for resuming file transfer in a distributed system based on MQTT communication is provided in accordance with an embodiment of the present invention;

[0059] Figure 2 In one embodiment of the present invention Figure 1 A schematic diagram of a distributed system file resumable transfer method based on MQTT communication, in which the file identifier and file path are stored in the distributed shared memory Redis;

[0060] Figure 3 In one embodiment of the present invention Figure 1 A schematic diagram of file response data of a distributed system file resumable transfer method based on MQTT communication is provided;

[0061] Figure 4 In one embodiment of the present invention Figure 1 A schematic diagram of the sent data of a distributed system file resumable transmission method based on MQTT communication is provided;

[0062] Figure 5 In one embodiment of the present invention Figure 1 A schematic diagram of a distributed system file resumable transfer method based on MQTT communication, whereby files are sent from a business system to a terminal device;

[0063] Figure 6 In one embodiment of the present invention Figure 1 A schematic diagram of a file download request for a distributed system file resumable download method based on MQTT communication is provided;

[0064] Figure 7 In one embodiment of the present invention Figure 1 A schematic diagram of request and response data of a distributed system file resumable transfer method based on MQTT communication is provided;

[0065] Figure 8 In one embodiment of the present invention Figure 1 A schematic diagram of a distributed system file resumable transfer method based on MQTT communication in which a target file is stored in the distributed file system;

[0066] Figure 9 In one embodiment of the present invention Figure 1A schematic diagram of a distributed system file resumable download method based on MQTT communication, in which files are downloaded from a terminal device in sub-packages to a system platform;

[0067] Figure 10 A module diagram of a distributed system file breakpoint resume transmission system based on MQTT communication provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0068] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0069] An embodiment of the present invention provides a method for resuming file transmission in a distributed system based on MQTT communication. The execution subject of the method for resuming file transmission in a distributed system based on MQTT communication includes but is not limited to at least one of electronic devices such as a server and a terminal that can be configured to execute the method provided by the embodiment of the present invention. In other words, the method for resuming file transmission in a distributed system based on MQTT communication can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0070] See Figure 1 FIG. 1 is a flow chart of a method for resuming file transfer in a distributed system based on MQTT communication according to an embodiment of the present invention. Figure 1 The distributed system file resumable transfer method based on MQTT communication described in includes:

[0071] S1. Transfer the file to be sent from the business system to the file service component, so that the file to be sent is uploaded to the distributed file system through the file service component, and after uploading the file to be sent to the distributed file system, store the file identifier and file path of the file to be sent in the distributed shared memory Redis.

[0072] In the embodiment of the present invention, the business system refers to a system for receiving files to be sent by users. Furthermore, the file service component refers to a file center service that can be deployed independently, providing file upload, file download, file preview and other services to the user end.

[0073] Furthermore, in an embodiment of the present invention, the distributed file system refers to FastDFS, which is an open source lightweight distributed file system that solves problems such as large data storage and load balancing. FastDFS manages files, and its functions include file storage, file synchronization, file access (file upload, file download, file deletion), etc.

[0074] Furthermore, an embodiment of the present invention stores the file identifier and file path of the file to be sent in the distributed shared memory Redis, so as to combine the distributed file storage middleware FastDFS and the distributed shared memory Redis, and designs the cache structure of Redis, in which the MD5 of the file content is used as the unique identifier of the file to realize content storage.

[0075] The distributed shared memory Redis refers to the Redis database.

[0076] In one embodiment of the present invention, storing the file identifier and file path of the file to be sent in the distributed shared memory Redis includes: extracting the file identifier and file path of the file to be sent from the distributed file system; returning the file identifier and file path of the file to be sent to the file service component, so that the file identifier and file path of the file to be sent are cached in the distributed shared memory Redis through the file service component.

[0077] See Figure 2 As shown, an embodiment of the present invention Figure 1 A schematic diagram of a distributed system file breakpoint resume method based on MQTT communication in which the file identifier and file path are stored in the distributed shared memory Redis. Figure 2 In [1], Redis Key is used to record the file identifier, the Md5 value of the file content is the file identifier, for example, e799bae41a379c3093100265ea52fae is the file identifier, and Redis Value is used to record the file path, for example, group1 / M00 / 04 / D0 / CgBuYGVcZJyAWz1HAACwt6q3odk729.xml is the file path.

[0078] S2. Query the file path from the distributed shared memory Redis using the file identifier, and collect the file content of the file to be sent from the distributed file system based on the file path.

[0079] In one embodiment of the present invention, the use of the file identifier to query the file path from the distributed shared memory Redis includes: sending a file acquisition request from the file service component to a preset MQTT server; in the preset MQTT server, based on the file acquisition request, extracting the file identifier of the file to be sent from the distributed file system; and querying the file path corresponding to the file identifier from the distributed shared memory Redis.

[0080] For example, since the platform's MQTT consumer service is deployed in a cluster, the service node data is >= 1, and the MQTT message consumption adopts a load balancing algorithm (polling). The request responded by the device is randomly received by one of the MQTT consumer services on the platform. At this time, it only needs to query Redis according to the fileMd5 value in the parameter. The Redis key is: file:md5:upload:{fileMd5} The file path (that is, the file path saved in FastDFS)

[0081] Furthermore, the embodiment of the present invention collects the file content of the file to be sent from the distributed file system based on the file path, so as to prepare for the subsequent setting of the number of sub-packaging of the file.

[0082] The file content includes the file itself and the file size.

[0083] S3. Calculate the number of subpackets of the file to be sent according to the file content, and based on the number of subpackets, send the file to be sent from the preset MQTT server to the preset terminal device, and in the preset terminal device, respond to the file to be sent to obtain file response data.

[0084] In one embodiment of the present invention, calculating the number of sub-packets of the file to be sent according to the file content includes: calculating the number of sub-packets of the file to be sent according to the file content using the following formula:

[0085]

[0086] Wherein, n represents the number of subpackets, D represents the file size in the file content, and D0 represents the standard subpacket size.

[0087] Furthermore, the embodiment of the present invention sends the file to be sent from the preset MQTT server to the preset terminal device based on the number of subpackets, so that when the device network information is poor and the device is disconnected, it waits for the device network to recover, re-triggers the request to continue transmitting the file, and sends the packOffset of the last transmission to the file service component. The file service component continues to transmit the remaining part of the file according to the packOffset field in the parameter, thereby realizing the file breakpoint resumption of the file to be sent.

[0088] The preset MQTT server refers to a server based on the MQTT service, and the MQTT service is based on the publish / subscribe mode for communication and data exchange.

[0089] In one embodiment of the present invention, the sending of the to-be-sent file from the preset MQTT server to the preset terminal device based on the number of sub-packets includes: splitting the to-be-sent file into multiple sub-packets based on the number of sub-packets; sending the multiple sub-packets from the preset MQTT server to the preset terminal device according to the sub-packetization order of the multiple sub-packets; detecting whether a current sub-packet response of the multiple sub-packets is received in the preset MQTT server within a preset timer time; and if no current sub-packet response of the multiple sub-packets is received in the preset MQTT server within the preset timer time, Query the last subpacket response received in the preset MQTT server, and identify the last file offset from the last subpacket response; determine the subpacket to be retransmitted among the multiple subpackets based on the last file offset; calculate the retransmission delay time of the subpacket to be retransmitted; based on the retransmission delay time, after retransmitting the subpacket to be retransmitted to the preset terminal device, send the last file offset to the file service component, so as to instruct the preset MQTT server to continue transmitting the remaining subpackets after the subpacket to be retransmitted among the multiple subpackets to the preset terminal device through the file service component.

[0090] In another embodiment of the present invention, the calculating the retransmission delay time of the subpacket to be retransmitted includes: calculating the retransmission delay time of the subpacket to be retransmitted using the following formula:

[0091] t=2rτ

[0092] Wherein, t represents the retransmission delay time, r represents a discrete integer set {0, 1, ..., 2 k -1}, k represents the number of packet retransmissions, and τ represents the one-way propagation delay from the preset MQTT server to the preset terminal device.

[0093] Exemplarily, when the subpacket is retransmitted for the first time, k=1, and the random number r is selected from {0, 1}; when the subpacket is retransmitted for the second time, k=2, and the random number r is selected from {0, 1, 3}.

[0094] See Figure 3 As shown, an embodiment of the present invention Figure 1 A schematic diagram of file response data of a distributed system file breakpoint resume method based on MQTT communication is provided. Figure 3 In the example, the file response data refers to a sub-packet response corresponding to each sub-packet when each sub-packet in the multiple sub-packets is received, and the file response data includes file Md5, packet size and current file offset.

[0095] S4. According to the file response data, the sent data of the file to be sent is identified, and the unsent data in the file to be sent is transmitted to the preset terminal device using the sent data to complete the first file breakpoint resume transmission of the distributed file system and obtain the first file breakpoint resume transmission result.

[0096] It should be noted that, in one embodiment of the present invention, the principle of identifying the sent data of the file to be sent based on the file response data is similar to the principle of determining the subpackets to be retransmitted among the multiple subpackets based on the last file offset, and will not be further explained here.

[0097] See Figure 4 As shown, an embodiment of the present invention Figure 1 A schematic diagram of the sent data of a distributed system file breakpoint resume transmission method based on MQTT communication is provided. Figure 4 In the figure, 42 / 45 indicates that the number of the multiple subpackets is 45 and the number of the sent data is 42.

[0098] See Figure 5 As shown, an embodiment of the present invention Figure 1 A schematic diagram of a distributed system file resumable transfer method based on MQTT communication is provided, showing a file being sent from a business system to a terminal device.

[0099] S5. Send a file download request from the file service component to the preset terminal device, so that the file download request is responded to in the preset terminal device to obtain request response data.

[0100] See Figure 6 As shown, an embodiment of the present invention Figure 1 A schematic diagram of a file download request for a distributed system file resumable download method based on MQTT communication is provided. Figure 6The file download request includes the file type, file offset, and package size.

[0101] See Figure 7 As shown, an embodiment of the present invention Figure 1 A schematic diagram of request and response data of a distributed system file breakpoint resume method based on MQTT communication is provided. Figure 7 The request response data includes the file Md5, the total file size, the package size, the current file offset, and the binary content of the file Base64.

[0102] S6. Based on the request response data, the target file corresponding to the file download request is stored in the distributed file system. After the target file is transferred from the distributed file system to the business system, the second file breakpoint resume of the distributed file system is completed to obtain a second file breakpoint resume result.

[0103] The embodiment of the present invention stores the target file corresponding to the file download request in the distributed file system based on the request response data, so that when the server downloads the file from the device, the device splits the file, converts the sub-package file into Base64 characters after serialization, and sends it to the target server in the form of a message through the MQTT protocol. After the target server receives the file sub-package data, it converts the Base64 characters into a Byte array and writes it into the FastDFS distributed file system, and writes the file information into the Redis cache structure to facilitate subsequent file merging, thereby realizing file transmission in a distributed system.

[0104] In one embodiment of the present invention, storing the target file corresponding to the file download request in the distributed file system based on the request response data includes: using the file identifier in the request response data as a file keyword to construct a hash storage structure of the target file; using the current file offset in the request response data as the hash address of the target file in the hash storage structure; and based on the hash address, using the hash storage structure to store the target file in the distributed file system.

[0105] For example, since the platform's Mqtt consumer service is clustered and deployed, the service node data is >= 1, and the Mqtt message consumption uses a load balancing algorithm (polling). The device's response request is randomly received by one of the platform's consumer services. At this time, it needs to construct a Hash storage structure with file:md5:download:{fileMd5} as the key based on the fileMd5 value in the parameter to save the content of each subpackage file. The data in the Hash structure uses the current file offset value as the HashKey, and the path where the subpackage file is saved in FastDFS is the HashValue.

[0106] See Figure 8 As shown, an embodiment of the present invention Figure 1 A schematic diagram of a distributed system file resuming method based on MQTT communication in which a target file is stored in the distributed file system. Figure 8 In the expression, key represents the current file offset of each sub-package file, value represents the file path of each sub-package file, and file:md5:download:{fileMd5} represents the file keyword.

[0107] Furthermore, the embodiment of the present invention transfers the target file from the distributed file system to the business system for use in a distributed system architecture, where distributed services and terminal devices perform file transfers via the MQTT protocol. Due to the uncontrollable network conditions of the terminal device and the limitation of network traffic, a large file needs to be split into multiple sub-packets for transmission. The distributed service is often deployed on the system side with multi-service redundant deployment and load balancing, which allows the files uploaded by the terminal device to be processed by different servers and finally merged into an original file.

[0108] In one embodiment of the present invention, transferring the target file from the distributed file system to the business system includes: after transferring the file path of the target file from the distributed file system to the distributed shared memory Redis, transferring each subpackage of the target file from the distributed file system to the file service component; collecting, in the business system, the file download progress of each subpackage transferred from the distributed file system to the file service component, and after transferring the file download progress from the file service component to the distributed shared memory Redis, determining whether the transfer of the target file from the distributed file system to the file service component is complete; when the transfer of the target file from the distributed file system to the file service component is complete, querying the file path of each subpackage based on the file identifier of each subpackage; sorting each subpackage based on the file path of each subpackage to obtain a subpackage sequence; splicing each subpackage in the subpackage sequence using a preset file operation function to obtain a complete version of the file; and transferring the complete version of the file from the file service component to the business system to complete the process of transferring the target file from the distributed file system to the business system.

[0109] In another embodiment of the present invention, determining whether the transmission of the target file from the distributed file system to the file service component is completed includes: when the sum of the current file offset and the packet size of each subpacket is greater than the total file size of the target file, determining that the transmission of the target file from the distributed file system to the file service component is completed.

[0110] Optionally, store the file sub-package path in Redis, and write the file download progress to Redis based on the current file writing progress, so that the business end can query the progress. Further, merge the files: wait until the last package is fully transmitted (the judgment logic for the last package transmission is: packOffset+packSize>fileSize), assemble RedisKey file:md5:download:{fileMd5} according to the fileMd5 of the file, query the paths of all sub-package files stored in the Hash, sort by key (that is, file offset), use file operation functions to sequentially splice the file byte arrays of all files into a complete file, and then output the file to the business system in a streaming manner for use.

[0111] See Figure 9 As shown, an embodiment of the present invention Figure 1 A schematic diagram of a distributed system file resumable download method based on MQTT communication is provided, in which files are downloaded in subpackages from a terminal device to a system platform.

[0112] It can be seen that the embodiment of the present invention stores the file identifier and file path of the file to be sent in the distributed shared memory Redis, so as to be used in combination with the distributed file storage middleware FastDFS and the distributed shared memory Redis, and designs the cache structure of Redis, in which the MD5 of the file content is used as the unique identifier of the file to achieve content storage. Furthermore, the embodiment of the present invention collects the file content of the file to be sent from the distributed file system based on the file path, so as to prepare for the subsequent setting of the number of sub-packets of the file. Furthermore, the embodiment of the present invention sends the file to be sent from the preset MQTT server to the preset terminal device based on the number of sub-packets, so as to be used for waiting for the device network to recover after the device network is restored and a request to continue transmitting the file is re-triggered when a disconnection occurs due to poor device network information, and the packOffset of the last transmission is sent to the file service component. The file service component continues to transmit the remaining part of the file according to the packOffset field in the parameter, thereby achieving the file breakpoint resumption of the file to be sent. The embodiment of the present invention is based on the request In response data, the target file corresponding to the file download request is stored in the distributed file system, so that when the server downloads the file from the device, the device splits the file, serializes the sub-package file and converts it into Base64 characters, and sends it to the target server in the form of a message through the MQTT protocol. After the target server receives the file sub-package data, it converts the Base64 characters into a Byte array and writes it into the FastDFS distributed file system, and writes the file information into the Redis cache structure to facilitate subsequent file merging, thereby realizing file transmission in a distributed system. Furthermore, the embodiment of the present invention transfers the target file from the distributed file system to the business system for use in a distributed system architecture. In the scenario where distributed services and terminal devices perform file transmission through the MQTT protocol, due to the uncontrollable network conditions of the terminal device and the limitation of network traffic, a large file needs to be split into multiple sub-packages for transmission, and the distributed service is often deployed on the system side with multiple service redundancy. The load balancing method allows the files uploaded by the terminal device to be processed by different servers and finally merged into an original file. Therefore, the distributed system file resuming method based on MQTT communication proposed in an embodiment of the present invention can achieve stable and reliable transmission of file uploads and downloads, and supports continuing the transmission from the last position after interruption, thereby realizing file resuming in a distributed environment.

[0113] like Figure 10 The figure shows a functional module diagram of a distributed system file breakpoint resume transmission system based on MQTT communication according to the present invention.

[0114] The distributed system file breakpoint-resume transmission system 1000 based on MQTT communication described in the present invention can be installed in an electronic device. Depending on the functions implemented, the distributed system file breakpoint-resume transmission system based on MQTT communication can include a file storage module 1001, a content acquisition module 1002, a packet sending module 1003, a first resume transmission module 1004, a request response module 1005, and a second resume transmission module 1006. The module described in the present invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can perform fixed functions, which are stored in the memory of the electronic device.

[0115] In the embodiment of the present invention, the functions of each module / unit are as follows:

[0116] The file storage module 1001 is used to transfer the file to be sent from the business system to the file service component, so as to upload the file to be sent to the distributed file system through the file service component, and after uploading the file to be sent to the distributed file system, store the file identifier and file path of the file to be sent in the distributed shared memory Redis;

[0117] The content collection module 1002 is configured to query the file path from the distributed shared memory Redis using the file identifier, and collect the file content of the file to be sent from the distributed file system based on the file path;

[0118] The sub-packet sending module 1003 is configured to calculate the number of sub-packets of the file to be sent according to the file content, and based on the number of sub-packets, send the file to be sent from the preset MQTT server to a preset terminal device, and in the preset terminal device, respond to the file to be sent to obtain file response data;

[0119] The first resuming module 1004 is configured to identify the sent data of the file to be sent based on the file response data, and use the sent data to transmit the unsent data in the file to be sent to the preset terminal device, so as to complete the first file breakpoint resume transmission of the distributed file system and obtain a first file breakpoint resume transmission result;

[0120] The request response module 1005 is configured to send a file download request from the file service component to the preset terminal device, so as to respond to the file download request in the preset terminal device and obtain request response data;

[0121] The second resume module 1006 is used to store the target file corresponding to the file download request in the distributed file system based on the request response data, and after transferring the target file from the distributed file system to the business system, complete the second file resume transmission of the distributed file system to obtain the second file resume transmission result.

[0122] In detail, the modules in the distributed system file breakpoint resume transmission system 1000 based on MQTT communication in the embodiment of the present invention adopt the same Figure 1 The same technical means are used as the method for resuming file breakpoint transmission in a distributed system based on MQTT communication, and can produce the same technical effects, so I will not go into details here.

[0123] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or hardware plus software functional modules.

[0124] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0125] Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims rather than the foregoing description, and all changes that come within the meaning and range of equivalents of the claims are intended to be embraced therein. Any reference to a figure in a claim should not be construed as limiting the claim to which it relates.

[0126] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0127] The foregoing description is intended only to provide specific embodiments of the present invention, which will enable those skilled in the art to understand and implement the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not intended to be limited to the embodiments shown herein, but is to be construed in the widest possible manner consistent with the principles and novel features disclosed herein.

Claims

1. A method for resuming file transfer in a distributed system based on MQTT communication, characterized in that: The method comprises: Transferring the file to be sent from the business system to the file service component, uploading the file to be sent to the distributed file system through the file service component, and after uploading the file to be sent to the distributed file system, storing the file identifier and file path of the file to be sent in the distributed shared memory Redis; Querying the file path from the distributed shared memory Redis using the file identifier, and collecting the file content of the to-be-sent file from the distributed file system based on the file path; Calculating the number of sub-packets of the file to be sent according to the file content, and sending the file to be sent from a preset MQTT server to a preset terminal device based on the number of sub-packets, and in the preset terminal device responding to the file to be sent to obtain file response data; Identifying, according to the file response data, sent data of the file to be sent, and using the sent data to transmit unsent data in the file to be sent to the preset terminal device, so as to complete the breakpoint resume transfer of the first file of the distributed file system and obtain a first file breakpoint resume transfer result; Sending a file download request from the file service component to the preset terminal device, so that the preset terminal device responds to the file download request and obtains request response data; Based on the request response data, the target file corresponding to the file download request is stored in the distributed file system, and after the target file is transferred from the distributed file system to the business system, the second file breakpoint resume of the distributed file system is completed to obtain the second file breakpoint resume result.

2. The method according to claim 1, characterized in that The step of storing the file identifier and the file path of the file to be sent in the distributed shared memory Redis includes: Extracting the file identifier and file path of the file to be sent from the distributed file system; The file identifier and the file path of the file to be sent are returned to the file service component, so that the file identifier and the file path of the file to be sent are cached in the distributed shared memory Redis through the file service component.

3. The method according to claim 1, characterized in that The using the file identifier to query the file path from the distributed shared memory Redis includes: Sending a file acquisition request from the file service component to a preset MQTT server; In the preset MQTT server, based on the file acquisition request, extracting the file identifier of the file to be sent from the distributed file system; The file path corresponding to the file identifier is queried from the distributed shared memory Redis.

4. The method according to claim 1, wherein Calculating the number of sub-packets of the file to be sent according to the file content includes: According to the file content, the number of sub-packets of the file to be sent is calculated using the following formula: Wherein, n represents the number of subpackets, D represents the file size in the file content, and D0 represents the standard subpacket size.

5. The method according to claim 1, wherein The sending of the to-be-sent file from the preset MQTT server to a preset terminal device based on the subpacket quantity includes: Splitting the file to be sent into multiple sub-packets based on the number of sub-packets; Sending the multiple subpackets from the preset MQTT server to a preset terminal device according to the subpacketization order of the multiple subpackets; Within a preset timer time, detecting whether a current sub-packet response of the plurality of sub-packets is received in the preset MQTT server; When no current sub-package response of the plurality of sub-packages is received from the preset MQTT server within the preset timer, querying the last sub-package response received from the preset MQTT server, and identifying the last file offset from the last sub-package response; Determining a to-be-retransmitted subpacket among the multiple subpackets based on the last file offset; Calculating the retransmission delay time of the subpacket to be retransmitted; Based on the retransmission delay time, after retransmitting the subpacket to be retransmitted to the preset terminal device, the last file offset is sent to the file service component, so as to instruct the preset MQTT server to continue transmitting the remaining subpackets after the subpacket to be retransmitted in the multiple subpackets to the preset terminal device through the file service component.

6. The method according to claim 5, characterized in that The calculating the retransmission delay time of the subpacket to be retransmitted includes: The retransmission delay time of the subpacket to be retransmitted is calculated using the following formula: t=2rτ Wherein, t represents the retransmission delay time, r represents a discrete integer set {0, 1, ..., 2 k -1}, k represents the number of packet retransmissions, and τ represents the one-way propagation delay from the preset MQTT server to the preset terminal device.

7. The method according to claim 1, characterized in that The storing, based on the request response data, the target file corresponding to the file download request in the distributed file system includes: Using the file identifier in the request response data as a file keyword, constructing a hash storage structure of the target file; Using the current file offset in the request response data as the hash address of the target file in the hash storage structure; Based on the hash address, the target file is stored in the distributed file system using the hash storage structure.

8. The method according to claim 1, characterized in that The transferring of the target file from the distributed file system to the business system includes: After transferring the file path of the target file from the distributed file system to the distributed shared memory Redis, each sub-package of the target file is transferred from the distributed file system to the file service component; After collecting the file download progress of each sub-packet transmitted from the distributed file system to the file service component in the business system and transmitting the file download progress from the file service component to the distributed shared memory Redis, determining whether the transmission of the target file from the distributed file system to the file service component is completed; When the transfer of the target file from the distributed file system to the file service component is completed, querying the file path of each sub-packet according to the file identifier of each sub-packet; Sort each subpackage based on the file path of each subpackage to obtain a subpackage sequence; Using a preset file operation function to splice each sub-package in the sub-package sequence to obtain a complete version of the file; The complete version of the file is transferred from the file service component to the business system to complete the process of transferring the target file from the distributed file system to the business system.

9. The method according to claim 8, characterized in that The determining whether the transfer of the target file from the distributed file system to the file service component is completed includes: When the sum of the current file offset and the packet size of each sub-packet is greater than the total file size of the target file, it is determined that the transmission of the target file from the distributed file system to the file service component is completed.

10. A distributed system file breakpoint resume transmission system based on MQTT communication, characterized in that: The system comprises: A file storage module is used to transfer a file to be sent from a business system to a file service component, so as to upload the file to be sent to a distributed file system through the file service component, and after uploading the file to be sent to the distributed file system, store the file identifier and file path of the file to be sent in a distributed shared memory Redis; A content collection module is used to query the file path from the distributed shared memory Redis using the file identifier, and based on the file path, collect the file content of the file to be sent from the distributed file system; A sub-packet sending module is configured to calculate the number of sub-packets of the file to be sent according to the file content, and based on the number of sub-packets, send the file to be sent from a preset MQTT server to a preset terminal device, and in the preset terminal device, respond to the file to be sent to obtain file response data; a first resuming transmission module, configured to identify, based on the file response data, sent data of the file to be sent, and use the sent data to transmit unsent data in the file to be sent to the preset terminal device, so as to complete the first file breakpoint resuming transmission of the distributed file system and obtain a first file breakpoint resuming transmission result; A request response module, configured to send a file download request from the file service component to the preset terminal device, so as to respond to the file download request in the preset terminal device and obtain request response data; The second resume transfer module is used to store the target file corresponding to the file download request in the distributed file system based on the request response data, and after transferring the target file from the distributed file system to the business system, complete the second file breakpoint resume transfer of the distributed file system to obtain the second file breakpoint resume transfer result.

Images