A file breakpoint resume transmission method based on MQTT communication of a distributed system

Abstract

The application relates to the field of MQTT protocols and discloses a distributed system file breakpoint resume transmission method and system based on MQTT communication, which comprises the following steps: storing the file identification and file path of a to-be-sent file into Redis; querying the file path from the distributed shared memory Redis by using the file identification, collecting the file content of the to-be-sent file from a distributed file system; calculating the packet quantity of the to-be-sent file, sending the to-be-sent file to a preset terminal device from an MQTT server; identifying the sent data, transmitting the unsent data in the to-be-sent file to the preset terminal device by using the sent data, so as to complete the first file breakpoint resume transmission of the distributed file system; storing the target file corresponding to a file download request into the distributed file system, and after the target file is transmitted from the distributed file system to a business system, the second file breakpoint resume transmission of the distributed file system is completed. The application can realize file breakpoint resume transmission in a distributed environment.

Description

Technical Field

[0001] This invention relates to the field of MQTT protocol, and in particular to a method and system for resuming interrupted file transfers in a distributed system based on MQTT communication. Background Technology

[0002] Existing IoT data acquisition systems communicate with devices using the MQTT protocol, while our terminal devices typically use 3G or 4G signals for communication, which have certain bandwidth limitations. Besides uploading monitoring data, which requires network signal and bandwidth, terminal devices frequently need to upload or download files in business applications. Furthermore, terminal devices have certain limitations on the size of each transmission (maximum 1KB per transmission), and it's necessary to coordinate with the device to resume interrupted file transfers when bandwidth or CPU usage is idle. Implementing these requirements is relatively simple in a single-process system, but very difficult in a distributed architecture because each file request or response is not fixed. Therefore, file reading, disassembling, and sending, as well as file packet reception and storage, all require... A comprehensive solution must consider both the normal reception and transmission of file packets during fragmented downloads and the prevention of repeated file packet uploads and requests from devices during interrupted downloads, thus avoiding bandwidth consumption. Secondly, IoT platforms are typically developed using distributed architectures. Besides supporting massive device connections and data collection, there are many business scenarios involving file uploads and downloads. In distributed platform applications, the processing of a single file may be split into multiple packets, with requests and responses processed on different servers. Without a complete solution, files may be partially stored in different server directories, resulting in incomplete files, or packet loss under poor network conditions, leading to wasted SIM card bandwidth through repeated transmissions. Therefore, a solution is urgently needed that enables stable and reliable file uploads and downloads, and supports resuming transmission from the previous position after interruption, achieving file interrupted downloads in a distributed environment. Summary of the Invention

[0003] To address the aforementioned issues, this invention provides a method and system for resuming interrupted file transfers in a distributed system based on MQTT communication. This method enables stable and reliable file uploads and downloads, and also supports resuming the transfer from the previous position after an interruption, thus achieving file resuming in a distributed environment.

[0004] In a first aspect, the present invention provides a method for resuming interrupted file transfers in a distributed system based on MQTT communication, comprising:

[0005] The file to be sent is transferred from the business system to the file service component, so that the file service component can upload the file to the distributed file system. After the file to be sent is uploaded to the distributed file system, the file identifier and file path of the file to be sent are stored in the distributed shared memory Redis.

[0006] The file path is queried from the distributed shared memory Redis using the file identifier, and the file content of the file to be sent is collected from the distributed file system based on the file path;

[0007] Based on the file content, the number of packets of the file to be sent is calculated. Based on the number of packets, the file to be sent is sent from a preset MQTT server to a preset terminal device. The preset terminal device responds to the file to be sent and obtains file response data.

[0008] Based on the file response data, the sent data of the file to be sent is identified, and the unsent data in the file to be sent is transmitted to the preset terminal device using the sent data to complete the first file breakpoint resume transmission of the distributed file system and obtain the first file breakpoint resume transmission result.

[0009] The file service component sends a file download request to the preset terminal device, and the preset terminal device responds to the file download request to obtain request response data;

[0010] Based on the request-response data, the target file corresponding to the file download request is stored in the distributed file system. After the target file is transferred from the distributed file system to the business system, the second file breakpoint resume transfer of the distributed file system is completed, and the second file breakpoint resume transfer result is obtained.

[0011] In one possible implementation of the first aspect, storing the file identifier and file path of the file to be sent in a distributed shared memory Redis includes:

[0012] Extract the file identifier and file path of the file to be sent from the distributed file system;

[0013] The file identifier and file path of the file to be sent are returned to the file service component, so that the file service component can cache the file identifier and file path of the file to be sent in the distributed shared memory Redis.

[0014] In one possible implementation of the first aspect, querying the file path from the distributed shared memory Redis using the file identifier includes:

[0015] The file service component sends a file retrieval request to the preset MQTT server;

[0016] In the preset MQTT server, based on the file retrieval request, the file identifier of the file to be sent is extracted from the distributed file system;

[0017] Query the file path corresponding to the file identifier from the distributed shared memory Redis.

[0018] In one possible implementation of the first aspect, calculating the number of packets of the file to be sent based on the file content includes:

[0019] Based on the file content, the number of packets in the file to be sent can be calculated using the following formula:

[0020]

[0021] Where n represents the number of sub-packages, D represents the file size in the file content, and D0 represents the standard sub-package size.

[0022] In one possible implementation of the first aspect, sending the file to be sent from a preset MQTT server to a preset terminal device based on the number of packets includes:

[0023] Based on the number of packets, the file to be sent is split into multiple packets;

[0024] According to the packet sub-packet order, the multiple packets are sent from the preset MQTT server to the preset terminal device;

[0025] Within a preset timer period, it is detected whether the current packet response of the multiple packets has been received in the preset MQTT server;

[0026] If no current packet response is received from the preset MQTT server within the preset timer period, query the previous packet response received from the preset MQTT server and identify the previous file offset from the previous packet response;

[0027] Based on the previous file offset, determine the packets to be retransmitted among the multiple packets;

[0028] Calculate the retransmission delay time of the packet to be retransmitted;

[0029] Based on the retransmission delay time, after retransmitting the packet to be retransmitted to the preset terminal device, the previous file offset is sent to the file service component, so that the file service component instructs the preset MQTT server to continue transmitting the remaining packets after the packet to be retransmitted in the multiple packets to the preset terminal device.

[0030] In one possible implementation of the first aspect, calculating the retransmission delay time of the packet to be retransmitted includes:

[0031] The retransmission delay time of the packet to be retransmitted is calculated using the following formula:

[0032] t=2rτ

[0033] Where t represents the retransmission delay time, and r represents the time from the discrete set of integers {0, 1, ..., 2}. k A number is randomly selected from {-1}, k represents the number of packet retransmissions, and τ represents the one-way propagation delay from the preset MQTT server to the preset terminal device.

[0034] In one possible implementation of the first aspect, storing the target file corresponding to the file download request in the distributed file system based on the request-response data includes:

[0035] The file identifier in the request-response data is used as the file key to construct the hash storage structure of the target file;

[0036] The current file offset in the request-response data is used as the hash address of the target file in the hash storage structure;

[0037] Based on the hash address, the target file is stored in the distributed file system using the hash storage structure.

[0038] In one possible implementation of the first aspect, transferring the target file from the distributed file system to the business system includes:

[0039] After transferring the file path of the target file from the distributed file system to the distributed shared memory Redis, each sub-packet of the target file is transferred from the distributed file system to the file service component;

[0040] After collecting the file download progress of each sub-packet from the distributed file system to the file service component in the business system, and transmitting the file download progress from the file service component to the distributed shared memory Redis, it is determined whether the transmission of the target file from the distributed file system to the file service component is complete.

[0041] When the transfer of the target file from the distributed file system to the file service component is completed, the file path of each sub-packet is queried according to the file identifier of each sub-packet;

[0042] Based on the file path of each sub-package, sort each sub-package to obtain a sub-package sequence;

[0043] By using preset file operation functions, each sub-packet in the sub-packet sequence is concatenated to obtain the complete file;

[0044] The complete version of the file is transferred from the file service component to the business system to complete the process of transferring the target file from the distributed file system to the business system.

[0045] In one possible implementation of the first aspect, determining whether the transfer of the target file from the distributed file system to the file service component is complete includes:

[0046] When the sum of the current file offset and the package size of each sub-packet is greater than the total file size of the target file, it is determined that the transfer of the target file from the distributed file system to the file service component is complete.

[0047] Secondly, this invention provides a distributed system file resuming system based on MQTT communication, the system comprising:

[0048] The file storage module is used to transfer the file to be sent from the business system to the file service component, so that the file to be sent can be uploaded to the distributed file system through the file service component, and after the file to be sent is uploaded to the distributed file system, the file identifier and file path of the file to be sent are stored in the distributed shared memory Redis;

[0049] The content acquisition module is used to query the file path from the distributed shared memory Redis using the file identifier, and to acquire the file content of the file to be sent from the distributed file system based on the file path;

[0050] The packet sending module is used to calculate the number of packets of the file to be sent based on the file content, and send the file to be sent from a preset MQTT server to a preset terminal device based on the number of packets. The preset terminal device responds to the file to be sent and obtains file response data.

[0051] The first resume module is used to identify the sent data of the file to be sent based on the file response data, and use the sent data to transfer the unsent data in the file to be sent to the preset terminal device, so as to complete the first file breakpoint resume transmission of the distributed file system and obtain the first file breakpoint resume transmission result.

[0052] The request-response module is used to send a file download request from the file service component to the preset terminal device, and to respond to the file download request in the preset terminal device to obtain request-response data;

[0053] The second resume module is used to store the target file corresponding to the file download request into the distributed file system based on the request response data, and after the target file is transferred from the distributed file system to the business system, complete the second file resume transmission of the distributed file system and obtain the second file resume transmission result.

[0054] Compared with existing technologies, the technical principles and beneficial effects of this solution are as follows:

[0055] This invention embodiment stores the file identifier and file path of the file to be sent in a distributed shared memory Redis, combining the distributed file storage middleware FastDFS and distributed shared memory Redis. By designing a Redis caching structure, the MD5 hash of the file content is used as the unique identifier for the file, enabling content storage. Furthermore, this invention embodiment collects the file content of the file to be sent from the distributed file system based on the file path, preparing for setting the number of file packets. Further, this invention embodiment sends the file to be sent from a preset MQTT server to a preset terminal device based on the number of packets. This allows for re-triggering a request to continue file transmission after the device network recovers in the event of a disconnection due to poor network information. The file service component sends the packOffset from the previous transmission to the file service component, which then continues transmitting the remaining portion of the file based on the packOffset field in the parameters, thereby achieving file resuming from interruption. This invention embodiment also utilizes the request-response mechanism... The data involves storing the target file corresponding to the file download request in the distributed file system. When the server downloads a file from the device, the device splits the file, serializes the split files into Base64 characters, and sends them to the target server via MQTT messages. Upon receiving the file split data, the target server converts the Base64 characters into a byte array and writes it to the FastDFS distributed file system, and also writes the file information to a Redis cache structure for subsequent file merging. This achieves file transfer in the distributed system. Furthermore, this embodiment of the invention transfers the target file from the distributed file system to the business system. In a distributed system architecture where distributed services and terminal devices transfer files via MQTT, due to uncontrollable network conditions and network traffic limitations of the terminal devices, a large file needs to be split into multiple packets for transmission. Distributed services are often deployed with redundant multi-service deployment and load balancing, allowing files uploaded by the terminal devices to be processed by different servers and ultimately merged back into a single original file. Therefore, the file breakpoint resume method and system based on MQTT communication proposed in this embodiment of the invention can achieve stable and reliable file upload and download, and also support the ability to continue transmission from the last position after interruption, thus realizing file breakpoint resume in a distributed environment. Attached Figure Description

[0056] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is a flowchart illustrating a method for resuming interrupted file transfers in a distributed system based on MQTT communication, according to an embodiment of the present invention.

[0059] Figure 2 As shown in one embodiment of the present invention Figure 1 A schematic diagram illustrating a method for resuming interrupted file transfers in a distributed system based on MQTT communication, where file identifiers and file paths are stored in a distributed shared memory Redis.

[0060] Figure 3 As shown in one embodiment of the present invention Figure 1 A schematic diagram of file response data provided for a file breakpoint resume method in a distributed system based on MQTT communication;

[0061] Figure 4 As shown in one embodiment of the present invention Figure 1 A schematic diagram of the sent data in a distributed system file breakpoint resume method based on MQTT communication is provided.

[0062] Figure 5 As shown in one embodiment of the present invention Figure 1 A schematic diagram illustrating a method for resuming interrupted file transfers in a distributed system based on MQTT communication, showing how a file is sent from a business system to a terminal device;

[0063] Figure 6 As shown in one embodiment of the present invention Figure 1 A schematic diagram of a file download request for a distributed system file resuming method based on MQTT communication;

[0064] Figure 7 As shown in one embodiment of the present invention Figure 1 A schematic diagram of request and response data for a file breakpoint resume method in a distributed system based on MQTT communication;

[0065] Figure 8 As shown in one embodiment of the present invention Figure 1 A schematic diagram illustrating the storage of the target file in the distributed file system as a method for resuming interrupted file transfers in a distributed system based on MQTT communication;

[0066] Figure 9 As shown in one embodiment of the present invention Figure 1A schematic diagram illustrating a distributed system file resuming method based on MQTT communication, showing how files are downloaded in packets from a terminal device to the system platform.

[0067] Figure 10 This is a schematic diagram of a distributed system file breakpoint resume system based on MQTT communication, provided as an embodiment of the present invention. Detailed Implementation

[0068] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0069] This invention provides a method for resuming interrupted file transfers in a distributed system based on MQTT communication. The execution entity of this method includes, but is not limited to, at least one of the following: a server, a terminal, or other electronic devices configured to execute the method provided in this invention. In other words, the method can be executed by software or hardware installed on a terminal device or a server device. The software may be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cluster of cloud servers. The server can be an independent server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0070] See Figure 1 The diagram shown is a flowchart illustrating a distributed system file resuming method based on MQTT communication according to an embodiment of the present invention. Figure 1 The method for resuming interrupted file transfers in a distributed system based on MQTT communication, as described in the document, includes:

[0071] S1. The file to be sent is transferred from the business system to the file service component, so that the file to be sent is uploaded to the distributed file system through the file service component, and after the file to be sent is uploaded to the distributed file system, the file identifier and file path of the file to be sent are stored in the distributed shared memory Redis.

[0072] In this embodiment of the invention, the business system refers to a system that receives files to be sent by users. Furthermore, the file service component refers to a file center service that can be deployed independently to provide file upload, file download, file preview and other services to users.

[0073] Furthermore, in this embodiment of the invention, the distributed file system refers to FastDFS, which is an open-source, lightweight distributed file system that solves problems such as large-scale data storage and load balancing. FastDFS manages files, and its functions include file storage, file synchronization, and file access (file upload, file download, file deletion).

[0074] Furthermore, in this embodiment of the invention, the file identifier and file path of the file to be sent are stored in a distributed shared memory Redis, which is used to combine the distributed file storage middleware FastDFS and the distributed shared memory Redis. By designing the Redis cache structure, the MD5 hash of the file content is used as the unique identifier of the file to realize content storage.

[0075] The distributed shared memory Redis mentioned here refers to the Redis database.

[0076] In one embodiment of the present invention, storing the file identifier and file path of the file to be sent in a distributed shared memory Redis includes: extracting the file identifier and file path of the file to be sent from the distributed file system; and returning the file identifier and file path of the file to be sent to the file service component, so that the file service component can cache the file identifier and file path of the file to be sent in the distributed shared memory Redis.

[0077] See Figure 2 As shown, this is one embodiment of the present invention. Figure 1 This diagram illustrates a method for resuming interrupted file transfers in a distributed system based on MQTT communication, where the file identifier and file path are stored in a distributed shared memory (Redis). Figure 2 In this context, the RedisKey is used to record the file identifier, and the MD5 value of the file content is the file identifier. For example, e799bae41a379c3093100265ea52fae is the file identifier. The RedisValue is used to record the file path, for example, group1 / M00 / 04 / D0 / CgBuYGVcZJyAWz1HAACwt6q3odk729.xml is the file path.

[0078] S2. Use the file identifier to query the file path from the distributed shared memory Redis, and based on the file path, collect the file content of the file to be sent from the distributed file system.

[0079] In one embodiment of the present invention, the step of querying the file path from the distributed shared memory Redis using the file identifier includes: sending a file retrieval request from the file service component to a preset MQTT server; extracting the file identifier of the file to be sent from the distributed file system based on the file retrieval request in the preset MQTT server; and querying the file path corresponding to the file identifier from the distributed shared memory Redis.

[0080] For example, since the platform's MQTT consumer service is deployed in a cluster, the number of service nodes is greater than or equal to 1. The MQTT message consumption adopts a load balancing algorithm (round-robin). The request responded by the device is randomly received by one of the platform's MQTT consumer services. At this time, it only needs to query the file path in Redis with the Redis key: file:md5:upload:{fileMd5} based on the fileMd5 value in the parameter (that is, the file path stored in FastDFS).

[0081] Furthermore, in this embodiment of the invention, the file content of the file to be sent is collected from the distributed file system based on the file path, in order to prepare for the subsequent setting of the number of packets for the file.

[0082] The file content includes the file itself and the file size.

[0083] S3. Calculate the number of packets of the file to be sent based on the file content. Based on the number of packets, send the file to be sent from a preset MQTT server to a preset terminal device. The preset terminal device responds to the file to be sent and obtains file response data.

[0084] In one embodiment of the present invention, calculating the number of packets of the file to be sent based on the file content includes: calculating the number of packets of the file to be sent using the following formula based on the file content:

[0085]

[0086] Where n represents the number of sub-packages, D represents the file size in the file content, and D0 represents the standard sub-package size.

[0087] Furthermore, in this embodiment of the invention, the file to be sent is sent from a preset MQTT server to a preset terminal device based on the number of packets. This is used to trigger a request to continue transmitting the file again after the device network is restored in the event of a disconnection due to poor device network information. The packOffset of the previous transmission is sent to the file service component, which then continues to transmit the remaining part of the file according to the packOffset field in the parameters, thereby realizing the file interruption resumption of the file to be sent.

[0088] The preset MQTT server refers to a server based on the MQTT service, which uses a publish / subscribe model for communication and data exchange.

[0089] In one embodiment of the present invention, sending the file to be sent from a preset MQTT server to a preset terminal device based on the number of packets includes: splitting the file to be sent into multiple packets based on the number of packets; sending the multiple packets from the preset MQTT server to the preset terminal device according to the packet splitting order of the multiple packets; detecting whether the current packet response of the multiple packets is received in the preset MQTT server within a preset timer period; and if the current packet response of the multiple packets is not received in the preset MQTT server within the preset timer period, ... The system queries the previous packet response received from the preset MQTT server and identifies the previous file offset from the previous packet response. Based on the previous file offset, it determines the packets to be retransmitted among the multiple packets. It calculates the retransmission delay time for the packets to be retransmitted. Based on the retransmission delay time, it retransmits the packets to be retransmitted to the preset terminal device and then sends the previous file offset to the file service component. The file service component instructs the preset MQTT server to continue transmitting the remaining packets after the packets to be retransmitted to the preset terminal device.

[0090] In another embodiment of the present invention, calculating the retransmission delay time of the packet to be retransmitted includes: calculating the retransmission delay time of the packet to be retransmitted using the following formula:

[0091] t=2rτ

[0092] Where t represents the retransmission delay time, and r represents the time from the discrete set of integers {0, 1, ..., 2}. k A number is randomly selected from {-1}, k represents the number of packet retransmissions, and τ represents the one-way propagation delay from the preset MQTT server to the preset terminal device.

[0093] For example, during the first packet retransmission, k=1 and the random number r is selected from {0, 1}. During the second packet retransmission, k=2 and the random number r is selected from {0, 1, 3}.

[0094] See Figure 3 As shown, this is one embodiment of the present invention. Figure 1 This diagram illustrates a file response data method for resuming interrupted file transfers in a distributed system based on MQTT communication. Figure 3 In this context, the file response data refers to the packet response made when each of the multiple packets is received, and the file response data includes the file MD5, packet size, and current file offset.

[0095] S4. Based on the file response data, identify the sent data of the file to be sent, and use the sent data to transfer the unsent data in the file to be sent to the preset terminal device to complete the first file breakpoint resume transmission of the distributed file system and obtain the first file breakpoint resume transmission result.

[0096] It should be noted that, in one embodiment of the present invention, the principle of identifying the sent data of the file to be sent based on the file response data is similar to the principle of determining the retransmission packets among the multiple packets based on the previous file offset, and will not be further explained here.

[0097] See Figure 4 As shown, this is one embodiment of the present invention. Figure 1 This diagram illustrates the sent data in a distributed system file resuming method based on MQTT communication. Figure 4 In this context, 42 / 45 indicates that the number of the multiple packets is 45 and the number of data sent is 42.

[0098] See Figure 5 As shown, this is one embodiment of the present invention. Figure 1 This diagram illustrates a method for resuming interrupted file transfers in a distributed system based on MQTT communication, where a file is sent from the business system to the terminal device.

[0099] S5. Send a file download request from the file service component to the preset terminal device, so that the preset terminal device responds to the file download request and obtains request response data.

[0100] See Figure 6 As shown, this is one embodiment of the present invention. Figure 1 This diagram illustrates a file download request using a distributed system file resuming method based on MQTT communication. Figure 6In the document, the file download request includes the file type, file offset, and package size.

[0101] See Figure 7 As shown, this is one embodiment of the present invention. Figure 1 This diagram illustrates the request and response data for a distributed system file resuming method based on MQTT communication. Figure 7 In the request response data, the file MD5 hash, total file size, package size, current file offset, and the binary content of the file's Base64 hash are included.

[0102] S6. Based on the request response data, store the target file corresponding to the file download request in the distributed file system, and after transferring the target file from the distributed file system to the business system, complete the second file breakpoint resume transmission of the distributed file system to obtain the second file breakpoint resume transmission result.

[0103] This invention, based on the request-response data, stores the target file corresponding to the file download request in the distributed file system. When the server downloads a file from the device, the device splits the file, serializes the split files into Base64 characters, and sends them to the target server as messages via the MQTT protocol. Upon receiving the file split data, the target server converts the Base64 characters into a byte array and writes it to the FastDFS distributed file system. It also writes the file information to a Redis cache structure for easy subsequent file merging, thus realizing file transfer in the distributed system.

[0104] In one embodiment of the present invention, storing the target file corresponding to the file download request in the distributed file system based on the request-response data includes: constructing a hash storage structure for the target file using the file identifier in the request-response data as the file key; using the current file offset in the request-response data as the hash address of the target file in the hash storage structure; and storing the target file in the distributed file system based on the hash address using the hash storage structure.

[0105] For example, since the platform's MQTT consumer service is deployed in a cluster with at least one service node, and MQTT message consumption uses a load balancing algorithm (round-robin), the requests responded to by the device are randomly received by one of the platform's consumer services. At this time, it needs to construct a hash storage structure with file:md5:download:{fileMd5} as the key based on the fileMd5 value in the parameter to store the content of each sub-file. The data in the hash structure uses the current file offset value as the hash key, and the path where the sub-file is stored in FastDFS is the hash value.

[0106] See Figure 8 As shown, this is one embodiment of the present invention. Figure 1 This diagram illustrates the storage of the target file in the distributed file system, as part of a distributed system file resuming method based on MQTT communication. Figure 8 In this context, key represents the current file offset of each sub-file, value represents the file path of each sub-file, and file:md5:download:{fileMd5} represents the file keyword.

[0107] Furthermore, in this embodiment of the invention, the target file is transferred from the distributed file system to the business system. In a distributed system architecture, where distributed services and terminal devices transmit files via the MQTT protocol, due to the uncontrollable network conditions of the terminal devices and the limitation of network traffic, a large file needs to be split into multiple packets for transmission. Distributed services are often deployed in a redundant and load-balanced manner on the system side, which allows files uploaded by terminal devices to be processed by different servers and eventually merged into a single original file.

[0108] In one embodiment of the present invention, the step of transferring the target file from the distributed file system to the business system includes: after transferring the file path of the target file from the distributed file system to the distributed shared memory Redis, transferring each sub-package of the target file from the distributed file system to the file service component; collecting the file download progress of each sub-package from the distributed file system to the file service component in the business system, and after transferring the file download progress from the file service component to the distributed shared memory Redis, determining whether the transfer of the target file from the distributed file system to the file service component is complete; when the transfer of the target file from the distributed file system to the file service component is complete, querying the file path of each sub-package according to the file identifier of each sub-package; sorting each sub-package based on the file path of each sub-package to obtain a sub-package sequence; concatenating each sub-package in the sub-package sequence using a preset file operation function to obtain a complete file; and transferring the complete file from the file service component to the business system to complete the process of transferring the target file from the distributed file system to the business system.

[0109] In another embodiment of the present invention, determining whether the transmission of the target file from the distributed file system to the file service component is completed includes: determining that the transmission of the target file from the distributed file system to the file service component is completed when the sum of the current file offset and the package size of each sub-packet is greater than the total file size of the target file.

[0110] Optionally, the file packet paths can be stored in Redis. Based on the current file writing progress, the file download progress can be written to Redis for the business system to query. Further, file merging: after the last packet is completely transmitted (the logic for determining the completeness of the last packet is: packOffset + packSize > fileSize), based on the file's fileMd5, assemble a RedisKey file:md5:download:{fileMd5}, query the paths of all packet files stored in the hash, sort by key (i.e., file offset), use file operation functions to concatenate the file byte arrays of all files sequentially into a complete file, and then output the file as a stream to the business system.

[0111] See Figure 9 As shown, this is one embodiment of the present invention. Figure 1 This diagram illustrates a distributed system file download method based on MQTT communication, which involves downloading files from a terminal device to the system platform in packets.

[0112] As can be seen, this embodiment of the invention stores the file identifier and file path of the file to be sent in a distributed shared memory Redis, which is used to combine the distributed file storage middleware FastDFS and distributed shared memory Redis. By designing a Redis cache structure, where the MD5 hash of the file content is used as the unique identifier of the file, the content is stored. Furthermore, this embodiment of the invention collects the file content of the file to be sent from the distributed file system based on the file path, to prepare for setting the number of file packets. Further, this embodiment of the invention sends the file to be sent from a preset MQTT server to a preset terminal device based on the number of packets. This is used to re-trigger a request to continue transmitting the file after the device network recovers in the event of a disconnection due to poor device network information, and sends the packOffset from the previous transmission to the file service component. The file service component then continues to transmit the remaining part of the file based on the packOffset field in the parameters, thereby achieving the resumption of file transmission from interruption. This embodiment of the invention, based on the request... The response data stores the target file corresponding to the file download request in the distributed file system. When the server downloads a file from the device, the device splits the file, serializes the split files into Base64 characters, and sends them to the target server via MQTT messages. Upon receiving the file split data, the target server converts the Base64 characters into a byte array and writes it to the FastDFS distributed file system, and also writes the file information to a Redis cache structure for subsequent file merging. This achieves file transfer in the distributed system. Furthermore, this embodiment of the invention transfers the target file from the distributed file system to the business system. In a distributed system architecture where distributed services and terminal devices transfer files via MQTT, due to uncontrollable network conditions and network traffic limitations of the terminal devices, a large file needs to be split into multiple packets for transmission. Distributed services are often deployed with redundant multi-service deployment and load balancing, allowing files uploaded by the terminal devices to be processed by different servers and ultimately merged back into a single original file. Therefore, the file breakpoint resume method for distributed systems based on MQTT communication proposed in this embodiment of the invention can achieve stable and reliable file upload and download, and also supports resuming transmission from the previous position after interruption, thus realizing file breakpoint resume in a distributed environment.

[0113] like Figure 10 The diagram shown is a functional block diagram of the distributed system file breakpoint resume system based on MQTT communication according to the present invention.

[0114] The distributed system file breakpoint resumption system 1000 based on MQTT communication described in this invention can be installed in an electronic device. Depending on the functions implemented, the distributed system file breakpoint resumption system based on MQTT communication may include a file storage module 1001, a content acquisition module 1002, a packet sending module 1003, a first resumption module 1004, a request-response module 1005, and a second resumption module 1006. The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and are stored in the memory of the electronic device.

[0115] In this embodiment of the invention, the functions of each module / unit are as follows:

[0116] The file storage module 1001 is used to transfer the file to be sent from the business system to the file service component, so that the file to be sent can be uploaded to the distributed file system through the file service component, and after the file to be sent is uploaded to the distributed file system, the file identifier and file path of the file to be sent are stored in the distributed shared memory Redis;

[0117] The content acquisition module 1002 is used to query the file path from the distributed shared memory Redis using the file identifier, and to acquire the file content of the file to be sent from the distributed file system based on the file path;

[0118] The packet sending module 1003 is used to calculate the number of packets of the file to be sent according to the file content, and send the file to be sent from a preset MQTT server to a preset terminal device based on the number of packets. The preset terminal device responds to the file to be sent and obtains file response data.

[0119] The first resume transmission module 1004 is used to identify the sent data of the file to be sent according to the file response data, and use the sent data to transmit the unsent data in the file to be sent to the preset terminal device, so as to complete the first file breakpoint resume transmission of the distributed file system and obtain the first file breakpoint resume transmission result.

[0120] The request response module 1005 is used to send a file download request from the file service component to the preset terminal device, and to respond to the file download request in the preset terminal device to obtain request response data;

[0121] The second resume module 1006 is used to store the target file corresponding to the file download request in the distributed file system based on the request response data, and after the target file is transferred from the distributed file system to the business system, complete the second file resume transmission of the distributed file system and obtain the second file resume transmission result.

[0122] In detail, the modules in the distributed system file breakpoint resume system 1000 based on MQTT communication described in this embodiment of the invention adopt the same approach as described above. Figure 1 The method described above uses the same technique as the distributed system file breakpoint resume method based on MQTT communication, and can produce the same technical effect, so it will not be repeated here.

[0123] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0124] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0125] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0126] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0127] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features of the invention herein.

Claims

1. A method for resuming interrupted file transfers in a distributed system based on MQTT communication, characterized in that, The method includes: The file to be sent is transferred from the business system to the file service component, so that the file service component can upload the file to the distributed file system. After the file to be sent is uploaded to the distributed file system, the file identifier and file path of the file to be sent are stored in the distributed shared memory Redis. The file path is queried from the distributed shared memory Redis using the file identifier, and the file content of the file to be sent is collected from the distributed file system based on the file path; Based on the file content, the number of packets of the file to be sent is calculated. Based on the number of packets, the file to be sent is sent from a preset MQTT server to a preset terminal device. The preset terminal device responds to the file to be sent and obtains file response data. Based on the file response data, the sent data of the file to be sent is identified, and the unsent data in the file to be sent is transmitted to the preset terminal device using the sent data to complete the first file breakpoint resume transmission of the distributed file system and obtain the first file breakpoint resume transmission result. The file service component sends a file download request to the preset terminal device, and the preset terminal device responds to the file download request to obtain request response data; Based on the request-response data, the target file corresponding to the file download request is stored in the distributed file system. After the target file is transferred from the distributed file system to the business system, the second file breakpoint resume transfer of the distributed file system is completed, and the second file breakpoint resume transfer result is obtained.

2. The method according to claim 1, characterized in that, The step of storing the file identifier and file path of the file to be sent in a distributed shared memory Redis includes: Extract the file identifier and file path of the file to be sent from the distributed file system; The file identifier and file path of the file to be sent are returned to the file service component, so that the file service component can cache the file identifier and file path of the file to be sent in the distributed shared memory Redis.

3. The method according to claim 1, characterized in that, The step of querying the file path from the distributed shared memory Redis using the file identifier includes: The file service component sends a file retrieval request to the preset MQTT server; In the preset MQTT server, based on the file retrieval request, the file identifier of the file to be sent is extracted from the distributed file system; Query the file path corresponding to the file identifier from the distributed shared memory Redis.

4. The method according to claim 1, characterized in that, The step of calculating the number of packets of the file to be sent based on the file content includes: Based on the file content, the number of packets in the file to be sent can be calculated using the following formula: Where n represents the number of sub-packages, D represents the file size in the file content, and D0 represents the standard sub-package size.

5. The method according to claim 1, characterized in that, The step of sending the file to be sent from a preset MQTT server to a preset terminal device based on the number of packets includes: Based on the number of packets, the file to be sent is split into multiple packets; According to the packet sub-packet order, the multiple packets are sent from the preset MQTT server to the preset terminal device; Within a preset timer period, it is detected whether the current packet response of the multiple packets has been received in the preset MQTT server; If no current packet response is received from the preset MQTT server within the preset timer period, query the previous packet response received from the preset MQTT server and identify the previous file offset from the previous packet response; Based on the previous file offset, determine the packets to be retransmitted among the multiple packets; Calculate the retransmission delay time of the packet to be retransmitted; Based on the retransmission delay time, after retransmitting the packet to be retransmitted to the preset terminal device, the previous file offset is sent to the file service component, so that the file service component instructs the preset MQTT server to continue transmitting the remaining packets after the packet to be retransmitted in the multiple packets to the preset terminal device.

6. The method according to claim 5, characterized in that, The calculation of the retransmission delay time of the packet to be retransmitted includes: The retransmission delay time of the packet to be retransmitted is calculated using the following formula: t=2rτ Where t represents the retransmission delay time, and r represents the time from the discrete set of integers {0, 1, ..., 2}. k A number is randomly selected from {-1}, k represents the number of packet retransmissions, and τ represents the one-way propagation delay from the preset MQTT server to the preset terminal device.

7. The method according to claim 1, characterized in that, The step of storing the target file corresponding to the file download request into the distributed file system based on the request-response data includes: The file identifier in the request-response data is used as the file key to construct the hash storage structure of the target file; The current file offset in the request-response data is used as the hash address of the target file in the hash storage structure; Based on the hash address, the target file is stored in the distributed file system using the hash storage structure.

8. The method according to claim 1, characterized in that, The step of transferring the target file from the distributed file system to the business system includes: After transferring the file path of the target file from the distributed file system to the distributed shared memory Redis, each sub-packet of the target file is transferred from the distributed file system to the file service component; After collecting the file download progress of each sub-packet from the distributed file system to the file service component in the business system, and transmitting the file download progress from the file service component to the distributed shared memory Redis, it is determined whether the transmission of the target file from the distributed file system to the file service component is complete. When the transfer of the target file from the distributed file system to the file service component is completed, the file path of each sub-packet is queried according to the file identifier of each sub-packet; Based on the file path of each sub-package, sort each sub-package to obtain a sub-package sequence; By using preset file operation functions, each sub-packet in the sub-packet sequence is concatenated to obtain the complete file; The complete version of the file is transferred from the file service component to the business system to complete the process of transferring the target file from the distributed file system to the business system.

9. The method according to claim 8, characterized in that, The step of determining whether the transfer of the target file from the distributed file system to the file service component is complete includes: When the sum of the current file offset and the package size of each sub-packet is greater than the total file size of the target file, it is determined that the transfer of the target file from the distributed file system to the file service component is complete.

10. A distributed system file resuming function based on MQTT communication, characterized in that, The system includes: The file storage module is used to transfer the file to be sent from the business system to the file service component, so that the file to be sent can be uploaded to the distributed file system through the file service component, and after the file to be sent is uploaded to the distributed file system, the file identifier and file path of the file to be sent are stored in the distributed shared memory Redis; The content acquisition module is used to query the file path from the distributed shared memory Redis using the file identifier, and to acquire the file content of the file to be sent from the distributed file system based on the file path; The packet sending module is used to calculate the number of packets of the file to be sent based on the file content, and send the file to be sent from a preset MQTT server to a preset terminal device based on the number of packets. The preset terminal device responds to the file to be sent and obtains file response data. The first resume module is used to identify the sent data of the file to be sent based on the file response data, and use the sent data to transfer the unsent data in the file to be sent to the preset terminal device, so as to complete the first file breakpoint resume transmission of the distributed file system and obtain the first file breakpoint resume transmission result. The request-response module is used to send a file download request from the file service component to the preset terminal device, and to respond to the file download request in the preset terminal device to obtain request-response data; The second resume module is used to store the target file corresponding to the file download request into the distributed file system based on the request response data, and after the target file is transferred from the distributed file system to the business system, complete the second file resume transmission of the distributed file system and obtain the second file resume transmission result.

Images