A massive small file transmission system and method based on IROS

Abstract

This invention provides a massive small file transfer system and method based on IROS, belonging to the field of data communication technology. It includes a file sending server and a file receiving server. The file sending server includes a sending directory, a small file sending module, a small file network sending module, and an intelligent robot operating system. The file receiving server includes a receiving directory, a small file receiving module, a small file network receiving module, and an intelligent robot operating system. By converting the inefficient random reading of small files to sequential reading, disk performance is effectively utilized. Combined with IROS's efficient message caching mechanism and by merging small messages into large messages, transmission bandwidth is effectively utilized during transmission, thus maximizing network transmission performance.

Description

Technical Field

[0001] This invention relates to a massive small file transfer system and method based on IROS, belonging to the field of data communication technology. Background Technology

[0002] Files are one of the most common methods for storing data and can be exchanged over a network. During network transfers, large files are generally transferred more efficiently than small files. For example, on a gigabit network, transferring a 100MB file via FTP takes just over a second, while transferring 10,000 individual 10KB files (also 100MB) via FTP often takes more than a minute – significantly slower. Even with the same file transfer tool, the same operating environment, and files of the same total size, the transfer efficiency of small files is noticeably slower due to the difference in the size of individual files. In many real-world scenarios, it is indeed necessary to transfer a large number of small files.

[0003] In the Linux operating system, files are stored in directories, and subdirectories can be nested within directories. Directories and files in Linux are indexed by an object called an inode. When reading a file, the `read` operation is called. The `read` operation reads the inode information to obtain the actual storage location of the file's content, and then reads the data from the disk. Writing to a file calls the `write` operation, which updates the inode information. Normally, inode information is cached in memory.

[0004] IROS is a new type of intelligent robot operating system, similar in function to ROS, but with higher performance, reliability, and security. IROS is essentially a robot middleware; it can be used not only on robots but also in PC and server environments, providing basic communication services such as messaging, services, and parameters.

[0005] The slow transfer speed of small files is due to several factors. From a network perspective, the transmission efficiency of small data packets is lower than that of large data packets, and small files are read as small data packets. From a disk perspective, the widely used, low-cost, and cost-effective mechanical hard drives (HDDs) are significantly more efficient in sequential read / write operations than in random read / write operations, and small files are processed through random read / write. These two factors are the fundamental reasons for the slow transfer speed of small files. Currently, there is a lack of methods to improve the transfer efficiency of massive numbers of small files using IROS. Summary of the Invention

[0006] The purpose of this invention is to provide a massive small file transfer system and method based on IROS. It innovates on disk read / write and network transmission aspects to address the reasons for slow small file transfer, thereby significantly improving the transfer efficiency of massive small files.

[0007] To achieve the above objectives, the present invention employs the following technical solution:

[0008] It includes a file sending server and a file receiving server. The file sending server includes a sending directory, a small file sending module, a small file network sending module, and an intelligent robot operating system. The file receiving server includes a receiving directory, a small file receiving module, a small file network receiving module, and an intelligent robot operating system.

[0009] The sending directory stores small files to be sent;

[0010] The small file sending module is used to read small files in the sending directory and publish the content of each file plus metadata as a message to the intelligent robot operating system deployed in the file sending server;

[0011] The intelligent robot operating system deployed in the file sending server caches the content of small files as messages in a specified topic and provides the functions of publishing and subscribing to topic messages.

[0012] The small file network sending module subscribes to messages from the intelligent robot operating system deployed on the file sending server, and encapsulates multiple subscribed messages into a single message and sends it to the small file network receiving module.

[0013] The small file network receiving module obtains the encapsulated message and splits the message into small file messages, which are then cached in a designated topic of the intelligent robot operating system deployed on the file receiving server.

[0014] The small file receiving module obtains small file messages from the intelligent robot operating system deployed in the file receiving server and restores them as corresponding small files in the receiving directory.

[0015] Preferably, the file sending server and the file receiving server are connected via an IP-based network, including a local area network (LAN) or a wide area network (WAN).

[0016] A method for transferring large numbers of small files based on IROS, including the following:

[0017] The small file sending module in the file sending server periodically traverses the sending directory. When it encounters a small file to be transmitted, it groups multiple small files together and adds metadata to the content of each file in each group as a message, which is then published to the intelligent robot operating system deployed in the file sending server.

[0018] The intelligent robot operating system deployed in the file sending server caches the content of small files as messages in a designated topic and provides the functions of publishing and subscribing to topic messages.

[0019] The small file network sending module subscribes to messages from a specified topic in the intelligent robot operating system deployed in the file sending server, and encapsulates multiple subscribed messages into a single message and sends it to the small file network receiving module.

[0020] The small file network receiving module receives the encapsulated message, splits the encapsulated message into independent small file messages, and stores them in the designated topic of the intelligent robot operating system deployed in the file receiving server;

[0021] The intelligent robot operating system deployed in the file receiving server caches the content of the received small files as messages in a designated topic and provides the functions of publishing and subscribing to topic messages.

[0022] The small file receiving module retrieves small file messages from a specified topic in the intelligent robot operating system deployed on the file receiving server and restores them as corresponding small files in the receiving directory.

[0023] Preferably, the small file sending module groups multiple small files together and publishes the content and metadata of each file in each group as a message to the intelligent robot operating system deployed on the file sending server. The specific method is as follows:

[0024] The small file sending module directly reads the inode information cached in the memory of the file sending server and analyzes the relevant small files in the sending directory;

[0025] Divide the small files into groups and call the read interface to read the small files in each group one by one;

[0026] The content of each small file in each group, along with its metadata, is cached as a message in the intelligent robot operating system topic on the file sending server.

[0027] Preferably, the specific content of grouping small files is as follows: based on the offset and size of the small files on the disk, multiple small files that are stored close to each other on the disk are grouped together.

[0028] Preferably, after encapsulating multiple subscribed messages into a single message, the message is further encrypted and compressed before being sent to the small file receiving module.

[0029] The advantages of this invention are as follows: By converting inefficient random reads of small files to sequential reads, the invention effectively utilizes disk performance; combined with IROS's efficient message caching mechanism and by merging small messages into large messages, it effectively utilizes transmission bandwidth during transmission, thus maximizing network transmission performance. Because it effectively solves the problem of slow transmission of massive amounts of small files, this invention significantly outperforms traditional methods in terms of transmission performance for massive amounts of small files, achieving performance hundreds of times that of FTP. Attached Figure Description

[0030] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0031] Figure 1 This is a schematic diagram of the main structure of Embodiment 1 of the present invention.

[0032] In the diagram: S1 is the file sending server, S2 is the file receiving server, D1 is the sending directory, and D2 is the receiving directory. Detailed Implementation

[0033] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0034] Example 1

[0035] A massive small file transfer system based on IROS includes a file sending server and a file receiving server. The file sending server includes a sending directory, a small file sending module, a small file network sending module, and an intelligent robot operating system. The file receiving server includes a receiving directory, a small file receiving module, a small file network receiving module, and the intelligent robot operating system. The sending directory stores small files to be sent. The small file sending module reads small files from the sending directory and publishes the content of each file, along with metadata, as a message to the intelligent robot operating system deployed on the file sending server. The intelligent robot operating system caches the content of the small files as messages in a designated topic and provides publish and subscribe functions for topic messages. The small file network sending module subscribes to messages from the intelligent robot operating system deployed on the file sending server and encapsulates multiple subscribed messages into a single message before sending it to the small file network receiving module. The small file network receiving module receives the encapsulated message, splits it into small file messages, and caches them in a designated topic within the intelligent robot operating system deployed on the file receiving server. The small file receiving module retrieves the small file messages from the intelligent robot operating system deployed on the file receiving server and restores them to the corresponding small files in the receiving directory. The file sending server and file receiving server are connected via an IP-based network, including a local area network (LAN) or a wide area network (WAN).

[0036] Example 2

[0037] A method for transferring large numbers of small files based on IROS, including the following:

[0038] The small file sending module in the file sending server periodically traverses the sending directory. When it encounters a small file to be transmitted, it groups multiple small files together and adds metadata to the content of each file in each group, publishing it as a message to the intelligent robot operating system deployed on the file sending server. The specific method by which the small file sending module groups multiple small files together and adds metadata to the content of each file in each group, publishing it as a message to the intelligent robot operating system deployed on the file sending server, is as follows:

[0039] The small file sending module directly reads the inode information cached in the file sending server's memory and analyzes the relevant small files in the sending directory.

[0040] Based on the offset and size of the small files on the disk, multiple small files that are close to each other on the disk are grouped together, and the read interface is called to read the small files in each group one by one.

[0041] The content of each small file in each group, along with its metadata, is cached as a message in the intelligent robot operating system topic on the file sending server.

[0042] The intelligent robot operating system deployed in the file sending server caches the content of small files as messages in a designated topic and provides the functions of publishing and subscribing to topic messages.

[0043] The small file network sending module subscribes to messages from a specified topic in the intelligent robot operating system deployed in the file sending server, and encapsulates multiple subscribed messages into a single message and sends it to the small file network receiving module.

[0044] The small file network receiving module receives the encapsulated message, splits the encapsulated message into independent small file messages, and stores them in the designated topic of the intelligent robot operating system deployed in the file receiving server.

[0045] The intelligent robot operating system deployed in the file receiving server caches the content of the received small files as messages in a designated topic and provides the functions of publishing and subscribing to topic messages.

[0046] The small file receiving module retrieves small file messages from a specified topic in the intelligent robot operating system deployed on the file receiving server and restores them as corresponding small files in the receiving directory.

[0047] Example 3

[0048] This disclosure provides a massive small file transfer device based on IROS, including a processor and a memory. Optionally, the device may further include a communication interface and a bus. The processor, communication interface, and memory can communicate with each other via the bus. The communication interface can be used for information transmission. The processor can invoke logical instructions in the memory to execute the massive small file transfer method based on IROS described above.

[0049] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0050] Memory, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as the program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor executes functional applications and data processing by running the program instructions / modules stored in the memory, thereby realizing the massive small file transfer method based on IROS in the above embodiments.

[0051] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory may include high-speed random access memory and may also include non-volatile memory.

[0052] Example 4

[0053] This disclosure provides a computer-readable storage medium storing computer-executable instructions configured to execute the above-described IROS-based method for transferring large numbers of small files.

[0054] The aforementioned computer-readable storage medium may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0055] The technical solutions of this disclosure can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in this disclosure. The aforementioned storage medium can be a non-transitory storage medium, including: a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and other media capable of storing program code; it can also be a transient storage medium.

[0056] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A massive small file transfer system based on IROS, characterized in that, It includes a file sending server and a file receiving server. The file sending server includes a sending directory, a small file sending module, a small file network sending module, and an intelligent robot operating system. The file receiving server includes a receiving directory, a small file receiving module, a small file network receiving module, and an intelligent robot operating system. The sending directory stores small files to be sent; The small file sending module is used to read small files in the sending directory and publish the content of each file plus metadata as a message to the intelligent robot operating system deployed in the file sending server; The intelligent robot operating system deployed in the file sending server caches the content of small files as messages in a specified topic and provides the functions of publishing and subscribing to topic messages. The small file network sending module subscribes to messages from the intelligent robot operating system deployed on the file sending server, and encapsulates multiple subscribed messages into a single message and sends it to the small file network receiving module. The small file network receiving module obtains the encapsulated message and splits the message into small file messages, which are then cached in a designated topic of the intelligent robot operating system deployed on the file receiving server. The small file receiving module obtains small file messages from the intelligent robot operating system deployed in the file receiving server and restores them as corresponding small files in the receiving directory.

2. The massive small file transfer system based on IROS according to claim 1, characterized in that, The file sending server and file receiving server are connected via an IP-based network, including a local area network (LAN) or a wide area network (WAN).

3. A method for transferring large amounts of small files based on IROS using the system described in any one of claims 1-2, characterized in that, Includes the following: The small file sending module in the file sending server periodically traverses the sending directory. When it encounters a small file to be transmitted, it groups multiple small files together and adds metadata to the content of each file in each group as a message, which is then published to the intelligent robot operating system deployed in the file sending server. The intelligent robot operating system deployed in the file sending server caches the content of small files as messages in a designated topic and provides the functions of publishing and subscribing to topic messages. The small file network sending module subscribes to messages from a specified topic in the intelligent robot operating system deployed in the file sending server, and encapsulates multiple subscribed messages into a single message and sends it to the small file network receiving module. The small file network receiving module receives the encapsulated message, splits the encapsulated message into independent small file messages, and stores them in the designated topic of the intelligent robot operating system deployed in the file receiving server; The intelligent robot operating system deployed in the file receiving server caches the content of the received small files as messages in a designated topic and provides the functions of publishing and subscribing to topic messages. The small file receiving module retrieves small file messages from a specified topic in the intelligent robot operating system deployed on the file receiving server and restores them as corresponding small files in the receiving directory.

4. The method for transferring large amounts of small files based on IROS according to claim 3, characterized in that, The small file sending module groups multiple small files together and adds metadata to the content of each file in each group, publishing it as a message to the intelligent robot operating system deployed on the file sending server. The specific method is as follows: The small file sending module directly reads the inode information cached in the memory of the file sending server and analyzes the relevant small files in the sending directory; Divide the small files into groups and call the read interface to read the small files in each group one by one; The content of each small file in each group, along with its metadata, is cached as a message in the intelligent robot operating system topic on the file sending server.

5. The method for transferring large amounts of small files based on IROS according to claim 4, characterized in that, The specific details of grouping small files are as follows: based on the offset and size of the small files on the disk, multiple small files that are stored close to each other on the disk are grouped together.

6. The method for transferring massive small files based on IROS according to claim 3, characterized in that, The process involves encapsulating multiple subscribed messages into a single message, encrypting and compressing it, and then sending it to the small file receiving module.

7. A massive small file transfer device based on IROS, comprising a processor and a memory storing program instructions, characterized in that, The processor is configured to execute the IROS-based method for transferring large numbers of small files as described in any one of claims 3 to 6 when running the program instructions.

8. A massive small file transfer device based on IROS, characterized in that, include: Product itself; The IROS-based large-scale small file transfer device as described in claim 7 is installed on the product body.

9. A storage medium storing program instructions, characterized in that, When the program instructions are executed, they perform the massive small file transfer method based on IROS as described in any one of claims 3 to 6.

Images