Rapid copying method and system for large-batch small files
By obtaining and sorting file location collections, judging and merging file fragments, and converting them into orderly I/O operations, the problem of inefficient copying of large batches of small files is solved, and more efficient disk copying speed is achieved.
Patent Information
- Application Number
- CN202411585636.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-07
- Publication Date
- 2025-03-28
AI Technical Summary
The prior art is inefficient when processing large batches of small files, which is mainly due to frequent random small I/O operations on the source disk, resulting in slow copying speed.
A quick copy method and system is proposed. By obtaining all location sets of files, it determines whether the file supports fast copying. It supports location sorting and fragmentation merging, and converts it into ordered I/O operations to reduce disk search time.
It effectively improves the copy efficiency of large batches of small files, and significantly improves the disk copy speed by reducing the disorder of I/O operations.
Smart Images

Figure CN119719040A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of file processing, and in particular to a method and system for quickly copying a large number of small files. Background Art
[0002] With the rapid development of information technology, the amount of data is showing an explosive growth trend. Among the many data processing requirements, copying large batches of small files has become a significant performance challenge.
[0003] At present, there are many ways to handle large batches of small file copies. On the one hand, users can directly copy through the copy and paste function provided by the operating system, but this method is often very inefficient when facing a large number of small files. On the other hand, there are also some commercial software on the market, such as FastCopy, which has adopted a series of measures to improve copy performance. For example, the strategy of reading first and then writing is adopted in the hope of optimizing the read and write process; the large memory cache technology is used to reduce frequent access to the disk; and the multi-threaded copy method is used to process the copy tasks of multiple files in parallel. However, although these methods have improved the performance of random small I / O operations to a certain extent, they still have not completely solved the efficiency problem of small file copying.
[0004] Research shows that the core problem of the performance bottleneck of copying a large number of small files lies in the random small I / O operations on the source disk. Due to the large number of small files and their relatively scattered distribution, the disk needs to frequently perform seek operations during the copy process, which greatly reduces the copy speed. Therefore, how to effectively reduce the I / O operations on the source disk and convert the disordered I / O operations into overall ordered I / O has become the key to improving the performance of copying a large number of small files. Summary of the invention
[0005] In order to solve the above technical problems existing in the prior art, the present invention proposes a method and system for quickly copying a large number of small files to solve the above technical problems.
[0006] According to a first aspect of the present invention, a method for quickly copying a large number of small files is proposed, comprising:
[0007] S1: Get a set F of all files in the path to be copied, and get a set S of all locations of any file in the set F on the disk;
[0008] S2: In response to the file supporting fast copy, all the location sets of the file are added to a new location set, and the new location set is reordered according to the starting position of the current fragmented file on the disk to obtain an ordered location set S';
[0009] S3: dynamically merge file fragments based on the location information and data length in the location set S' to obtain a new location set S'';
[0010] S4: Take out the fragment elements in the set S" in turn, read the data block whose starting offset on the disk is the starting offset of the element and whose length is the length of the element, and write the data to the corresponding target path until all file fragments are processed.
[0011] In some specific embodiments, S2 further includes: in response to the file not supporting fast copying, recording the file into a new copy file set F', and sequentially taking out each file for copying until all files are processed.
[0012] In some specific embodiments, the method for determining whether the file supports fast copy in S2 specifically includes:
[0013] S21: determining whether the file size is 0, if it is 0, fast copy is not supported;
[0014] S22: Determine whether the file can be parsed into the location set, if not parsable, fast copy is not supported;
[0015] S23: Check two ordered elements in the position set S. If the starting position of the previous element is greater than 0 and the starting position of the next element is greater than the starting position of the previous element, fast copying is supported. Otherwise, fast copying is not supported.
[0016] In some specific embodiments, S3 specifically includes: for any two adjacent elements in the ordered position set S', if the starting offset of the previous element plus its length plus the length between the two elements is greater than or equal to the starting offset of the latter element, then the two elements are merged to obtain a new position set S".
[0017] According to a second aspect of the present invention, a computer-readable storage medium is provided, on which one or more computer programs are stored. When the one or more computer programs are executed by a computer processor, the above method is implemented.
[0018] According to a third aspect of the present invention, a system for quickly copying a large number of small files is provided, comprising:
[0019] A file acquisition unit, configured to acquire a set F of all files whose paths need to be copied, and for any file in the set F, acquire a set S of all locations of the file on the disk;
[0020] A fast copy judgment unit, configured to, in response to the file supporting fast copy, add all the location sets of the file to a new location set, and reorder the new location set according to the starting position of the current fragmented file on the disk to obtain an ordered location set S';
[0021] A dynamic merging unit, configured to dynamically merge file fragments based on the position information and data length in the position set S' to obtain a new position set S'';
[0022] The data writing unit is configured to sequentially retrieve the fragment elements in the set S", read a data block whose starting offset on the disk is the starting offset of the element and whose length is the length of the element, and write the data to a corresponding target path until all file fragments are processed.
[0023] In some specific embodiments, the fast copy determination unit is further configured to record the file into a new copy file set F' in response to the file not supporting fast copy, and sequentially take out each file for copying until all files are processed.
[0024] In some specific embodiments, the method for determining whether a file supports fast copy in the fast copy determination unit specifically includes:
[0025] S21: determining whether the file size is 0, if it is 0, fast copy is not supported;
[0026] S22: Determine whether the file can be parsed into the location set, if not parsable, fast copy is not supported;
[0027] S23: Check two ordered elements in the position set S. If the starting position of the previous element is greater than 0 and the starting position of the next element is greater than the starting position of the previous element, fast copying is supported. Otherwise, fast copying is not supported.
[0028] In some specific embodiments, the dynamic merging unit specifically includes: for any two adjacent elements in the ordered position set S', if the starting offset of the previous element plus its length plus the length between the two elements is greater than or equal to the starting offset of the latter element, then the two elements are merged to obtain a new position set S".
[0029] In some specific embodiments, when executing the copy operation, based on a large memory cache and multi-threaded processing, part of the data to be copied is cached and multiple threads are started simultaneously to process different tasks. The cache size and the number of threads are dynamically adjusted based on the size of the file to be copied and the system resources.
[0030] The present invention proposes a method and system for fast copying of large batches of small files. Through file screening and filtering, files suitable for the fast copy method are screened, and files supporting fast copying are processed separately from ordinary copy files, thereby improving the overall copy efficiency; an adaptive merging mechanism dynamically merges file fragments according to the data length (MAXGAP) between two fragments in the disk, merges small file fragments to the maximum extent, reduces the number of file IO times, and the IO of the disk is sequential IO as a whole, reduces the disk seek time, and maximizes the disk copy speed. The present application can effectively solve the problem of slow copying of large batches of small files, and provide a more efficient and flexible file copying solution. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated into and constitute a part of this specification. The accompanying drawings illustrate the embodiments and together with the description are used to explain the principles of the present invention. Other embodiments and many expected advantages of the embodiments will be readily appreciated as they become better understood by reference to the following detailed description. Other features, objects and advantages of the present application will become more apparent by reading the detailed description of the non-limiting embodiments made with reference to the following drawings:
[0032] Figure 1 is a flow chart of a method for quickly copying a large number of small files according to an embodiment of the present application;
[0033] Figure 2 This is a flowchart of a method for quickly copying a large number of small files according to a specific embodiment of the present application;
[0034] Figure 3 is a flowchart of a method for determining whether a small file supports fast copying according to a specific embodiment of the present application;
[0035] Figure 4 This is a framework diagram of a system for quickly copying a large number of small files according to an embodiment of the present application;
[0036] Figure 5a-5c is a test result diagram of a specific embodiment of the present application;
[0037] Figure 6 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application. DETAILED DESCRIPTION
[0038] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.
[0039] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0040] Figure 1 FIG. 1 is a flow chart showing a method for quickly copying a large number of small files according to an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:
[0041] S1: Get a set F of all files in the path to be copied, and get a set S of all locations of any file in the set F on the disk.
[0042] S2: In response to the file supporting fast copy, all location sets of the file are added to a new location set, and the new location set is reordered according to the starting position of the current fragmented file on the disk to obtain an ordered location set S'.
[0043] In a specific embodiment, in response to a file not supporting fast copying, the file is recorded in a new copy file set F', and each file is taken out in turn for copying until all files are processed. Files that do not support fast copying are processed separately, ensuring the integrity of the entire copying process, so that different types of files can be copied in a suitable manner, further improving the overall copying efficiency.
[0044] In a specific embodiment, the method for determining whether a file supports fast copy in S2 specifically includes:
[0045] S21: determining whether the file size is 0, if it is 0, fast copy is not supported;
[0046] S22: Determine whether the file can be parsed into the location set, if not parsable, fast copy is not supported;
[0047] S23: Check the two ordered elements in the position set S. If the starting position of the previous element is greater than 0 and the starting position of the next element is greater than the starting position of the previous element, then fast copy is supported. Otherwise, fast copy is not supported. Through multi-level judgment conditions, files that support fast copy are accurately screened out, avoiding incorrect processing of files that are not suitable for fast copy, improving the accuracy and effectiveness of the fast copy process, and thus optimizing the copy performance.
[0048] S3: Dynamically merge file fragments based on the location information and data length in the location set S' to obtain a new location set S".
[0049] In a specific embodiment, for any two adjacent elements in the ordered position set S', if the starting offset of the previous element plus its length plus the length between the two elements is greater than or equal to the starting offset of the latter element, the two elements are merged to obtain a new position set S". File fragments are dynamically merged based on position information and data length, which minimizes the number of file I / O times, integrates small file fragments into larger continuous data blocks, makes disk I / O closer to sequential I / O, and significantly improves the disk copy speed.
[0050] S4: Take out the fragment elements in the set S" in turn, read the data block whose starting offset on the disk is the starting offset of the element and whose length is the length of the element, and write the data to the corresponding target path until all file fragments are processed.
[0051] This method obtains a file set and its location set, sorts the positions of files that support fast copying, merges the fragments, and then copies them. It can convert disordered small file I / O operations into ordered large I / O operations, reduce disk seek time, and improve the copying efficiency of large batches of small files.
[0052] Figure 2 A flowchart of a method for quickly copying a large number of small files according to a specific embodiment of the present application is shown. Figure 2 As shown, the specific steps include:
[0053] Step #1: Get the set of all files in the path to be copied, denoted as F = {F1, F2, F...Fn};
[0054] Step #2: For any file Fn in the file set F obtained in step #1, obtain the set of all locations of the file on the disk, denoted as S = {S1, S2, S...Sn}
[0055] Step #3: Determine the files and their location set in step #2, by Figure 3 In the "Method flow for determining whether a small file supports fast copy", determine whether fast copy is supported. If so, add all the location sets of the file to the new location set S' and jump to step #4. Otherwise, record the file to the new common copy file set F' and directly jump to step #7.
[0056] Step #4: For the new set S'={S'1, S'2, S'...S'n} obtained in step #3, reorder the set according to [S'n].StartOffset to obtain an ordered set S', and jump to step #5
[0057] Step #5: For the ordered set S' obtained in step #4, for any two ordered elements S'n and S'n+1, if [S'n].StartOffset+[S'n].Length+MAXGAP>=[S'n+1].StartOffset, merge the two elements S'n and S'n+1 to finally obtain a new set S"={S"1,S"2,S"......S"n}, and jump to step #6;
[0058] Step #6: For the set S”={S”1, S”2, S”.....S”n} obtained in step #5, take out the elements S”n one by one, read the data block with the starting offset of [S”n].StartOffset and the length of [S”n].Length on the disk, and write the data in the data block to the corresponding target path until all the file fragments are processed and the process ends;
[0059] Step #7: For the new file set F'={F'1, F'2, F'.....F'n}, remove the element F'n in turn and perform a normal file copy until all files are processed and the process ends.
[0060] Figure 3 A flowchart of a method for determining whether a small file supports fast copying according to a specific embodiment of the present application is shown. Figure 3 As shown, the determination method specifically includes the following steps:
[0061] Step 1: Check if the file size is 0. If not, jump to step #2. Otherwise, fast copy is not supported.
[0062] Step 2: Determine whether the file can resolve the location set. If yes, jump to step #3. Otherwise, fast copy is not supported.
[0063] Step 3: Get the location set S = {S1, S2, S...Sn} of the file on the disk, and take out the two ordered elements Sn and Sn+1 in turn. If [Sn].StartOffset>0 and [Sn+1].StartOffset>[Sn].StartOffset, the file supports fast copying, otherwise it does not support fast copying.
[0064] In the above embodiment, it is assumed that the set F = {F1, F2, F...Fn} is the set of small files to be copied, where Fn represents a small file. It is assumed that the set S = {S1, S2, S...Sn} is the set of positions of small files to be copied on the disk, where Sn contains two elements: [Sn].StartOffset represents the starting position of the current fragment file on the disk, and [Sn].Length represents the ending position of the current fragment file on the disk, that is, the length of the fragment file. MAXGAP is the data length between two fragments on the disk.
[0065] Figure 4 FIG. 1 shows an architecture diagram of a system for quickly copying a large number of small files according to an embodiment of the present application. Figure 4 As shown, the system includes a file acquisition unit 401, a fast copy judgment unit 402, a dynamic merging unit 403 and a data writing unit 404, wherein the file acquisition unit 401 is configured to acquire a set F of all files of a path to be copied, and for any file in the set F, acquire a set S of all locations of the file on the disk; the fast copy judgment unit 402 is configured to add all location sets of the file to a new location set in response to the file supporting fast copy, and reorder the new location set according to the starting position of the current fragment file on the disk to obtain an ordered location set S'; the dynamic merging unit 403 is configured to dynamically merge file fragments based on the position information in the location set S' and the data length to obtain a new location set S''; the data writing unit 404 is configured to sequentially take out the fragment elements in the set S'', read a data block whose starting offset on the disk is the starting offset of the element and whose length is the length of the element, and write the data to the corresponding target path until all file fragments are processed.
[0066] Continue to refer Figure 5a-5c , Figure 5a-5c is a test result diagram of a specific embodiment of the present application, such as Figure 5a , Experimental environment: the source disk is a mechanical disk, the target disk is an SSD, the total number of nodes is 270565 (the total number of files is 150334, the total number of folders is 120231), and the total file size is 26.9G; Experiment 1: Based on Windows copying, the total time is 37 minutes; Experiment 2: Based on FastCopy copying, the total time is 16 minutes (such as Figure 5b ), Experiment 3: Based on the large batch of small files fast copy solution of this application, the total time taken is 7 minutes (such as Figure 5c ), it can be seen that the large batch small file fast copying method of the present application can effectively improve the copying performance of massive small files.
[0067] Reference below Figure 6, which shows a schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application. Figure 6 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0068] like Figure 6 As shown, the computer system includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0069] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.
[0070] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above functions defined in the method of the present application are executed. It should be noted that the computer-readable storage medium of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wireline, optical cable, RF, etc., or any suitable combination of the foregoing.
[0071] Computer program code for performing the operations of the present application may be written in one or more programming languages or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0072] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0073] The modules involved in the embodiments of the present application may be implemented by software or by hardware.
[0074] As another aspect, the present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or it may exist independently and not be assembled into the electronic device. The above computer-readable storage medium carries one or more programs. When the above one or more programs are executed by the electronic device, the electronic device: obtains a set F of all files of the path to be copied, and obtains a set S of all locations of any file in the set F on the disk; in response to the file supporting fast copying, adds all the location sets of the file to a new location set, reorders the new location set according to the starting position of the current fragmented file on the disk to obtain an ordered location set S'; dynamically merges file fragments based on the location information and data length in the location set S' to obtain a new location set S"; sequentially takes out the fragment elements in the set S", reads a data block whose starting offset on the disk is the starting offset of the element and whose length is the length of the element, and writes the data to the corresponding target path until all file fragments are processed.
[0075] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.
Claims
1. A method for quickly copying a large number of small files, characterized in that: include: S1: Obtain a set F of all files in the path to be copied, and obtain a set S of all locations of any file in the set F on the disk; S2: in response to the file supporting fast copy, all the location sets of the file are added to a new location set, and the new location set is reordered according to the starting position of the current fragmented file on the disk to obtain an ordered location set S'; S3: dynamically merge file fragments based on the location information and data length in the location set S' to obtain a new location set S". S4: Take out the fragment elements in the set S" in sequence, read the data block whose starting offset on the disk is the starting offset of the element and whose length is the length of the element, and write the data to the corresponding target path until all file fragments are processed.
2. The method for quickly copying a large number of small files according to claim 1, characterized in that: The step S2 further includes: in response to the file not supporting fast copying, recording the file into a new copy file set F', and sequentially taking out each file for copying until all files are processed.
3. The method for quickly copying a large number of small files according to claim 1 or 2, characterized in that: The method for determining whether the file supports fast copy in S2 specifically includes: S21: Determine whether the file size is 0, if it is 0, fast copy is not supported; S22: Determine whether the file can be parsed into the location set, if not, fast copy is not supported; S23: Check two ordered elements in the position set S. If the starting position of the previous element is greater than 0 and the starting position of the next element is greater than the starting position of the previous element, fast copying is supported. Otherwise, fast copying is not supported.
4. The method for quickly copying a large number of small files according to claim 1, characterized in that: The S3 specifically includes: for any two adjacent elements in the ordered position set S', if the starting offset of the previous element plus its length plus the length between the two elements is greater than or equal to the starting offset of the latter element, then the two elements are merged to obtain a new position set S".
5. A computer-readable storage medium having one or more computer programs stored thereon, characterized in that: When the one or more computer programs are executed by a computer processor, the method according to any one of claims 1 to 4 is implemented.
6. A fast copy system for large batches of small files, characterized in that: include: A file acquisition unit, configured to acquire a set F of all files whose paths need to be copied, and for any file in the set F, acquire a set S of all locations of the file on the disk; a fast copy judgment unit, configured to, in response to the file supporting fast copy, add all the location sets of the file to a new location set, and reorder the new location set according to the starting position of the current fragmented file on the disk to obtain an ordered location set S'; A dynamic merging unit, configured to dynamically merge file fragments based on the position information and data length in the position set S' to obtain a new position set S''; The data writing unit is configured to sequentially retrieve the fragment elements in the set S", read a data block whose starting offset on the disk is the starting offset of the element and whose length is the length of the element, and write the data to a corresponding target path until all file fragments are processed.
7. The fast copy system for large batches of small files according to claim 6, characterized in that: The fast copy determination unit is further configured to record the file into a new copy file set F' in response to the file not supporting fast copy, and sequentially take out each file for copying until all files are processed.
8. The fast copying system for large batches of small files according to claim 6, characterized in that: The method for determining whether the file supports fast copy in the fast copy determination unit specifically includes: S21: Determine whether the file size is 0, if it is 0, fast copy is not supported; S22: Determine whether the file can be parsed into the location set, if not, fast copy is not supported; S23: Check two ordered elements in the position set S. If the starting position of the previous element is greater than 0 and the starting position of the next element is greater than the starting position of the previous element, fast copying is supported. Otherwise, fast copying is not supported.
9. The fast copy system for large batches of small files according to claim 6, characterized in that: The dynamic merging unit specifically includes: for any two adjacent elements in the ordered position set S', if the starting offset of the previous element plus its length plus the length between the two elements is greater than or equal to the starting offset of the latter element, then the two elements are merged to obtain a new position set S".
10. The fast copy system for large batches of small files according to claim 6, characterized in that: In the copy operation, based on large memory cache and multi-threaded processing, part of the data to be copied is cached and multiple threads are started simultaneously to process different tasks. The cache size and the number of threads are adjusted dynamically based on the size of the file to be copied and the system.