File copying method and device, equipment, medium and product

Abstract

The invention discloses a file copying method, device and equipment, a medium and a product. The method comprises the steps of generating a corresponding to-be-transmitted fragment information list according to file copying progress information of a to-be-copied target file; and transmitting the fragments in the to-be-transmitted fragment information list to a receiver in parallel according to the copy control parameters until the target file is completely copied. According to the scheme, whether the breakpoint exists in the file copying process or not is determined according to the file copying progress information of the target file, so that the to-be-transmitted fragment information list used for guiding fragment transmission is generated, then the fragments in the to-be-transmitted fragment information list are transmitted to the receiver in a parallel transmission mode, breakpoint continuous transmission can be supported, and the transmission efficiency is improved. According to the file copying method and device, the fault tolerance of file copying is improved, the file copying transmission efficiency can be effectively improved while the data correctness and consistency are guaranteed through the parallel transmission design of the fragments, and the customization requirement of file copying can be met through the copying control parameters.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a file copying method, device, equipment, medium and product. Background Art

[0002] At present, cloud computing environments often encounter the copying of static large files or very large files such as mirror data, backup data, and disk data. These data are important production data. Therefore, while seeking data correctness and consistency, how to effectively improve the transmission efficiency of file copying is a technical problem that needs to be solved. Summary of the invention

[0003] The present invention provides a file copying method, device, equipment, medium and product, so as to effectively improve the transmission efficiency of file copying while ensuring the correctness and consistency of data.

[0004] According to one aspect of the present invention, a file copying method is provided, the method comprising:

[0005] Generate a corresponding list of fragment information to be transferred according to the file copy progress information of the target file to be copied;

[0006] According to the copy control parameters, each fragment in the fragment information list to be transmitted is transmitted to the receiving party in parallel until all the target files are copied.

[0007] According to another aspect of the present invention, there is provided a file copying device, the device comprising:

[0008] A list generation module, used to generate a corresponding list of to-be-transmitted fragment information according to the file copy progress information of the target file to be copied;

[0009] The file copy module is used to transmit each fragment in the fragment information list to be transmitted to the receiving party in parallel according to the copy control parameters until all the target files are copied.

[0010] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0011] at least one processor; and

[0012] a memory communicatively connected to the at least one processor; wherein,

[0013] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the file copying method described in any embodiment of the present invention.

[0014] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the file copy method described in any embodiment of the present invention when executed.

[0015] According to another aspect of the present invention, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the file copying method according to any embodiment of the present invention is implemented.

[0016] The technical solution of the embodiment of the present invention generates a corresponding list of fragment information to be transmitted according to the file copy progress information of the target file to be copied; and transmits each fragment in the list of fragment information to be transmitted to the receiver in parallel according to the copy control parameters until all the target files are copied. This solution can determine whether there is a breakpoint in the file copy process according to the file copy progress information of the target file, thereby generating a list of fragment information to be transmitted for guiding the fragment transmission, and then transmitting each fragment in the list of fragment information to be transmitted to the receiver by parallel transmission, which can support breakpoint resumption and increase the fault tolerance of file copying. The parallel transmission design of the fragments can effectively improve the transmission efficiency of file copying while ensuring data correctness and consistency; in addition, the customized requirements of file copying can be realized by using the copy control parameters.

[0017] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 is a flowchart of a file copying method provided according to Embodiment 1 of the present invention;

[0020] Figure 2 is a flowchart of a file copying method provided according to Embodiment 2 of the present invention;

[0021] Figure 3 is a structural diagram of a file copying system provided according to Embodiment 3 of the present invention;

[0022] Figure 4is a program execution flow chart of a control channel provided according to Embodiment 3 of the present invention;

[0023] Figure 5 is a program execution flow chart of a data channel provided according to Embodiment 3 of the present invention;

[0024] Figure 6 is a schematic diagram of a file transmission effect provided according to Embodiment 3 of the present invention;

[0025] Figure 7 is a schematic diagram of file copy progress information changes provided according to Embodiment 3 of the present invention;

[0026] Figure 8 is a structural schematic diagram of a file copying device provided according to a fourth embodiment of the present invention;

[0027] Fig. 9 It is a structural schematic diagram of an electronic device for implementing the file copying method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] Embodiment 1

[0031] Figure 1This is a flowchart of a file copying method provided in the first embodiment of the present invention. This embodiment is applicable to the case of parallel transmission of static large files in pieces. The method can be executed by a file copying device. The file copying device can be implemented in the form of hardware and / or software. The file copying device can be configured in an electronic device. Figure 1 As shown, a file copying method provided in this embodiment includes the following steps:

[0032] S110 . Generate a corresponding list of to-be-transmitted fragment information according to the file copy progress information of the target file to be copied.

[0033] Among them, the target file can refer to a large static file to be copied. The characteristics of this type of file are large data volume and will not be rewritten during the transmission process. It is often used in scenarios such as data backup, replication, and migration. For example, the target file can be a mirror data file, backup data file, and disk data file involved in a cloud computing environment.

[0034] The file copy progress information can be used to characterize the overall data copy transmission progress of the target file. The file copy progress information can include the completed progress range of the data copy, such as the completed progress range of 100-1000, which can be used to indicate that the data in the offset range of (100, 1000) (in bytes) in the target file has been copied. It can be understood that by reading the file copy progress information, it is possible to clearly understand whether there are breakpoints in the copy process of the target file, which is a comparison condition for subsequently generating the corresponding list of fragment information to be transmitted.

[0035] The list of information about the fragments to be transmitted can be understood as a list of information for storing the fragments of the target file to be transmitted, and the list can include the fragment ranges of each fragment to be transmitted. A fragment can refer to a number of data blocks obtained by dividing the target file according to the set fragment size, and each fragment has a fragment range corresponding to it, which is used to represent the actual data range of the corresponding fragment. For example, the fragment range (20,100) represents 80 bytes with an offset of 20.

[0036] In an embodiment of the present invention, the file copy progress information of the current target file to be copied can be obtained to determine whether there is a breakpoint in the current file copy process, and then generate a list of fragment information to be transmitted for guiding the subsequent fragment transmission. In a specific embodiment, a complete fragment information list can be initialized and constructed according to the file size and fragment size of the target file, and then each initial fragment in the fragment information list is matched with the obtained file copy progress information. If the offset range recorded in the file copy progress information includes a certain / certain initial fragments, it indicates that the corresponding initial fragment has been copied. Therefore, after removing the initial fragment that has been copied from the aforementioned fragment information list, the corresponding fragment information list to be transmitted can be obtained.

[0037] S120 , transmitting each fragment in the list of fragment information to be transmitted to the receiving party in parallel according to the copy control parameters until all the target files are copied.

[0038] The copy control parameters may refer to the relevant parameters for controlling the data copy task of the target file, such as but not limited to: the number of concurrent operations, compression method, encryption method, speed limit size, fragment size, etc.

[0039] In an embodiment of the present invention, the copy control parameters pre-configured for the data copy task of the target file, such as the number of concurrent operations, compression method, and encryption method, can be obtained, and then multiple processes can be concurrently started according to the above copy control parameters to perform the reading, transmission, and writing operations of the shard data, so as to make full use of the system bandwidth resources, so as to quickly transmit each shard in the list of shard information to be transmitted to the receiving party until the target file is completely copied. In a specific embodiment, the pipe technology of the Linux system can be used to implement the reading, transmission, and writing operations of the shard data.

[0040] Further, based on the above-mentioned embodiment of the invention, when the sender and the receiver corresponding to the target file are different hosts, the method further includes:

[0041] Before executing the file copy process, a session connection and a data transmission connection are established between the sender and the receiver.

[0042] Specifically, the file copy method provided by the embodiment of the present invention supports the following two data copy scenarios:

[0043] 1) Direct copying between local disks, that is, the sender and receiver of the target file are the same host, and the data copy operation is completed in the same storage medium (such as hard disk) in the host;

[0044] 2) Cross-device copying, specifically including: ① completing data copy operations between different storage media in the same host; ② cross-host (i.e., cross-network), that is, the sender and receiver corresponding to the target file are different hosts.

[0045] Among them, network factors need to be considered in cross-host scenarios, that is, before copying the target file, it is necessary to first establish a session connection and a data transmission connection between the sender and the receiver, wherein the session connection is a connection for message communication, and the data transmission connection is a connection for file data transmission. The session connection and the data transmission connection can adopt some existing communication protocols or self-developed communication services, which are not specifically limited in this embodiment.

[0046] The technical solution of the embodiment of the present invention generates a corresponding list of fragment information to be transmitted according to the file copy progress information of the target file to be copied; and transmits each fragment in the list of fragment information to be transmitted to the receiver in parallel according to the copy control parameters until all the target files are copied. This solution can determine whether there is a breakpoint in the file copy process according to the file copy progress information of the target file, thereby generating a list of fragment information to be transmitted for guiding the fragment transmission, and then transmitting each fragment in the list of fragment information to be transmitted to the receiver by parallel transmission, which can support breakpoint resumption and increase the fault tolerance of file copying. The parallel transmission design of the fragments can effectively improve the transmission efficiency of file copying while ensuring data correctness and consistency; in addition, the customized requirements of file copying can be realized by using the copy control parameters.

[0047] Embodiment 2

[0048] Figure 2 The flowchart of a file copying method provided in the second embodiment of the present invention is further optimized and expanded based on the above implementation, and can be combined with various optional technical solutions in the above implementation. Figure 2 As shown, a file copying method provided in the second embodiment specifically includes the following steps:

[0049] S210: Read file copy progress information from a file copy progress file configured for a target file.

[0050] The file copy progress file may refer to a file used to record the file copy progress information throughout the file copy process. The file may be stored in the same folder as the target file and may be deleted after the target file is successfully copied.

[0051] In an embodiment of the present invention, in order to determine whether there is a breakpoint in the copying process of the target file, thereby generating a list of fragment information to be transmitted for subsequent guidance of fragment transmission, the corresponding file copy progress information can be first read from the file copy progress file configured corresponding to the target file to determine the fragments of the current target file that have been copied.

[0052] S220: Initialize and construct an initial list of fragment information to be transmitted according to the file size and fragment size corresponding to the target file.

[0053] In an embodiment of the present invention, in order to generate a subsequent list of fragment information to be transmitted, an initial list of fragment information to be transmitted can be initialized based on the file size of the target file and the fragment size specified by the user (if the user does not specify, the default fragment size is used), wherein the initial list of fragment information to be transmitted includes several initial fragments obtained after logically dividing the target file according to the fragment size, and the fragment range corresponding to each initial fragment represents the actual data range of the fragment.

[0054] S230: Match the completed progress range recorded in the file copy progress information with the fragment range of each initial fragment in the initial list of fragment information to be transmitted, and remove the initial fragments that have been copied in the initial list of fragment information to be transmitted, so as to generate a list of fragment information to be transmitted.

[0055] In an embodiment of the present invention, the fragment range of each initial fragment in the initial list of fragment information to be transmitted can be compared one by one with the completed progress range recorded in the file copy progress information. If the completed progress range contains the fragment range corresponding to a certain initial fragment, indicating that the corresponding initial fragment has been copied, the fragment range corresponding to the corresponding initial fragment is removed from the initial list of fragment information to be transmitted, thereby obtaining a list of fragment information to be transmitted.

[0056] S240: Create a temporary file corresponding to the target file at the receiving end.

[0057] The temporary file may refer to a sparse file (sparse file) created at the receiver of the target file for receiving the fragmented data.

[0058] In the embodiment of the present invention, a temporary sparse file for receiving the slice data of the target file can be created at the receiving end. The use of the sparse file can save storage space and improve the writing speed of the slice data.

[0059] S250, reading the list of fragment information to be transmitted and the copy control parameters; wherein the copy control parameters include at least one of the following: concurrent number, compression method, encryption method, and speed limit.

[0060] In an embodiment of the present invention, the copy control parameters can be passed in from the command line or the environment variables. When these parameters are not passed in, the program can use the default parameter values, for example: the concurrency is 1 (when the number of CPU cores is 1) or 2 (when the number of CPU cores is greater than 1), the compression method is lz4, the speed limit is 50MB / s, and the fragment size is 36MB.

[0061] S260. Concurrently transmit and write each fragment in the fragment information list to be transmitted into a temporary file using the created communication channel; wherein the communication channel includes at least the following information: fragment data reading control parameters, copy control parameters, transmission connection mode, and fragment data writing control parameters.

[0062] The communication pipeline may at least include a pipeline in the Linux system, which can be used to implement inter-process communication. The read control parameters of the sliced ​​data may include but are not limited to: the input file of the sliced ​​data (i.e., the source file corresponding to the target file), the number of data blocks to be skipped specified in the input file, the size of the data block, the number of data blocks to be copied, etc. The write control parameters of the sliced ​​data may include but are not limited to: the output file of the sliced ​​data (i.e., the temporary file corresponding to the target file), the number of data blocks to be skipped specified in the output file, etc.

[0063] In an embodiment of the present invention, based on each shard in the list of shard information to be transmitted, multiple processes can be started concurrently to respectively execute the data copy task of each shard, and the data copy task of each shard can be implemented using the pipe command (pipe) in the Linux system. The following is an example of a pipeline construction of shard data: read source file shard data | compress | encrypt | speed limit | transmission connection | decrypt | decompress | write shard data to a temporary file of the target file, where "|" represents a pipe symbol.

[0064] S270: After the segments are written into the temporary files, the file copy progress information is updated.

[0065] In an embodiment of the present invention, after each segment is transferred and written into the temporary file corresponding to the target file, the file copy progress information will be updated so that when the data copy task is re-initiated next time, a new list of segment information to be transferred can be generated according to the breakpoint, so that the segment data that has been transferred does not need to be copied repeatedly.

[0066] Further, based on the above-mentioned embodiment of the invention, S270 specifically includes the following steps:

[0067] S2701. After all the shard data in the shard is written into the temporary file, a target progress range corresponding to the shard is generated;

[0068] S2702, matching the target progress range with the completed progress range in the file copy progress information, and if the target progress range and the completed progress range do not overlap at all, adding the target progress range to the file copy progress information; if the target progress range and the completed progress range overlap, merging the target progress range and the completed progress range;

[0069] S2703: Save the updated file copy progress information to the file copy progress file corresponding to the target file.

[0070] In an embodiment of the present invention, the updating process of the file copy progress information is as follows: after the shards are written into the temporary file, a corresponding target progress range can be generated, that is, the shard range of the copied shard data; then the target progress range is compared with the completed progress range recorded in the file copy progress information, and if the two do not overlap, the target progress range is added to the file copy progress information; if the two overlap, the ranges of the two are merged, for example: the target progress range is (0, 4) and the completed progress range is (2, 6), then the result of merging the two is (0, 6); finally, the updated file copy progress information is saved in the file copy progress file corresponding to the target file.

[0071] Furthermore, based on the above-mentioned embodiment of the invention, updating the file copy progress information further includes: adding a mutex lock to the operation of updating the file copy progress information.

[0072] In an embodiment of the present invention, since the data copy tasks of each shard are executed concurrently, it is necessary to add mutual exclusion locks to the operations of updating the file copy progress information in turn, so as to avoid conflicts in the operations of updating the file copy progress information between the data copy tasks and ensure the accuracy of the data copy.

[0073] S280: Determine whether the updated completed progress range in the file copy progress information is consistent with the file range of the source file corresponding to the target file.

[0074] In an embodiment of the present invention, after the file copy progress information is updated, the progress information of the target file can be verified, that is, the updated completed progress range is compared with the file range of the source file corresponding to the target file to determine whether the two are consistent, thereby determining whether the target file has been completely copied.

[0075] S290, if yes, determine that the target file copy is complete, and rename the file name of the temporary file to the file name of the target file; if no, continue the copy process of the remaining segments until the target file copy is complete.

[0076] In an embodiment of the present invention, if the updated completed progress range is consistent with the file range of the source file corresponding to the target file, it indicates that the target file has been completely copied. At this time, the file name of the temporary file can be renamed to the file name of the target file, and the file copy progress file can be deleted. If the two are inconsistent, it indicates that there are still fragments that have not been copied, then the data copy process of the corresponding fragments will continue until the target file is completely copied.

[0077] The technical solution of the embodiment of the present invention can determine whether there is a breakpoint in the file copying process by reading the file copying progress information of the target file, thereby generating a list of fragment information to be transmitted for guiding the fragment transmission, and then each fragment in the list of fragment information to be transmitted is concurrently transmitted and written to a temporary file using the created communication pipeline, and the file copying process is completed after verifying the progress information of the target file. It can support breakpoint resumption and increase the fault tolerance of file copying. The parallel transmission design of the fragments can effectively improve the transmission efficiency of the file copy while ensuring the correctness and consistency of the data. In addition, by utilizing copy control parameters such as the number of concurrency, compression method, encryption method, and speed limit size, customized file copying requirements can be achieved.

[0078] Embodiment 3

[0079] Figure 3 This is a schematic diagram of the structure of a file copy system provided by Embodiment 3 of the present invention. Figure 3 As shown, the file copy system includes the following two components: a control channel 31 and a data channel 32. The above two components are specifically introduced below.

[0080] Control channel 31 is a management and scheduling program with the following functions:

[0081] ①Task scheduling: control the number of concurrent operations, compression method, encryption method, speed limit, shard size and other parameters, and then call the data channel 32 to handle the execution process of reading, sending, receiving, writing, etc.;

[0082] ②Information management: generate and transmit task information, progress information, etc.;

[0083] ③Logical shard calculation: According to the offset and shard size, the target file will be logically divided into multiple shards according to the shard size. These logical shards will be scheduled separately according to the offset and shard size to complete the data copy process of "reading, transferring, and writing" in parallel.

[0084] For example, if the target file F.img is 1000MB in size and the shard size is 64MB, there will be 16 logical shards, with shards 1 to 15 all being 64MB in size, and the last shard 16 being 40MB in size. These 16 logical shards will concurrently execute "read, transfer, write" according to the established concurrency rules. It is worth noting that in the scenario of local copy (i.e., performing file copy operations on the same host), "concurrency" may not play a big role. It is mainly used to deal with scenarios across machines, hardware, and networks.

[0085] Figure 4 This is the program execution flow chart of the control channel. Figure 4 As shown, by executing the step "Parsing copy control parameters", control parameters such as the number of concurrent operations, compression method, encryption method, speed limit size, and fragment size are determined. These parameters can be passed in from the command line or environment variables. When these parameters are not passed in, the program can use the default parameter values.

[0086] By executing the step "establishing a session connection object", the message communication method between the sender and the receiver is determined. A session connection will be established between hosts, and message-related transactions will be directly operated on the same host. The session connection established between hosts is not the focus of this solution. For example, the following session connection methods can be used:

[0087] 1) You can directly use the simple SSH (Secure Shell) protocol;

[0088] 2) You can develop your own session service. For example, the service listens to a fixed port address (such as tcp 8018) and provides a standard RESTFUL API interface (other interfaces are also acceptable, and this solution does not limit them, such as grpc, zmq and other transmission protocols), and messages are exchanged across hosts through this protocol.

[0089] The operation object of the data channel is determined by executing the step "loading the data channel", which responds to the subsequent step "executing the transmission program of the data channel" and plays a scheduling role.

[0090] By executing the step "loading the file copy progress information of the target file", it is determined whether there is a breakpoint in the copy process of the target file, which serves as a comparison condition for the subsequent step "generating a list of fragment information to be transmitted".

[0091] The list of slices that the target file needs to transfer at this moment is determined by executing the step "generating a list of slice information to be transferred". The slice size comes from the above-mentioned copy control parameter. If not specified, the default value is used. In one embodiment, the slice size can be set to a multiple of a power of 2, and the default value is 36MB.

[0092] In one embodiment, the process of generating the list of fragment information to be transmitted is as follows:

[0093] 1) Based on the file size and shard size of the target file, build a complete initial list of shard information to be transmitted. The corresponding pseudo code can be: offset = 0; total_shards = []; while offset <file_size:total_shards.append(offset,min(filesize,offset+block_size));offset+=block_size。

[0094] 2) Match the completed progress range recorded in the file copy progress information with the fragment range of each initial fragment in the initial list of fragment information to be transmitted, and remove the initial fragments that have been copied in the initial list of fragment information to be transmitted to generate a list of fragment information to be transmitted. For example: the completed progress range is (0, 75497472), and the fragment range of an initial fragment is (0, 37748736). Obviously, the fragment range of the initial fragment is fully included in the completed progress range, so the initial fragment needs to be removed.

[0095] In one embodiment, the file copy progress information includes two parts of information about the target file: ① filesize, a number used to indicate the file size in bytes; ② progress, a list whose elements are tuples used to indicate the offset range of the copied data. The following is an example of file copy progress information for a target file with a file size of 137MB:

[0096]

[0097] In the above example, the completed progress range includes: (0, 75497472) and (113246208, 43654912).

[0098] In one embodiment, the elements in the initial list of fragment information to be transmitted may be a tuple, which is used to represent the fragment range corresponding to the initial fragment, and the count starts from 0. Taking the file size of 137MB and the fragment size of 36MB as an example, the initial list of fragment information to be transmitted may be represented as follows:

[0099]

[0100]

[0101] In certain extreme cases, for example, the file size is 36MB+1B, then the range of a certain segment is (37748736, 37748737), indicating that the actual data range is a byte at offset 37748736, that is, the last byte.

[0102] In certain extreme cases, for example, the file size is 1B, there is only one fragment, and the fragment range is (0, 1), which means that the actual data range is a byte at offset 0, that is, the first byte.

[0103] Taking the 137MB file in the above example as an example, by comparing the completed progress range recorded in the file copy progress information with the initial list of fragment information to be transmitted, the corresponding list of fragment information to be transmitted can be generated, that is, fragment 3 (75497472, 113246208) has not been transmitted yet, that is, it is the fragment to be transmitted.

[0104] The task of transmitting the sharded data is started by executing the step "Execute the transmission program of the data channel", which is an asynchronous task.

[0105] After each shard is transferred, the file copy progress information will be updated and saved to the file copy progress file in the same folder as the target file. The file copy progress file is saved with a special file name, for example, if the target file name is test.1, the file copy progress file name is test.1.fscp. Since the transfer of each shard can be performed concurrently, it is necessary to add a mutex lock (such as Python's lock method) to the operation of updating the file copy progress information, so as to avoid conflicts between the operations of updating the file copy progress information between the data copy tasks, so as to ensure the accuracy of data copying.

[0106] In one embodiment, the update process of the file copy progress information may be: comparing the target progress range corresponding to the transferred completed segments with the completed progress range recorded in the file copy progress information, and if the two do not overlap, adding the target progress range to the file copy progress information; if the two overlap, merging the ranges of the two. The update process may involve the following four situations:

[0107] ① If the file copy progress information is empty, directly add the target progress range corresponding to the shard in the file copy progress information;

[0108] ② If the target progress range corresponding to the shard does not overlap with the completed progress range recorded in the file copy progress information, then the target progress range is added to the file copy progress information;

[0109] ③ If the target progress range corresponding to the shard is exactly adjacent to the completed progress range recorded in the file copy progress information, the two ranges are merged; for example, if the progress ranges of (0,4) and (4,8) are adjacent, they are merged into (0,8);

[0110] ④ If the target progress range corresponding to the shard partially overlaps with the completed progress range recorded in the file copy progress information, the two ranges are merged; for example, if the progress ranges of (0,4) and (2,6) overlap, they are merged into (0,6); for example, if the progress ranges of (0,4) and (1,3) are inclusive, they are merged into (0,4).

[0111] When the file transfer is successful, the file copy progress file corresponding to the target file is deleted. Otherwise, the file copy progress file is retained so that a new list of shard information to be transferred can be generated according to the breakpoint when the data copy task is re-initiated next time, so that the transferred shard data does not need to be copied repeatedly.

[0112] After the control channel 31 executes the step "execute the data channel transmission program", the data channel 32 starts working. Specifically, the data channel 32 is a data transmission program, which integrates many modules and tools for processing data, such as data reading tools, data transmission tools, data writing tools, speed limiting tools, compression tools, encryption tools, etc. The data channel 32 can be dynamically expanded according to the actual application scenario needs, and mainly has the following functional modules:

[0113] ①Sending module: read data, process data, and send data;

[0114] ②Receiving module: receiving data, processing data, and writing data;

[0115] ③ Verification module: After the file transfer is completed, check the file size;

[0116] ④ Processing module, including speed limit, compression, encryption and other functions, which are used in the "data processing" link in the above-mentioned sending module and receiving module.

[0117] Figure 5 This is the program execution flow chart of the data channel. Figure 5 As shown, the data transmission mode is determined by executing the step "establishing a data transmission connection object". Cross-host copying uses the network transmission protocol, while same-host copying directly reads and writes local files. The data transmission connection established between cross-hosts is not the focus of this solution. For example, the following data transmission connection mode can be used:

[0118] 1) You can directly use the simple SSH protocol;

[0119] 2) You can develop your own session service. For example, the service listens to a fixed port address (such as tcp 8019) and provides a standard RESTFUL API interface (other interfaces are also acceptable, and this solution does not limit them, such as grpc, zmq and other transmission protocols), and messages are exchanged across hosts through this protocol.

[0120] By executing the step "creating a temporary file corresponding to the target file", a temporary sparse file is created on the receiving side for receiving the fragmented data of the target file.

[0121] By executing the step of “reading the list of fragment information to be transmitted”, the fragment data is started to be transmitted concurrently. Among them, the control parameters such as the number of concurrent operations, compression method, encryption method, speed limit size, etc. are from the copy control parameters of the control channel 31.

[0122] In one embodiment, the transmission process of the shard data can use the pipe technology in the Linux system. The following is an example of the pipeline construction of the shard data: read the shard data of the source file|compress|encrypt|speed limit|transmission connection|decryption|decompression|write the shard data to the temporary file of the target file, where "|" represents the pipe symbol.

[0123] By executing data transmission tasks in this pipeline manner, various advanced function plug-ins and transmission protocols can be easily customized. At the same time, the pipeline consumes less resources and has high processing efficiency.

[0124] There are two situations involved in the step "stop transmission": one is that when a certain fragment fails to be transmitted, the transmission of other fragment data is immediately terminated and the transmission is stopped; the other is that when all fragments are fully transmitted, the transmission is stopped.

[0125] Verify the file copy progress information. If the completed progress range in the file copy progress information is consistent with the file range of the source file, the target file copy is complete. Check the file size. If the actual target file size is equal to the source file size, the target file copy is complete.

[0126] In one embodiment, if the progress information of the target file is all completed, but the size of the target file is inconsistent with that of the source file, unexpected external intervention may occur, such as the user performing certain operations on the target file. In this case, the target file needs to be deleted and the file copy process needs to be restarted.

[0127] When the target file is successfully transferred, the temporary file is renamed to the target file. Otherwise, no processing is done until the next restart to continue writing the shard data so that the data of the temporary file is gradually complete.

[0128] In order to enable those skilled in the art to better understand the embodiments of the present invention, the following is a specific example to illustrate the file copy method in the embodiments of the present invention. Taking cross-host copy as an example, and both the session connection and the data transmission connection use the SSH protocol, the file transmission effect example is as follows: Figure 6 As shown in the figure, the local host is the machine that initiates the copy request, and the file to be copied is on this machine; the remote host is the machine that receives the file to be copied, and the file will be copied to this machine.

[0129] File copying involves the following key processes:

[0130] 1) Create a sparse file on the remote host: dd if= / dev / zero of= / tmp / test.1.data count=0seek=143654912bs=1; where the test.1.data file is a temporary file used to store the fragmented data during the transmission process. After the file transfer is completed, it will be renamed to the specified target file name / tmp / test.1.

[0131] 2) Write the fragment data through the data transmission connection ssh, and use the lz4 compression method and pipe speed limit: dd if= / tmp / fscp.test / test.1skip=0count=576bs=65536|lz4|pv-L 1048576|sshroot@qc_gd1'lz4-d|seek_write_file--ifile---oseek 0--ofile / tmp / test.1.data'; The interpretation from left to right is to read the first fragment data (dd) on the local host, then use lz4 compression, use pipe speed limit (pv), transmit data through the ssh transmission protocol, use lz4 decompression on the remote host, and write the decompressed data to the sparse file according to the fragment offset. This process can start multiple processes concurrently by fragment to perform "reading, transmitting, and writing" of each fragment data to improve bandwidth utilization and accelerate file transfer. If you are copying data within the same host, just remove the ssh network transfer command.

[0132] 3) After each slice is transferred, the file copy progress information will be updated and written to the same folder as the target file. The file path is / tmp / test.1.fscp. Assuming that the order of slice transfer is: slice 2->slice 1–>slice 4–>slice 3, the file copy progress information will change as follows: Figure 7 shown.

[0133] 4) After the shards are transferred, the actual file size is verified. If the verification passes, the file " / tmp / test.1.fscp" storing the file copy progress information is deleted, and the temporary file is renamed to the specified file, that is, ' / tmp / test.1.data' is renamed to ' / tmp / test.1'.

[0134] The file copying method provided by the embodiment of the present invention has at least the following beneficial effects:

[0135] 1) The transmission design of logical sharding ensures the correctness and consistency of file copying;

[0136] 2) The concurrent transmission design of logical sharding effectively improves the transmission efficiency of file copying;

[0137] 3) Supports the ability to resume breakpoint transfer, which increases the fault tolerance of file copying. That is, there is no need to recalculate data consistency for the transferred part, which reduces CPU computing power and improves transmission efficiency.

[0138] 4) Customized file copying requirements can be achieved by using copy control parameters such as concurrent number, compression method, encryption method, speed limit, etc.

[0139] 5) The pipeline-based inter-process communication design is conducive to customized functions and modular expansion capabilities, while also improving data processing efficiency.

[0140] Embodiment 4

[0141] Figure 8 This is a schematic diagram of the structure of a file copying device provided by Embodiment 4 of the present invention. Figure 8 As shown, the device comprises:

[0142] A list generation module 41 is used to generate a corresponding list of to-be-transmitted fragment information according to the file copy progress information of the target file to be copied;

[0143] The file copy module 42 is used to transmit the fragments in the to-be-transmitted fragment information list to the receiving party in parallel according to the copy control parameters until all target files are copied.

[0144] The technical solution of the embodiment of the present invention generates a corresponding list of fragment information to be transmitted according to the file copy progress information of the target file to be copied through a list generation module; the file copy module transmits each fragment in the list of fragment information to be transmitted to the receiver in parallel according to the copy control parameters until all the target files are copied. This solution can determine whether there is a breakpoint in the file copy process according to the file copy progress information of the target file, thereby generating a list of fragment information to be transmitted for guiding the fragment transmission, and then transmitting each fragment in the list of fragment information to be transmitted to the receiver by parallel transmission, which can support breakpoint resumption and increase the fault tolerance of file copying. The parallel transmission design of the fragments can effectively improve the transmission efficiency of file copying while ensuring data correctness and consistency; in addition, the customized requirements of file copying can be realized by using the copy control parameters.

[0145] Further, based on the above-mentioned embodiment of the invention, the list generation module 41 includes:

[0146] A progress information reading unit, used to read the file copy progress information from the file copy progress file configured for the target file;

[0147] An initial list building unit, used to initialize and build an initial list of fragment information to be transmitted according to the file size and fragment size corresponding to the target file;

[0148] The list generating unit is used to match the completed progress range recorded in the file copy progress information with the fragment range of each initial fragment in the initial list of fragment information to be transmitted, and remove the initial fragments that have been copied in the initial list of fragment information to be transmitted to generate a list of fragment information to be transmitted.

[0149] Further, based on the above-mentioned embodiment of the invention, the file copy module 42 includes:

[0150] A temporary file creation unit, used to create a temporary file corresponding to the target file at the receiving end;

[0151] A list and parameter reading unit, used to read the list of fragment information to be transmitted and the copy control parameters; wherein the copy control parameters include at least one of the following: concurrent number, compression method, encryption method, speed limit size;

[0152] The fragment transmission unit is used to concurrently transmit and write each fragment in the fragment information list to be transmitted into a temporary file using the created communication channel; wherein the communication channel includes at least the following information: fragment data reading control parameters, copy control parameters, transmission connection mode, and fragment data writing control parameters;

[0153] A progress information updating unit, used to update the file copy progress information after the slice is written into the temporary file;

[0154] A file verification unit, used to determine whether the updated completed progress range in the file copy progress information is consistent with the file range of the source file corresponding to the target file;

[0155] The verification processing unit is used to determine that the target file copy is completed if yes, and rename the file name of the temporary file to the file name of the target file; if no, continue the copy process of the remaining segments until the target file copy is completed.

[0156] Further, based on the above-mentioned embodiment of the invention, the progress information updating unit is specifically used for:

[0157] After all the shard data in the shard is written into the temporary file, the target progress range corresponding to the shard is generated;

[0158] Match the target progress range with the completed progress range in the file copy progress information. If the target progress range and the completed progress range do not overlap at all, add the target progress range to the file copy progress information; if the target progress range and the completed progress range overlap, merge the target progress range and the completed progress range;

[0159] The updated file copy progress information is saved in the file copy progress file corresponding to the target file.

[0160] Further, based on the above-mentioned embodiment of the invention, the progress information updating unit is further used for:

[0161] Add a mutex lock for the operation of updating file copy progress information.

[0162] Furthermore, based on the above-mentioned embodiment of the invention, the file copying device further includes:

[0163] The connection establishment module is used to establish a session connection and a data transmission connection between the sender and the receiver before executing the file copy process when the sender and the receiver corresponding to the target file are different hosts.

[0164] The file copying device provided in the embodiment of the present invention can execute the file copying method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0165] Embodiment 5

[0166] Fig. 9A schematic diagram of an electronic device 50 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0167] like Fig. 9 As shown, the electronic device 50 includes at least one processor 51, and a memory connected to the at least one processor 51 in communication, such as a read-only memory (ROM) 52, a random access memory (RAM) 53, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 51 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 52 or the computer program loaded from the storage unit 58 to the random access memory (RAM) 53. In the RAM 53, various programs and data required for the operation of the electronic device 50 can also be stored. The processor 51, the ROM 52, and the RAM 53 are connected to each other via a bus 54. An input / output (I / O) interface 55 is also connected to the bus 54.

[0168] A number of components in the electronic device 50 are connected to the I / O interface 55, including: an input unit 56, such as a keyboard, a mouse, etc.; an output unit 57, such as various types of displays, speakers, etc.; a storage unit 58, such as a disk, an optical disk, etc.; and a communication unit 59, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 59 allows the electronic device 50 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0169] The processor 51 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 51 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 51 executes the various methods and processes described above, such as a file copy method.

[0170] In some embodiments, the file copy method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 58. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 50 via the ROM 52 and / or the communication unit 59. When the computer program is loaded into the RAM 53 and executed by the processor 51, one or more steps of the file copy method described above may be performed. Alternatively, in other embodiments, the processor 51 may be configured to perform the file copy method in any other appropriate manner (e.g., by means of firmware).

[0171] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0172] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0173] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0174] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0175] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0176] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0177] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0178] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

[0179] Embodiment 6

[0180] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the file copying method provided in any embodiment of the present application.

[0181] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

Claims

1. A file copying method, characterized in that: The method comprises: Generate a corresponding list of fragment information to be transferred according to the file copy progress information of the target file to be copied; The segments in the to-be-transmitted segment information list are transmitted in parallel to the receiving party according to the copy control parameters until all the target files are copied.

2. The method according to claim 1, characterized in that The step of generating a corresponding list of to-be-transmitted fragment information according to the file copy progress information of the target file to be copied comprises: Reading the file copy progress information from the file copy progress file configured for the target file; Initialize and construct an initial list of fragment information to be transmitted according to the file size and fragment size corresponding to the target file; The completed progress range recorded in the file copy progress information is matched with the fragment range of each initial fragment in the initial list of fragment information to be transmitted, and the initial fragments that have been copied in the initial list of fragment information to be transmitted are removed to generate the list of fragment information to be transmitted.

3. The method according to claim 1, characterized in that: The step of transmitting each fragment in the list of fragment information to be transmitted to the receiving party in parallel according to the copy control parameter until all the target files are copied includes: Creating a temporary file corresponding to the target file at the receiving side; Read the list of fragment information to be transmitted and the copy control parameters; wherein the copy control parameters include at least one of the following: concurrent number, compression method, encryption method, speed limit; Concurrently transmit and write each of the fragments in the list of fragment information to be transmitted into the temporary file using the created communication channel; wherein the communication channel includes at least the following information: read control parameters of fragment data, the copy control parameters, transmission connection mode, and write control parameters of fragment data; After the slice is written into the temporary file, updating the file copy progress information; Determine whether the updated completed progress range in the file copy progress information is consistent with the file range of the source file corresponding to the target file; If so, it is determined that the target file is copied completely, and the file name of the temporary file is renamed to the file name of the target file; if not, the copy process of the remaining segments is continued until the target file is copied completely.

4. The method according to claim 3, characterized in that After the slice is written into the temporary file, updating the file copy progress information includes: After all the shard data in the shard are written into the temporary file, a target progress range corresponding to the shard is generated; Matching the target progress range with the completed progress range in the file copy progress information; if the target progress range and the completed progress range do not overlap at all, adding the target progress range to the file copy progress information; if the target progress range and the completed progress range overlap, merging the target progress range and the completed progress range; The updated file copy progress information is saved in the file copy progress file corresponding to the target file.

5. The method according to claim 3, characterized in that: The updating of the file copy progress information further includes: A mutex is added to the operation of updating the file copy progress information.

6. The method according to claim 1, characterized in that When the sender and the receiver corresponding to the target file are different hosts, the method further includes: Before executing the file copy process, a session connection and a data transmission connection are established between the sender and the receiver.

7. A file copying device, characterized in that: The device comprises: A list generation module, used to generate a corresponding list of to-be-transmitted fragment information according to the file copy progress information of the target file to be copied; The file copy module is used to transmit each fragment in the to-be-transmitted fragment information list to the receiving party in parallel according to the copy control parameters until all the target files are copied.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the file copying method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the file copying method according to any one of claims 1 to 6 when executed.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the file copying method according to any one of claims 1 to 6 is implemented.

Images