A file copying method, device, server and storage medium

Abstract

The application provides a file copying method and device, a server and a storage medium, and relates to the technical field of networks, and solves the technical problems of slow file copying speed and low efficiency in related technologies. The method comprises the following steps: obtaining a file copying request sent by a terminal, wherein the file copying request comprises a file name of a to-be-copied file, a position of the to-be-copied file and a creation time of the to-be-copied file; determining a first position and a second position based on the file name of the to-be-copied file, the position of the to-be-copied file and the creation time of the to-be-copied file, wherein the first position is a storage position of file data of the to-be-copied file, and the second position is a storage position of metadata of the to-be-copied file; obtaining the file data of the to-be-copied file based on the first position and obtaining the metadata of the to-be-copied file based on the second position; and generating a target copied file based on the file data of the to-be-copied file and the metadata of the to-be-copied file.

Description

TECHNICAL FIELD

[0001] The present application relates to the network technical field, and particularly relates to a file copying method and device, a server and a storage medium. BACKGROUND

[0002] At present, in a distributed storage system, when file copying is performed, the metadata of an original file is first copied into a new file, the file data of the original file is obtained based on the copied metadata, and the file data of the original file is copied into the new file, so that the copying of the original file is completed, and the new file obtained is the copied file.

[0003] However, in the above method, since the metadata of some files is relatively large, the metadata of the original file is first copied, and then the file data of the original file is copied, which causes slow file copying speed and low efficiency. SUMMARY

[0004] The present application provides a file copying method and device, a server and a storage medium, which solve the technical problem of slow file copying speed and low efficiency caused by the fact that the metadata of some files is relatively large in the prior art.

[0005] In a first aspect, the present application provides a file copying method, which includes: obtaining a file copying request sent by a terminal, the file copying request including a file name of a to-be-copied file, a location of the to-be-copied file and a creation time of the to-be-copied file; determining a first location and a second location based on the file name of the to-be-copied file, the location of the to-be-copied file and the creation time of the to-be-copied file, the first location being a storage location of file data of the to-be-copied file, and the second location being a storage location of metadata of the to-be-copied file, the metadata of the to-be-copied file being used to represent attribute information of the to-be-copied file, and the attribute information of the to-be-copied file being used to indicate access and management of the file data of the to-be-copied file; obtaining the file data of the to-be-copied file based on the first location, and obtaining the metadata of the to-be-copied file based on the second location; and generating a target copied file based on the file data of the to-be-copied file and the metadata of the to-be-copied file.

[0006] Optionally, the generating of the target copied file based on the file data of the to-be-copied file and the metadata of the to-be-copied file includes: writing the file data of the to-be-copied file into file data of the target copied file, and writing target metadata of the to-be-copied file into metadata of the target copied file, the target metadata including attribute information of the metadata of the to-be-copied file other than the file name of the to-be-copied file, the location of the to-be-copied file and the creation time of the to-be-copied file.

[0007] In the present application, the server writes the file data of the to-be-copied file into the file data of the target copy file, so that the file data of the target copy file and the to-be-copied file are the same. The server writes the target metadata of the to-be-copied file into the metadata of the target copy file, and the target metadata includes attribute information of the metadata of the to-be-copied file except for the file name, the location and the creation time of the to-be-copied file. Thus, the attribute information of the metadata of the target copy file except for the file name, the location and the creation time is consistent with that of the to-be-copied file. In this way, the copying of the to-be-copied file is completed.

[0008] Optionally, the method further includes: adding a first operation into a target mapping queue, the first operation being the writing of the target metadata of the to-be-copied file into the metadata of the target copy file, the target mapping queue being used to indicate respective files corresponding to one or more operations; receiving a service request sent by the terminal, the service request being used to request processing of the metadata of the target copy file; in the case that the first operation is not completed, adding the service request into a request waiting queue, the request waiting queue being used to store a next arrived service request when a current operation of a file is not executed; in the case that the first operation is completed, deleting the service request from the request waiting queue, deleting the first operation from the target mapping queue and adding a second operation into the target mapping queue, the second operation being the processing of the target copy file based on the service request; and executing the second operation based on the target mapping queue.

[0009] In the present application, the server adds the first operation to the target mapping queue, the terminal sends a service request to the server, the server receives the service request sent by the terminal, in the case where the first operation is not completed, if the service request is executed at the same time, since the first operation and the service request process the metadata of the target copy file at the same time, there may be a conflict, at this time, the server adds the service request to the request waiting queue, that is, temporarily does not execute the service request, in the case where the first operation is completed, the first operation and the service request are not executed at the same time, that is, there is no operation conflict, at this time, the server processes the target copy file based on the service request, at the same time, since the request waiting queue is used to store the next arrived service request when the previous operation is not executed, when the service request is executed, the server deletes the service request from the request waiting queue to avoid the server repeatedly acquiring the service request in the request waiting queue, since the second operation is located in the target mapping queue, and the target mapping queue is used to indicate one or more operations corresponding to the file respectively, therefore, the server can determine that the service request requests to process the target copy file based on the target mapping queue, and then the server executes the second operation, that is, the server executes the second operation on the target copy file, in this way, it can be ensured that there is no operation conflict when the file data and the metadata of the to-be-copied file are acquired at the same time and the target copy file is generated, and the effectiveness of the access and management of the target copy file is reliably ensured while the file is quickly copied.

[0010] Optionally, the file copy request further includes a file system to which the to-be-copied file belongs, the file system to which the to-be-copied file belongs is an asynchronous file system or a synchronous file system, and the method further includes: in the case where the file system to which the to-be-copied file belongs is a synchronous file system, determining the second position based on a file name of the to-be-copied file, a position of the to-be-copied file, and a creation time of the to-be-copied file; acquiring the metadata of the to-be-copied file based on the second position; and acquiring the file data of the to-be-copied file based on the metadata of the to-be-copied file.

[0011] In the present application, in the case that the file system to which the to-be-copied file belongs is a synchronous file system, it is indicated that the user selects not to perform an asynchronous copy operation on the to-be-copied file. At this time, the server determines the second position, i.e., the storage position of the to-be-copied metadata, based on the file name of the to-be-copied file, the position of the to-be-copied file, and the creation time of the to-be-copied file. The server obtains the metadata of the to-be-copied file based on the second position. Since the metadata of the to-be-copied file is used to represent attribute information of the to-be-copied file, the attribute information of the to-be-copied file is used to indicate access and management of file data of the to-be-copied file, it is indicated that the attribute information of the to-be-copied file (i.e., the metadata of the to-be-copied file) includes the storage position of the file data of the to-be-copied file. Therefore, the server can obtain the to-be-copied file data based on the to-be-copied metadata, and further generate the target copy file based on the to-be-copied metadata and the to-be-copied file data, thereby ensuring the flexibility and self-selection of file copying.

[0012] In a second aspect, the present application provides a file copying device, comprising: an obtaining module, a determining module, and a processing module. The obtaining module is configured to obtain a file copy request sent by a terminal, wherein the file copy request comprises a file name of a to-be-copied file, a position of the to-be-copied file, and a creation time of the to-be-copied file. The determining module is configured to determine a first position and a second position based on the file name of the to-be-copied file, the position of the to-be-copied file, and the creation time of the to-be-copied file. The first position is a storage position of file data of the to-be-copied file, and the second position is a storage position of metadata of the to-be-copied file. The metadata of the to-be-copied file is used to represent attribute information of the to-be-copied file, and the attribute information of the to-be-copied file is used to indicate access and management of file data of the to-be-copied file. The obtaining module is further configured to obtain the file data of the to-be-copied file based on the first position, and obtain the metadata of the to-be-copied file based on the second position. The processing module is configured to generate a target copy file based on the file data of the to-be-copied file and the metadata of the to-be-copied file.

[0013] Optionally, the processing module is specifically configured to write the file data of the to-be-copied file into file data of the target copy file, and write target metadata of the to-be-copied file into metadata of the target copy file. The target metadata comprises attribute information of the metadata of the to-be-copied file, except for the file name of the to-be-copied file, the position of the to-be-copied file, and the creation time of the to-be-copied file.

[0014] Optionally, the apparatus further comprises a receiving module; the processing module is further configured to add the first operation into a target mapping queue, the first operation being the operation of writing the target metadata of the to-be-copied file into the metadata of the target copied file, the first operation corresponding to the target copied file, the target mapping queue being used to indicate respective files corresponding to one or more operations; the receiving module is configured to receive a service request sent by the terminal, the service request being used to request processing of the metadata of the target copied file; the processing module is further configured to, in a case where the first operation is not completed, add the service request into a request waiting queue, the request waiting queue being used to store a next arrived service request when a current operation of a file is not executed; the processing module is further configured to, in a case where the first operation is completed, delete the service request from the request waiting queue and delete the first operation from the target mapping queue and add a second operation into the target mapping queue, the second operation being an operation of processing the target copied file based on the service request; and the processing module is further configured to execute the second operation based on the target mapping queue.

[0015] Optionally, the file copy request further comprises a file system to which the to-be-copied file belongs, the file system to which the to-be-copied file belongs being an asynchronous file system or a synchronous file system; the determining module is further configured to, in a case where the file system to which the to-be-copied file belongs is a synchronous file system, determine the second location based on a file name of the to-be-copied file, a location of the to-be-copied file and a creation time of the to-be-copied file; the obtaining module is further configured to obtain the metadata of the to-be-copied file based on the second location; and the obtaining module is further configured to obtain file data of the to-be-copied file based on the metadata of the to-be-copied file.

[0016] In a third aspect, the present application provides a server, comprising: a processor and a memory configured to store processor-executable instructions; wherein the processor is configured to execute the instructions to implement any of the optional file copy methods in the first aspect.

[0017] In a fourth aspect, the present application provides a computer-readable storage medium, the computer-readable storage medium storing instructions, when the instructions in the computer-readable storage medium are executed by a server, the server is enabled to execute any of the optional file copy methods in the first aspect.

[0018] The file copying method, device, server and storage medium provided by the present application, the terminal sends a file copying request to the server, the server obtains the file copying request sent by the terminal, the server determines a first position and a second position based on the file name of the file to be copied, the position of the file to be copied and the creation time of the file to be copied, the server obtains the file data of the file to be copied based on the first position; the server obtains the metadata of the file to be copied based on the second position, the server generates a target copy file based on the file data of the file to be copied and the metadata of the file to be copied, in this way, the metadata of the file to be copied does not need to be obtained first and then the file data of the file to be copied, the file data and the metadata of the file to be copied can be obtained at the same time, and then the target copy file corresponding to the file to be copied can be quickly and accurately generated, thereby ensuring the speed and efficiency of file copying. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced.

[0020] Figure 1 The network architecture schematic diagram of the file copying system provided by the embodiments of the present application;

[0021] Figure 2 The flowchart schematic diagram of the file copying method provided by the embodiments of the present application;

[0022] Figure 3 The flowchart schematic diagram of another file copying method provided by the embodiments of the present application;

[0023] Figure 4 The flowchart schematic diagram of another file copying method provided by the embodiments of the present application;

[0024] Figure 5 The flowchart schematic diagram of another file copying method provided by the embodiments of the present application;

[0025] Figure 6 The internal structure schematic diagram of a server provided by the embodiments of the present application;

[0026] Figure 7 The structure schematic diagram of the above target mapping queue and the above request waiting queue provided by the embodiments of the present application;

[0027] Figure 8 The structure schematic diagram of a file copying device provided by the embodiments of the present application;

[0028] Figure 9 The structure schematic diagram of another file copying device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0029] The file copying method and device provided by the embodiments of the present application, the server and the storage medium will be described in detail below with reference to the accompanying drawings.

[0030] The terms "first" and "second" and the like in the description of the present application and the accompanying drawings are used to distinguish different objects, rather than to describe a specific order of the objects, for example, the first operation and the second operation are used to distinguish different operations, rather than to describe a specific order of the operations.

[0031] In addition, the terms "include" and "have" and any variations thereof mentioned in the description of the present application are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but can optionally include other steps or units not listed, or can optionally include other steps or units inherent to the process, method, product or device.

[0032] It should be noted that in the embodiments of the present application, the words "exemplary" or "for example" are used to represent an example, illustration or description. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the use of "exemplary" or "for example" is intended to present the relevant concept in a specific manner.

[0033] In the description of the present application, "a plurality of" means two or more, unless otherwise specified.

[0034] Based on the description in the background, in the related art, the data of the application is usually backed up periodically, and the copying of the dataset directory is completed. The copying command flow first creates a new file, and then copies the file data by reading and writing. It often takes more time to copy a directory set containing a large amount of data, especially in AI, machine learning business, the application dataset often contains a large number of small files in the directory. For small files, the file data copying time is relatively small, and the creation time of a large number of small files is relatively high. Based on this, the embodiments of the present application provide a file copying method, device, server and storage medium, which do not need to obtain the metadata of the file to be copied first and then obtain the file data of the file to be copied, can simultaneously obtain the file data and metadata of the file to be copied, and then quickly and accurately generate the target copy file corresponding to the file to be copied, thereby ensuring the speed and efficiency of file copying.

[0035] The file copying method, device, server and storage medium provided by the embodiments of the present application can be applied to a file copying system, such as Figure 1As shown, the file copying system includes a terminal 101, a server 102, and a storage device 103. Generally, in actual applications, the connections between the above devices can be wireless connections, and in order to intuitively represent the connection relationship between the devices, Figure 1 The connection between the terminal 101 and the server 102 is shown by a solid line.

[0036] The terminal 101 is configured to send a file processing request to the server 102 to access or manage a file stored in the storage device 103, and the file processing request is a file copying request, a file deletion request, or a file modification request.

[0037] The server 102 is configured to receive the file processing request sent by the terminal 101, and process the file stored in the storage device 103 based on the file processing request.

[0038] The storage device 103 is configured to store one or more files, wherein each file includes metadata of the file and file data of the file, the metadata of the file is used to represent attribute information of the file, and the attribute information of the file is used to indicate access and management of the file data of the file.

[0039] Exemplarily, the terminal executing the file copying method provided by the embodiment of the present application can be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) \ virtual reality (VR) device, and the like. The specific form of the terminal is not specially limited in the embodiment of the present application. The terminal can perform human-computer interaction with a user through one or more of a keyboard, a touchpad, a touch screen, a remote controller, voice interaction, a handwriting device, and the like.

[0040] Exemplarily, the server executing the file copying method provided by the embodiment of the present application can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN) services, and basic cloud computing services such as big data and artificial intelligence platforms.

[0041] The file copying method, device, server and storage medium provided by the embodiments of the present application are applied to the scene of fast file copying in a distributed storage system. Specifically, the server can simultaneously copy the metadata of a to-be-copied file and the file data of the to-be-copied file, so as to quickly and accurately obtain a copy file corresponding to the to-be-copied file and complete the copying of the to-be-copied file.

[0042] As shown in Figure 2 The file copying method provided by the embodiments of the present application can include S101-S105.

[0043] S101, the terminal sends a file copying request to the server.

[0044] The file copying request includes the file name of the to-be-copied file, the location of the to-be-copied file and the creation time of the to-be-copied file.

[0045] Optionally, the terminal sends the file copying request to the server through a cephFS (ceph file storage, which is a subsystem of ceph (distributed storage system) and is used for file storage, access and management).

[0046] S102, the server acquires the file copying request sent by the terminal.

[0047] In an implementation manner of the embodiments of the present application, after the server receives the file copying request sent by the terminal, a new file is created, the file data of the to-be-copied file and the metadata of the to-be-copied file (specifically, the file data of the to-be-copied file and the metadata of the to-be-copied file are stored in rados, and rados is a distributed object storage system composed of multiple OSDs (object storage cluster, object storage cluster, used for managing data stored in a disk space)) are read from a storage device, and the file data of the to-be-copied file and the metadata of the to-be-copied file are written into the new file, thereby completing the copying of the to-be-copied file.

[0048] Optionally, the above data is stored in rados according to a storage block of 4M bytes.

[0049] Optionally, the server includes an MDS (metadata server cluster, metadata server, used for caching and synchronizing distributed metadata), and the reading and writing of the above metadata of the to-be-copied file are completed by the MDS.

[0050] In an implementation manner of the embodiments of the present application, the creation of the new file is completed by an asynchronous operation module in the server.

[0051] S103, the server determines the first location and the second location based on a file name of the file to be copied, a location of the file to be copied, and a creation time of the file to be copied.

[0052] The first location is a storage location of file data of the file to be copied, and the second location is a storage location of metadata of the file to be copied. The metadata of the file to be copied is used to represent attribute information of the file to be copied, and the attribute information of the file to be copied is used to indicate access and management of the file data of the file to be copied.

[0053] In an implementation manner of the embodiment of the application, the server generates a file handle of the file to be copied based on the file name of the file to be copied, the location of the file to be copied, and the creation time of the file to be copied. The file handle is used to indicate the storage location of the file data of the file to be copied. The server obtains the storage location of the metadata of the file to be copied based on the file name of the file to be copied.

[0054] Optionally, the MDS includes a metadata processing module and a cache module. The server caches the file handle in the memory of the server, and sends a file creation request to the MDS. The metadata processing module in the MDS generates metadata of a new file based on the file creation request and the metadata of the file to be copied.

[0055] In an implementation manner of the embodiment of the application, the cache module further stores the metadata of the new file cached in the memory of the MDS in the rados, that is, completes the persistence of the metadata of the new file.

[0056] In an implementation manner of the embodiment of the application, when the cache module caches the metadata of the new file in the memory of the MDS, the server generates a journal log corresponding to the new file to ensure the transactionality of the metadata operation.

[0057] S104, the server obtains file data of the file to be copied based on the first location, and obtains metadata of the file to be copied based on the second location.

[0058] In an implementation manner of the embodiment of the application, the MDS in the server obtains the metadata of the file to be copied from the rados based on the storage location of the metadata to be copied. The server obtains the file data of the file to be copied from the rados based on the storage location of the file data to be copied.

[0059] S105, the server generates a target copy file based on the file data of the file to be copied and the metadata of the file to be copied.

[0060] It should be understood that the target copy file includes the file data and metadata of the target copy file.

[0061] Combination Figure 2 ,like Figure 3 As shown, in one implementation, the server generates a target copy file based on the file data and metadata of the file to be copied, including S1051.

[0062] S1051. The server writes the file data of the file to be copied into the file data of the target file to be copied, and writes the target metadata of the file to be copied into the metadata of the target file to be copied.

[0063] The target metadata includes attribute information of the metadata of the file to be copied, excluding the file name, location, and creation time of the file to be copied.

[0064] It should be understood that the server writes the file data of the file to be copied into the file data of the target file to be copied, so that the file data of the target file to be copied and the file to be copied are identical. The server writes the target metadata of the file to be copied into the metadata of the target file to be copied. The target metadata includes the attribute information of the metadata of the file to be copied, except for the file name, the location of the file to be copied, and the creation time of the file to be copied. This makes the other attribute information of the metadata of the target file to be copied consistent with that of the file to be copied. In this way, the copying of the file to be copied can be completed.

[0065] It is understandable that, since the filename, location, and creation time of the target copy file are all different from those of the file to be copied, the server writes the target metadata into the metadata of the target copy file, rather than writing all the metadata of the file to be copied into the metadata of the target copy file.

[0066] The technical scheme provided by the above embodiments can bring at least the following beneficial effects: as can be seen from S101-S105, the terminal sends a file copy request to the server, the server obtains the file copy request sent by the terminal, the server determines a first position and a second position based on the file name of the file to be copied, the position of the file to be copied and the creation time of the file to be copied, the server obtains the file data of the file to be copied based on the first position, the server obtains the metadata of the file to be copied based on the second position, and the server generates a target copy file based on the file data of the file to be copied and the metadata of the file to be copied. In this way, the metadata of the file to be copied does not need to be obtained before the file data of the file to be copied is obtained, the file data and the metadata of the file to be copied can be obtained at the same time, and then the target copy file corresponding to the file to be copied can be quickly and accurately generated, thereby ensuring the speed and efficiency of file copying.

[0067] In combination Figure 3 As shown in the foregoing embodiments, in an implementation manner, the file copy method further includes S106-S111. Figure 4

[0068] S106, the server adds the first operation to a target mapping queue.

[0069] The first operation is writing the target metadata of the file to be copied into the metadata of the target copy file, and the target mapping queue is used to indicate the file corresponding to each operation.

[0070] For example, the target mapping queue stores the operation corresponding to the target copy file in the form of a key-value pair.

[0071] Optionally, the target mapping queue includes at least one service request and a hash value corresponding to each service request, and the hash value corresponding to one service request is used to represent the path name of one file, and the path name of one file is used to represent the storage position of the file.

[0072] It can be understood that the server adds the first operation to an operation mapping queue, and the operation mapping queue is used to indicate the file corresponding to each operation, so that the server can determine the file corresponding to each operation based on the operation mapping queue.

[0073] S107, the terminal sends a service request to the server.

[0074] The service request is used to request processing of the metadata of the target copy file.

[0075] Optionally, the service request is used to request processing of the metadata of the target copy file in the form of access, deletion and modification.

[0076] ​S108, the server receives the service request sent by the terminal.

[0077] S109, in the case that the first operation is not completed, the server adds the service request to a request waiting queue.

[0078] The request waiting queue is used to store the operation corresponding to the next arrived service request when the current operation of a file is not completed.

[0079] It should be understood that in the case that the first operation is not completed, if the service request is executed simultaneously, since the first operation and the service request process the metadata of the target copy file at the same time, there may be a conflict, at this time, the server adds the service request to the request waiting queue, that is, temporarily does not execute the service request.

[0080] For example, it is assumed that the first operation is used to delete the metadata of the target copy file, the first operation is not completed, that is, the metadata of the to-be-copied file has not been written into the metadata of the target copy file, at this time, there is no metadata of the target copy file, so the first operation cannot be executed, that is, simultaneous execution of the first operation and the service request will cause a conflict.

[0081] S110, in the case that the first operation is completed, the service request is deleted from the request waiting queue, the first operation is deleted from the target mapping queue, and the second operation is added to the target mapping queue.

[0082] The second operation is processing the target copy file based on the service request.

[0083] In combination with the above S109, it should be understood that in the case that the first operation is completed, the simultaneous execution of the first operation and the service request does not occur, that is, there is no operation conflict, at this time, the server processes the target copy file based on the service request, and since the request waiting queue is used to store the operation corresponding to the next arrived service request when the current operation of a file is not completed, when the service request is executed, the server deletes the service request from the request waiting queue to avoid the server repeatedly obtaining the service request in the request waiting queue.

[0084] S111, the server executes the second operation based on the target mapping queue.

[0085] It can be understood that since the second operation is in the target mapping queue, and the target mapping queue is used to indicate the respective files corresponding to one or more operations, the server can determine that the service request requests processing the target copy file based on the target mapping queue, and then the server executes the second operation, that is, the server executes the second operation on the target copy file.

[0086] In an implementation form of the embodiment of the application, the first operation is completed by the MDS in the server, and when the first operation is completed, the MDS sends a first operation completion response to the server, and the server wakes up the service request in the request waiting queue to perform a subsequent step.

[0087] In the embodiment of the application, the server adds the first operation to the target mapping queue, the terminal sends a service request to the server, and the server receives the service request sent by the terminal. If the service request is executed at the same time when the first operation is not completed, there may be a conflict because the first operation and the service request process the metadata of the target copy file at the same time. At this time, the server adds the service request to the request waiting queue, that is, temporarily does not execute the service request. If the first operation is completed, the first operation and the service request are not executed at the same time, that is, there is no operation conflict. At this time, the server processes the target copy file based on the service request. At the same time, because the request waiting queue is used to store the next incoming service request when the previous operation is not executed, when the service request is executed, the server deletes the service request from the request waiting queue to avoid the server repeatedly obtaining the service request from the request waiting queue. Because the second operation is located in the target mapping queue, and the target mapping queue is used to indicate one or more operations corresponding to the file, the server can determine that the service request requests to process the target copy file based on the target mapping queue, and then the server executes the second operation, that is, the server executes the second operation on the target copy file. In this way, it can be ensured that there is no operation conflict when the file data and the metadata of the to-be-copied file are obtained at the same time and the target copy file is generated, and the effectiveness of the access and management of the target copy file is reliably ensured while the file is quickly copied.

[0088] In an implementation form of the embodiment of the application, the file copy request further includes a file system to which the to-be-copied file belongs. The file system to which the to-be-copied file belongs is an asynchronous file system or a synchronous file system. Figure 1 As shown in an implementation form, the file copy method further includes S112-S114. Figure 5

[0089] S112, in a case where the file system to which the to-be-copied file belongs is a synchronous file system, determining a second position based on a file name of the to-be-copied file, a position of the to-be-copied file, and a creation time of the to-be-copied file.

[0090] ​It should be understood that, in the case that the file system to which the to-be-copied file belongs is an asynchronous file system, it is explained that the user selects an asynchronous copy operation on the to-be-copied file, that is, the user selects to use a manner of simultaneously obtaining file data of the to-be-copied file and metadata of the to-be-copied file, and generating a target copy file, at this time, the server determines the first position and the second position based on a file name of the to-be-copied file, a position of the to-be-copied file, and a creation time of the to-be-copied file; in the case that the file system to which the to-be-copied file belongs is a synchronous file system, it is explained that the user does not select an asynchronous copy operation on the to-be-copied file, at this time, the server determines the second position, that is, the storage position of the to-be-copied metadata, based on the file name of the to-be-copied file, the position of the to-be-copied file, and the creation time of the to-be-copied file.

[0091] In an implementation manner of the embodiment of the present application, the server determines the first position and the second position based on the file name of the to-be-copied file, the position of the to-be-copied file, and the creation time of the to-be-copied file. <filesystem>The async command sets the file system creation operation to be asynchronous; specifically, the server modifies the fsmap (a mapping topology diagram containing multiple file system attributes), and the monitor (used to store the topology diagram of the cluster) synchronizes the fsmap to each cluster through the paxos protocol; the server determines the file system to which the requested file in the user request belongs based on the fsmap, and if it belongs to an asynchronous file system, the file creation request is processed asynchronously, otherwise it is processed synchronously (i.e. in the case of a synchronous file system to which the to-be-copied file belongs, the server determines the storage location of the to-be-copied metadata based on the file name of the to-be-copied file, the location of the to-be-copied file, and the creation time of the to-be-copied file, the server obtains the to-be-copied metadata based on the storage location of the to-be-copied metadata, and obtains the to-be-copied file data based on the to-be-copied metadata, and then generates the target copy file based on the to-be-copied metadata and the to-be-copied file data).

[0092] S113, the server obtains the metadata of the to-be-copied file based on the second location.

[0093] S114, the server obtains the file data of the to-be-copied file based on the metadata of the to-be-copied file.

[0094] In combination with the above S112, it should be understood that in the case of a synchronous file system to which the to-be-copied file belongs, the server determines the storage location of the to-be-copied metadata based on the file name of the to-be-copied file, the location of the to-be-copied file, and the creation time of the to-be-copied file, the server obtains the to-be-copied metadata based on the storage location of the to-be-copied metadata, since the metadata of the to-be-copied file is used to represent the attribute information of the to-be-copied file, the attribute information of the to-be-copied file is used to indicate access and management of the file data of the to-be-copied file, therefore the server can obtain the to-be-copied file data based on the to-be-copied metadata, and then can generate the target copy file based on the to-be-copied metadata and the to-be-copied file data, thereby ensuring the flexibility and self-selection of file copying.

[0095] It can be understood that the attribute information of the to-be-copied file is used to indicate access and management of the file data of the to-be-copied file, which means that the storage location of the file data of the to-be-copied file is included in the attribute information of the to-be-copied file (i.e. in the metadata of the to-be-copied file), so that the server can access the file data of the to-be-copied file based on the storage location of the file data of the to-be-copied file.

[0096] Specifically, the server obtains the storage location of the file data of the to-be-copied file from the metadata of the to-be-copied file, and obtains the file data of the to-be-copied file based on the storage location of the file data of the to-be-copied file.

[0097] In the embodiment of the present application, in the case that the file system to which the to-be-copied file belongs is a synchronous file system, it is illustrated that the user selects not to perform the asynchronous copy operation on the to-be-copy file. At this time, the server determines the second position, i.e., the storage position of the to-be-copied metadata, based on the file name of the to-be-copied file, the position of the to-be-copied file, and the creation time of the to-be-copied file. The server obtains the metadata of the to-be-copied file based on the second position. Since the metadata of the to-be-copied file is used to represent the attribute information of the to-be-copied file, the attribute information of the to-be-copied file is used to indicate the access and management of the file data of the to-be-copied file, it is illustrated that the storage position of the file data of the to-be-copied file is included in the attribute information of the to-be-copied file (i.e., in the metadata of the to-be-copied file). Therefore, the server can obtain the to-be-copied file data based on the to-be-copied metadata, and further generate the target copy file based on the to-be-copied metadata and the to-be-copied file data, thereby ensuring the flexibility and self-selection of file copying.

[0098] As shown in Figure 6 , it is an internal structure schematic diagram of a server provided by the embodiment of the present application, which comprises a master server module, an MDS module, and a rados module.

[0099] The master server module comprises an asynchronous request module and a master server cache module. The asynchronous request module is used to determine the storage position of the to-be-copied file data based on the file name of the to-be-copied file, the position of the to-be-copied file, and the creation time of the to-be-copied file. The asynchronous request module is also used to send a metadata construction request to the MDS module to construct the metadata of the target copy file. The master server cache module is used to cache the storage position of the to-be-copied file data, and is also used to cache the metadata of the target copy file and the storage position of the to-be-copied file data. The master server module is also used to obtain the to-be-copied file data based on the storage position of the to-be-copied file data. In this way, the server can generate the metadata of the target copy file and the file data of the target copy file.

[0100] The MDS module comprises a metadata processing module and an MDS cache module. The metadata processing module is used to construct the metadata of the target copy file. The MDS cache module is used to store the metadata of the target copy file in the memory of the MDS, and is also used to send the metadata of the target copy file to the master server cache module. The MDS cache module is also used to store the metadata of the target copy file in the rados module to realize the persistence of the target copy file.

[0101] a rados module, configured to store the file data of the target copy file, the metadata of the target copy file, the file data of the to-be-copied file, and the metadata of the to-be-copied file, and provide a read, write, or delete service of the file data or the metadata.

[0102] As shown in Figure 7 Fig. 1 is a structural diagram of the target mapping queue and the request waiting queue provided by an embodiment of the present application.

[0103] The target mapping queue includes a plurality of files and respective operations corresponding to each file.

[0104] The request waiting queue includes a plurality of service requests, wherein each service request includes an operation of the service request, a processing file of the service request, and a request time of the service request, and the plurality of service requests in the request waiting queue are arranged in chronological order.

[0105] It should be understood that when the operation of a file in the target mapping queue is completed, the server obtains the service request corresponding to the file from the request waiting queue, and when there are a plurality of service requests corresponding to the file, the server obtains the earliest service request from the plurality of service requests corresponding to the file, and determines the operation corresponding to the service request based on the service request and adds the operation to the target mapping queue, wherein the operation is located at a position corresponding to the file in the target mapping queue.

[0106] The embodiments of the present application can divide the functional modules of the server and the like according to the above method examples, for example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be realized in the form of hardware or in the form of a software functional module. It should be noted that the division of the modules in the embodiments of the present application is illustrative, and is only a logical functional division, and another division mode can be used in actual implementation.

[0107] In the case of dividing each functional module according to each function, Figure 8 Fig. 2 shows a possible structural diagram of the file copy device involved in the above embodiments, as shown in Figure 8 The file copy device 20 can include an obtaining module 201, a determining module 202, and a processing module 203.

[0108] The obtaining module 201 is configured to obtain a file copy request sent by a terminal, wherein the file copy request includes a file name of a to-be-copied file, a location of the to-be-copied file, and a creation time of the to-be-copied file.

[0109] The determining module 202 is configured to determine a first position and a second position based on a file name of the file to be copied, a position of the file to be copied, and a creation time of the file to be copied, the first position being a storage position of file data of the file to be copied, and the second position being a storage position of metadata of the file to be copied, the metadata of the file to be copied being used to represent attribute information of the file to be copied, and the attribute information of the file to be copied being used to indicate access and management of the file data of the file to be copied.

[0110] The obtaining module 201 is further configured to obtain the file data of the file to be copied based on the storage position of the file data of the file to be copied, and obtain the metadata of the file to be copied based on the storage position of the metadata of the file to be copied.

[0111] The processing module 203 is configured to generate a target copy file based on the file data of the file to be copied and the metadata of the file to be copied.

[0112] Optionally, the processing module 203 is specifically configured to write the file data of the file to be copied into file data of the target copy file, and write target metadata of the file to be copied into metadata of the target copy file, the target metadata including attribute information of the metadata of the file to be copied except for the file name of the file to be copied, the position of the file to be copied, and the creation time of the file to be copied.

[0113] Optionally, the apparatus further includes a receiving module 204.

[0114] The processing module 203 is further configured to add the first operation, which is the writing of the target metadata of the file to be copied into the metadata of the target copy file, into a target mapping queue, the first operation corresponding to the target copy file, and the target mapping queue being used to indicate respective files corresponding to one or more operations.

[0115] The receiving module 204 is configured to receive a service request sent by the terminal, the service request being used to request processing of the metadata of the target copy file.

[0116] The processing module 203 is further configured to add the service request into a request waiting queue in a case where the first operation is not completed, the request waiting queue being used to store a next arrived service request when a current operation of a file is not executed.

[0117] The processing module 203 is further configured to, in a case where the first operation is completed, delete the service request from the request waiting queue, delete the first operation from the target mapping queue, and add a second operation to the target mapping queue, the second operation being an operation of processing the target copy file based on the service request, the second operation corresponding to the target copy file.

[0118] The processing module 203 is further configured to execute the second operation based on the target mapping queue.

[0119] Optionally, the file copy request further includes a file system to which the to-be-copied file belongs, the file system to which the to-be-copied file belongs being an asynchronous file system or a synchronous file system.

[0120] The determining module 202 is further configured to, in a case where the file system to which the to-be-copied file belongs is a synchronous file system, determine the second location based on a file name of the to-be-copied file, a location of the to-be-copied file, and a creation time of the to-be-copied file.

[0121] The obtaining module 201 is further configured to obtain metadata of the to-be-copied file based on the second location.

[0122] The obtaining module 201 is further configured to obtain file data of the to-be-copied file based on the metadata of the to-be-copied file.

[0123] In a case where an integrated unit is adopted, Figure 9 A possible structural schematic diagram of the file copy device involved in the above embodiments is shown. As shown in Figure 9 The file copy device 30 can include a processing module 301 and a communication module 302. The processing module 301 can be configured to control and manage the actions of the file copy device 30. The communication module 302 can be configured to support the communication of the file copy device 30 with other entities. Optionally, as shown in Figure 9 The file copy device 30 can further include a storage module 303 configured to store the program codes and data of the file copy device 30.

[0124] The processing module 301 can be a processor or a controller. The communication module 302 can be a transceiver, a transceiver circuit, or a communication interface, etc. The storage module 303 can be a memory.

[0125] When the processing module 301 is a processor, the communication module 302 is a transceiver, and the storage module 303 is a memory, the processor, the transceiver, and the memory can be connected through a bus. The bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0126] It should be understood that the size of the sequence number of each process described above in various embodiments of the present application does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0127] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0128] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the system, device and unit described above can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0129] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment scheme.

[0130] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by using a software program, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium that can be accessed by a computer or data storage device including one or more servers, data centers, etc. integrated with the medium. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0131] The above description is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.< / filesystem>

Claims

1. A file copying method characterized by, The method comprises: obtaining a file copy request sent by a terminal, the file copy request comprising a file name of a file to be copied, a location of the file to be copied, and a creation time of the file to be copied; determining a first location and a second location based on the file name of the file to be copied, the location of the file to be copied, and the creation time of the file to be copied, the first location being a storage location of file data of the file to be copied, and the second location being a storage location of metadata of the file to be copied, the metadata of the file to be copied being used to represent attribute information of the file to be copied, the attribute information of the file to be copied being used to indicate access and management of the file data of the file to be copied; obtaining the file data of the file to be copied based on the first location, and obtaining the metadata of the file to be copied based on the second location; generating a target copy file based on the file data of the file to be copied and the metadata of the file to be copied.

2. The file copying method according to claim 1, characterized by, The generating of the target copy file based on the file data of the file to be copied and the metadata of the file to be copied comprises: writing the file data of the file to be copied into file data of the target copy file, and writing target metadata of the file to be copied into metadata of the target copy file, the target metadata comprising attribute information of the metadata of the file to be copied other than the file name of the file to be copied, the location of the file to be copied, and the creation time of the file to be copied.

3. The file copying method according to claim 2, characterized by, The method further comprises: adding a first operation to a target mapping queue, the first operation being the writing of the target metadata of the file to be copied into the metadata of the target copy file, the target mapping queue being used to indicate respective files corresponding to one or more operations; receiving a service request sent by the terminal, the service request being used to request processing of the metadata of the target copy file; in a case where the first operation is not completed, adding the service request to a request waiting queue, the request waiting queue being used to store a next incoming service request when a current operation of a file is not executed; in a case where the first operation is completed, deleting the service request from the request waiting queue, deleting the first operation from the target mapping queue, and adding a second operation to the target mapping queue, the second operation being processing of the target copy file based on the service request; performing the second operation based on the target mapping queue.

4. The file copying method according to any one of claims 1 to 3, characterized by, The file copy request further comprises a file system to which the file to be copied belongs, the file system to which the file to be copied belongs being an asynchronous file system or a synchronous file system, and the method further comprises: in a case where the file system to which the file to be copied belongs is a synchronous file system, determining the second location based on the file name of the file to be copied, the location of the file to be copied, and the creation time of the file to be copied; obtaining the metadata of the file to be copied based on the second location. Obtaining file data of the to-be-copied file based on metadata of the to-be-copied file.

5. A file copying apparatus characterized by comprising: Comprise: An obtaining module, a determining module and a processing module; The obtaining module is configured to obtain a file copy request sent by a terminal, the file copy request comprising a file name of a to-be-copied file, a location of the to-be-copied file and a creation time of the to-be-copied file; The determining module is configured to determine a first location and a second location based on the file name of the to-be-copied file, the location of the to-be-copied file and the creation time of the to-be-copied file, the first location being a storage location of file data of the to-be-copied file, and the second location being a storage location of metadata of the to-be-copied file, the metadata of the to-be-copied file being used to represent attribute information of the to-be-copied file, the attribute information of the to-be-copied file being used to indicate access and management of the file data of the to-be-copied file; The obtaining module is further configured to obtain the file data of the to-be-copied file based on the first location, and obtain the metadata of the to-be-copied file based on the second location; The processing module is configured to generate a target copy file based on the file data of the to-be-copied file and the metadata of the to-be-copied file.

6. The file copy device according to claim 5, wherein The processing module is specifically configured to write the file data of the to-be-copied file into file data of the target copy file, and write target metadata of the to-be-copied file into metadata of the target copy file, the target metadata comprising attribute information of the metadata of the to-be-copied file other than the file name of the to-be-copied file, the location of the to-be-copied file and the creation time of the to-be-copied file.

7. The file copying apparatus according to claim 6, characterized by The device further comprises a receiving module; The processing module is further configured to add a first operation into a target mapping queue, the first operation being the writing of the target metadata of the to-be-copied file into the metadata of the target copy file, the first operation corresponding to the target copy file, and the target mapping queue being used to indicate respective files corresponding to one or more operations; The receiving module is configured to receive a service request sent by the terminal, the service request being used to request processing of the metadata of the target copy file; The processing module is further configured to, in a case where the first operation is not completed, add the service request into a request waiting queue, the request waiting queue being used to store a next incoming service request when a current operation of a file is not executed; The processing module is further configured to, in a case where the first operation is completed, delete the service request from the request waiting queue, delete the first operation from the target mapping queue and add a second operation into the target mapping queue, the second operation being processing of the target copy file based on the service request; The processing module is further configured to execute the second operation based on the target mapping queue.

8. The file copying apparatus according to any one of claims 5 to 7, characterized by, The file copy request further comprises a file system to which the file to be copied belongs, and the file system to which the file to be copied belongs is an asynchronous file system or a synchronous file system. The determining module is further configured to, when the file system to which the file to be copied belongs is a synchronous file system, determine the second location based on a file name of the file to be copied, a position of the file to be copied, and a creation time of the file to be copied. The obtaining module is further configured to obtain metadata of the file to be copied based on the second location. The obtaining module is further configured to obtain file data of the file to be copied based on the metadata of the file to be copied.

9. A server, characterized by The server comprises: a processor; a memory configured to store instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the file copy method according to any one of claims 1-4.

10. A computer-readable storage medium having stored thereon instructions, the instructions comprising, When the instructions in the computer readable storage medium are executed by the server, the server is enabled to perform the file copy method according to any one of claims 1-4.

Images