Data acquisition method and device, equipment, storage medium and program product

Abstract

The invention relates to a data acquisition method and device, equipment, a storage medium and a program product. The method comprises the steps of obtaining first metadata information of a target file in response to a data obtaining instruction for the target file, and obtaining second metadata information of a source file according to a metadata index of the source file contained in the first metadata information under the condition of determining that the target file is in a data copying state, the source file is a file for copying data to the target file, and obtaining target data from the source file according to the second metadata information and the data obtaining instruction. By adopting the method, the data acquisition efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data acquisition method, device, equipment, storage medium and program product. Background Art

[0002] In current storage systems, in order to ensure the balance of data storage, data is generally divided into fixed-size data blocks and stored in different disks. Accordingly, when acquiring data, the disk location of the data is queried based on the metadata information of the data to be acquired, and then the data to be acquired is acquired.

[0003] However, in current distributed file systems, file copy operations are often performed. For the target file obtained through the copy operation, the existing data acquisition method cannot directly perform the data acquisition operation, but needs to obtain data from the target file after the data copy process is completed, thus reducing the efficiency of data acquisition. Summary of the invention

[0004] Based on this, it is necessary to provide a data acquisition method, device, equipment, storage medium and program product that can improve data acquisition efficiency in response to the above technical problems.

[0005] In a first aspect, the present application provides a data acquisition method, comprising:

[0006] In response to a data acquisition instruction for a target file, acquiring first metadata information of the target file;

[0007] When it is determined that the target file is in a data copy state, obtaining second metadata information of the source file according to the metadata index of the source file included in the first metadata information; wherein the source file is a file that copies data to the target file;

[0008] According to the second metadata information and the data acquisition instruction, the target data is acquired from the source file.

[0009] In an embodiment of the present application, when the target file is in a data copying state, compared to the related art, in which the target data needs to be obtained from the target file after the data copying process is completed, the above method is adopted to obtain the target data from the source file by adopting the metadata index of the source file contained in the first metadata information, without waiting for the data copying process, which can improve the efficiency of obtaining the target data while ensuring the accuracy of the target data.

[0010] In one embodiment, determining that the target file is in a data copy state includes:

[0011] When the first metadata information includes a copy identifier, it is determined that the target file is in a data copy state.

[0012] In the embodiment of the present application, the status of the target file is determined by the existence of the copy identifier, so that the accuracy and efficiency of the file status identification can be guaranteed.

[0013] In one of the embodiments, in response to the data acquisition instruction for the target file, before acquiring the first metadata information of the target file, the method further includes:

[0014] According to the file copy instruction for the source file, the metadata index and data attribute information of the source file are determined, and the first file storage path of the target file is extracted from the file copy instruction; according to the data attribute information of the source file, the target file is created under the first file storage path, and a copy identifier and the metadata index of the source file are added to the first metadata information of the target file.

[0015] In the embodiment of the present application, by adding the copy identifier and the metadata index of the source file to the first metadata information, the data acquisition instruction for the target file during the data copying process can be directed to the source file, thereby ensuring the efficiency of acquiring the target data.

[0016] In one embodiment, determining the metadata index and data attribute information of the source file according to the file copy instruction for the source file includes:

[0017] Extract a second file storage path of the source file from a file copy instruction for the source file; obtain a metadata index of the source file based on the second file storage path; obtain second metadata information of the source file based on the metadata index of the source file, and obtain data attribute information of the source file from the second metadata information.

[0018] In the embodiment of the present application, by adopting the metadata index of the source file, the data attribute information of the source file is obtained from the second metadata information, so that the reliability of the acquisition of the data attribute information can be guaranteed.

[0019] In one embodiment, the method further comprises:

[0020] After the adding process is completed, the target storage space corresponding to the target file is determined according to the remaining memory of the storage system; the address information of the target storage space is added to the first metadata information of the target file, and the data in the source file is copied to the target storage space.

[0021] In the embodiment of the present application, by determining the target storage space corresponding to the target file according to the remaining memory of the storage system, the rationality of the memory space allocation of the storage system can be ensured.

[0022] In one embodiment, copying data in a source file to a target storage space includes:

[0023] According to the second metadata information of the source file, determine the number of data blocks corresponding to the source file; according to the number of data blocks and the current data processing status of the storage system, determine the number of task processing that the storage system can undertake; construct task processing number copy tasks; execute each copy task in parallel to copy each data block corresponding to the source file to the target storage space.

[0024] In the embodiment of the present application, each data block is copied to the target storage space by executing each copy task in parallel, which can effectively improve the efficiency of data copying.

[0025] In one embodiment, the method further comprises:

[0026] After the copy operation is completed, the copy identifier in the first metadata information is cleared.

[0027] In the embodiment of the present application, by clearing the copy identifier in the first metadata information after the copy operation is completed, the accuracy and efficiency of file status identification can be ensured.

[0028] In one embodiment, the method further comprises:

[0029] When it is determined that the target file is in a non-data copy state, the target data is acquired from the target file according to the first metadata information and the data acquisition instruction.

[0030] In the embodiment of the present application, when the target file is in a non-data copy state, the target data is obtained in the target file through the data acquisition instruction and the first metadata information, so that the accuracy of the data acquisition can be guaranteed.

[0031] In a second aspect, the present application further provides a data acquisition device, comprising:

[0032] A first acquisition module, configured to acquire first metadata information of a target file in response to a data acquisition instruction for the target file;

[0033] A second acquisition module is used to acquire second metadata information of the source file according to the metadata index of the source file included in the first metadata information when it is determined that the target file is in a data copy state; wherein the source file is a file that copies data to the target file;

[0034] The third acquisition module is used to acquire target data from the source file according to the second metadata information and the data acquisition instruction.

[0035] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0036] In response to a data acquisition instruction for a target file, acquiring first metadata information of the target file;

[0037] When it is determined that the target file is in a data copy state, obtaining second metadata information of the source file according to the metadata index of the source file included in the first metadata information; wherein the source file is a file that copies data to the target file;

[0038] According to the second metadata information and the data acquisition instruction, the target data is acquired from the source file.

[0039] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0040] In response to a data acquisition instruction for a target file, acquiring first metadata information of the target file;

[0041] When it is determined that the target file is in a data copy state, obtaining second metadata information of the source file according to the metadata index of the source file included in the first metadata information; wherein the source file is a file that copies data to the target file;

[0042] According to the second metadata information and the data acquisition instruction, the target data is acquired from the source file.

[0043] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0044] In response to a data acquisition instruction for a target file, acquiring first metadata information of the target file;

[0045] When it is determined that the target file is in a data copy state, obtaining second metadata information of the source file according to the metadata index of the source file included in the first metadata information; wherein the source file is a file that copies data to the target file;

[0046] According to the second metadata information and the data acquisition instruction, the target data is acquired from the source file.

[0047] The above-mentioned data acquisition method, device, equipment, storage medium and program product acquire the first metadata information of the target file by responding to the data acquisition instruction for the target file, and acquire the second metadata information of the source file for copying data to the target file according to the metadata index of the source file contained in the first metadata information when determining that the target file is in the data copy state; then, acquire the target data from the source file according to the second metadata information and the data acquisition instruction. When the target file is in the data copy state, compared with the related art that needs to acquire the target data from the target file after the data copy process is completed, the above-mentioned method acquires the target data from the source file by using the metadata index of the source file contained in the first metadata information, without waiting for the data copy process, and can improve the acquisition efficiency of the target data on the basis of ensuring the accuracy of the target data. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1A A schematic diagram of a flow chart of a data acquisition method in an embodiment;

[0050] Figure 1B A schematic diagram of obtaining target data in one embodiment;

[0051] Figure 2 A schematic diagram of a process for constructing a target file in an embodiment;

[0052] Figure 3 A schematic diagram of a process for determining information in one embodiment;

[0053] Figure 4 A schematic diagram of a data copy process in an embodiment;

[0054] Figure 5A A schematic diagram of a process of copying data in parallel in one embodiment;

[0055] Figure 5B A schematic diagram of copying data in parallel in one embodiment;

[0056] Figure 6 is a flow chart of a data acquisition method in another embodiment;

[0057] Figure 7 is a structural block diagram of a data acquisition device in one embodiment;

[0058] Figure 8 is a structural block diagram of a data acquisition device in another embodiment;

[0059] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0061] In current storage systems, in order to ensure the balance of data storage, data is generally divided into fixed-size data blocks and stored in different disks. Accordingly, when acquiring data, the disk location of the data is queried based on the metadata information of the data to be acquired, and then the data to be acquired is acquired.

[0062] However, in current distributed file systems, file copy operations are often performed. For the target file obtained through the copy operation, the existing data acquisition method cannot directly perform the data acquisition operation, but needs to obtain data from the target file after the data copy process is completed, thus reducing the efficiency of data acquisition.

[0063] Based on this, in an exemplary embodiment, Figure 1A As shown, a data acquisition method is provided, which is described by taking the method applied to a data acquisition device in a server as an example, and specifically includes the following steps:

[0064] S101, in response to a data acquisition instruction for a target file, acquiring first metadata information of the target file.

[0065] Among them, the target file is used to represent the file where the data acquisition object is located; the data acquisition instruction refers to the instruction used to acquire data, which may include but is not limited to information such as the data acquisition path; the first metadata information is the metadata information corresponding to the target file.

[0066] The data in a distributed file system consists of two parts: metadata information and file data. File data refers to the content of a file; metadata information is data that describes the relevant information of the file data. Operations such as file creation, deletion, path search, attribute information reading and setting are the search and reading and writing of metadata. Specifically, in a distributed file system, in order to ensure the balance of memory usage, the complete file data can be divided into fixed-size data blocks, and each data block can be stored on different disks of different nodes in the system; then, according to the specific location of the data block in the storage system, storage node, and storage disk, the file layout corresponding to the file is obtained.

[0067] Optionally, after obtaining the data acquisition instruction, if there is no path information in the data acquisition instruction, the identification information of the target file contained in the data acquisition instruction can be used as an index to search between the storage locations of each file and the metadata information, thereby obtaining the first metadata information of the target file; if there is path information in the data acquisition instruction, the first metadata information of the target file can be obtained directly based on the path information.

[0068] S102: When it is determined that the target file is in a data copy state, second metadata information of the source file is obtained according to the metadata index of the source file included in the first metadata information.

[0069] Among them, the metadata index is used to represent the index information of locating metadata information; the source file is a file that copies data to the target file, that is, the source file is copied to obtain the target file; the target file is in a data copy state to represent that the target file is in the process of copying data from the source file; the second metadata information is the metadata information corresponding to the source file.

[0070] Optionally, it is possible to determine whether the target file is in a data copy state based on the first metadata information. Exemplarily, it is possible to determine whether the target file is in a data copy state based on the amount of data corresponding to the file layout in the first metadata information, that is, if the amount of data corresponding to the file layout is small, it is determined that the target file is in a data copy state; if the amount of data corresponding to the file layout is large, it is determined that the target file is in a non-data copy state. The target file being in a non-data copy state is used to indicate that the data of the target file has been copied and there is no need to copy data from the source file.

[0071] Alternatively, when the first metadata information includes a copy identifier, it is determined that the target file is in a data copy state, wherein the copy identifier is used to indicate that the file is in a state of copying data from a source file.

[0072] It can be understood that in order to ensure the accuracy and efficiency of state identification, a copy identifier can be added to the first metadata information when the target file is in a data copy state in which data is copied from a source file; accordingly, if the copy identifier is queried from the first metadata information, it proves that the target file is in a data copy state; if the copy identifier is not queried from the first metadata information, it proves that the target file is in a non-data copy state.

[0073] Since the target file is in a data copy state, it is not possible to directly perform data acquisition operations in the target file, so the data acquisition operation can be performed from the source file corresponding to the target file. Optionally, the metadata index of the source file can be obtained from the first metadata information; then, based on the metadata index of the source file, the second metadata information of the source file is obtained.

[0074] In addition, since the target file can directly perform a data acquisition operation when it is in a non-data copy state, when it is determined that the target file is in a non-data copy state, the target data is acquired from the target file according to the first metadata information and the data acquisition instruction, wherein the target data is the data to be acquired.

[0075] Exemplarily, the data storage location of the target data in the target file can be determined from the first metadata information according to the data acquisition requirement in the data acquisition instruction, and then the target data corresponding to the data acquisition instruction can be acquired according to the data storage location.

[0076] S103: Acquire target data from the source file according to the second metadata information and the data acquisition instruction.

[0077] Optionally, when it is determined that the target file is in a data copy state, the data storage location of the target data under the source file can be determined from the second metadata information according to the data acquisition requirements in the data acquisition instruction, and then the target data corresponding to the data acquisition instruction can be acquired according to the data storage location.

[0078] For example, refer to Figure 1B In the process of copying data in the source file A to the target file B, it is detected that the first metadata information corresponding to the target file B contains the copy identifier X B In this case, the data acquisition instruction for the target file B can be directed to the source file A according to the metadata index A' corresponding to the source file in the first metadata information, so as to obtain the target data from the source file A.

[0079] In the above data acquisition method, the first metadata information of the target file is acquired in response to the data acquisition instruction for the target file, and when it is determined that the target file is in a data copy state, the second metadata information of the source file for copying data to the target file is acquired according to the metadata index of the source file contained in the first metadata information; then, the target data is acquired from the source file according to the second metadata information and the data acquisition instruction. When the target file is in a data copy state, compared with the related art, which requires acquiring the target data from the target file after the data copy process is completed, the above method acquires the target data from the source file by using the metadata index of the source file contained in the first metadata information, without waiting for the data copy process, and can improve the acquisition efficiency of the target data on the basis of ensuring the accuracy of the target data.

[0080] In order to ensure the reliability of the target file construction, based on the above embodiment, in this embodiment, an optional method for constructing the target file is provided, such as Figure 2 As shown, the specific steps include:

[0081] S201, determining a metadata index and data attribute information of a source file according to a file copy instruction for a source file, and extracting a first file storage path of a target file from the file copy instruction.

[0082] Among them, the file copy instruction refers to the instruction for copying a file, which may include but is not limited to the storage paths of the source file and the target file; the data attribute information refers to the attribute information of the source file, such as file size, file access rights, file group, file access time and file modification time; the first file storage path refers to the storage path of the target file to be constructed.

[0083] Optionally, after obtaining the file copy instruction for the source file, the file identifier of the source file can be used as an index, and the metadata index of the source file can be queried from the association relationship between each file and each metadata index to obtain the data attribute information of the source file, and then determine the data attribute information of the source file; or, the metadata index and data attribute information of the source file can be determined based on the second file storage path of the source file contained in the file copy instruction.

[0084] It is understandable that when copying data, it is necessary to pre-specify the directory where the copied target file is located. Therefore, the first file storage path corresponding to the target file can be directly extracted in the file copy instruction.

[0085] S202: Create a target file in a first file storage path according to data attribute information of the source file, and add a copy identifier and a metadata index of the source file to first metadata information of the target file.

[0086] Optionally, in order to ensure the reliability of data copying, a target file with the same attributes as the source file may be created under the first file storage path based on the data attribute information of the source file, for example, a target file with the same file size and file access rights as the source file may be created.

[0087] Furthermore, in order to direct the data acquisition instruction for the target file to the source file during the data copying process, it is necessary to add a copy mark and a metadata index of the source file at a preset fixed position in the first metadata information before copying the data.

[0088] In the embodiment of the present application, by adding the copy identifier and the metadata index of the source file to the first metadata information, the data acquisition instruction for the target file during the data copying process can be directed to the source file, thereby ensuring the efficiency of acquiring the target data.

[0089] In order to ensure the reliability of data attribute information acquisition, based on the above embodiment, in this embodiment, an optional method for determining information is provided, such as Figure 3 As shown, the specific steps include:

[0090] S301, extracting a second file storage path of the source file from a file copy instruction for the source file.

[0091] The second file storage path refers to the storage path corresponding to the metadata of the source file.

[0092] It is understandable that when copying data, it is necessary to pre-specify the directory where the source file to be copied is located. Therefore, the second file storage path corresponding to the source file can be directly extracted in the file copy instruction.

[0093] Alternatively, the file identification information of the source file in the file copy instruction may be used as an index for query to obtain the second file storage path corresponding to the source file.

[0094] S302: Obtain a metadata index of the source file according to the second file storage path.

[0095] Optionally, after the second file storage path is determined, a search may be performed according to the second file storage path to obtain a metadata index of the source file.

[0096] S303: Acquire second metadata information of the source file according to the metadata index of the source file, and acquire data attribute information of the source file from the second metadata information.

[0097] Optionally, second metadata information of the source file may be located in the storage system according to the metadata index of the source file; subsequently, data attribute information of the source file may be obtained from the second metadata information according to the data type identifier of the metadata.

[0098] In the embodiment of the present application, by adopting the metadata index of the source file, the data attribute information of the source file is obtained from the second metadata information, so that the reliability of the acquisition of the data attribute information can be guaranteed.

[0099] In order to ensure the rationality of the memory space allocation of the storage system, based on the above embodiment, in this embodiment, an optional method of data copy is provided, such as Figure 4 As shown, the specific steps include:

[0100] S401, after the adding process is completed, determine the target storage space corresponding to the target file according to the remaining memory of the storage system.

[0101] Among them, the storage system is the system where the source file and the target file are located; the remaining memory is the remaining memory in each disk in the storage system at the current moment; and the target storage space is the storage space used to store each data block corresponding to the target file.

[0102] Optionally, after adding the copy identifier and the metadata index of the source file to the first metadata information, the target storage space corresponding to each data block in the target file can be determined from each disk of the storage system based on the remaining memory of the storage system and with load balancing as the goal.

[0103] S402: Add the address information of the target storage space to the first metadata information of the target file, and copy the data in the source file to the target storage space.

[0104] Optionally, after determining the target storage space corresponding to each data block in the target file, a file layout corresponding to the target file can be generated based on address information of the target storage space, and the file layout corresponding to the target file can be added to the first metadata information of the target file.

[0105] Furthermore, in order to ensure the efficiency of data copying, the data in the source file can be copied to the target storage space in parallel; then, after the copy operation is completed, in order to accurately locate the file status of the target file, the copy status of the first metadata information needs to be modified.

[0106] Exemplarily, if a copy mark is added to the first metadata information when the target file is in a data copy state of copying data from the source file, it is necessary to clear the copy mark in the first metadata information after the copy operation is completed. For example, after detecting that the file copy operation is completed, the mark clearing operation can be performed according to the preset mark position.

[0107] In the embodiment of the present application, by determining the target storage space corresponding to the target file according to the remaining memory of the storage system, the rationality of the memory space allocation of the storage system can be ensured.

[0108] In order to improve the data copying efficiency, based on the above embodiment, in this embodiment, an optional method of copying data in parallel is provided, such as Figure 5A As shown, the specific steps include:

[0109] S501: Determine the number of data blocks corresponding to the source file according to second metadata information of the source file.

[0110] The number of data blocks is the number of data blocks.

[0111] Optionally, the file layout of the source file is extracted from the second metadata information of the source file; then, the number of data blocks corresponding to the source file can be determined according to the number of addresses in the file layout of the source file. For example, when only one data block is stored in a location, the number of addresses in the file layout can be directly used as the number of data blocks corresponding to the source file.

[0112] S502: Determine the number of task processing that the storage system can undertake based on the number of data blocks and the current data processing status of the storage system.

[0113] The current data processing status refers to the data processing volume of the storage system at the current moment; the task processing quantity refers to the number of tasks that the storage system can undertake for data copying.

[0114] Optionally, a task processing quantity that is no greater than the number of data blocks may be determined based on the maximum data processing capacity of the storage system, the gap between the current data processing situation, and the number of data blocks of the source file.

[0115] For example, when the maximum data processing capacity of the storage system is significantly different from the current data processing capacity, a corresponding copy task can be constructed for each data block. In this case, the number of data blocks can be directly used as the number of task processing. The copy task is a task for performing data copy processing.

[0116] When the gap between the maximum data processing capacity of the storage system and the current data processing situation is small, it is necessary to determine the number of task processing tasks that the storage system can undertake based on the task volume gap and the system performance required for each data copy task.

[0117] S503, constructing task processing quantity copy tasks.

[0118] Optionally, after the task processing quantity is determined, the task processing quantity copy tasks may be constructed according to the predetermined target storage space for storing each data block in the target file.

[0119] It is understandable that when multiple data blocks correspond to the same copy task, the copy order between the data blocks can be determined according to the priorities corresponding to the data blocks; then, the copy task is constructed according to the target storage space where the first data block in the copy order is to be stored.

[0120] S504, executing the copy tasks in parallel to copy the data blocks corresponding to the source file to the target storage space.

[0121] Optionally, after the copy tasks are constructed, the copy tasks can be executed in parallel to copy each data block in the source file to the corresponding target storage space. Figure 5B , each data block in the source file can be delivered to a corresponding copy task; then, the copy tasks are executed in parallel to copy the data blocks in the source file to the target file in parallel.

[0122] It can be understood that, when there is a corresponding copy task for each data block, for any copy task, the copy task will be automatically cleared after the data copy process corresponding to the copy task is completed; when multiple data blocks correspond to the same copy task, after the copy processing of the current data block is completed, the copy task needs to be modified according to the target storage space where the next data block is to be stored.

[0123] In the embodiment of the present application, by executing the copy tasks in parallel to copy each data block to the target storage space, the efficiency of data copying can be effectively improved.

[0124] Figure 6 FIG. 1 is a flow chart of a data acquisition method in another embodiment. Based on the above embodiment, this embodiment provides an optional example of a data acquisition method. Figure 6 The specific implementation process is as follows:

[0125] S601, extracting a second file storage path of the source file from a file copy instruction for the source file, and acquiring a metadata index of the source file according to the second file storage path.

[0126] S602: Acquire second metadata information of the source file according to the metadata index of the source file, and acquire data attribute information of the source file from the second metadata information.

[0127] S603: Extract a first file storage path of the target file from the file copy instruction.

[0128] S604: Create a target file in the first file storage path according to the data attribute information of the source file, and add a copy identifier and a metadata index of the source file to the first metadata information of the target file.

[0129] S605: Determine the target storage space corresponding to the target file according to the remaining memory of the storage system.

[0130] S606: Add the address information of the target storage space to the first metadata information of the target file, and determine the number of data blocks corresponding to the source file according to the second metadata information of the source file.

[0131] S607, determining the number of task processing that the storage system can undertake according to the number of data blocks and the current data processing status of the storage system.

[0132] S608, constructing task processing quantity copy tasks.

[0133] S609, executing the copy tasks in parallel to copy the data blocks corresponding to the source file to the target storage space.

[0134] S610: In response to a data acquisition instruction for a target file, first metadata information of the target file is acquired.

[0135] S611, determine whether the first metadata information includes a copy identifier, if so, execute S612; if not, execute S615.

[0136] S612: Acquire second metadata information of the source file according to the metadata index of the source file included in the first metadata information.

[0137] S613: Acquire target data from the source file according to the second metadata information and the data acquisition instruction.

[0138] S614: After the copy operation is completed, clear the copy identifier in the first metadata information.

[0139] S615: Acquire target data from the target file according to the first metadata information and the data acquisition instruction.

[0140] The specific process of the above S601-S615 can refer to the description of the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0141] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0142] Based on the same inventive concept, the embodiment of the present application also provides a data acquisition device for implementing the data acquisition method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more data acquisition device embodiments provided below can refer to the limitations on the data acquisition method above, and will not be repeated here.

[0143] In an exemplary embodiment, Figure 7 As shown, a data acquisition device 1 is provided, comprising: a first acquisition module 10, a second acquisition module 20 and a third acquisition module 30, wherein:

[0144] A first acquisition module 10, configured to acquire first metadata information of a target file in response to a data acquisition instruction for the target file;

[0145] The second acquisition module 20 is used to acquire second metadata information of the source file according to the metadata index of the source file included in the first metadata information when it is determined that the target file is in a data copy state; wherein the source file is a file that copies data to the target file;

[0146] The third acquisition module 30 is used to acquire target data from the source file according to the second metadata information and the data acquisition instruction.

[0147] In an exemplary embodiment, the second acquisition module 20 is specifically configured to:

[0148] When the first metadata information includes a copy identifier, it is determined that the target file is in a data copy state.

[0149] In an exemplary embodiment, Figure 8 As shown, the data acquisition device 1 further includes a data copy module 40, wherein the data copy module 40 includes:

[0150] A path determination unit 41 is used to determine the metadata index and data attribute information of the source file according to the file copy instruction for the source file, and extract the first file storage path of the target file from the file copy instruction;

[0151] The file construction unit 42 is used to create a target file in the first file storage path according to the data attribute information of the source file, and add a copy mark and a metadata index of the source file to the first metadata information of the target file.

[0152] In an exemplary embodiment, the path determination unit 41 is specifically configured to:

[0153] Extract a second file storage path of the source file from a file copy instruction for the source file; obtain a metadata index of the source file based on the second file storage path; obtain second metadata information of the source file based on the metadata index of the source file, and obtain data attribute information of the source file from the second metadata information.

[0154] In an exemplary embodiment, Figure 8 As shown, the data copy module 40 also includes:

[0155] The space determination unit 43 is used to determine the target storage space corresponding to the target file according to the remaining memory of the storage system after the adding process is completed;

[0156] The data copy unit 44 is used to add the address information of the target storage space to the first metadata information of the target file, and copy the data in the source file to the target storage space.

[0157] In an exemplary embodiment, the data copy unit 44 is specifically used for:

[0158] According to the second metadata information of the source file, determine the number of data blocks corresponding to the source file; according to the number of data blocks and the current data processing status of the storage system, determine the number of task processing that the storage system can undertake; construct task processing number copy tasks; execute each copy task in parallel to copy each data block corresponding to the source file to the target storage space.

[0159] In an exemplary embodiment, Figure 8 As shown, the data copy module 40 also includes:

[0160] The identification clearing unit 45 is used to clear the copy identification in the first metadata information after the copy operation is completed.

[0161] In an exemplary embodiment, the third acquisition module 30 is further configured to:

[0162] When it is determined that the target file is in a non-data copy state, the target data is acquired from the target file according to the first metadata information and the data acquisition instruction.

[0163] Each module in the above data acquisition device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0164] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig. 9 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store source file data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data acquisition method is implemented.

[0165] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0166] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0167] In response to a data acquisition instruction for a target file, acquiring first metadata information of the target file;

[0168] When it is determined that the target file is in a data copy state, obtaining second metadata information of the source file according to the metadata index of the source file included in the first metadata information; wherein the source file is a file that copies data to the target file;

[0169] According to the second metadata information and the data acquisition instruction, the target data is acquired from the source file.

[0170] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0171] When the first metadata information includes a copy identifier, it is determined that the target file is in a data copy state.

[0172] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0173] According to the file copy instruction for the source file, the metadata index and data attribute information of the source file are determined, and the first file storage path of the target file is extracted from the file copy instruction; according to the data attribute information of the source file, the target file is created under the first file storage path, and a copy identifier and the metadata index of the source file are added to the first metadata information of the target file.

[0174] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0175] Extract a second file storage path of the source file from a file copy instruction for the source file; obtain a metadata index of the source file based on the second file storage path; obtain second metadata information of the source file based on the metadata index of the source file, and obtain data attribute information of the source file from the second metadata information.

[0176] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0177] After the adding process is completed, the target storage space corresponding to the target file is determined according to the remaining memory of the storage system; the address information of the target storage space is added to the first metadata information of the target file, and the data in the source file is copied to the target storage space.

[0178] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0179] According to the second metadata information of the source file, determine the number of data blocks corresponding to the source file; according to the number of data blocks and the current data processing status of the storage system, determine the number of task processing that the storage system can undertake; construct task processing number copy tasks; execute each copy task in parallel to copy each data block corresponding to the source file to the target storage space.

[0180] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0181] After the copy operation is completed, the copy identifier in the first metadata information is cleared.

[0182] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0183] When it is determined that the target file is in a non-data copy state, the target data is acquired from the target file according to the first metadata information and the data acquisition instruction.

[0184] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0185] In response to a data acquisition instruction for a target file, acquiring first metadata information of the target file;

[0186] When it is determined that the target file is in a data copy state, obtaining second metadata information of the source file according to the metadata index of the source file included in the first metadata information; wherein the source file is a file that copies data to the target file;

[0187] According to the second metadata information and the data acquisition instruction, the target data is acquired from the source file.

[0188] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0189] When the first metadata information includes a copy identifier, it is determined that the target file is in a data copy state.

[0190] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0191] According to the file copy instruction for the source file, the metadata index and data attribute information of the source file are determined, and the first file storage path of the target file is extracted from the file copy instruction; according to the data attribute information of the source file, the target file is created under the first file storage path, and a copy identifier and the metadata index of the source file are added to the first metadata information of the target file.

[0192] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0193] Extract a second file storage path of the source file from a file copy instruction for the source file; obtain a metadata index of the source file based on the second file storage path; obtain second metadata information of the source file based on the metadata index of the source file, and obtain data attribute information of the source file from the second metadata information.

[0194] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0195] After the adding process is completed, the target storage space corresponding to the target file is determined according to the remaining memory of the storage system; the address information of the target storage space is added to the first metadata information of the target file, and the data in the source file is copied to the target storage space.

[0196] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0197] According to the second metadata information of the source file, determine the number of data blocks corresponding to the source file; according to the number of data blocks and the current data processing status of the storage system, determine the number of task processing that the storage system can undertake; construct task processing number copy tasks; execute each copy task in parallel to copy each data block corresponding to the source file to the target storage space.

[0198] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0199] After the copy operation is completed, the copy identifier in the first metadata information is cleared.

[0200] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0201] When it is determined that the target file is in a non-data copy state, the target data is acquired from the target file according to the first metadata information and the data acquisition instruction.

[0202] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0203] In response to a data acquisition instruction for a target file, acquiring first metadata information of the target file;

[0204] When it is determined that the target file is in a data copy state, obtaining second metadata information of the source file according to the metadata index of the source file included in the first metadata information; wherein the source file is a file that copies data to the target file;

[0205] According to the second metadata information and the data acquisition instruction, the target data is acquired from the source file.

[0206] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0207] When the first metadata information includes a copy identifier, it is determined that the target file is in a data copy state.

[0208] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0209] According to the file copy instruction for the source file, the metadata index and data attribute information of the source file are determined, and the first file storage path of the target file is extracted from the file copy instruction; according to the data attribute information of the source file, the target file is created under the first file storage path, and a copy identifier and the metadata index of the source file are added to the first metadata information of the target file.

[0210] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0211] Extract a second file storage path of the source file from a file copy instruction for the source file; obtain a metadata index of the source file based on the second file storage path; obtain second metadata information of the source file based on the metadata index of the source file, and obtain data attribute information of the source file from the second metadata information.

[0212] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0213] After the adding process is completed, the target storage space corresponding to the target file is determined according to the remaining memory of the storage system; the address information of the target storage space is added to the first metadata information of the target file, and the data in the source file is copied to the target storage space.

[0214] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0215] According to the second metadata information of the source file, determine the number of data blocks corresponding to the source file; according to the number of data blocks and the current data processing status of the storage system, determine the number of task processing that the storage system can undertake; construct task processing number copy tasks; execute each copy task in parallel to copy each data block corresponding to the source file to the target storage space.

[0216] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0217] After the copy operation is completed, the copy identifier in the first metadata information is cleared.

[0218] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0219] When it is determined that the target file is in a non-data copy state, the target data is acquired from the target file according to the first metadata information and the data acquisition instruction.

[0220] It should be noted that the data involved in this application (including but not limited to source file data, etc.) are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0221] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0222] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0223] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data acquisition method, characterized in that: The method comprises: In response to a data acquisition instruction for a target file, acquiring first metadata information of the target file; When it is determined that the target file is in a data copy state, obtaining second metadata information of the source file according to the metadata index of the source file included in the first metadata information; wherein the source file is a file that copies data to the target file; According to the second metadata information and the data acquisition instruction, target data is acquired from the source file.

2. The method according to claim 1, characterized in that: The determining that the target file is in a data copy state includes: When the first metadata information includes a copy identifier, it is determined that the target file is in a data copy state.

3. The method according to claim 2, characterized in that In response to a data acquisition instruction for a target file, before acquiring first metadata information of the target file, the method further includes: Determine the metadata index and data attribute information of the source file according to the file copy instruction for the source file, and extract the first file storage path of the target file from the file copy instruction; The target file is created in the first file storage path according to the data attribute information of the source file, and the copy identifier and the metadata index of the source file are added to the first metadata information of the target file.

4. The method according to claim 3, characterized in that The step of determining the metadata index and data attribute information of the source file according to the file copy instruction for the source file includes: Extracting a second file storage path of the source file from the file copy instruction for the source file; According to the second file storage path, obtaining a metadata index of the source file; According to the metadata index of the source file, second metadata information of the source file is obtained, and data attribute information of the source file is obtained from the second metadata information.

5. The method according to claim 3, characterized in that: The method further comprises: After the adding process is completed, the target storage space corresponding to the target file is determined according to the remaining memory of the storage system; The address information of the target storage space is added to the first metadata information of the target file, and the data in the source file is copied to the target storage space.

6. The method according to claim 5, characterized in that The copying of the data in the source file to the target storage space includes: Determine the number of data blocks corresponding to the source file according to the second metadata information of the source file; Determine the number of task processing that the storage system can undertake according to the number of data blocks and the current data processing status of the storage system; Construct the task processing quantity copy tasks; The copy tasks are executed in parallel to copy the data blocks corresponding to the source file to the target storage space.

7. The method according to claim 5, characterized in that The method further comprises: After the copy operation is completed, the copy identifier in the first metadata information is cleared.

8. The method according to claim 1, characterized in that The method further comprises: When it is determined that the target file is in a non-data copy state, target data is acquired from the target file according to the first metadata information and the data acquisition instruction.

9. A data acquisition device, characterized in that: The device comprises: A first acquisition module, configured to acquire first metadata information of a target file in response to a data acquisition instruction for the target file; A second acquisition module is used to acquire second metadata information of the source file according to the metadata index of the source file contained in the first metadata information when it is determined that the target file is in a data copy state; wherein the source file is a file that copies data to the target file; The third acquisition module is used to acquire target data from the source file according to the second metadata information and the data acquisition instruction.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images