File merging method and device and medium

Abstract

The invention discloses a file merging method and device and a medium, relates to the technical field of file merging, is used for realizing file merging, and provides a file merging method and a new file data index mode aiming at the problem that file merging can be realized only by adopting a native HDFS (Hadoop Distributed File System) mode traditionally, so that multi-pool mapping of merged files can be realized. In addition, according to the method, file merging is achieved through a transaction mechanism of a distributed file system. At present, a plurality of data rollback schemes in case of faults exist for transactions, so that when the method is used for carrying out disk falling on file merging through the transactions, if the faults occur midway, data can be rolled back through a rollback method of a transaction mechanism, and the data is rolled back to be before file merging. In conclusion, the method can effectively solve two challenges confronted by non-native HDFS file merging, so that a solution except the native HDFS is expanded for the implementation of file merging.

Description

Technical Field

[0001] The present invention relates to the technical field of file merging, and in particular to a file merging method, device and medium. Background Art

[0002] With the development of the Internet and big data technology, the amount of data has exploded. In a distributed file system, since it takes time to find the metadata and read the file content of each file, numerous small files will cause a large amount of metadata storage and management overhead, which will reduce the access performance of the system. Therefore, in data analysis and processing scenarios, it is often necessary to merge multiple data files for batch processing and analysis.

[0003] Currently, file merging can be achieved in HDFS (a highly fault-tolerant distributed file system) distributed storage. However, as the overall big data system is moving towards a storage-computing separation architecture that separates computing from storage, the native HDFS itself also has problems such as single point failure, storage capacity limitations, global locks, and metadata scalability. If other methods other than native HDFS are used to achieve file merging, there are two main challenges:

[0004] 1. Due to the differences in data and metadata organization of different distributed file storage backends, how to achieve data organization mapping after file merging.

[0005] 2. The metadata layer faces cross-metadata server processing operations. How to deal with failures in the middle?

[0006] Therefore, technicians in this field now urgently need a file merging method to replace the native HDFS to realize file merging in a distributed file system. Summary of the invention

[0007] The purpose of the present invention is to provide a file merging method, device and medium, which are used to provide a file merging solution implemented in a non-native HDFS manner.

[0008] In order to solve the above technical problems, the present invention provides a file merging method, comprising:

[0009] When receiving a file merge request, the target file is merged and stored on disk;

[0010] Updating metadata attributes of the target file and the file merge transaction result file;

[0011] The metadata attributes of the file include: file type; the file types include: common file, merged file and merged file; the metadata attributes of the merged file and the merged file also include: file data index attribute;

[0012] The file data index attributes include: storage pool index mapping information and unique identification number index information;

[0013] The storage pool index mapping information is stored in the storage pool mapping table in the form of a key-value pair, wherein the storage pool index mapping information uses the index unique code as a key and the storage pool information as a value; the storage pool information includes: the storage pool unique code, the stripe pair size and the redundancy strategy;

[0014] The unique identification number index information is stored in the unique identification number index table in the form of a key-value pair, wherein the unique identification number index information uses the file offset as the key and the data segment information as the value; the data segment information is in the form of a key-value pair, wherein the unique identification number is used as the key and the data length array is used as the value; the data length array includes the data length and the index unique code.

[0015] In a possible embodiment, performing a file merging transaction on the target file and storing it on disk includes:

[0016] According to the preset merging granularity, a corresponding number of the target files are regarded as a batch, and the file merging tasks of each batch are cyclically performed;

[0017] The method further comprises:

[0018] If an exception occurs when the file merging transaction is written to the disk, the operation of the abnormal file merging transaction is replayed.

[0019] In a possible embodiment, after updating the metadata attributes of the target file and the file merge transaction result file, the method further includes:

[0020] Deleting the metadata attribute of the merged file;

[0021] If there is no abnormality in the storage of the file merging transaction, the local file merging transaction is deleted, and the next batch of file merging transactions are stored in the disk.

[0022] In a possible embodiment, deleting the metadata attribute of the merged file includes:

[0023] When the metadata server receives a merge source file deletion request, the metadata attribute of the merge source file is read to obtain the file offset and the data length; wherein the merge source file is the merged file targeted by this read request;

[0024] Acquire index information from the storage pool mapping table and the unique identification number index table according to the file offset and the data length; wherein the index information includes: the unique identification number, the file offset, the data length and the storage pool information;

[0025] Merging the storage pool information of all the merged source files according to the index information, and striping the specific location of the data according to the merged storage pool information;

[0026] Sending a data deletion request to the storage pool according to the specific location of the data;

[0027] After the storage pool completes the deletion of all data, the deletion result is returned as the result of the merged source file deletion request.

[0028] In a possible embodiment, the method further includes:

[0029] When receiving a merge source file read request initiated by a business client, checking whether the read-write area hits the cache; wherein the merge source file is the merged file targeted by this read request;

[0030] If no hit is found, reading the metadata attribute of the merged source file to obtain the file offset and the data length;

[0031] Obtaining a read-write area index from the storage pool mapping table and the unique identification number index table according to the file offset and the data length; wherein the read-write area index includes: the unique identification number, the file offset, the data length and the storage pool information;

[0032] Merging the storage pool information of all the merged source files according to the read-write area index, and striping the specific location of the data according to the merged storage pool information;

[0033] The data is read according to the specific location of the data, and after all the data are read, all the read data are combined as the return result of the merged source file read request.

[0034] In a possible embodiment, after the file type of the target file is updated to the merged file, the method further includes:

[0035] The file attribute of the merged file on the user side is set to be hidden.

[0036] In a possible embodiment, the storage pool mapping table and the unique identification number index table are both composed of multiple sub-tables;

[0037] The storage pool index mapping information and the unique identification number index information of each of the merged file and the merged file are allocated to the designated sub-table for storage in a hash manner.

[0038] In order to solve the above technical problems, the present invention further provides a file merging device, comprising:

[0039] The transaction storage module is used to store the target file in a file merging transaction after receiving a file merging request;

[0040] A metadata updating module, used for updating metadata attributes of the target file and the file merging transaction result file;

[0041] The metadata attributes of the file include: file type; the file types include: common file, merged file and merged file; the metadata attributes of the merged file and the merged file also include: file data index attribute;

[0042] The file data index attributes include: storage pool index mapping information and unique identification number index information;

[0043] The storage pool index mapping information is stored in the storage pool mapping table in the form of a key-value pair, wherein the storage pool index mapping information uses the index unique code as a key and the storage pool information as a value; the storage pool information includes: the storage pool unique code, the stripe pair size and the redundancy strategy;

[0044] The unique identification number index information is stored in the unique identification number index table in the form of a key-value pair, wherein the unique identification number index information uses the file offset as the key and the data segment information as the value; the data segment information is in the form of a key-value pair, wherein the unique identification number is used as the key and the data length array is used as the value; the data length array includes the data length and the index unique code.

[0045] In order to solve the above technical problems, the present invention further provides a file merging device, comprising:

[0046] Memory for storing computer programs;

[0047] A processor is used to implement the steps of the file merging method described above when executing the computer program.

[0048] In order to solve the above technical problem, the present invention further provides a non-volatile storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the file merging method as described above are implemented.

[0049] A file merging method provided by the present invention proposes a new file data indexing method, and adds a file type and a file data indexing attribute to the file metadata attributes. Among them, the file type mainly identifies whether the file is a file related to merging (i.e., the merged file and the files to be merged), and the file data indexing attribute mainly records the basic indexing information of the files related to merging. Specifically, the file data indexing attribute includes two types of information: storage pool index mapping information and unique identifier (ino) index information. The storage pool index mapping information is mainly used to optimize memory / space occupancy, and is composed of key-value pairs such as index unique code (id)-storage pool information; the storage pool information includes information such as storage pool id, striping pair size, and redundancy policy. Subsequently, the corresponding storage pool information can be quickly and conveniently found based on the index id. The ino index information is mainly used to record the indexing information of the files related to merging, and the data format is a key-value pair of <file offset, <ino, [data length, storage pool unique code]>>. Based on the above file data indexing method, the data range position of the files related to merging can be directly mapped to any storage pool, thereby realizing multi-pool mapping of the merged file, and solving the problem of how to realize data organization mapping after file merging when the merged file is implemented in a way other than the native HDFS.

[0050] In addition, in this method, file merging is implemented by using the transaction mechanism of the distributed file system. Currently, there are various data rollback solutions when a failure occurs for transactions. Therefore, when this method writes the merged file to disk through a transaction, if a failure occurs midway, the data can be rolled back through the rollback method of the transaction mechanism, so that the data is rolled back to before the file merging, solving the problem of how to handle a failure that occurs midway when the merged file is implemented in a way other than the native HDFS.

[0051] In summary, this method proposes a file merging method that does not use the native HDFS, which can effectively solve the two major challenges faced by non-native HDFS file merging, thereby expanding a solution other than the native HDFS for the implementation of file merging.

[0052] The file merging device and the non-volatile storage medium provided by the present invention correspond to the above method, and have the same effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0054] Figure 1A flowchart of a file merging method provided by an embodiment of the present invention;

[0055] Figure 2 A flowchart of a merged file deletion request processing provided by an embodiment of the present invention;

[0056] Figure 3 A flowchart of a merged file read request process provided by an embodiment of the present invention;

[0057] Figure 4 A flowchart of another file merging method provided by an embodiment of the present invention;

[0058] Figure 5 A structural diagram of a file merging device provided by an embodiment of the present invention;

[0059] Figure 6 A structural diagram of another file merging device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] The core of the present invention is to provide a file merging method, device and medium.

[0062] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0063] In the related art, the mainstream solution for distributed file systems to implement the file merging function is to enable the distributed file system to support the HDFS (a highly fault-tolerant distributed file system) protocol. However, since native HDFS itself also has problems such as single point failure, storage capacity limitation, global lock, metadata scalability, etc., those skilled in the art are also exploring another way to implement the file merging function.

[0064] Currently, due to the differences in the organization of backend data and metadata in distributed file storage, file merging in native HDFS currently faces two major challenges:

[0065] 1. The data layer originally calculates the object position by the file unique identification number (ino) and the file offset. So how should the data organization mapping be implemented after the files are merged?

[0066] 2. The metadata layer faces cross-metadata server processing operations. What should be done if a failure occurs in the middle?

[0067] To solve the above problems, the present invention provides a file merging method, such as Figure 1 As shown, including:

[0068] S11: After receiving the file merge request, the target file is merged and stored in the disk.

[0069] S12: Update metadata attributes of the target file and the file merge transaction result file.

[0070] The metadata attributes of the file include: file type; the file types include: common file, merged file and merged file; the metadata attributes of the merged file and the merged file also include: file data index attribute.

[0071] The file data index attributes include: storage pool index mapping information and unique identification number (ino) index information.

[0072] The storage pool index mapping information is stored in the storage pool mapping table in the form of key-value pairs. The storage pool index mapping information uses the index unique code as the key and the storage pool information as the value. The storage pool information includes: the storage pool unique code (id), stripe pair size, redundancy strategy and other information.

[0073] The ino index information is also stored in the ino index table in key-value format. The ino index information uses the file offset (offset) as the key and the data segment information as the value. The data segment information is in key-value format, with ino as the key and the data length array as the value. The data length array includes the data length (len) and the index id (pool_index). Specifically, the data format of the ino index information is <offst,<ino,

len,pool_index

[0074] It is not difficult to see from the above that the present invention provides a new file data indexing method. For any merged file, the corresponding data segment information can be queried from the ino index table based on the file offset of the merged file.<ino,

len,pool_index

[0075] Specifically, two types of attribute information, file type and file data index attribute, are added to the file metadata attributes. Among them, the file type mainly identifies whether the file is a merge-related file (i.e., a merged file and a merged file), and the file data index attribute mainly records the basic information of the merge-related files (ordinary files do not have file data index attributes, or the file data index attributes are empty).

[0076] The file data index attribute is further composed of storage pool index mapping information and ino index information. The storage pool index mapping information is mainly used to optimize memory / space usage. The storage pool index information is recorded through a map structure (a data structure) such as <index id, storage pool information>. The corresponding storage pool information can be found later using the index id. The storage pool information includes information related to the file and the storage pool, including but not limited to the above-mentioned: storage pool id, striping pair size, and redundancy strategy. The ino index information is used to record the index information of the merged related files. The data format is <offst,<ino,

len, pool_index

[0077] Furthermore, in step S11, the method implements the specific execution of file merging through the transaction mechanism of the distributed file system, and writes the merged file to the disk through the transaction to achieve persistent storage. It is easy to understand that the transaction mechanism, as a mature mechanism in the database, is mainly used to prevent data inconsistencies when data updates occur synchronously. In the related art, there are currently a variety of data rollback solutions for transactions when failures occur. Therefore, when the method merges files to the disk through a transaction, if a failure occurs midway, the data can be rolled back through the rollback method of the transaction mechanism, so that the data is rolled back to before the file merge, thereby achieving the processing of the midway failure problem of the file merge.

[0078] In addition, for step S12, the target file is the "merged file" in this file merge, so after the file merge transaction is stored on disk, the file type of the target file should be updated from a normal file to a "merged file". Similarly, the file merge transaction result file in step S12, that is, the result file output by the file merge transaction, is the "merged file" in this file merge, and the file type of the target file should be updated to a "merged file".

[0079] In summary, the file merging method provided by the present invention can effectively solve the two major challenges faced when using non-native HDFS to implement file merging - how to implement data organization mapping and how to handle failures in the middle of file merging, making it possible to implement a non-native HDFS file merging solution proposed by the method, and providing a file merging solution other than native HDFS. At the same time, it is also helpful to avoid a series of problems existing in native HDFS, and is conducive to promoting the further development of the big data storage industry.

[0080] On the other hand, the above embodiment illustrates the specific implementation of the method using the transaction mechanism to realize the merge file operation, thereby utilizing the various mature data rollback solutions currently for the transaction mechanism to perform data rollback when an exception occurs in the file merge, so as to restore the data to the version before the merge. Based on this, this embodiment provides a solution for handling failures in the middle of file merge, and the above step S11 specifically includes:

[0081] According to the preset merging granularity, a corresponding number of target files are taken as a batch, and the file merging transactions of each batch are performed cyclically.

[0082] Correspondingly, such as Figure 1 As shown, the method also includes:

[0083] S13: If an exception occurs when the file merge transaction is written to the disk, the abnormal file merge transaction is replayed.

[0084] It should be noted that this embodiment does not limit the above-mentioned merging granularity, and it should be selected according to actual needs. The merging granularity represents how many merged files are merged to obtain a merged file when a file is merged. In a possible example, the merging granularity is 100, that is, every 100 merged files are merged into 1 merged file.

[0085] In this embodiment, the operation playback mechanism commonly supported by the current distributed file system is used to implement the operation playback of the faulty transaction, so as to roll back the data to before the file merge, and avoid problems such as data inconsistency caused by a failure in the middle of the file merge. In addition, this solution performs file merges in batches with a preset granularity; when the merge of the previous batch of files is completed normally, the next batch of file merge transactions are started to be stored on the disk; if an exception occurs in the file merge transaction, the data is rolled back through operation playback; thereby achieving orderly and efficient file merges and avoiding confusion caused by exceptions in the middle.

[0086] Furthermore, based on the above embodiment, this embodiment also provides a possible implementation scheme, such as Figure 1 As shown, after step S12, the method further includes:

[0087] S14: Delete the metadata attributes of the merged file.

[0088] S15: If there is no abnormality in the file merging transaction being written to the disk, the local file merging transaction is deleted, and the next batch of file merging transactions are written to the disk.

[0089] It should be noted that one of the main purposes of file merging is to integrate multiple data files to simplify the data processing process and improve data processing efficiency. Similarly, the above purpose is also applicable to the management and processing of metadata of each file in a distributed file system. Therefore, after the merged file updates the file metadata attributes, this embodiment also deletes the metadata of the merged file to reduce the data in the metadata server, realize the simplification of metadata, and improve the processing efficiency of metadata.

[0090] Further, based on the above embodiment, this embodiment also provides another suitable implementation scheme for the data of the above-mentioned merged file that has not been deleted. After the file type of the target file is updated to the merged file, such as Figure 1 As shown, the method also includes:

[0091] S16: The file attribute of the merged file on the user side is set to be hidden.

[0092] As can be seen from the above, one purpose of file merging is to reduce the number of files, and convert the processing of multiple small files into the processing of one large file, which is conducive to simplifying the data processing process and improving processing efficiency. In the above embodiment, after the metadata attributes of the merged file are updated, the metadata attributes of the merged file can be deleted, thereby reducing the amount of metadata managed in the metadata server. This embodiment processes the data object of the merged file itself, and hides the merged file on the user side so that the user will not perceive the merged file on the front end. These files are presented to the user in the distributed file system. Only the merged files are presented to the user, which is conducive to simplifying the number of files displayed on the user side and improving the user experience.

[0093] Further, based on the metadata deletion solution of the merged file provided in the above embodiment, this embodiment also provides an implementation scheme for deleting the merged file, such as Figure 2 As shown, the method also includes:

[0094] S21: When the metadata server receives the merge source file deletion request, it reads the metadata attributes of the merge source file to obtain the file offset and data length.

[0095] The merge source file is the merged file targeted by this read request.

[0096] S22: Obtain index information from the storage pool mapping table and the unique identification number index table according to the file offset and the data length.

[0097] The index information includes: unique identification number (ino), file offset (offset), data length (len) and storage pool information.

[0098] S23: Merge the storage pool information of all merged source files according to the index information, and stripe the specific location of the data according to the merged storage pool information.

[0099] S24: Send a data deletion request to the storage pool according to the specific location of the data.

[0100] S25: After the storage pool completes the deletion of all data, the deletion result is returned as the result of merging the source file deletion request.

[0101] Through the above-mentioned process of this embodiment, the merge source file deletion request initiated by the corresponding business client can be responded to. This embodiment uses a new file data index mechanism provided by the above-mentioned file merging method to realize the determination of each merged file data object corresponding to the merged source file, thereby deleting the specified invalid data object in the multi-storage pool, which is an application scheme of the file merging method provided by the present invention when deleting merged files.

[0102] Furthermore, in addition to the above-mentioned merged file deletion, another important access scenario for merged files is read requests. Based on this, this embodiment also provides a merged file read request processing solution that is adapted to a new file data indexing method provided by the above-mentioned embodiment of the method, such as Figure 3 As shown, the method also includes:

[0103] S31: When receiving a merge source file read request initiated by a business client, checking whether the read / write area hits the cache.

[0104] S32: If there is no hit, read the metadata attributes of the merged source file to obtain the file offset and data length.

[0105] S33: Obtain a read / write area index from the storage pool mapping table and the unique identification number index table according to the file offset and the data length.

[0106] S34: Merge the storage pool information of all merged source files according to the read-write area index, and stripe the specific location of the data according to the merged storage pool information.

[0107] S35: Read the data according to the specific location of the data, and after all the data are read, combine all the read data as the return result of the merged source file read request.

[0108] It is not difficult to see from the above steps that the method for locating and reading data objects in this embodiment is the same as the method for locating and deleting data objects in the above embodiment, which is to obtain the file offset and data length by reading the metadata attributes of the merged source file. Among them, the file offset is used as the key in the ino index table to retrieve the corresponding value, that is,<ino,

len,pool_index

[0109] Based on the processing of several common access scenarios such as read requests and delete requests implemented by the file data indexing method provided by the present invention provided in the above embodiments, it is explained that the file data indexing method provided by the present invention is indeed a practical data organization mapping method after file merging, and supports single-file ino striping mapping to multi-ino and multi-storage pool striping mapping, which can truly solve the challenge of "how to implement data organization mapping after merging" faced when using non-native HDFS file merging methods.

[0110] In addition, this embodiment also provides a possible implementation scheme:

[0111] The storage pool mapping table and the unique identification number index table are both composed of multiple sub-tables. The storage pool index mapping information and the unique identification number index information of each merging file and the merged file are allocated to the designated sub-table for storage in a hash manner.

[0112] That is, in the solution provided by this embodiment, when the ino index table and the storage pool mapping table need to be used to store the ino index information and the storage pool index mapping information, in order to avoid a single table being too large, multiple tables are set in the database to store the file data index attributes of all files respectively. When the file data index attributes are actually stored, each file maps the file data index attributes to the corresponding table in a hashing manner for storage.

[0113] It should be noted that this embodiment does not limit the number of sub-tables in the above-mentioned ino index table and storage pool mapping table, and the number of sub-tables contained in the two tables can be the same or different. However, this embodiment provides a possible implementation scheme, in which the database contains 1024 tables for storing the file data index attributes of each file.

[0114] On the other hand, it should be noted that the file merging solution provided in the above embodiment is only a basic file merging process. In practical applications, other steps can be added according to different needs. For example, this embodiment provides a permission verification solution before file merging, such as Figure 4 As shown, the method also includes:

[0115] S10: When receiving a file merge request, perform basic parameter verification to determine whether the merged file and the merged file have read permissions. If no permission is given, subsequent operations are prohibited and operation no permission information is returned to the user side.

[0116] The permission verification scheme before file merging provided in this embodiment verifies the permission of the requesting party before formally merging the files, thereby preventing unauthorized requesting parties from interfering with the normal operation of the distributed file system, and preventing operations such as file merging involving large amounts of data changes from being triggered without permission, thereby ensuring the reliability and security of the distributed file system operation.

[0117] In addition, if Figure 4 As shown, according to different actual needs, other necessary operations may be added to the file merging process, such as Ranger verification, access control list verification, file locking (flock and local lock), quota application and other operations, which are not limited in this embodiment.

[0118] Ranger validation: A centralized security management framework that mainly solves authorization and auditing issues and can perform fine-grained data access control on components of the Hadoop (a distributed system infrastructure) ecosystem such as HDFS.

[0119] In the above embodiment, a file merging method is described in detail, and the present invention also provides a corresponding embodiment of a file merging device. It should be noted that the present invention describes the embodiment of the device part from two perspectives, one is based on the functional module perspective, and the other is based on the hardware perspective.

[0120] Based on the functional module perspective, such as Figure 5 As shown, this embodiment provides a file merging device, including:

[0121] The transaction storage module 11 is used to store the target file in a file merging transaction after receiving a file merging request.

[0122] The metadata updating module 12 is used to update the metadata attributes of the target file and the file merging transaction result file.

[0123] The metadata attributes of the file include: file type; the file types include: common file, merged file and merged file; the metadata attributes of the merged file and the merged file also include: file data index attribute.

[0124] The file data index attributes include: storage pool index mapping information and unique identification number index information.

[0125] The storage pool index mapping information is stored in the storage pool mapping table in the form of key-value pairs, with the index unique code as the key and the storage pool information as the value; the storage pool information includes: storage pool unique code, stripe pair size and redundancy strategy.

[0126] The unique identification number index information is stored in the unique identification number index table in the form of a key-value pair, where the unique identification number index information uses the file offset as the key and the data segment information as the value; the data segment information is in the form of a key-value pair, with the unique identification number as the key and the data length array as the value; the data length array includes the data length and the index unique code.

[0127] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.

[0128] Figure 6 A structural diagram of a file merging device provided in another embodiment of the present invention is shown in FIG. Figure 6 As shown, a file merging device includes: a memory 20, for storing a computer program;

[0129] The processor 21 is used to implement the steps of a file merging method in the above embodiment when executing a computer program.

[0130] The file merging device provided in this embodiment may include but is not limited to a mobile terminal, a personal computer, a workstation, etc.

[0131] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of a digital signal processor (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0132] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 20 is at least used to store the following computer program 201, wherein, after the computer program is loaded and executed by the processor 21, it can implement the relevant steps of a file merging method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 202 may include Windows, Unix, Linux, etc. Data 203 may include, but is not limited to, a file merging method, etc.

[0133] In some embodiments, a file merging device may further include a display screen 22 , an input / output interface 23 , a communication interface 24 , a power supply 25 , and a communication bus 26 .

[0134] Those skilled in the art will understand that Figure 6 The structure shown in the figure does not constitute a limitation of a file merging device, and may include more or fewer components than those shown in the figure.

[0135] A file merging device provided by an embodiment of the present invention includes a memory and a processor. When the processor executes a program stored in the memory, it can implement the following method: a file merging method.

[0136] In addition to the embodiments of the file merging method and device provided in the above embodiments, the present invention also provides an embodiment corresponding to a computer program product. A computer program product includes a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of the file merging method described in any of the above embodiments can be implemented.

[0137] Since the embodiments of the computer program product part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the computer program product part, which will not be repeated here.

[0138] Finally, the present invention also provides an embodiment corresponding to a non-volatile storage medium. The non-volatile storage medium stores a computer program, and when the computer program is executed by a processor, the steps recorded in the above method embodiment are implemented.

[0139] It is understandable that if the method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc. Various media that can store program codes.

[0140] The above is a detailed introduction to a file merging method, device and medium provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the present invention.

[0141] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A file merging method, characterized in that: include: When receiving a file merge request, the target file is merged and stored on disk; Updating metadata attributes of the target file and the file merge transaction result file; The metadata attributes of the file include: file type; the file types include: common file, merged file and merged file; the metadata attributes of the merged file and the merged file also include: file data index attribute; The file data index attributes include: storage pool index mapping information and unique identification number index information; The storage pool index mapping information is stored in the storage pool mapping table in the form of a key-value pair, wherein the storage pool index mapping information uses the index unique code as a key and the storage pool information as a value; the storage pool information includes: the storage pool unique code, the stripe pair size and the redundancy strategy; The unique identification number index information is stored in the unique identification number index table in the form of a key-value pair, wherein the unique identification number index information uses the file offset as the key and the data segment information as the value; the data segment information is in the form of a key-value pair, wherein the unique identification number is used as the key and the data length array is used as the value; the data length array includes the data length and the index unique code.

2. The file merging method according to claim 1, characterized in that: The file merging transaction of the target file into disk includes: According to the preset merging granularity, a corresponding number of the target files are regarded as a batch, and the file merging tasks of each batch are cyclically performed; The method further comprises: If an exception occurs when the file merging transaction is written to the disk, the operation of the abnormal file merging transaction is replayed.

3. The file merging method according to claim 2, characterized in that: After updating the metadata attributes of the target file and the file merge transaction result file, the method further includes: Deleting the metadata attribute of the merged file; If there is no abnormality in the storage of the file merging transaction, the local file merging transaction is deleted, and the next batch of file merging transactions are stored in the disk.

4. The file merging method according to claim 3, characterized in that: Deleting the metadata attribute of the merged file includes: When the metadata server receives a merge source file deletion request, the metadata attribute of the merge source file is read to obtain the file offset and the data length; wherein the merge source file is the merged file targeted by this read request; Acquire index information from the storage pool mapping table and the unique identification number index table according to the file offset and the data length; wherein the index information includes: the unique identification number, the file offset, the data length and the storage pool information; Merging the storage pool information of all the merged source files according to the index information, and striping the specific location of the data according to the merged storage pool information; Sending a data deletion request to the storage pool according to the specific location of the data; After the storage pool completes the deletion of all data, the deletion result is returned as the result of the merged source file deletion request.

5. The file merging method according to claim 1, characterized in that: The method further comprises: When receiving a merge source file read request initiated by a business client, checking whether the read-write area hits the cache; wherein the merge source file is the merged file targeted by this read request; If no hit is found, reading the metadata attribute of the merged source file to obtain the file offset and the data length; Obtaining a read-write area index from the storage pool mapping table and the unique identification number index table according to the file offset and the data length; wherein the read-write area index includes: the unique identification number, the file offset, the data length and the storage pool information; Merging the storage pool information of all the merged source files according to the read-write area index, and striping the specific location of the data according to the merged storage pool information; The data is read according to the specific location of the data, and after all the data are read, all the read data are combined as the return result of the merged source file read request.

6. The file merging method according to claim 1, characterized in that: After the file type of the target file is updated to the merged file, the method further includes: The file attribute of the merged file on the user side is set to be hidden.

7. The file merging method according to any one of claims 1 to 6, characterized in that: The storage pool mapping table and the unique identification number index table are both composed of multiple sub-tables; The storage pool index mapping information and the unique identification number index information of each of the merged file and the merged file are allocated to the designated sub-table for storage in a hash manner.

8. A file merging device, characterized in that: include: The transaction storage module is used to store the target file in a file merging transaction after receiving a file merging request; A metadata updating module, used for updating metadata attributes of the target file and the file merging transaction result file; The metadata attributes of the file include: file type; the file types include: common file, merged file and merged file; the metadata attributes of the merged file and the merged file also include: file data index attribute; The file data index attributes include: storage pool index mapping information and unique identification number index information; The storage pool index mapping information is stored in the storage pool mapping table in the form of a key-value pair, wherein the storage pool index mapping information uses the index unique code as a key and the storage pool information as a value; the storage pool information includes: the storage pool unique code, the stripe pair size and the redundancy strategy; The unique identification number index information is stored in the unique identification number index table in the form of a key-value pair, wherein the unique identification number index information uses the file offset as the key and the data segment information as the value; the data segment information is in the form of a key-value pair, wherein the unique identification number is used as the key and the data length array is used as the value; the data length array includes the data length and the index unique code.

9. A file merging device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the file merging method as described in any one of claims 1 to 7 when executing the computer program.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the file merging method according to any one of claims 1 to 7 are implemented.

Images