File merging method, device, electronic device, storage medium, and program product

Abstract

The disclosed embodiments disclose a file merging method, device, electronic device, storage medium and program product, the method comprising: receiving a small file merging request from a user; the small file merging request comprising identification information of a candidate file to be merged and type information of the candidate file to be merged; obtaining metadata of multiple small files to be merged from an HDFS storage system based on the identification information and type information; merging the metadata of the multiple small files to be merged; sending a file merging request carrying the merged metadata to the HDFS storage system, so that the HDFS storage system manages the multiple files to be merged using the merged metadata. This technical solution can reduce the pressure on the NameNode when managing the metadata of the multiple small files, and can reduce the reading and writing overhead of the metadata of the multiple small files.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and in particular to a file merging method, device, electronic device, storage medium, and program product. Background Art

[0002] In the era of big data, with the rapid rise and widespread adoption of internet technology, the amount of data collected across diverse fields has reached unprecedented levels. Simultaneously, the way data is generated, stored, and processed has undergone revolutionary changes. People's work and daily lives can now be represented digitally, and data usage and querying are extremely frequent.

[0003] Spark is a fast, versatile computing engine designed specifically for large-scale data processing, and is currently forming a rapidly developing and widely used ecosystem. Spark can perform a wide range of operations, including SQL queries, text processing, and machine learning. Spark also provides a large number of libraries, including Spark Core, Spark SQL, Spark Streaming, MLlib, and GraphX. However, when writing to Hive or directly to HDFS using Spark SQL or Spark Streaming, an excessive number of small files can place significant pressure on NameNode memory management, affecting the stable operation of the entire cluster. Therefore, how to address the increased pressure on NameNode memory management caused by storing too many small files in the HDFS storage system is a major issue facing those skilled in the art. Summary of the Invention

[0004] Embodiments of the present disclosure provide a file merging method, device, electronic device, storage medium, and program product.

[0005] In a first aspect, an embodiment of the present disclosure provides a file merging method, comprising:

[0006] Receive a small file merging request from a user; the small file merging request includes identification information of candidate files to be merged and type information of the candidate files to be merged;

[0007] Obtain metadata of multiple small files to be merged from the HDFS storage system based on the identification information and type information;

[0008] Merging the metadata of the multiple small files to be merged;

[0009] A file merge request carrying the merged metadata is sent to the HDFS storage system, so that the HDFS storage system manages the multiple files to be merged using the merged metadata.

[0010] Furthermore, metadata of multiple small files to be merged is obtained from the HDFS storage system based on the identification information and type information, including:

[0011] Obtain metadata of candidate files to be merged that meet the identification information and type information from the HDFS storage system;

[0012] Determining, based on the metadata of the candidate files to be merged, the candidate files to be merged whose storage size is smaller than a preset threshold as the small files to be merged;

[0013] Obtain metadata of the multiple small files to be merged.

[0014] Furthermore, the identification information includes a storage directory of the candidate file to be merged; and / or the type information includes a file type whose corresponding storage block in the HDFS storage system is an independent content.

[0015] Furthermore, merging the metadata of the plurality of small files to be merged includes:

[0016] The metadata of the multiple small files to be merged are merged into the metadata of a merged file, and the data storage block information in the merged metadata includes all the data storage block information in the metadata of the multiple small files to be merged.

[0017] Furthermore, receiving a small file merge request from a user includes:

[0018] In response to a user's small file merging operation, outputting file directory information to which the user has authority;

[0019] Based on the user's selection operation on the file directory information, the small file merging request is triggered; wherein, in the small file merging request, the file directory information operated by the user is used as the identification information of the candidate file to be merged, and the file type selected by the user is used as the type information.

[0020] Furthermore, the merged metadata includes the association relationship between the file identifier of the merged file and the file identifiers of the multiple small files to be merged; and / or, the merged metadata includes the mapping relationship between the file identifiers of the multiple small files to be merged and the data storage block information.

[0021] In a second aspect, an embodiment of the present disclosure provides a file merging method, which includes:

[0022] Receive a file merge request from a client; the file merge request includes file information of multiple small files to be merged and merged metadata; the file contents of the multiple small files to be merged are stored in a data storage block;

[0023] Deleting metadata of the plurality of small files to be merged from the corresponding file information management structure;

[0024] The merged metadata is added to the file information management structure as metadata of a merged file, and the data storage blocks where the file contents of the multiple small files to be merged are located are managed in the form of a merged file; the merged metadata includes information on the data storage blocks where the file contents of the multiple small files to be merged are located.

[0025] Furthermore, it also includes:

[0026] Receiving a file metadata request from the client; the file metadata request includes identification information of a candidate file to be merged and type information of the candidate file to be merged;

[0027] Based on the file directory corresponding to the identification information, metadata of the candidate files to be merged that meet the type information are returned to the client.

[0028] Furthermore, it also includes:

[0029] Obtaining an access request from a client for the merged file; the access request includes a file identifier of the merged file;

[0030] Acquire metadata of the merged file based on the file identifier of the merged file;

[0031] Based on the information of the data storage block in the metadata of the merged file, the file content of the merged file is read from the data storage block and returned to the client.

[0032] Furthermore, it also includes:

[0033] Obtaining an access request from a client to a target small file in the small files to be merged; the access request includes a file identifier of the target small file;

[0034] Determining whether the file identifier is associated with a file identifier of a merged file;

[0035] When the file identifier is associated with the file identifier of a merged file, obtaining metadata of the merged file;

[0036] Determine the target storage block information corresponding to the target small file based on the data storage block information in the metadata of the merged file;

[0037] The file content of the target small file is read from the target storage block based on the information of the target storage block and returned to the client.

[0038] In a third aspect, an embodiment of the present disclosure provides a file merging method, which includes:

[0039] The client receives a small file merging request from a user; the small file merging request includes identification information of candidate files to be merged and type information of the candidate files to be merged;

[0040] The client obtains metadata of multiple small files to be merged from the HDFS storage system based on the identification information and type information;

[0041] The client merges the metadata of the multiple small files to be merged, and sends a file merge request carrying the merged metadata to the HDFS storage system;

[0042] The HDFS storage system receives a file merge request from a client; the file merge request includes file information of multiple small files to be merged and metadata after the merge; the file contents of the multiple small files to be merged are stored in the data storage block;

[0043] The HDFS storage system deletes the metadata of the multiple small files to be merged from the corresponding file information management structure, and adds the merged metadata as the metadata of a merged file to the file information management structure, and manages the data storage blocks where the file contents of the multiple small files to be merged are located in the form of a merged file; the merged metadata includes information about the data storage blocks where the file contents of the multiple small files to be merged are located.

[0044] Furthermore, the client obtains metadata of multiple small files to be merged from the HDFS storage system based on the identification information and type information, including:

[0045] The client obtains metadata of the candidate files to be merged that meet the identification information and type information from the HDFS storage system;

[0046] The client determines, based on the metadata of the candidate files to be merged, the candidate files to be merged whose storage size is smaller than a preset threshold as the small files to be merged;

[0047] The client obtains metadata of the multiple small files to be merged.

[0048] Furthermore, the client merges the metadata of the multiple small files to be merged, including:

[0049] The client merges the metadata of the multiple small files to be merged into the metadata of a merged file, and the data storage block information in the merged metadata includes all the data storage block information in the metadata of the multiple small files to be merged.

[0050] Furthermore, the client receives a small file merge request from the user, including:

[0051] The client outputs file directory information to which the user has authority in response to the user's small file merging operation;

[0052] The client triggers the small file merging request based on the user's selection operation on the file directory information; wherein, in the small file merging request, the file directory information operated by the user is used as the identification information of the candidate file to be merged, and the file type selected by the user is used as the type information.

[0053] Furthermore, the merged metadata includes the association relationship between the file identifier of the merged file and the file identifiers of the multiple small files to be merged; and / or, the merged metadata includes the mapping relationship between the file identifiers of the multiple small files to be merged and the data storage block information.

[0054] Furthermore, the client receives a small file merge request from the user, including:

[0055] The client outputs file directory information to which the user has authority in response to the user's small file merging operation;

[0056] The client triggers the small file merging request based on the user's selection operation on the file directory information; wherein, in the small file merging request, the file directory information operated by the user is used as the identification information of the candidate file to be merged, and the file type selected by the user is used as the type information.

[0057] Furthermore, it also includes:

[0058] The HDFS storage system receives the file metadata request from the client; the file metadata request includes identification information of the candidate file to be merged and type information of the candidate file to be merged;

[0059] The HDFS storage system returns metadata of candidate files to be merged that meet the type information under the file directory corresponding to the identification information to the client.

[0060] Furthermore, it also includes:

[0061] The HDFS storage system obtains an access request from the client for the merged file; the access request includes a file identifier of the merged file;

[0062] The HDFS storage system obtains metadata of the merged file based on the file identifier of the merged file;

[0063] The HDFS storage system reads the file content of the merged file from the data storage block based on the information of the data storage block in the metadata of the merged file, and returns the content to the client.

[0064] Furthermore, it also includes:

[0065] The HDFS storage system obtains a client's access request for a target small file in the small files to be merged; the access request includes a file identifier of the target small file;

[0066] The HDFS storage system determines whether the file identifier is associated with the file identifier of the merged file;

[0067] When the file identifier is associated with the file identifier of the merged file, the HDFS storage system obtains metadata of the merged file;

[0068] The HDFS storage system determines the target storage block information corresponding to the target small file based on the data storage block information in the metadata of the merged file;

[0069] The HDFS storage system reads the file content of the target small file from the target storage block based on the information of the target storage block and returns it to the client.

[0070] In a fourth aspect, an embodiment of the present disclosure provides a file merging device, including:

[0071] A first receiving module is configured to receive a small file merging request from a user; the small file merging request includes identification information of candidate files to be merged and type information of the candidate files to be merged;

[0072] An acquisition module is configured to acquire metadata of a plurality of small files to be merged from an HDFS storage system based on the identification information and the type information;

[0073] A merging module, configured to merge metadata of the plurality of small files to be merged;

[0074] The sending module is configured to send a file merge request carrying the merged metadata to the HDFS storage system, so that the HDFS storage system uses the merged metadata to manage the multiple files to be merged.

[0075] In a fifth aspect, an embodiment of the present disclosure provides a file merging device, including:

[0076] The second receiving module is configured to receive a file merging request from a client; the file merging request includes file information of multiple small files to be merged and merged metadata; the file contents of the multiple small files to be merged are stored in the data storage block;

[0077] a deletion module configured to delete the metadata of the plurality of small files to be merged from the corresponding file information management structure;

[0078] An adding module is configured to add the merged metadata as the metadata of a merged file to the file information management structure, and manage the data storage blocks where the file contents of the multiple small files to be merged are located in the form of a merged file; the merged metadata includes information on the data storage blocks where the file contents of the multiple small files to be merged are located.

[0079] The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the functions.

[0080] In one possible design, the apparatus includes a memory and a processor. The memory is configured to store one or more computer instructions that enable the apparatus to perform the corresponding method, and the processor is configured to execute the computer instructions stored in the memory. The apparatus may also include a communication interface for communicating with other devices or a communication network.

[0081] In a sixth aspect, an embodiment of the present disclosure provides a file merging system, comprising: a client and an HDFS storage system; wherein,

[0082] The client receives a small file merge request from a user; the small file merge request includes identification information of candidate files to be merged and type information of the candidate files to be merged; the client obtains metadata of multiple small files to be merged from the HDFS storage system based on the identification information and type information; merges the metadata of the multiple small files to be merged, and sends a file merge request carrying the merged metadata to the HDFS storage system;

[0083] The HDFS storage system receives a file merge request from a client; the file merge request includes file information of multiple small files to be merged and metadata after merging; the file contents of the multiple small files to be merged are stored in a data storage block; the HDFS storage system deletes the metadata of the multiple small files to be merged from the corresponding file information management structure, and adds the merged metadata as metadata of a merged file to the file information management structure, and manages the data storage block where the file contents of the multiple small files to be merged are located in the form of a merged file; the merged metadata includes information about the data storage block where the file contents of the multiple small files to be merged are located.

[0084] In a seventh aspect, embodiments of the present disclosure provide an electronic device, comprising a memory and a processor, wherein the memory is configured to store one or more computer instructions that support any of the aforementioned devices in executing the aforementioned corresponding methods, and the processor is configured to execute the computer instructions stored in the memory. Any of the aforementioned devices may also include a communication interface for communicating with other devices or a communication network.

[0085] In an eighth aspect, an embodiment of the present disclosure provides a computer-readable storage medium for storing computer instructions used by any of the above-mentioned devices, which includes computer instructions involved in executing any of the above-mentioned methods.

[0086] In a ninth aspect, an embodiment of the present disclosure provides a computer program product comprising computer instructions, which, when executed by a processor, are used to implement the steps of the method described in any of the above aspects.

[0087] The technical solutions provided by the embodiments of the present disclosure may have the following beneficial effects:

[0088] Through the embodiments of the present disclosure, the metadata of multiple small files are merged on the client and sent to the HDFS storage system. The NameNode of the HDFS storage system can store the merged metadata and delete the original metadata of the multiple small files. The data blocks containing the file contents corresponding to the multiple small files remain as they are without being merged. After receiving the merged metadata, the NameNode manages the multiple small files in a manner similar to managing a large file. Even if there are too many small files, merging the metadata of the multiple small files through the embodiments of the present disclosure can reduce the pressure on the NameNode when managing the metadata of the multiple small files and reduce the reading and writing overhead of the metadata of the multiple small files.

[0089] It should be understood that the foregoing general description and the following detailed description are merely exemplary and explanatory and are not restrictive of the embodiments of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Other features, objectives and advantages of the embodiments of the present disclosure will become more apparent through the following detailed description of non-limiting embodiments in conjunction with the accompanying drawings. In the accompanying drawings:

[0091] Figure 1 A flowchart showing a method for merging files according to an embodiment of the present disclosure is shown;

[0092] Figure 2 A flowchart showing a file merging method according to another embodiment of the present disclosure is shown;

[0093] Figure 3 A flowchart showing a file merging method according to another embodiment of the present disclosure is shown;

[0094] Figure 4 Shows an overall flow chart of a file merging method according to an embodiment of the present disclosure;

[0095] Figure 5 A structural block diagram of a file merging device according to an embodiment of the present disclosure is shown;

[0096] Figure 6 A structural block diagram of a file merging device according to another embodiment of the present disclosure is shown;

[0097] Figure 7 A structural block diagram of a file merging system according to an embodiment of the present disclosure is shown;

[0098] Figure 8 It is a structural diagram of a computer system suitable for implementing the file merging method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0099] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. In addition, for the sake of clarity, parts not related to the description of the exemplary embodiments are omitted in the accompanying drawings.

[0100] In the embodiments of the present disclosure, it should be understood that terms such as "including" or "having" are intended to indicate the existence of features, numbers, steps, behaviors, components, parts, or a combination thereof disclosed in this specification, and are not intended to exclude the possibility of one or more other features, numbers, steps, behaviors, components, parts, or a combination thereof existing or being added.

[0101] It should also be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present disclosure can be combined with each other. The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0102] The Hadoop Distributed File System (HDFS) is widely used in large-scale computing due to its high reliability, efficiency, and scalability. A distributed file system, consisting of a NameNode and multiple DataNodes, is a crucial component of a cluster structure. As the size of a cluster's data continues to grow, the NameNode's resident memory also increases with the amount of data. Therefore, the NameNode's heap memory size must be continuously adjusted to accommodate the growing memory space. However, the NameNode heap space cannot be increased indefinitely. For a cluster with 200 million data blocks, the total NameNode memory occupies approximately 113GB; each small file occupies one data block. Merging small files can effectively reduce the number of data blocks.

[0103] Currently, there are several commonly used small file merging solutions:

[0104] (1) Storing small files in Hbase improves file storage efficiency through file merging and decomposition. The disadvantage of this solution is that as the number of files increases, a large number of Hbase merging and decomposition operations will occupy a large amount of system resources, seriously affecting system performance. Rewriting small files into large files puts a large amount of read and write pressure on the system. In addition, Hbase only supports simple character types and does not support other types such as pictures well, which requires users to handle them separately.

[0105] (2) Another file merging solution is to use Sequence File to merge files. Sequence File is used to store binary key-value files. Usually, when using Sequence File to store small files, the file name is stored in the key (key value) and the file content is stored in the value. The biggest disadvantage of this method is that since the key value is an unsorted file, the random reading efficiency is low and the entire file needs to be traversed to read. In addition, this method does not support file append operations. Therefore, the small files before merging must be cached on the server, so the security of the files cannot be guaranteed.

[0106] To this end, the embodiment of the present disclosure proposes a file merging solution, in which a user initiates a small file merging request for small files stored in HDFS. The user can specify the file information of the small files to be merged; the file information may include the identification information of the candidate files to be merged and the type information of the candidate files to be merged. After receiving the small file merging request from the user, the client obtains the metadata of the small files to be merged from the HDFS storage system based on the identification information and the type information. The metadata may include but is not limited to file identification, storage location, storage type, size, offset, etc. The client can merge the metadata of multiple small files to be merged and send the merged metadata to the HDFS storage system so that the HDFS storage system uses the merged metadata to manage the multiple small files to be merged. In this way, the metadata of multiple small files are merged on the client and sent to the HDFS storage system. The NameNode of the HDFS storage system can store the merged metadata and delete the original metadata of the multiple small files. The data blocks where the file contents corresponding to the multiple small files are located remain as they are without being merged. After receiving the merged metadata, the NameNode manages the multiple small files in a manner similar to that of managing a large file. Even if there are too many small files, merging the metadata of the multiple small files in the manner of the embodiment of the present disclosure can reduce the pressure on the NameNode when managing the metadata of the multiple small files, and can reduce the reading and writing overhead of the metadata of the multiple small files.

[0107] Figure 1 A flowchart of a file merging method according to an embodiment of the present disclosure is shown as follows: Figure 1 As shown, the file merging method includes the following steps:

[0108] In step S101, a small file merging request is received from a user; the small file merging request includes identification information of candidate files to be merged and type information of the candidate files to be merged;

[0109] In step S102, metadata of multiple small files to be merged is obtained from the HDFS storage system based on the identification information and type information;

[0110] In step S103, the metadata of the plurality of small files to be merged are merged;

[0111] In step S104, a file merge request carrying the merged metadata is sent to the HDFS storage system, so that the HDFS storage system manages the multiple files to be merged using the merged metadata.

[0112] As mentioned above, Spark is a fast, versatile computing engine designed specifically for large-scale data processing, and it has formed a rapidly developing and widely used ecosystem. Spark can perform a wide range of operations, including SQL queries, text processing, and machine learning. Spark also provides a large number of libraries, including Spark Core, Spark SQL, Spark Streaming, MLlib, and GraphX. However, when writing to Hive or directly to HDFS using Spark SQL or Spark Streaming, excessive small files can place significant pressure on NameNode memory management, affecting the stable operation of the entire cluster. Therefore, how to address the increased pressure on NameNode memory management caused by storing too many small files in the HDFS storage system is a major issue facing those skilled in the art.

[0113] In view of the above problems, in this embodiment, a file merging method is proposed, in which a user initiates a small file merging request for small files stored in HDFS. The user can specify the file information of the small files to be merged; the file information may include the identification information of the candidate files to be merged and the type information of the candidate files to be merged. After receiving the user's small file merging request, the client obtains the metadata of the small files to be merged from the HDFS storage system based on the identification information and type information. The metadata may include but is not limited to file identification, storage location, storage type, size, offset, etc. The client can merge the metadata of multiple small files to be merged and send the merged metadata to the HDFS storage system so that the HDFS storage system uses the merged metadata to manage the multiple small files to be merged. In this way, the metadata of multiple small files are merged on the client and sent to the HDFS storage system. The NameNode of the HDFS storage system can store the merged metadata and delete the original metadata of the multiple small files. The data blocks where the file contents corresponding to the multiple small files are located remain as they are without being merged. After receiving the merged metadata, the NameNode manages the multiple small files in a manner similar to that of managing a large file. Even if there are too many small files, merging the metadata of the multiple small files in the manner of the embodiment of the present disclosure can reduce the pressure on the NameNode when managing the metadata of the multiple small files, and can reduce the reading and writing overhead of the metadata of the multiple small files.

[0114] In one embodiment of the present disclosure, the file merging method may be applicable to be executed on a client of a distributed file system.

[0115] In one embodiment of the present disclosure, after the execution of the task started by the big data processing and analysis engine, such as Spark, is completed, the data can be written to a disk file. The disk file can be a disk file in a distributed file system. In the big data processing and analysis engine, a job is assigned to multiple different tasks executed in parallel, and each task generates multiple files during the execution process. Therefore, for one job, multiple tasks executed in parallel will generate multiple disk files, and when there are many parallel tasks, more smaller disk files may be generated. In some existing technologies, in order to reduce the excessive number of small files in the distributed file system, multiple small files in the distributed file system are read out by Namenode, and then the multiple small files are merged and rewritten into a large file. This method is likely to cause greater pressure on Namenode when there are many small files, and the reading and writing overhead of too many small files will also increase.

[0116] Therefore, in the embodiment of the present disclosure, the user checks the small files stored in the distributed storage system. If it is found that there are too many small files in a certain directory or directories, these small files can be merged through the client request. Alternatively, the user can also regularly initiate a request to merge small files. The user can provide the identification information and type information of the small files to be merged. In the case that the user cannot determine which are the small files, the identification information of the candidate files to be merged and the type information that can be merged can be specified. The client can obtain the metadata of the small files to be merged from the HDFS storage system based on the identification information and type information of the candidate files to be merged specified by the user.

[0117] In one embodiment of the present disclosure, the identification information of the candidate files to be merged may be a target identifier including multiple candidate files to be merged. The candidate files to be merged may or may not be the files to be merged. The user submits a request to merge the small files that meet the identification information and type information by providing the identification information and type information of the candidate files to be merged. The client or the HDFS storage system may determine the small files to be merged that meet the identification information and type information, thereby obtaining the metadata of the multiple small files to be merged from the HDFS storage system.

[0118] In one embodiment of the present disclosure, after a client obtains metadata of multiple small files to be merged from an HDFS storage system, the metadata of the multiple small files to be merged can be merged. The merged metadata at least stores the storage block information where the file contents of the multiple small files to be merged are located. The file contents of the small files to be merged can still be accessed through the merged metadata. After the merged metadata is sent to the HDFS storage system, the HDFS storage system stores and manages it as the metadata of a large file, and the metadata corresponding to the multiple small files to be merged can be directly deleted. In this case, when it is necessary to access the file contents of the multiple small files to be merged, the merged metadata can be directly read to determine the storage block information where the file contents of the small files to be accessed are located, and the corresponding file content can be obtained from the storage block. In this way, only the metadata of the small files are merged, and the corresponding storage blocks remain unchanged. Therefore, during the merging process, only the metadata of the multiple small files need to be read, and the corresponding storage blocks do not need to be read. This will not put a lot of pressure on the HDFS storage system and will not generate a lot of reading overhead.

[0119] In one embodiment of the present disclosure, step S102, i.e., the step of obtaining metadata of multiple small files to be merged from the HDFS storage system based on the identification information and type information, further includes the following steps:

[0120] Obtain metadata of candidate files to be merged that meet the identification information and type information from the HDFS storage system;

[0121] Determining, based on the metadata of the candidate files to be merged, the candidate files to be merged whose storage size is smaller than a preset threshold as the small files to be merged;

[0122] Obtain metadata of the multiple small files to be merged.

[0123] In this embodiment, the client can obtain metadata of candidate files to be merged that meet the identification information and type information provided by the user from the HDFS storage system. For example, the client can obtain metadata of all files in the directory specified by the user whose file types match the type information.

[0124] Since the file size is stored in the metadata of each file, the size of each candidate file to be merged can be determined based on its metadata. If the size of the candidate file to be merged is less than a preset threshold, the candidate file to be merged is considered a small file to be merged; otherwise, the candidate file to be merged is considered not a small file to be merged. The preset threshold can be set based on actual needs and is not limited here.

[0125] In one embodiment of the present disclosure, in the HDFS storage system, in order to facilitate the management and backup of files, the concept of storage blocks (blocks) is introduced. The storage block here is the smallest storage unit in the HDFS storage system. HDFS defines the size of a storage block as 64MB by default. When a file is uploaded to HDFS, if the file size is larger than the set storage block size, the file will be divided and stored into multiple storage blocks. Multiple storage blocks can be stored on different DataNodes. During the whole process, the HDFS storage system will ensure that one storage block is stored on one DataNode. However, it is worth noting that if the size of a file does not reach 64MB, the file will not occupy the entire storage block space. The NameNode in HDFS will record on which DataNode each storage block of the file in the above-mentioned file storage blocks is stored. This information is generally also called metadata (MetaInfo). When a job is submitted to a big data processing and analysis engine such as Spark for execution, a large amount of data may be transmitted during the process. The data generated by the Spark task will be partitioned and then generate multiple files. The size of each file may be smaller than the size of the storage block. In this case, a file stored on a storage block cannot occupy the entire space of the disk storage block, resulting in wasted space and excessive metadata size, which squeezes the NameNode memory. Therefore, in the embodiment of the present disclosure, a preset threshold can be set based on the storage block size of the distributed file system, for example, set to the storage block size, and candidate files to be merged that are smaller than the storage block size are determined to be small files to be merged.

[0126] After determining which small files are to be merged, the metadata of these small files to be merged are merged into the metadata of a large file, and the merged metadata is provided to the HDFS storage system for storage and management.

[0127] In one embodiment of the present disclosure, the identification information includes the storage directory of the candidate file to be merged; and / or the type information includes the file type of the corresponding storage block in the HDFS storage system as independent content.

[0128] In this embodiment, since some small files may not exist independently, similar to the case where a compressed file is split into multiple files for storage, such files cannot be merged using the method of the embodiment of the present disclosure. Therefore, in the embodiment of the present disclosure, the file type of the small files to be merged can be a file whose content in the storage block is independent, that is, the storage block content corresponding to the file does not depend on any other files.

[0129] In one embodiment of the present disclosure, step S103, i.e., the step of merging the metadata of the plurality of small files to be merged, further includes the following steps:

[0130] The metadata of the multiple small files to be merged are merged into the metadata of a merged file, and the data storage block information in the merged metadata includes all the data storage block information in the metadata of the multiple small files to be merged.

[0131] In this embodiment, the metadata in multiple small files to be merged can be merged into the metadata of one file, the duplicate content in the multiple files to be merged can be deleted, and the non-duplicate content can be retained and organized according to the structure of the metadata of one file. The storage block information used to indicate the location of the file content in the multiple files to be merged needs to be retained in the merged metadata. When accessing the file content of one of the small files to be merged, the storage block where the file content of the small file to be merged is located can be found from the merged metadata, and then the corresponding file content can be obtained from the storage block for access.

[0132] In one embodiment of the present disclosure, step S101, i.e., the step of receiving a small file merging request from a user, further includes the following steps:

[0133] In response to a user's small file merging operation, outputting file directory information to which the user has authority;

[0134] Based on the user's selection operation on the file directory information, the small file merging request is triggered; wherein, in the small file merging request, the file directory information operated by the user is used as the identification information of the candidate file to be merged, and the file type selected by the user is used as the type information.

[0135] In this embodiment, when a user wants to request to merge small files stored in the HDFS storage system at an appropriate time, a small file merge operation can be initiated based on the interface provided on the client. After the client detects the small file merge operation, information such as the file directory to which the user has authority can be output on the client interface. The file directory can be, for example, one of the directories or a directory tree structure. The user can select the output file directory, thereby triggering the client to generate a small file merge request. The small file merge request can include the file directory selected by the user, and use the file directory as identification information of the candidate file to be merged; in addition, the user can select the file type to be merged, and the small file merge request also includes type information of the file type.

[0136] In one embodiment of the present disclosure, the merged metadata includes an association relationship between the file identifier of the merged file and the file identifiers of the multiple small files to be merged; and / or, the merged metadata includes a mapping relationship between the file identifiers of the multiple small files to be merged and data storage block information.

[0137] In this embodiment, in addition to storing the storage block information corresponding to the file contents of each of the multiple small files to be merged, the merged metadata also includes the file identifier of the merged file corresponding to the merged metadata, and the mapping relationship between the file identifier of the merged file and the file identifiers of the multiple small files to be merged.

[0138] In addition, the storage block information stored in the merged metadata can also establish a mapping relationship between it and the corresponding small file to be merged. Through this mapping relationship, the storage block information where the file content of the small file to be merged is located can be determined based on the file identifier of the merged file and the file identifier of the small file to be merged.

[0139] Figure 2 A flowchart of a file merging method according to another embodiment of the present disclosure is shown. Figure 2 As shown, the file merging method includes the following steps S201-S203:

[0140] In step S201, a file merging request is received from a client; the file merging request includes file information of multiple small files to be merged and merged metadata; the file contents of the multiple small files to be merged are stored in a data storage block;

[0141] In step S202, the metadata of the plurality of small files to be merged are deleted from the corresponding file information management structure;

[0142] In step S203, the merged metadata is added to the file information management structure as the metadata of a merged file, and the data storage block where the file contents of the multiple small files to be merged are located is managed in the form of a merged file; the merged metadata includes information on the data storage block where the file contents of the multiple small files to be merged are located.

[0143] As mentioned above, Spark is a fast, versatile computing engine designed specifically for large-scale data processing, and it has formed a rapidly developing and widely used ecosystem. Spark can perform a wide range of operations, including SQL queries, text processing, and machine learning. Spark also provides a large number of libraries, including Spark Core, Spark SQL, Spark Streaming, MLlib, and GraphX. However, when writing to Hive or directly to HDFS using Spark SQL or Spark Streaming, excessive small files can place significant pressure on NameNode memory management, affecting the stable operation of the entire cluster. Therefore, how to address the increased pressure on NameNode memory management caused by storing too many small files in the HDFS storage system is a major issue facing those skilled in the art.

[0144] In view of the above problems, in this embodiment, a file merging method is proposed, in which a user initiates a small file merging request for small files stored in HDFS. The user can specify the file information of the small files to be merged; the file information may include the identification information of the candidate files to be merged and the type information of the candidate files to be merged. After receiving the user's small file merging request, the client obtains the metadata of the small files to be merged from the HDFS storage system based on the identification information and type information. The metadata may include but is not limited to file identification, storage location, storage type, size, offset, etc. The client can merge the metadata of multiple small files to be merged and send the merged metadata to the HDFS storage system so that the HDFS storage system uses the merged metadata to manage the multiple small files to be merged. In this way, the metadata of multiple small files are merged on the client and sent to the HDFS storage system. The NameNode of the HDFS storage system can store the merged metadata and delete the original metadata of the multiple small files. The data blocks where the file contents corresponding to the multiple small files are located remain as they are without being merged. After receiving the merged metadata, the NameNode manages the multiple small files in a manner similar to that of managing a large file. Even if there are too many small files, merging the metadata of the multiple small files in the manner of the embodiment of the present disclosure can reduce the pressure on the NameNode when managing the metadata of the multiple small files, and can reduce the reading and writing overhead of the metadata of the multiple small files.

[0145] In one embodiment of the present disclosure, the file merging method may be applicable to be executed on an HDFS storage system.

[0146] In one embodiment of the present disclosure, after the execution of the task started by the big data processing and analysis engine, such as Spark, is completed, the data can be written to a disk file. The disk file can be a disk file in a distributed file system. In the big data processing and analysis engine, a job is assigned to multiple different tasks executed in parallel, and each task generates multiple files during the execution process. Therefore, for one job, multiple tasks executed in parallel will generate multiple disk files, and when there are many parallel tasks, more smaller disk files may be generated. In some existing technologies, in order to reduce the excessive number of small files in the distributed file system, multiple small files in the distributed file system are read out by Namenode, and then the multiple small files are merged and rewritten into a large file. This method is likely to cause greater pressure on Namenode when there are many small files, and the reading and writing overhead of too many small files will also increase.

[0147] To this end, after the client obtains the metadata of multiple small files to be merged from the HDFS storage system, it can merge the metadata of the multiple small files to be merged. The merged metadata at least stores the storage block information where the file contents of the multiple small files to be merged are located. The file contents of the small files to be merged can still be accessed through the merged metadata. After the merged metadata is sent to the HDFS storage system, the HDFS storage system stores and manages it as the metadata of a large file, and the metadata corresponding to the multiple small files to be merged can be directly deleted.

[0148] After the HDFS storage system receives the above-mentioned file merge request, it deletes the metadata of the multiple small files to be merged in the file information management structure, and adds the merged metadata to the file information management structure as the metadata of a merged file. The file information management structure can be managed by the Namenode in the HDFS storage system, for example. After the Namenode receives the file merge request, it can delete the metadata of the multiple small files to be merged in the corresponding directory in the file information management structure, and add the merged metadata to the corresponding directory. At the Namenode, this is equivalent to merging the multiple small files to be merged into a merged file. In addition, the file contents of the multiple small files to be merged are stored in storage blocks, which are distributed on different Datanode nodes. The storage blocks where the file contents of the multiple small files to be merged are located do not undergo any changes. Instead, the information of these storage blocks is written into the merged metadata as the storage blocks corresponding to the merged file, thereby realizing the merging of multiple small files, reducing the maintenance pressure of the metadata of these small files on the Namenode, and reducing the reading and writing overhead of these small files.

[0149] In Namenode, the metadata of multiple small files to be merged is replaced by the merged metadata, and the data storage blocks corresponding to the multiple small files to be merged are managed as the data storage blocks of a merged file. Although the data storage blocks of the merged file are distributed in multiple different data storage blocks, it can still be managed as a file, thereby achieving the purpose of merging small files.

[0150] In this case, when you need to access the file contents of the multiple small files to be merged, you can directly read the merged metadata, determine the storage block information where the file contents of the small files to be accessed are located, and then obtain the corresponding file contents from the storage block. In this way, only the metadata of the small files are merged, and the corresponding storage blocks remain unchanged. Therefore, during the merging process, you only need to read the metadata of multiple small files, and do not need to read the corresponding storage blocks. This will not put a lot of pressure on the HDFS storage system, and will not incur a lot of read overhead.

[0151] In one embodiment of the present disclosure, the method further comprises the following steps:

[0152] Receiving a file metadata request from the client; the file metadata request includes identification information of a candidate file to be merged and type information of the candidate file to be merged;

[0153] Based on the file directory corresponding to the identification information, metadata of the candidate files to be merged that meet the type information are returned to the client.

[0154] In this embodiment, the client can obtain metadata of candidate files to be merged that meet the identification information and type information provided by the user from the HDFS storage system. For example, the client can obtain metadata of all files in the directory specified by the user whose file types match the type information.

[0155] The identification information of the candidate files to be merged can be a target identifier including multiple candidate files to be merged. The candidate files to be merged can be or not be the files to be merged. The user provides the identification information and type information of the candidate files to be merged, and requests to merge small files that match the identification information and type information. The client or HDFS storage system can determine the small files to be merged that match the identification information and type information, and thereby obtain metadata of the multiple small files to be merged from the HDFS storage system.

[0156] After the HDFS storage system receives the file metadata request from the client, it obtains the metadata of the corresponding candidate file to be merged from the corresponding file directory based on the identification information. The candidate file to be merged is a file whose file type matches the type information in the file metadata request. The metadata of the candidate file to be merged can be returned to the client.

[0157] Because the file size is stored in the metadata of each file, the client can determine the size of each candidate file to be merged based on the metadata of the candidate file to be merged. If the size of the candidate file to be merged is less than a preset threshold, the candidate file to be merged is considered to be a small file to be merged. Otherwise, the candidate file to be merged is considered not to be a small file to be merged. The preset threshold can be set based on actual needs and is not limited here.

[0158] In one embodiment of the present disclosure, the method further comprises the following steps:

[0159] Obtaining an access request from a client for the merged file; the access request includes a file identifier of the merged file;

[0160] Acquire metadata of the merged file based on the file identifier of the merged file;

[0161] Based on the information of the data storage block in the metadata of the merged file, the file content of the merged file is read from the data storage block and returned to the client.

[0162] In this optional implementation, in addition to storing the storage block information corresponding to the file contents of each of the multiple small files to be merged, the merged metadata also includes the file identifier of the merged file corresponding to the merged metadata, and the mapping relationship between the file identifier of the merged file and the file identifiers of the multiple small files to be merged.

[0163] In addition, the storage block information stored in the merged metadata can also establish a mapping relationship between it and the corresponding small file to be merged. Through this mapping relationship, the storage block information where the file content of the small file to be merged is located can be determined based on the file identifier of the merged file and the file identifier of the small file to be merged.

[0164] The client can access data for the merged file. After sending a file access request to the HDFS storage system, the HDFS storage system retrieves the corresponding metadata from the file information management structure based on the file identifier of the merged file in the file access request, and reads the file content of the merged file from the data storage block information recorded in the metadata. It should be noted that the metadata records the information of all data storage blocks of multiple small files to be merged before they are merged, and based on the information of the data storage blocks, the file content corresponding to each small file to be merged can be read.

[0165] In one embodiment of the present disclosure, the method further comprises the following steps:

[0166] Obtaining an access request from a client to a target small file in the small files to be merged; the access request includes a file identifier of the target small file;

[0167] Determining whether the file identifier is associated with a file identifier of a merged file;

[0168] When the file identifier is associated with the file identifier of a merged file, obtaining metadata of the merged file;

[0169] Determine the target storage block information corresponding to the target small file based on the data storage block information in the metadata of the merged file;

[0170] The file content of the target small file is read from the target storage block based on the information of the target storage block and returned to the client.

[0171] In this embodiment, as described above, in addition to storing the storage block information corresponding to the file contents of each of the multiple small files to be merged, the merged metadata also includes the file identifier of the merged file corresponding to the merged metadata, and the mapping relationship between the file identifier of the merged file and the file identifiers of the multiple small files to be merged.

[0172] Therefore, the client can also access the file content of the target small file based on the file identifier of one of the original multiple small files to be merged. After the HDFS storage system receives a file access request carrying the file identifier of the target small file, if the file identifier corresponding to the file identifier of the merged file is found in the file information management structure, the metadata of the merged file is obtained based on the file identifier of the merged file, the information of the data storage block corresponding to the file identifier of the target small file is matched from the metadata of the merged file, and the file content of the target small file is read from the corresponding data storage block based on the information of the data storage block, thereby returning the file content to the client.

[0173] Figure 2The technical terms and technical features involved in the embodiments shown and related Figure 1 The technical terms and technical features mentioned in the embodiments shown and related are the same or similar. Figure 2 The explanation and description of the technical terms and technical features involved in the embodiments shown and related can refer to the above Figure 1 The explanations of the illustrated and related embodiments will not be repeated here.

[0174] Figure 3 A flowchart of a file merging method according to another embodiment of the present disclosure is shown. Figure 3 As shown, the file merging method includes the following steps S301-S305:

[0175] In step S301, the client obtains metadata of multiple small files to be merged from the HDFS storage system based on the identification information and type information;

[0176] In step S302, the distributed file system returns the file information in the distributed file directory involved in the data writing operation corresponding to the data writing success event in response to the file information reading request;

[0177] In step S303, the client merges the metadata of the multiple small files to be merged, and sends a file merge request carrying the merged metadata to the HDFS storage system;

[0178] In step S304, the HDFS storage system receives a file merge request from the client; the file merge request includes file information of multiple small files to be merged and merged metadata; the file contents of the multiple small files to be merged are stored in the data storage block;

[0179] In step S305, the HDFS storage system deletes the metadata of the multiple small files to be merged from the corresponding file information management structure, and adds the merged metadata as the metadata of a merged file to the file information management structure, and manages the data storage blocks where the file contents of the multiple small files to be merged are located in the form of a merged file; the merged metadata includes information on the data storage blocks where the file contents of the multiple small files to be merged are located.

[0180] As mentioned above, Spark is a fast, versatile computing engine designed specifically for large-scale data processing, and it has formed a rapidly developing and widely used ecosystem. Spark can perform a wide range of operations, including SQL queries, text processing, and machine learning. Spark also provides a large number of libraries, including Spark Core, Spark SQL, Spark Streaming, MLlib, and GraphX. However, when writing to Hive or directly to HDFS using Spark SQL or Spark Streaming, excessive small files can place significant pressure on NameNode memory management, affecting the stable operation of the entire cluster. Therefore, how to address the increased pressure on NameNode memory management caused by storing too many small files in the HDFS storage system is a major issue facing those skilled in the art.

[0181] In view of the above problems, in this embodiment, a file merging method is proposed, in which a user initiates a small file merging request for small files stored in HDFS. The user can specify the file information of the small files to be merged; the file information may include the identification information of the candidate files to be merged and the type information of the candidate files to be merged. After receiving the user's small file merging request, the client obtains the metadata of the small files to be merged from the HDFS storage system based on the identification information and type information. The metadata may include but is not limited to file identification, storage location, storage type, size, offset, etc. The client can merge the metadata of multiple small files to be merged and send the merged metadata to the HDFS storage system so that the HDFS storage system uses the merged metadata to manage the multiple small files to be merged. In this way, the metadata of multiple small files are merged on the client and sent to the HDFS storage system. The NameNode of the HDFS storage system can store the merged metadata and delete the original metadata of the multiple small files. The data blocks where the file contents corresponding to the multiple small files are located remain as they are without being merged. After receiving the merged metadata, the NameNode manages the multiple small files in a manner similar to that of managing a large file. Even if there are too many small files, merging the metadata of the multiple small files in the manner of the embodiment of the present disclosure can reduce the pressure on the NameNode when managing the metadata of the multiple small files, and can reduce the reading and writing overhead of the metadata of the multiple small files.

[0182] In one embodiment of the present disclosure, the file merging method may be applicable to be executed on a system including a client and an HDFS storage system.

[0183] In one embodiment of the present disclosure, step S302, i.e., the step in which the client obtains metadata of multiple small files to be merged from the HDFS storage system based on the identification information and type information, further includes the following steps:

[0184] The client obtains metadata of the candidate files to be merged that meet the identification information and type information from the HDFS storage system;

[0185] The client determines, based on the metadata of the candidate files to be merged, the candidate files to be merged whose storage size is smaller than a preset threshold as the small files to be merged;

[0186] The client obtains metadata of the multiple small files to be merged.

[0187] In this embodiment, the client can obtain metadata of candidate files to be merged that meet the identification information and type information provided by the user from the HDFS storage system. For example, the client can obtain metadata of all files in the directory specified by the user whose file types match the type information.

[0188] Since the file size is stored in the metadata of each file, the size of each candidate file to be merged can be determined based on its metadata. If the size of the candidate file to be merged is less than a preset threshold, the candidate file to be merged is considered a small file to be merged; otherwise, the candidate file to be merged is considered not a small file to be merged. The preset threshold can be set based on actual needs and is not limited here.

[0189] In one embodiment of the present disclosure, in the HDFS storage system, in order to facilitate the management and backup of files, the concept of storage blocks (blocks) is introduced. The storage block here is the smallest storage unit in the HDFS storage system. HDFS defines the size of a storage block as 64MB by default. When a file is uploaded to HDFS, if the file size is larger than the set storage block size, the file will be divided and stored into multiple storage blocks. Multiple storage blocks can be stored on different DataNodes. During the whole process, the HDFS storage system will ensure that one storage block is stored on one DataNode. However, it is worth noting that if the size of a file does not reach 64MB, the file will not occupy the entire storage block space. The NameNode in HDFS will record on which DataNode each storage block of the file in the above-mentioned file storage blocks is stored. This information is generally also called metadata (MetaInfo). When a job is submitted to a big data processing and analysis engine such as Spark for execution, a large amount of data may be transmitted during the process. The data generated by the Spark task will be partitioned and then generate multiple files. The size of each file may be smaller than the size of the storage block. In this case, a file stored on a storage block cannot occupy the entire space of the disk storage block, resulting in wasted space and excessive metadata size, which squeezes the NameNode memory. Therefore, in the embodiment of the present disclosure, a preset threshold can be set based on the storage block size of the distributed file system, for example, set to the storage block size, and candidate files to be merged that are smaller than the storage block size are determined to be small files to be merged.

[0190] After determining which small files are to be merged, the metadata of these small files to be merged are merged into the metadata of a large file, and the merged metadata is provided to the HDFS storage system for storage and management.

[0191] In one embodiment of the present disclosure, the identification information includes the storage directory of the candidate file to be merged; and / or the type information includes the file type of the corresponding storage block in the HDFS storage system as independent content.

[0192] In this embodiment, since some small files may not exist independently, similar to the case where a compressed file is split into multiple files for storage, such files cannot be merged using the method of the embodiment of the present disclosure. Therefore, in the embodiment of the present disclosure, the file type of the small files to be merged can be a file whose content in the storage block is independent, that is, the storage block content corresponding to the file does not depend on any other files.

[0193] In one embodiment of the present disclosure, in step S303, the step of merging the metadata of the plurality of small files to be merged by the client further includes the following steps:

[0194] The client merges the metadata of the multiple small files to be merged into the metadata of a merged file, and the data storage block information in the merged metadata includes all the data storage block information in the metadata of the multiple small files to be merged.

[0195] In this embodiment, the metadata in multiple small files to be merged can be merged into the metadata of one file, the duplicate content in the multiple files to be merged can be deleted, and the non-duplicate content can be retained and organized according to the structure of the metadata of one file. The storage block information used to indicate the location of the file content in the multiple files to be merged needs to be retained in the merged metadata. When accessing the file content of one of the small files to be merged, the storage block where the file content of the small file to be merged is located can be found from the merged metadata, and then the corresponding file content can be obtained from the storage block for access.

[0196] In one embodiment of the present disclosure, step S301, i.e., the step where the client receives a small file merge request from a user, further includes the following steps:

[0197] The client outputs file directory information to which the user has authority in response to the user's small file merging operation;

[0198] The client triggers the small file merging request based on the user's selection operation on the file directory information; wherein, in the small file merging request, the file directory information operated by the user is used as the identification information of the candidate file to be merged, and the file type selected by the user is used as the type information.

[0199] In this embodiment, when a user wants to request to merge small files stored in the HDFS storage system at an appropriate time, a small file merge operation can be initiated based on the interface provided on the client. After the client detects the small file merge operation, information such as the file directory to which the user has authority can be output on the client interface. The file directory can be, for example, one of the directories or a directory tree structure. The user can select the output file directory, thereby triggering the client to generate a small file merge request. The small file merge request can include the file directory selected by the user, and use the file directory as identification information of the candidate file to be merged; in addition, the user can select the file type to be merged, and the small file merge request also includes type information of the file type.

[0200] In one embodiment of the present disclosure, the merged metadata includes an association relationship between the file identifier of the merged file and the file identifiers of the multiple small files to be merged; and / or, the merged metadata includes a mapping relationship between the file identifiers of the multiple small files to be merged and data storage block information.

[0201] In this embodiment, in addition to storing the storage block information corresponding to the file contents of each of the multiple small files to be merged, the merged metadata also includes the file identifier of the merged file corresponding to the merged metadata, and the mapping relationship between the file identifier of the merged file and the file identifiers of the multiple small files to be merged.

[0202] In addition, the storage block information stored in the merged metadata can also establish a mapping relationship between it and the corresponding small file to be merged. Through this mapping relationship, the storage block information where the file content of the small file to be merged is located can be determined based on the file identifier of the merged file and the file identifier of the small file to be merged.

[0203] In one embodiment of the present disclosure, the method further comprises the following steps:

[0204] The HDFS storage system receives the file metadata request from the client; the file metadata request includes identification information of the candidate file to be merged and type information of the candidate file to be merged;

[0205] The HDFS storage system returns metadata of candidate files to be merged that meet the type information under the file directory corresponding to the identification information to the client.

[0206] In this embodiment, the client can obtain metadata of candidate files to be merged that meet the identification information and type information provided by the user from the HDFS storage system. For example, the client can obtain metadata of all files in the directory specified by the user whose file types match the type information.

[0207] The identification information of the candidate files to be merged can be a target identifier including multiple candidate files to be merged. The candidate files to be merged can be or not be the files to be merged. The user provides the identification information and type information of the candidate files to be merged, and requests to merge small files that match the identification information and type information. The client or HDFS storage system can determine the small files to be merged that match the identification information and type information, and thereby obtain metadata of the multiple small files to be merged from the HDFS storage system.

[0208] After the HDFS storage system receives the file metadata request from the client, it obtains the metadata of the corresponding candidate file to be merged from the corresponding file directory based on the identification information. The candidate file to be merged is a file whose file type matches the type information in the file metadata request. The metadata of the candidate file to be merged can be returned to the client.

[0209] Because the file size is stored in the metadata of each file, the client can determine the size of each candidate file to be merged based on the metadata of the candidate file to be merged. If the size of the candidate file to be merged is less than a preset threshold, the candidate file to be merged is considered to be a small file to be merged. Otherwise, the candidate file to be merged is considered not to be a small file to be merged. The preset threshold can be set based on actual needs and is not limited here.

[0210] In one embodiment of the present disclosure, the method further comprises the following steps:

[0211] The HDFS storage system obtains an access request from the client for the merged file; the access request includes a file identifier of the merged file;

[0212] The HDFS storage system obtains metadata of the merged file based on the file identifier of the merged file;

[0213] The HDFS storage system reads the file content of the merged file from the data storage block based on the information of the data storage block in the metadata of the merged file, and returns the content to the client.

[0214] In this optional implementation, in addition to storing the storage block information corresponding to the file contents of each of the multiple small files to be merged, the merged metadata also includes the file identifier of the merged file corresponding to the merged metadata, and the mapping relationship between the file identifier of the merged file and the file identifiers of the multiple small files to be merged.

[0215] In addition, the storage block information stored in the merged metadata can also establish a mapping relationship between it and the corresponding small file to be merged. Through this mapping relationship, the storage block information where the file content of the small file to be merged is located can be determined based on the file identifier of the merged file and the file identifier of the small file to be merged.

[0216] The client can access data for the merged file. After sending a file access request to the HDFS storage system, the HDFS storage system retrieves the corresponding metadata from the file information management structure based on the file identifier of the merged file in the file access request, and reads the file content of the merged file from the data storage block information recorded in the metadata. It should be noted that the metadata records the information of all data storage blocks of multiple small files to be merged before they are merged, and based on the information of the data storage blocks, the file content corresponding to each small file to be merged can be read.

[0217] In one embodiment of the present disclosure, the method further comprises the following steps:

[0218] The HDFS storage system obtains a client's access request for a target small file in the small files to be merged; the access request includes a file identifier of the target small file;

[0219] The HDFS storage system determines whether the file identifier is associated with the file identifier of the merged file;

[0220] When the file identifier is associated with the file identifier of the merged file, the HDFS storage system obtains metadata of the merged file;

[0221] The HDFS storage system determines the target storage block information corresponding to the target small file based on the data storage block information in the metadata of the merged file;

[0222] The HDFS storage system reads the file content of the target small file from the target storage block based on the information of the target storage block and returns it to the client.

[0223] In this embodiment, as described above, in addition to storing the storage block information corresponding to the file contents of each of the multiple small files to be merged, the merged metadata also includes the file identifier of the merged file corresponding to the merged metadata, and the mapping relationship between the file identifier of the merged file and the file identifiers of the multiple small files to be merged.

[0224] Therefore, the client can also access the file content of the target small file based on the file identifier of one of the original multiple small files to be merged. After the HDFS storage system receives a file access request carrying the file identifier of the target small file, if the file identifier corresponding to the file identifier of the merged file is found in the file information management structure, the metadata of the merged file is obtained based on the file identifier of the merged file, the information of the data storage block corresponding to the file identifier of the target small file is matched from the metadata of the merged file, and the file content of the target small file is read from the corresponding data storage block based on the information of the data storage block, thereby returning the file content to the client.

[0225] Figure 3 The technical terms and technical features involved in the embodiments shown and related Figure 1-Figure 2 The technical terms and technical features mentioned in the embodiments shown and related are the same or similar. Figure 2 The explanation and description of the technical terms and technical features involved in the embodiments shown and related can refer to the above Figure 1-Figure 2 The explanations of the illustrated and related embodiments will not be repeated here.

[0226] Figure 4 FIG. 1 shows an overall flow chart of a file merging method according to an embodiment of the present disclosure. Figure 4 As shown, the user performs a small file merge operation through the client. After the client detects the small file merge operation, it requests the relevant directory files from the HDFS storage system. For example, it can be all directories that the user has permission to, or it can be a directory structure specified by the user. After the client receives the file directory returned by the HDFS storage system, it displays it to the user. The user can select one or more subdirectories from which the small file merge operation is to be performed, as well as the type information of the files to be merged. The client sends a metadata request to HDFS. HDFS returns the metadata of the candidate small files to be merged under the subdirectory selected by the user whose file types and type information match to the client. The client selects the small files to be merged that can be merged from these candidate small files to be merged, and then merges the metadata of these small files to be merged into the metadata of a merged file. The metadata of the merged file is sent to the HDFS storage system. The HDFS storage system writes the metadata of the merged file into the file information management structure, and deletes the metadata of the small files to be merged from the file information management structure, completing the merge operation of multiple small files to be merged.

[0227] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein.

[0228] Figure 5 The structure block diagram of the file merging device according to one embodiment of the present disclosure is shown. The device can be implemented as part or all of an electronic device through software, hardware or a combination of both. Figure 5 As shown, the file merging device includes:

[0229] The first receiving module 501 is configured to receive a small file merging request from a user; the small file merging request includes identification information of candidate files to be merged and type information of the candidate files to be merged;

[0230] An acquisition module 502 is configured to acquire metadata of multiple small files to be merged from an HDFS storage system based on the identification information and type information;

[0231] A merging module 503 is configured to merge metadata of the plurality of small files to be merged;

[0232] The sending module 504 is configured to send a file merge request carrying the merged metadata to the HDFS storage system, so that the HDFS storage system manages the multiple files to be merged using the merged metadata.

[0233] In one embodiment of the present disclosure, the file merging device may be adapted to be executed on a client.

[0234] Figure 6 FIG2 shows a structural block diagram of a file merging device according to another embodiment of the present disclosure, which can be implemented as part or all of an electronic device through software, hardware, or a combination of both. Figure 6 As shown, the file merging device includes:

[0235] The second receiving module 601 is configured to receive a file merging request from a client; the file merging request includes file information of multiple small files to be merged and merged metadata; the file contents of the multiple small files to be merged are stored in the data storage block;

[0236] A deletion module 602 is configured to delete metadata of the plurality of small files to be merged from a corresponding file information management structure;

[0237] The adding module 603 is configured to add the merged metadata as the metadata of a merged file to the file information management structure, and manage the data storage block where the file contents of the multiple small files to be merged are located in the form of a merged file; the merged metadata includes information on the data storage block where the file contents of the multiple small files to be merged are located.

[0238] In one embodiment of the present disclosure, the file merging device may be adapted to be executed on an HDFS storage system.

[0239] Figure 7 FIG1 shows a structural block diagram of a file merging system according to an embodiment of the present disclosure. The system can be implemented as part or all of an electronic device through software, hardware, or a combination of both. Figure 7 As shown, the file merging system includes: a client 701 and an HDFS storage system 702;

[0240] The client 701 receives a small file merge request from a user; the small file merge request includes identification information of candidate files to be merged and type information of the candidate files to be merged; the client 701 obtains metadata of multiple small files to be merged from the HDFS storage system 702 based on the identification information and type information; merges the metadata of the multiple small files to be merged, and sends a file merge request carrying the merged metadata to the HDFS storage system 702;

[0241] The HDFS storage system 702 receives a file merge request from the client 701; the file merge request includes file information of multiple small files to be merged and metadata after merging; the file contents of the multiple small files to be merged are stored in a data storage block; the HDFS storage system 702 deletes the metadata of the multiple small files to be merged from the corresponding file information management structure, and adds the merged metadata as metadata of a merged file to the file information management structure, and manages the data storage block where the file contents of the multiple small files to be merged are located in the form of a merged file; the merged metadata includes information of the data storage block where the file contents of the multiple small files to be merged are located.

[0242] In one embodiment of the present disclosure, the file merging system may be applicable to a process in which a user merges small files on an HDFS storage system through a client.

[0243] The technical features involved in the above-mentioned device embodiments and their corresponding explanations and descriptions are the same, corresponding or similar to the technical features involved in the method embodiments described above and their corresponding explanations and descriptions. For the technical features involved in the above-mentioned device embodiments and their corresponding explanations and descriptions, reference may be made to the technical features involved in the above-mentioned method embodiments and their corresponding explanations and descriptions, and this disclosure will not repeat them here.

[0244] The present disclosure also discloses an electronic device, which includes a memory and a processor; wherein:

[0245] The memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement any of the above method steps.

[0246] Figure 8 It is a structural diagram of a computer system suitable for implementing the file merging method according to an embodiment of the present disclosure.

[0247] like Figure 8As shown, the computer system 800 includes a processing unit 801, which can execute various processes in the above-mentioned embodiments according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage unit 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the computer system 800 are also stored in the RAM 803. The processing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0248] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed so that a computer program read therefrom can be installed into the storage section 808 as needed. Among them, the processing unit 801 can be implemented as a processing unit such as a CPU, a GPU, a TPU, an FPGA, an NPU, etc.

[0249] In particular, according to embodiments of the present disclosure, the methods described above can be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program tangibly embodied on a computer-readable medium, the computer program comprising program code for executing the data transmission method. In such embodiments, the computer program can be downloaded and installed from a network via the communication portion 809 and / or installed from removable media 811.

[0250] The embodiments of the present disclosure further disclose a computer program product, which includes a computer program / instruction, and implements any of the above method steps when the computer program / instruction is executed by a processor.

[0251] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the diagram or block diagram can represent a module, program segment or part of the code, and the module, program segment or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, as well as the combination of boxes in the block diagram and / or flow chart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0252] The units or modules involved in the embodiments described in this disclosure may be implemented in software or hardware. The units or modules described may also be provided in a processor, and the names of these units or modules do not, in certain circumstances, constitute limitations on the units or modules themselves.

[0253] As another aspect, embodiments of the present disclosure further provide a computer-readable storage medium. This computer-readable storage medium may be included in the apparatus described in the above embodiments, or may be a standalone computer-readable storage medium not incorporated into the apparatus. The computer-readable storage medium stores one or more programs, which are used by one or more processors to execute the methods described in the embodiments of the present disclosure.

[0254] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also encompass other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the inventive concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A file merging method, comprising: Receive small file merge requests from users; The small file merging request includes identification information of the candidate files to be merged and type information of the candidate files to be merged; Obtain metadata of multiple small files to be merged from the HDFS storage system based on the identification information and type information; Merging the metadata of the plurality of small files to be merged into metadata of a merged file, wherein the data storage block information in the merged metadata includes all the data storage block information in the metadata of the plurality of small files to be merged; The merged metadata includes an association relationship between the file identifier of the merged file and the file identifiers of the plurality of small files to be merged; the merged metadata includes a mapping relationship between the file identifiers of the plurality of small files to be merged and data storage block information; A file merge request carrying the merged metadata is sent to the HDFS storage system, so that the HDFS storage system uses the merged metadata to manage the multiple files to be merged; wherein the merged metadata is stored, the original metadata of the multiple small files to be merged are deleted, and the data blocks containing the file contents corresponding to the multiple small files to be merged remain intact without being merged.

2. The method according to claim 1, wherein Obtaining metadata of multiple small files to be merged from the HDFS storage system based on the identification information and type information, including: Obtain metadata of candidate files to be merged that meet the identification information and type information from the HDFS storage system; Determining, based on the metadata of the candidate files to be merged, the candidate files to be merged whose storage size is smaller than a preset threshold as the small files to be merged; Obtain metadata of the multiple small files to be merged.

3. The method according to claim 1 or 2, wherein: The identification information includes the storage directory of the candidate file to be merged; and / or the type information includes the file type of the independent content corresponding to the storage block in the HDFS storage system.

4. The method according to claim 1 or 2, wherein: Receive small file merge requests from users, including: In response to a user's small file merging operation, outputting file directory information to which the user has authority; Based on the user's selection operation on the file directory information, the small file merging request is triggered; wherein, in the small file merging request, the file directory information operated by the user is used as the identification information of the candidate file to be merged, and the file type selected by the user is used as the type information.

5. A file merging method, wherein: include: Receive file merge requests from clients; The file merging request includes file information of multiple small files to be merged and metadata after merging; The file contents of the multiple small files to be merged are stored in a data storage block; wherein the merged metadata is obtained by merging the metadata of the multiple small files to be merged into the metadata of a merged file, and the data storage block information in the merged metadata includes all the data storage block information in the metadata of the multiple small files to be merged; the merged metadata includes the association relationship between the file identifier of the merged file and the file identifiers of the multiple small files to be merged; the merged metadata includes the mapping relationship between the file identifiers of the multiple small files to be merged and the data storage block information; Deleting metadata of the plurality of small files to be merged from the corresponding file information management structure; The merged metadata is added to the file information management structure as the metadata of a merged file, and the data storage block where the file contents of the multiple small files to be merged are located is managed in the form of a merged file; wherein the merged metadata is stored, the original metadata of the multiple small files to be merged are deleted, and the data blocks where the file contents corresponding to the multiple small files to be merged are located remain the same without being merged.

6. The method according to claim 5, wherein: Also includes: Receiving a file metadata request from the client; The file metadata request includes identification information of the candidate file to be merged and type information of the candidate file to be merged; Based on the file directory corresponding to the identification information, metadata of the candidate files to be merged that meet the type information are returned to the client.

7. The method according to claim 5 or 6, wherein: Also includes: Obtaining a client's access request to the merged file; The access request includes the file identifier of the merged file; Acquire metadata of the merged file based on the file identifier of the merged file; Based on the information of the data storage block in the metadata of the merged file, the file content of the merged file is read from the data storage block and returned to the client.

8. The method according to claim 5 or 6, wherein: Also includes: Obtaining a client's access request to a target small file in the small files to be merged; The access request includes the file identifier of the target small file; Determining whether the file identifier is associated with a file identifier of a merged file; When the file identifier is associated with the file identifier of a merged file, obtaining metadata of the merged file; Determine the target storage block information corresponding to the target small file based on the data storage block information in the metadata of the merged file; The file content of the target small file is read from the target storage block based on the information of the target storage block and returned to the client.

9. A method for merging files, wherein: include: The client receives small file merge requests from users; The small file merging request includes identification information of the candidate files to be merged and type information of the candidate files to be merged; The client obtains metadata of multiple small files to be merged from the HDFS storage system based on the identification information and type information; The client merges the metadata of the multiple small files to be merged into metadata of a merged file, and the data storage block information in the merged metadata includes all data storage block information in the metadata of the multiple small files to be merged; The merged metadata includes an association relationship between the file identifier of the merged file and the file identifiers of the multiple small files to be merged; the merged metadata includes a mapping relationship between the file identifiers of the multiple small files to be merged and data storage block information, and a file merge request carrying the merged metadata is sent to the HDFS storage system; The HDFS storage system receives a file merge request from the client; The file merging request includes file information of multiple small files to be merged and metadata after merging; The file contents of the plurality of small files to be merged are stored in the data storage block; The HDFS storage system deletes the metadata of the multiple small files to be merged from the corresponding file information management structure, and adds the merged metadata as the metadata of a merged file to the file information management structure, and manages the data storage blocks where the file contents of the multiple small files to be merged are located in the form of a merged file; the merged metadata includes information about the data storage blocks where the file contents of the multiple small files to be merged are located; wherein the merged metadata is stored, the original metadata of the multiple small files to be merged are deleted, and the data blocks where the file contents corresponding to the multiple small files to be merged are located remain as they are without being merged.

10. The method according to claim 9, wherein: The client obtains metadata of multiple small files to be merged from the HDFS storage system based on the identification information and type information, including: The client obtains metadata of the candidate files to be merged that meet the identification information and type information from the HDFS storage system; The client determines, based on the metadata of the candidate files to be merged, the candidate files to be merged whose storage size is smaller than a preset threshold as the small files to be merged; The client obtains metadata of the multiple small files to be merged.

11. The method according to claim 9 or 10, wherein: The client receives small file merge requests from users, including: The client outputs file directory information to which the user has authority in response to the user's small file merging operation; The client triggers the small file merging request based on the user's selection operation on the file directory information; wherein, in the small file merging request, the file directory information operated by the user is used as the identification information of the candidate file to be merged, and the file type selected by the user is used as the type information.

12. The method according to claim 9 or 10, wherein: Also includes: The HDFS storage system receives the file metadata request from the client; The file metadata request includes identification information of the candidate file to be merged and type information of the candidate file to be merged; The HDFS storage system returns metadata of candidate files to be merged that meet the type information under the file directory corresponding to the identification information to the client.

13. The method according to claim 9 or 10, wherein: Also includes: The HDFS storage system obtains a client's access request to the merged file; The access request includes the file identifier of the merged file; The HDFS storage system obtains metadata of the merged file based on the file identifier of the merged file; The HDFS storage system reads the file content of the merged file from the data storage block based on the information of the data storage block in the metadata of the merged file, and returns the content to the client.

14. The method according to claim 9 or 10, wherein: Also includes: The HDFS storage system obtains a client's access request for the target small file in the small files to be merged; The access request includes the file identifier of the target small file; The HDFS storage system determines whether the file identifier is associated with the file identifier of the merged file; When the file identifier is associated with the file identifier of the merged file, the HDFS storage system obtains metadata of the merged file; The HDFS storage system determines the target storage block information corresponding to the target small file based on the data storage block information in the metadata of the merged file; The HDFS storage system reads the file content of the target small file from the target storage block based on the information of the target storage block and returns it to the client.

15. A file merging device, comprising: A first receiving module is configured to receive a small file merging request from a user; The small file merging request includes identification information of the candidate files to be merged and type information of the candidate files to be merged; An acquisition module is configured to acquire metadata of a plurality of small files to be merged from an HDFS storage system based on the identification information and the type information; a merging module configured to merge the metadata of the plurality of small files to be merged into metadata of a merged file, wherein the data storage block information in the merged metadata includes all the data storage block information in the metadata of the plurality of small files to be merged; The merged metadata includes an association relationship between the file identifier of the merged file and the file identifiers of the plurality of small files to be merged; the merged metadata includes a mapping relationship between the file identifiers of the plurality of small files to be merged and data storage block information; The sending module is configured to send a file merge request carrying the merged metadata to the HDFS storage system, so that the HDFS storage system can use the merged metadata to manage the multiple files to be merged; wherein the merged metadata is stored, the original metadata of the multiple small files to be merged are deleted, and the data blocks containing the file contents corresponding to the multiple small files to be merged remain intact without being merged.

16. A file merging device, comprising: A second receiving module is configured to receive a file merging request from a client; The file merging request includes file information of multiple small files to be merged and metadata after merging; The file contents of the multiple small files to be merged are stored in a data storage block; wherein the merged metadata is obtained by merging the metadata of the multiple small files to be merged into the metadata of a merged file, and the data storage block information in the merged metadata includes all the data storage block information in the metadata of the multiple small files to be merged; the merged metadata includes the association relationship between the file identifier of the merged file and the file identifiers of the multiple small files to be merged; the merged metadata includes the mapping relationship between the file identifiers of the multiple small files to be merged and the data storage block information; a deletion module configured to delete the metadata of the plurality of small files to be merged from the corresponding file information management structure; An adding module is configured to add the merged metadata as the metadata of a merged file to the file information management structure, and manage the data storage blocks where the file contents of the multiple small files to be merged are located in the form of a merged file; the merged metadata includes information on the data storage blocks where the file contents of the multiple small files to be merged are located; wherein the merged metadata is stored, the original metadata of the multiple small files to be merged are deleted, and the data blocks where the file contents corresponding to the multiple small files to be merged are located remain as they are without being merged.

17. A file merging system comprising: Client and HDFS storage system; among them, The client receives a small file merge request from a user; the small file merge request includes identification information of a candidate file to be merged and type information of the candidate file to be merged; the client obtains metadata of multiple small files to be merged from the HDFS storage system based on the identification information and type information; and merges the metadata of the multiple small files to be merged into metadata of a merged file, and the data storage block information in the merged metadata includes all data storage block information in the metadata of the multiple small files to be merged; the merged metadata includes the association relationship between the file identifier of the merged file and the file identifiers of the multiple small files to be merged; the merged metadata includes the mapping relationship between the file identifiers of the multiple small files to be merged and the data storage block information, and sends the file merge request carrying the merged metadata to the HDFS storage system; The HDFS storage system receives a file merge request from a client; the file merge request includes file information of multiple small files to be merged and metadata after merging; the file contents of the multiple small files to be merged are stored in a data storage block; the HDFS storage system deletes the metadata of the multiple small files to be merged from the corresponding file information management structure, and adds the merged metadata as the metadata of a merged file to the file information management structure, and manages the data storage block where the file contents of the multiple small files to be merged are located in the form of a merged file; the merged metadata includes information of the data storage block where the file contents of the multiple small files to be merged are located; wherein the merged metadata is stored, the original metadata of the multiple small files to be merged are deleted, and the data blocks where the file contents corresponding to the multiple small files to be merged are located remain as they are without being merged.

18. An electronic device comprising a memory and a processor; wherein: The memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the steps of the method according to any one of claims 1 to 14.

19. A computer-readable storage medium having computer instructions stored thereon, wherein: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 14 are implemented.

20. A computer program product comprising a computer program / instruction, which implements the steps of the method according to any one of claims 1 to 14 when executed by a processor.

Images