An optimization method and terminal for reducing metadata quantity

Abstract

The application provides an optimization method and terminal for reducing the amount of metadata, comprising writing each to-be-stored file into a local file directory; obtaining a specified hdfs partition to which the to-be-stored file will be written in advance, and calculating the average file size of the specified hdfs partition; merging the to-be-stored files with memory occupation less than the average file size into at least one large file with total memory occupation greater than or equal to the average file size; and uploading the large file and the to-be-stored file with memory occupation greater than or equal to the average file size to the specified hdfs partition in sequence. The application can avoid the problem of unavailability of part of data in the file merging scenario, effectively reduce the amount of metadata, and further reduce the memory resource occupation of the distributed file system, thereby improving the system efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet, in particular to an optimization method for reducing the number of metadata and a terminal. BACKGROUND

[0002] In the current Internet system, in order to support the mass storage of data, a distributed deployment mode is usually adopted to support the storage of files. Among them, hdfs is the most common distributed file storage format in the big data scenario. However, usually there are many big data tasks, and many small files to be stored will be generated in the process of task execution. These small files to be stored will be stored in hdfs by generating one or more metadata information first. In order to realize the fast query of files, the metadata information is usually cached in the memory of the distributed file system.

[0003] However, with the increase of the number of tasks, the number of small files to be stored also increases sharply, which leads to the increase of the storage information of metadata, and the memory occupied by the distributed file system becomes larger and larger. Finally, the memory of the distributed file system will need to be expanded constantly.

[0004] In order to solve the above problems, the existing mode usually merges the small files to be stored through a timing task after the task is executed and the small files to be stored are written into hdfs. However, in this way, the external cannot access these small files to be stored in the process of merging, so this method is not applicable in the case of high availability. SUMMARY

[0005] The technical problem to be solved by the present application is to provide an optimization method for reducing the number of metadata and a terminal, which can avoid the problem of unavailability of part of data in the file merging scenario, effectively reduce the number of metadata, and thus reduce the memory resource occupation of the distributed file system, and improve the system efficiency.

[0006] In order to solve the above technical problems, the technical scheme adopted by the present application is:

[0007] An optimization method for reducing the number of metadata, comprising the steps of:

[0008] S1, writing each file to be stored into a local file directory;

[0009] S2, pre-acquiring a specified hdfs partition to which the file to be stored will be written, and calculating the average file size of the specified hdfs partition;

[0010] S3, merging the file to be stored with a memory occupation less than the average file size into at least one large file with a total memory occupation greater than or equal to the average file size.

[0011] S4. Upload the large file and the file to be stored that occupies more or less memory than the average file size to the designated HDFS partition in sequence.

[0012] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is as follows:

[0013] An optimized terminal for reducing metadata includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps:

[0014] S1. Write each file to be stored to the local file directory;

[0015] S2. Pre-obtain the specified HDFS partition where the file to be stored will be written, and calculate the average file size of the specified HDFS partition;

[0016] S3. Merge the files to be stored that occupy less memory than the average file size into at least one large file whose total memory occupation is greater than or equal to the average file size;

[0017] S4. Upload the large file and the file to be stored that occupies more or less memory than the average file size to the designated HDFS partition in sequence.

[0018] The beneficial effects of this invention are as follows: It provides an optimized method and terminal for reducing the amount of metadata. After the task is completed, instead of prioritizing writing to HDFS files, it first writes to a separate local file directory. Simultaneously, based on the pre-obtained specified HDFS partition where the files generated by the task will be written, and according to the average file size within the specified HDFS partition, files in the local file directory smaller than the average file size are merged into a larger file. Then, the larger file and the files in the local file directory larger than or equal to the average file size are uploaded together to the specified HDFS partition. In this way, the problem of some data being unavailable in file merging scenarios can be avoided. At the same time, since the number of merged files is reduced, the amount of metadata generated when storing in HDFS can be effectively reduced, thereby reducing the memory resource consumption of the distributed file system and improving system efficiency. Attached Figure Description

[0019] Figure 1 This is a flowchart of an optimization method for reducing the amount of metadata according to an embodiment of the present invention;

[0020] Figure 2 This is a schematic diagram of the structure of an optimized terminal for reducing the amount of metadata according to an embodiment of the present invention.

[0021] Explanation of reference numerals

[0022] 1. An optimization terminal for reducing the amount of metadata; 2. a memory; 3. a processor. DETAILED DESCRIPTION

[0023] To make the technical contents, purposes and effects of the present application clear, the following will be described in combination with embodiments and the accompanying drawings.

[0024] Please refer to Figure 1 An optimization method for reducing the amount of metadata, comprising the steps of:

[0025] S1, writing each to-be-stored file into a local file directory;

[0026] S2, pre-acquiring a specified hdfs partition to which the to-be-stored file will be written, and calculating the average file size of the specified hdfs partition;

[0027] S3, merging the to-be-stored files with memory occupancy less than the average file size into at least one large file with total memory occupancy greater than or equal to the average file size;

[0028] S4, sequentially uploading the large file and the to-be-stored file with memory occupancy greater than or equal to the average file size to the specified hdfs partition.

[0029] From the above description, the beneficial effects of the present application are as follows: an optimization method for reducing the amount of metadata is provided, when the task is completed, the hdfs file is no longer written preferentially, but is written into an independent local file directory first, at the same time, the specified hdfs partition to which the file generated by the pre-acquired task will be written is acquired, and according to the average file size in the specified hdfs partition, the to-be-stored files smaller than the average file size in the local file directory are merged into a large file, and then the large file and the to-be-stored files greater than or equal to the average file size in the local file directory are uploaded to the specified hdfs partition, in this way, the problem of unavailable data in the file merging scenario can be avoided, at the same time, due to the reduced number of merged files, the amount of metadata generated when stored in the hdfs can be effectively reduced, thereby reducing the memory resource occupation of the distributed file system and improving the system efficiency.

[0030] Further, the step S2 is specifically:

[0031] S21, periodically counting the average file size in each hdsf partition;

[0032] S22, the center node of the distributed system calculates the md5 value of the to-be-stored file in advance through a hash algorithm, and determines the specified hdsf file partition to which the to-be-stored file is written according to the md5 value result;

[0033] S23, the average file size of the specified hdsf partition is obtained.

[0034] As can be seen from the above description, the average file size in each hdfs partition is calculated periodically in advance, so that the average file size of the specified hdfs partition can be quickly called when the to-be-stored file is compared with the average file size in the subsequent process, and the comparison efficiency is improved.

[0035] Further, the step S21 further includes:

[0036] The files in each hdsf partition whose occupied memory is less than or equal to the preset size are excluded from the calculation of the average file size.

[0037] As can be seen from the above description, the files smaller than the preset size, for example, 10K, are excluded from the calculation of the average file size of the corresponding hdsf partition, so as to prevent the average file size from being too small due to too many small files, that is, the calculation method of removing the extreme value.

[0038] Further, the step S3 further includes:

[0039] When the number of to-be-stored files whose occupied memory is less than the average file size remaining in the local file directory is less than the preset number, the remaining to-be-stored files are directly merged into a random large file.

[0040] As can be seen from the above description, when the to-be-stored files whose occupied memory is less than the average file size are successively merged into one large file, and the remaining to-be-stored files are less than the preset number, that is, only one or two to-be-stored files whose occupied memory is less than the average file size remain, the to-be-stored files of the one or two to-be-stored files are not particularly large after merging, and therefore, they only need to be merged into a random large file, without wasting time and resources to further merge the one or two to-be-stored files, thereby further reducing the amount of subsequent metadata information and reducing the memory resource occupation of the distributed file system.

[0041] Further, the step S4 further includes:

[0042] S5, the specified hdfs partition generates metadata information with a number equal to the sum of the large file and the number of to-be-stored files whose occupied memory is greater than or equal to the average file size, and stores the uploaded large file and to-be-stored file whose occupied memory is greater than or equal to the average file size.

[0043] From the above description, it can be seen that the file written into the hdfs partition needs to be stored through the corresponding metadata information, that is, the generation amount of the metadata information is actually related to the file amount written, and the to-be-stored files with the memory occupation smaller than the average file size are combined into large files, so that the file amount written into the hdfs partition can be greatly reduced, and the metadata information amount is reduced, thereby effectively reducing the memory resource occupation of the distributed file system and improving the system efficiency.

[0044] Please refer to Figure 2 An optimization terminal for reducing the metadata amount, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the following steps when executing the computer program:

[0045] S1, write each to-be-stored file into a local file directory;

[0046] S2, pre-acquire a specified hdfs partition to which the to-be-stored file will be written, and calculate the average file size of the specified hdfs partition;

[0047] S3, combine the to-be-stored files with the memory occupation smaller than the average file size into at least one large file with the total memory occupation greater than or equal to the average file size;

[0048] S4, sequentially upload the large file and the to-be-stored file with the memory occupation greater than or equal to the average file size to the specified hdfs partition.

[0049] From the above description, the beneficial effects of the present application are that: based on the same technical concept, cooperating with the above-mentioned optimization method for reducing the metadata amount, an optimization terminal for reducing the metadata amount is provided, when the task is executed, the hdfs file is no longer written preferentially, but is written into an independent local file directory, at the same time, according to the specified hdfs partition to which the file generated by the pre-acquired task will be written, and according to the average file size in the specified hdfs, the to-be-stored files smaller than the average file size in the local file directory are combined into large files, and then the large files and the to-be-stored files greater than or equal to the average file size in the local file directory are uploaded to the specified hdfs partition, in this way, the problem of unavailable data in the file combination scenario can be avoided, and since the number of combined files is reduced, the amount of metadata generated when stored in the hdfs can be effectively reduced, thereby reducing the memory resource occupation of the distributed file system and improving the system efficiency.

[0050] Further, the step S2 is specifically:

[0051] S21, periodically calculate the average file size in each hdsf partition;

[0052] S22, the center node of the distributed system calculates the md5 value of the to-be-stored file in advance by a hash algorithm, and judges the specified hdsf file partition to which the to-be-stored file is written according to the md5 value result;

[0053] S23, the average file size of the specified hdsf partition is obtained.

[0054] From the above description, it can be known that the average file size in each hdfs partition is calculated periodically in advance, so that the average file size of the specified hdfs partition can be quickly called when the to-be-stored file is compared with the average file size in the subsequent process, and the comparison efficiency is improved.

[0055] Further, the step S21 further includes:

[0056] The files in each hdsf partition whose occupied memory is less than or equal to the preset size are excluded from the calculation of the average file size.

[0057] From the above description, it can be known that the files smaller than the preset size, for example, 10K, are excluded from the calculation of the average file size of the corresponding hdsf partition, so as to prevent the average file size from being too small due to too many small files, that is, the calculation method of removing the extreme value.

[0058] Further, the step S3 further includes:

[0059] When the number of to-be-stored files whose occupied memory is less than the average file size remaining in the local file directory is less than the preset number, the remaining to-be-stored files are directly merged into a random large file.

[0060] From the above description, it can be known that when the to-be-stored files whose occupied memory is less than the average file size are successively merged into one by one, the to-be-stored files remaining less than the preset number, for example, only one or two to-be-stored files whose occupied memory is less than the average file size, the to-be-stored files of the one or two to-be-stored files are not particularly large after merging, and therefore, they only need to be merged into a random large file, without wasting time and resources to further merge the one or two to-be-stored files separately, thereby further reducing the amount of subsequent metadata information and reducing the memory resource occupation of the distributed file system.

[0061] Further, the step S4 further includes:

[0062] S5, the designated hdfs partition generates metadata information same as the sum of the number of the large files and the number of the to-be-stored files occupying memory greater than or equal to the average file, and stores the uploaded large files and the to-be-stored files occupying memory greater than or equal to the average file size.

[0063] From the above description, it can be seen that the file written to the hdfs partition needs to be stored through the corresponding metadata information, that is, the generation amount of the metadata information is actually related to the amount of the written file. After the to-be-stored files occupying memory less than the average file size are merged into large files, the number of the files written to the hdfs partition can be greatly reduced, and the amount of the metadata information is reduced, thereby effectively reducing the memory resource occupation of the distributed file system and improving the system efficiency.

[0064] The application provides an optimization method and a terminal for reducing the amount of metadata, which are mainly applied to the data storage scene of an Internet distributed file system, and will be specifically described below in combination with specific embodiments.

[0065] Please refer to Figure 1 , the embodiment one of the application is:

[0066] In the data storage scene of the existing distributed file system, when a task is executed, it is assumed that 10 files are generated, of which 4 files occupy memory of 500K, and the other 6 files occupy memory of 100K. The 10 files will be directly written into the hdfs partition, and 10 pieces of metadata information will be generated at the same time.

[0067] The optimization method for reducing the amount of metadata in the embodiment, as shown in Figure 1 , comprises the following steps:

[0068] S1, write each to-be-stored file generated after the current task is executed into a local file directory.

[0069] That is, instead of being directly written into the hdfs partition, the to-be-stored files are preferentially stored in the local file directory.

[0070] S2, preacquire a designated hdfs partition to which the to-be-stored files will be written, and calculate the average file size of the designated hdfs partition.

[0071] S3, merge the to-be-stored files occupying memory less than the average file size into at least one large file occupying total memory greater than or equal to the average file size.

[0072] For example, if the average file size of the designated hdfs partition is calculated to be 300K, then 6 files of 100K each will be merged into 2 files of 300K each, and the remaining 4 files of 500K each will not need to be merged because their memory usage is greater than the average file size of 300K.

[0073] S4, sequentially upload the large files and the to-be-stored files with memory usage greater than or equal to the average file size to the designated hdfs partition.

[0074] That is, sequentially upload the 2 files of 300K each and the 4 files of 500K each to the designated hdfs partition for storage, and at this time, only 6 pieces of metadata information are generated, which is greatly reduced compared to 10 pieces of metadata information in the existing traditional method.

[0075] That is, in the present embodiment, after the task is executed, the hdfs file is no longer written preferentially, but is written to an independent local file directory first, and then according to the designated hdfs partition to which the file generated by the task is to be written, and according to the average file size in the designated hdfs partition, the to-be-stored files smaller than the average file size in the local file directory are merged into large files, and then the large files and the to-be-stored files greater than or equal to the average file size in the local file directory are uploaded to the designated hdfs partition. By this way, the problem of unavailability of part of the data in the file merging scenario can be avoided, and since the number of files after merging is reduced, the number of metadata generated when storing in the hdfs can be effectively reduced, thereby reducing the memory resource occupation of the distributed file system and improving the system efficiency.

[0076] Embodiment two of the present application is:

[0077] An optimization method for reducing the number of metadata, based on the above embodiment one, in the present embodiment, step S2 is specifically:

[0078] S21, periodically calculate the average file size in each hdsf partition, and exclude the files with memory usage less than or equal to a preset size from the calculation of the average file size.

[0079] In the present embodiment, the preset size can be limited to 10K, and since 10K is very small and can be ignored, the files with memory usage less than or equal to 10K in the originally stored files in each partition are excluded from the calculation of the average file size, so as to prevent the average file size from being too small due to too many small files, that is, the calculation method of removing the extreme value. In other equivalent embodiments, the specific value of the preset size can be limited according to actual needs.

[0080] S22, the center node of the distributed system calculates the md5 value of the to-be-stored file in advance through a hash algorithm, and judges the specified hdfs file partition to which the to-be-stored file is written according to the md5 value result

[0081] S23, the average file size of the specified hdfs partition is obtained.

[0082] That is, in the embodiment, the average file size in each hdfs partition is calculated periodically in advance, so that the average file size of the specified hdfs partition can be quickly called when the to-be-stored file is compared with the average file size in the subsequent step, and the comparison efficiency is effectively improved.

[0083] Embodiment three of the application is:

[0084] An optimization method for reducing the number of metadata, based on the above-mentioned embodiment one or embodiment two, in the embodiment, step S3 further comprises:

[0085] When the remaining memory occupation in the local file directory is less than the average file size, and the number of to-be-stored files is less than the preset number, the remaining number of to-be-stored files is directly merged into a random large file.

[0086] For example, it is assumed that 10 to-be-stored files are generated after the current task is completed, of which 5 to-be-stored files occupy 500K of memory, and the other 5 to-be-stored files occupy 100K of memory, and the average file size of the specified hdfs partition is calculated to be 200K, then according to the above-mentioned merging step, the 5 to-be-stored files of 100K are merged into two large files of 200K of memory each, and the remaining one to-be-stored file of 100K can be randomly merged into one of the 200K large files to obtain a 300K large file, and the subsequent step S4 is to sequentially write the 5 to-be-stored files of 500K, one 200K large file and one 300K large file into the specified hdfs partition for storage, and the generated metadata information is 7, which is one less than the metadata information of 8 generated by directly writing the remaining 100K to-be-stored file into the specified hdfs partition.

[0087] That is, in the embodiment, when the to-be-stored files with memory occupation less than the average file size are successively merged into one large file, and the remaining to-be-stored files with memory occupation less than the average file size are only one or two, the to-be-stored files of the one or two to-be-stored files are not particularly large, and therefore only need to be merged into a random large file, without wasting time and resources to further merge the one or two to-be-stored files, thereby further reducing the amount of subsequent metadata information and reducing the memory resource occupation of the distributed file system.

[0088] Meanwhile, in the embodiment, step S4 is followed by:

[0089] S5, the hdfs partition is specified to generate metadata information same as the number of large files and the number of to-be-stored files occupying memory greater than or equal to the average file size, and the large files and the to-be-stored files occupying memory greater than or equal to the average file size are stored.

[0090] That is, after the file is written into the hdfs partition, the corresponding metadata information needs to be stored, that is, the amount of generated metadata information is actually related to the amount of written files. After the to-be-stored files occupying memory less than the average file size are combined into large files, the number of files written into the hdfs partition can be greatly reduced, and the amount of metadata information can be reduced, thereby effectively reducing the memory resource occupation of the distributed file system and improving the system efficiency.

[0091] It is worth noting that the reason why not all to-be-stored files occupying memory less than the average file size are combined into one large file is that the file cannot be too large, and too large will cause the writing and reading time of the entire file to increase, and for the same reason, the to-be-stored file cannot be too small, and too small will cause the to-be-stored file to be too many and the amount of generated metadata information to increase.

[0092] In addition, in the embodiment, after the client completes the combination of the to-be-stored files into large files and writes them into the specified hdfs partition, the generated metadata information is the metadata information corresponding to the combined large files, and is directly written into the metadata memory of the central node of the distributed file system. When needed, the corresponding large file can be directly obtained by reading the metadata in the memory, and the file query is realized.

[0093] Please refer to Figure 2 Embodiment four of the present application is:

[0094] An optimization terminal 1 for reducing the amount of metadata, comprising a memory 2, a processor 3, and a computer program stored on the memory 2 and executable on the processor 3, wherein the processor 3 executes the computer program to complete the steps of the optimization method for reducing the amount of metadata in any of embodiments one to three.

[0095] In summary, the optimization method and terminal for reducing the amount of metadata provided by the present application can avoid the problem of unavailable data in the file combination scenario, effectively reduce the amount of metadata, and further reduce the memory resource occupation of the distributed file system and improve the system efficiency.

[0096] The above description is only an embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent transformation or direct or indirect application in related technical fields based on the content of the specification and drawings is also included in the patent protection scope of the present application.

Claims

1. An optimization method for reducing the amount of metadata, characterized in that, The method comprises the steps of: S1, writing each to-be-stored file into a local file directory; S2, pre-acquiring a specified hdfs partition to which the to-be-stored file is to be written, and calculating an average file size of the specified hdfs partition; S3, merging the to-be-stored files with memory occupancy less than the average file size into at least one large file with total memory occupancy greater than or equal to the average file size; S4, sequentially uploading the large file and the to-be-stored files with memory occupancy greater than or equal to the average file size to the specified hdfs partition; The step S2 is specifically: S21, periodically counting the average file size in each hdfs partition; S22, pre-calculating the md5 value of the to-be-stored file by a hash algorithm through a central node of a distributed system, and judging the specified hdfs partition to which the to-be-stored file is written according to the md5 value result; S23, acquiring the average file size of the specified hdfs partition; The step S3 further comprises: When the number of the to-be-stored files with memory occupancy less than the average file size remaining in the local file directory is less than a preset number, directly merging the remaining to-be-stored files into a random large file; The step S4 further comprises: S5, the specified hdfs partition generates metadata information with a number same as the sum of the large file and the number of the to-be-stored files with memory occupancy greater than or equal to the average file, and stores the uploaded large file and the to-be-stored files with memory occupancy greater than or equal to the average file size.

2. The optimization method of claim 1, wherein, The step S21 further comprises: Excluding the files with memory occupancy less than or equal to a preset size in each hdfs partition from the calculation of the average file size.

3. An optimization terminal for reducing the amount of metadata, characterized by A computer program product, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: S1, writing each to-be-stored file into a local file directory; S2, pre-acquiring a specified hdfs partition to which the to-be-stored file is to be written, and calculating an average file size of the specified hdfs partition; S3, merging the to-be-stored files with memory occupancy less than the average file size into at least one large file with total memory occupancy greater than or equal to the average file size; S4, sequentially uploading the large file and the to-be-stored files with memory occupancy greater than or equal to the average file size to the specified hdfs partition; The step S2 is specifically: S21, periodically counting the average file size in each hdfs partition; S22, pre-calculating the md5 value of the to-be-stored file by a hash algorithm through a central node of a distributed system, and judging the specified hdfs partition to which the to-be-stored file is written according to the md5 value result; S23, acquiring the average file size of the specified hdfs partition; The step S3 further comprises: When the remaining occupied memory in the local file directory is less than the preset number, directly merge the remaining number of the to-be-stored files into a random one of the large files; The step S4 further includes: S5, the designated hdfs partition generates metadata information same as the sum of the number of the large files and the number of the to-be-stored files whose occupied memory is greater than or equal to the average file size, and stores the uploaded large files and the to-be-stored files whose occupied memory is greater than or equal to the average file size.

4. The optimization terminal for reducing the amount of metadata according to claim 3, characterized by, The step S21 further includes: The files whose occupied memory is less than or equal to the preset size in each hdfs partition are excluded from the calculation of the average file size.

Images