File processing method and device based on distributed file system

Abstract

The present disclosure relates to a file processing method and device based on a distributed file system, and a computer storable medium, and relates to the field of computer technology. The file processing method based on a distributed file system includes: selecting multiple candidate file directories from multiple file directories in the distributed file system; determining the unevenness rate between multiple business files included in each candidate file directory, wherein the unevenness rate represents the degree of discreteness between the file sizes of the multiple business files; and performing uniform processing on at least one candidate file directory among the multiple candidate file directories according to the unevenness rate corresponding to the multiple candidate file directories, so that the multiple business files included in the at least one candidate file directory are evenly stored in each data block of the distributed file system.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a file processing method and device based on a distributed file system, and a computer storable medium. Background Art

[0002] As the business of an enterprise continues to develop, the amount of data is also growing. The data generated by each business unit corresponding to each business is usually extracted or processed into the data warehouse through different business channels. The business files of the data warehouse are mainly stored in the distributed file system. Due to the differences in the forms of the businesses, the file sizes of the businesses stored in the data warehouse will be uneven, which often leads to a decrease in the storage performance of the distributed file system and causes the skew problem in distributed computing.

[0003] In the related art, for each file directory in a distributed file system, when the average file size of multiple business files included in the file directory is smaller than the block size of a data block, the block size is used as the file size to optimize the multiple business files, so that the split parts of the business files with a file size larger than the block size are merged into the business files with a file size smaller than the block size. That is, by reducing the number of files and blocks by merging small files, the unevenness of the file directory is reduced. Summary of the invention

[0004] In the related art, by calculating the average file sizes of multiple business files in a file directory and comparing the average file size with the block size of the data block, it is impossible to identify all file directories with uneven file sizes of business files, which has certain limitations.

[0005] In response to the above technical problems, the present disclosure proposes a solution that can accurately identify uneven file directories, reduce the probability of data skew problems in distributed file systems caused by uneven problems, and improve the storage performance of distributed file systems.

[0006] According to a first aspect of the present disclosure, a file processing method based on a distributed file system is provided, comprising: selecting multiple candidate file directories from multiple file directories in the distributed file system; determining an unevenness rate between multiple business files included in each candidate file directory, wherein the unevenness rate represents the degree of discreteness between file sizes of the multiple business files; and performing uniformity processing on at least one candidate file directory among the multiple candidate file directories according to the unevenness rate corresponding to the multiple candidate file directories, so that the multiple business files included in the at least one candidate file directory are uniformly stored in each data block of the distributed file system.

[0007] In some embodiments, determining the unevenness rate between multiple business files included in each candidate file directory includes: calculating the relative average deviation of the file sizes of the multiple business files included in each candidate file directory as the unevenness rate corresponding to each candidate file directory.

[0008] In some embodiments, calculating the relative average deviation of the file sizes of multiple business files included in each candidate file directory includes: calculating the average file size of the multiple business files as the total average; determining that the business file among the multiple business files whose file size is smaller than the total average is the first business file; determining that the business file among the multiple business files whose file size is greater than or equal to the total average is the second business file; respectively determining the number of the first business files and the number of the second business files; calculating the average file size of the first business file and the average file size of the second business file as the first mean and the second mean, respectively; calculating the relative average deviation based on the total average, the first mean, the second mean, the number of the first business files and the number of the second business files.

[0009] In some embodiments, performing uniform processing on at least one of the multiple candidate file directories based on the non-uniform rates corresponding to the multiple candidate file directories includes: sorting the non-uniform rates corresponding to the multiple candidate file directories; and performing uniform processing on each of the top M candidate file directories with the largest non-uniform rates, where M is a positive integer.

[0010] In some embodiments, performing homogenization processing on at least one of the multiple candidate file directories according to the non-uniformity rates corresponding to the multiple candidate file directories includes: performing homogenization processing on each candidate file directory having a non-uniformity rate greater than a non-uniformity rate threshold.

[0011] In some embodiments, selecting multiple candidate file directories includes: selecting one or more candidate business units from multiple business units served by the distributed file system; determining the sum of the number of business files generated by each candidate business unit and the number of data blocks used to store the business files generated by each candidate business unit as a first value; selecting the first N business units with the largest first value from the one or more candidate business units as target business units, where N is a positive integer; for each target business unit, selecting multiple file directories corresponding to each target business unit as the multiple candidate file directories.

[0012] In some embodiments, the distributed file system includes multiple management nodes, and selecting one or more candidate business units includes: determining the sum of the number of business files maintained by each management node and the number of data blocks used to store the business files maintained by each management node as a second value; determining the sum of the maximum number of business files that can be maintained by each management node and the number of data blocks used to store the maximum number of business files as a third value; selecting one or more management nodes whose ratio of the second value to the third value is greater than a ratio threshold as one or more target management nodes; and selecting one or more business units corresponding to the one or more target management nodes as the one or more candidate business units.

[0013] In some embodiments, determining the sum of the number of business files maintained by each management node and the number of data blocks used to store the business files maintained by each management node as the second value includes: counting the number of business files maintained by each management node based on the correspondence between the file identifier of the business file and the node identifier of the management node to which the business file belongs; calculating the number of data blocks used to store the business files maintained by each management node based on the file identifier of the business file, the correspondence between the file size of the business file and the node identifier of the management node to which the business file belongs, and the block size of the data block; performing a sum operation on the number of business files maintained by each management node and the number of data blocks used to store the business files maintained by each management node to obtain the second value.

[0014] In some embodiments, determining the sum of the number of business files generated by each candidate business unit and the number of data blocks used to store the business files generated by each candidate business unit as a first value includes: counting the number of business files generated by each candidate business unit based on the correspondence between the file identifier of the business file and the group identifier of the group to which the business file belongs, and the correspondence between the group identifier and the unit identifier of the candidate business unit; calculating the number of data blocks used to store the business files generated by each candidate business unit based on the file identifier of the business file, the correspondence between the file size of the business file and the group identifier of the group to which the business file belongs, the correspondence between the group identifier and the unit identifier of the candidate business unit, and the block size of the data block; performing a sum operation on the number of business files generated by each candidate business unit and the number of data blocks used to store the business files generated by each candidate business unit to obtain the first value.

[0015] In some embodiments, for each target business unit, multiple file directories corresponding to each target business unit are selected as the multiple candidate file directories, including: for each target business unit, based on the correspondence between the group identifier and the unit identifier of the target business unit, and the correspondence between the group identifier and the directory identifier of the file directory, multiple file directories corresponding to each target business unit are selected as the multiple candidate file directories.

[0016] In some embodiments, performing homogenization processing on at least one of the multiple candidate file directories includes: for each candidate file directory in the at least one candidate file directory, merging the multiple business files included in each candidate file directory to obtain a business file to be processed; determining the number of multiple target business files after the homogenization processing of each candidate file directory based on the block size of the data block and the sum of the file sizes of the multiple business files included in each candidate file directory; and performing a split operation on the business file to be processed based on the determined number of multiple target business files to obtain the multiple target business files, wherein the difference between the file sizes of any two target business files is within a difference range.

[0017] According to a second aspect of the present disclosure, a file processing device based on a distributed file system is provided, comprising: a selection module, configured to select multiple candidate file directories from multiple file directories in the distributed file system; a determination module, configured to determine the unevenness rate between multiple business files included in each candidate file directory, wherein the unevenness rate represents the degree of discreteness between the file sizes of the multiple business files; and a uniformization processing module, configured to perform uniformization processing on at least one candidate file directory among the multiple candidate file directories according to the unevenness rate corresponding to the multiple candidate file directories, so that the multiple business files included in the at least one candidate file directory are uniformly stored in each data block of the distributed file system.

[0018] According to a third aspect of the present disclosure, a device is provided, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute the file processing method based on a distributed file system described in any of the above embodiments based on instructions stored in the memory.

[0019] According to a fourth aspect of the present disclosure, a computer storable medium is provided, on which computer program instructions are stored, and when the instructions are executed by a processor, the file processing method based on a distributed file system described in any of the above embodiments is implemented.

[0020] In the above embodiment, uneven file directories can be accurately identified, the probability of data skew problems in the distributed file system caused by unevenness is reduced, and the storage performance of the distributed file system is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0022] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0023] Figure 1 is a flowchart illustrating a file processing method based on a distributed file system according to some embodiments of the present disclosure;

[0024] Figure 2 is a flowchart illustrating selection of multiple candidate file directories according to some embodiments of the present disclosure;

[0025] Figure 3 is a flowchart illustrating selection of one or more candidate business units according to some embodiments of the present disclosure;

[0026] Figure 4 is a block diagram showing a file processing device based on a distributed file system according to some embodiments of the present disclosure;

[0027] Figure 5 is a block diagram showing a file processing device based on a distributed file system according to some embodiments of the present disclosure;

[0028] Figure 6 is a block diagram illustrating a computer system for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0029] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0030] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0031] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0032] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0033] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0034] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0035] Figure 1 is a flowchart illustrating a file processing method based on a distributed file system according to some embodiments of the present disclosure.

[0036] like Figure 1 As shown, the file processing method based on the distributed file system includes steps S110 to S130. For example, the distributed file system may be a Hadoop distributed file system (Hadoop Distributed File System, HDFS).

[0037] In step S110, a plurality of candidate file directories are selected from a plurality of file directories in the distributed file system.

[0038] In some embodiments, the Figure 2 Select multiple candidate file directories in the manner shown.

[0039] Figure 2 is a flowchart illustrating selecting multiple candidate file directories according to some embodiments of the present disclosure.

[0040] like Figure 2 As shown, selecting multiple candidate file directories includes steps S111 to S114.

[0041] In step S111, one or more candidate service units are selected from a plurality of service units served by the distributed file system.

[0042] For example, a distributed file system includes multiple management nodes (NameNode), which can be Figure 3 The method shown is used to select one or more candidate business units.

[0043] Figure 3 is a flowchart illustrating the selection of one or more candidate business units according to some embodiments of the present disclosure.

[0044] like Figure 3 As shown, selecting one or more candidate business units includes steps S1110 to S1113.

[0045] In step S1110, the sum of the number of service files maintained by each management node and the number of data blocks used to store the service files maintained by each management node is determined as a second value.

[0046] For example, step S1110 may be implemented as follows.

[0047] First, according to the correspondence between the file identifier of the business file and the node identifier of the management node to which the business file belongs, the number of business files maintained by each management node is counted. The correspondence between the file identifier of the business file and the node identifier of the management node to which the business file belongs is part of the metadata information of all business files in the distributed file system.

[0048] Generally, metadata information of all business files in a distributed file system is stored in the FsImage image file. The metadata information includes the file identifier of the business file, the file directory information to which the business file belongs (for example, the directory identifier of the file directory), the file size of the business file, the location information of the business file (the storage path of the business file), the number of copies of the business file, the permission information of the business file, the user to whom the business file belongs, the group identifier of the group to which the business file belongs, and the information of the management node where the business file is located (the node identifier of the management node).

[0049] Secondly, the number of blocks used to store the data blocks of the business files maintained by each management node is calculated based on the file identifier of the business file, the corresponding relationship between the file size of the business file and the node identifier of the management node to which the business file belongs, and the block size of the data block. For example, for each management node, the file size of each business file corresponding to it is obtained. For each business file of the management node, the smallest integer greater than or equal to the quotient of the file size and the block size is used as the number of blocks of the data blocks of each business file maintained by the management node.

[0050] A sum operation is performed on the number of business files maintained by each management node and the number of data blocks used to store the business files maintained by each management node to obtain the second value.

[0051] In step S1111, the sum of the maximum number of service files maintainable by each management node and the number of data blocks used to store the maximum number of service files is determined as a third value.

[0052] In step S1112, one or more management nodes whose ratio of the second value to the third value is greater than a ratio threshold are selected as one or more target management nodes. For example, the ratio threshold is 0.8.

[0053] In step S1113, one or more service units corresponding to one or more target management nodes are selected as one or more candidate service units.

[0054] In a distributed file system, the storage space of a management node is limited. The smaller the remaining storage space of a management node, the greater the impact of the file directory maintained by the management node on the performance of the distributed file system, and the more it is necessary to optimize the file directory maintained by the management node. In the above embodiment, the business unit corresponding to the management node whose ratio of the second value to the third value is greater than the ratio threshold is screened out as a candidate business unit, and each business unit under the management node that needs to be optimized can be accurately screened out, so as to accurately select the file directory for uniform processing.

[0055] return Figure 2 In step S112, the sum of the number of business files generated by each candidate business unit and the number of data blocks used to store the business files generated by each candidate business unit is determined as a first value.

[0056] For example, step S112 may be implemented as follows.

[0057] First, according to the correspondence between the file identifier of the business file and the group identifier of the group to which the business file belongs, and the correspondence between the group identifier and the unit identifier of the candidate business unit, the number of business files generated by each candidate business unit is counted. The correspondence between the file identifier of the business file and the group identifier of the group to which the business file belongs is a part of the metadata information of all business files in the distributed file system. The correspondence between the group identifier and the unit identifier of the candidate business unit is a pre-stored correspondence.

[0058] Then, according to the file identifier of the business file, the correspondence between the file size of the business file and the group identifier of the group to which the business file belongs, the correspondence between the group identifier and the unit identifier of the candidate business unit, and the block size of the data block, the number of blocks used to store the data blocks of the business file generated by each candidate business unit is calculated. The method of calculating the number of blocks here is similar to the above, and will not be repeated here.

[0059] Finally, a sum operation is performed on the number of service files generated by each candidate service unit and the number of data blocks used to store the service files generated by each candidate service unit to obtain the first value.

[0060] In step S113, the first N business units with the largest first values ​​are selected from one or more candidate business units as target business units, where N is a positive integer.

[0061] In step S114, for each target business unit, multiple file directories corresponding to each target business unit are selected as multiple candidate file directories. For example, for each target business unit, based on the correspondence between the group identifier and the unit identifier of the target business unit, and the correspondence between the group identifier and the directory identifier of the file directory, multiple file directories corresponding to each target business unit are selected as multiple candidate file directories. The correspondence between the group identifier and the unit identifier of the target business unit is part of the metadata information of all business files in the distributed file system.

[0062] Generally, a business unit with a larger sum of the number of business files and the number of corresponding data blocks is an important business unit with a larger business volume. In the above embodiment, by screening out the file directories under the business unit with a larger sum of the number of business files and the number of corresponding data blocks as candidate file directories, the unevenness of the file directories under some important business units can be accurately solved.

[0063] return Figure 1 In step S120, the unevenness rate between the multiple business files included in each candidate file directory is determined. The unevenness rate represents the degree of dispersion between the file sizes of the multiple business files included in each candidate file directory. The degree of dispersion can also be called the degree of unevenness.

[0064] In some embodiments, a relative average deviation of the file sizes of a plurality of business files included in each candidate file directory is calculated as the unevenness rate corresponding to each candidate file directory.

[0065] For example, the relative average deviation of the file sizes of the multiple business files included in each candidate file directory is calculated in the following manner.

[0066] First, the average of the file sizes of multiple business files is calculated as the total average. For example, the total average is A. Where n is the total number of business files, A i is the file size of the i-th business file. i and n are both positive integers.

[0067] Secondly, a business file whose file size is smaller than the total average among the multiple business files is determined as the first business file.

[0068] Thirdly, a business file whose file size is greater than or equal to the total average value among the multiple business files is determined as the second business file.

[0069] Then, the number of the first business files and the number of the second business files are determined respectively. For example, the number of the first business files and the number of the second business files are N1 and N2 respectively, and both N1 and N2 are positive integers less than or equal to n.

[0070] Then, the average value of the file size of the first business file and the average value of the file size of the second business file are calculated as the first mean and the second mean, respectively. For example, the first mean and the second mean are A1 and A2, respectively. Among them, A j is the file size of the j-th business file. Among them, A k is the file size of the kth business file. j and k are both positive integers

[0071] Finally, the relative average deviation is calculated based on the total average, the first mean, the second mean, the number of the first business files, and the number of the second business files. For example, the relative average deviation is The relative average deviation R is the unevenness rate.

[0072] In step S130, at least one of the plurality of candidate file directories is subjected to a uniform processing according to the non-uniformity ratio corresponding to the plurality of candidate file directories. The plurality of business files included in the at least one candidate file directory after the uniform processing are uniformly stored in each data block of the distributed file system. That is, for the candidate file directory subjected to the uniform processing, the number of the plurality of business files included in the candidate file directory subjected to the uniform processing is the same as the number of the data blocks used to store the plurality of business files subjected to the uniform processing.

[0073] In some embodiments, the uniformization process is performed on at least one of the multiple candidate file directories according to the non-uniformity rates corresponding to the multiple candidate file directories in the following manner.

[0074] First, the uneven rates corresponding to the plurality of candidate file directories are sorted, for example, from large to small or from small to large.

[0075] Then, each candidate file directory among the first M candidate file directories with the largest unevenness rate is subjected to uniformization processing. M is a positive integer. For example, M can be set according to actual conditions.

[0076] In other embodiments, the uniformity processing of at least one candidate file directory among the multiple candidate file directories can be implemented according to the non-uniformity rates corresponding to the multiple candidate file directories in the following manner. The uniformity processing is performed on each candidate file directory whose non-uniformity rate is greater than the non-uniformity rate threshold. For example, the non-uniformity rate threshold can be set according to the non-uniformity rates corresponding to the multiple candidate file directories, or can be set according to the importance of the business units to which the multiple candidate file directories belong.

[0077] In some embodiments, at least one candidate file directory among the multiple candidate file directories may be homogenized in the following manner.

[0078] First, for each candidate file directory in at least one candidate file directory, multiple business files included in each candidate file directory are merged to obtain a business file to be processed.

[0079] Then, according to the block size of the data block and the sum of the file sizes of the multiple business files included in each candidate file directory (i.e., the file size of the business file to be processed), the number of multiple target business files after the uniform processing of each candidate file directory is determined. For example, the smallest integer greater than or equal to the quotient between the file size of the business file to be processed and the block size is determined as the number of multiple target business files.

[0080] Finally, according to the determined number of the plurality of target business files, the business file to be processed is split to obtain a plurality of target business files. The difference between the file sizes of any two target business files is within the difference range.

[0081] In a distributed system, data skew refers to the situation in which the data processed by some partitions or nodes during parallel processing is significantly higher than that of other partitions or nodes, causing the data processing tasks of this part to be much larger than other tasks, thus becoming the slowest part of this stage, and then becoming the bottleneck of the entire job execution, and even directly leading to job failure. That is, the greater the unevenness between business files in the file directory, the more likely it is that data skew will occur, thereby reducing the storage performance of the distributed file system.

[0082] In the above embodiment, according to the non-uniformity rate corresponding to each candidate file directory, the non-uniform file directory can be accurately identified. By performing a uniform process on at least one relatively non-uniform candidate file directory among multiple candidate file directories, the business files are evenly stored in the data blocks of the distributed file system, which can reduce the probability of data skew problems in the distributed file system caused by the non-uniformity problem, and improve the storage performance of the distributed file system.

[0083] Figure 4 is a block diagram showing a file processing device based on a distributed file system according to some embodiments of the present disclosure.

[0084] like Figure 4 As shown, the file processing device 4 based on the distributed file system includes a selection module 41, a determination module 42 and a uniform processing module 43.

[0085] The selection module 41 is configured to select multiple candidate file directories from multiple file directories in the distributed file system, for example, Figure 1Step S110 is shown.

[0086] The determination module 42 is configured to determine the unevenness ratio between the multiple business files included in each candidate file directory, for example, by performing the following steps: Figure 1 Step S120 is shown. The non-uniformity rate represents the degree of dispersion between the file sizes of multiple business files.

[0087] The uniform processing module 43 is configured to perform uniform processing on at least one candidate file directory among the multiple candidate file directories according to the non-uniformity rate corresponding to the multiple candidate file directories, so that the multiple business files included in the at least one candidate file directory are uniformly stored in each data block of the distributed file system, for example, performing the following steps: Figure 1 Step S130 is shown.

[0088] Figure 5 is a block diagram showing a file processing device based on a distributed file system according to some embodiments of the present disclosure.

[0089] like Figure 5 As shown, the file processing device 5 based on the distributed file system includes a memory 51; and a processor 52 coupled to the memory 51. The memory 51 is used to store instructions for executing the corresponding embodiments of the file processing method based on the distributed file system. The processor 52 is configured to execute the file processing method based on the distributed file system in any of the embodiments of the present disclosure based on the instructions stored in the memory 51.

[0090] Figure 6 is a block diagram illustrating a computer system for implementing some embodiments of the present disclosure.

[0091] like Figure 6 As shown, the computer system 60 may be embodied in the form of a general purpose computing device. The computer system 60 includes a memory 610, a processor 620, and a bus 600 that connects the various system components.

[0092] The memory 610 may include, for example, a system memory, a non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, and other programs. The system memory may include a volatile storage medium, such as a random access memory (RAM) and / or a cache memory. The non-volatile storage medium may store, for example, instructions for executing at least one corresponding embodiment of the file processing method based on a distributed file system. The non-volatile storage medium includes, but is not limited to, a disk memory, an optical memory, a flash memory, etc.

[0093] The processor 620 can be implemented by a general processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistors, etc. Discrete hardware components. Accordingly, each module such as the judgment module and the determination module can be implemented by a central processing unit (CPU) running instructions in a memory that execute corresponding steps, or can be implemented by a dedicated circuit that executes corresponding steps.

[0094] The bus 600 may use any of a variety of bus architectures, including, but not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, and a Peripheral Component Interconnect (PCI) bus.

[0095] The computer system 60 may also include an input / output interface 630, a network interface 640, a storage interface 650, etc. These interfaces 630, 640, 650, the memory 610, and the processor 620 may be connected via a bus 600. The input / output interface 630 may provide a connection interface for input / output devices such as a display, a mouse, and a keyboard. The network interface 640 may provide a connection interface for various networked devices. The storage interface 650 may provide a connection interface for external storage devices such as a floppy disk, a USB flash drive, and an SD card.

[0096] Here, various aspects of the present disclosure are described with reference to flowcharts and / or block diagrams of methods, devices, and computer program products according to embodiments of the present disclosure. It should be understood that each frame of the flowchart and / or block diagram and the combination of frames can be implemented by computer-readable program instructions.

[0097] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable device to produce a machine, so that the processor executes the instructions to produce means for implementing the functions specified in one or more blocks in the flowchart and / or block diagram.

[0098] These computer-readable program instructions may also be stored in a computer-readable memory, which cause the computer to work in a specific manner to produce an article of manufacture, including instructions for implementing the functions specified in one or more blocks in the flowchart and / or block diagram.

[0099] The present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects.

[0100] Through the file processing method and device based on the distributed file system and the computer storable medium in the above-mentioned embodiments, uneven file directories can be accurately identified, the probability of data skew problems in the distributed file system caused by uneven problems can be reduced, and the storage performance of the distributed file system can be improved.

[0101] So far, the file processing method and device based on the distributed file system and the computer storable medium according to the present disclosure have been described in detail. In order to avoid obscuring the concept of the present disclosure, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed herein.

Claims

1. A file processing method based on a distributed file system, comprising: Selecting multiple candidate file directories from multiple file directories in the distributed file system; Determine a non-uniformity rate among a plurality of business files included in each candidate file directory, wherein the non-uniformity rate represents a degree of dispersion among file sizes of the plurality of business files; According to the unevenness rate corresponding to the multiple candidate file directories, at least one candidate file directory among the multiple candidate file directories is homogenized so that the multiple business files included in the at least one candidate file directory are evenly stored in each data block of the distributed file system.

2. The file processing method based on a distributed file system according to claim 1, wherein: Determining the unevenness rate between the multiple business files included in each candidate file directory includes: A relative average deviation of the file sizes of the multiple business files included in each candidate file directory is calculated as the unevenness rate corresponding to each candidate file directory.

3. The file processing method based on a distributed file system according to claim 2, wherein: Calculating the relative average deviation of the file sizes of the multiple business files included in each candidate file directory includes: Calculating an average of the file sizes of the multiple business files as a total average; Determine, among the multiple business files, a business file whose file size is smaller than the total average value as a first business file; Determine, among the multiple business files, a business file whose file size is greater than or equal to the total average value as a second business file; respectively determining the number of the first business files and the number of the second business files; Calculate an average value of the file sizes of the first business files and an average value of the file sizes of the second business files, as a first mean value and a second mean value, respectively; The relative average deviation is calculated according to the total average, the first mean, the second mean, the number of the first business files, and the number of the second business files.

4. The file processing method based on a distributed file system according to claim 1, wherein: According to the non-uniformity rates corresponding to the multiple candidate file directories, performing uniform processing on at least one candidate file directory among the multiple candidate file directories includes: sorting the uneven rates corresponding to the plurality of candidate file directories; A uniform processing is performed on each of the first M candidate file directories with the largest non-uniformity, where M is a positive integer.

5. The file processing method based on a distributed file system according to claim 1, wherein: According to the non-uniformity rates corresponding to the multiple candidate file directories, performing uniform processing on at least one candidate file directory among the multiple candidate file directories includes: A uniform processing is performed on each candidate file directory whose uniformity is greater than the uniformity threshold.

6. The file processing method based on a distributed file system according to claim 1, wherein: Select multiple candidate file directories including: Selecting one or more candidate business units from a plurality of business units served by the distributed file system; Determine the sum of the number of service files generated by each candidate service unit and the number of data blocks used to store the service files generated by each candidate service unit as a first value; Selecting first N business units with the largest first values ​​from the one or more candidate business units as target business units, where N is a positive integer; For each target business unit, a plurality of file directories corresponding to each target business unit are selected as the plurality of candidate file directories.

7. The file processing method based on a distributed file system according to claim 6, wherein: The distributed file system includes multiple management nodes, and selecting one or more candidate business units includes: Determine the sum of the number of service files maintained by each management node and the number of data blocks used to store the service files maintained by each management node as a second value; Determine the sum of the maximum number of service files that can be maintained by each management node and the number of data blocks used to store the maximum number of service files as a third value; Select one or more management nodes whose ratio of the second value to the third value is greater than a ratio threshold as one or more target management nodes; One or more service units corresponding to the one or more target management nodes are selected as the one or more candidate service units.

8. The file processing method based on a distributed file system according to claim 7, wherein: Determining the sum of the number of service files maintained by each management node and the number of data blocks used to store the service files maintained by each management node as the second value includes: According to the correspondence between the file identifier of the business file and the node identifier of the management node to which the business file belongs, counting the number of business files maintained by each management node; Calculate the number of data blocks used to store the business files maintained by each management node according to the file identifier of the business file, the correspondence between the file size of the business file and the node identifier of the management node to which the business file belongs, and the block size of the data block; A sum operation is performed on the number of business files maintained by each management node and the number of data blocks used to store the business files maintained by each management node to obtain the second value.

9. The file processing method based on a distributed file system according to claim 6, wherein: Determining the sum of the number of service files generated by each candidate service unit and the number of data blocks used to store the service files generated by each candidate service unit as the first value includes: According to the correspondence between the file identifier of the business file and the group identifier of the group to which the business file belongs, and the correspondence between the group identifier and the unit identifier of the candidate business unit, the number of business files generated by each candidate business unit is counted; Calculate the number of data blocks used to store the business file generated by each candidate business unit according to the file identifier of the business file, the correspondence between the file size of the business file and the group identifier of the group to which the business file belongs, the correspondence between the group identifier and the unit identifier of the candidate business unit, and the block size of the data block; A sum operation is performed on the number of service files generated by each candidate service unit and the number of data blocks used to store the service files generated by each candidate service unit to obtain the first value.

10. The file processing method based on a distributed file system according to claim 6, wherein: For each target business unit, multiple file directories corresponding to each target business unit are selected as the multiple candidate file directories including: For each target business unit, multiple file directories corresponding to each target business unit are selected as the multiple candidate file directories based on the correspondence between the group identifier and the unit identifier of the target business unit and the correspondence between the group identifier and the directory identifier of the file directory.

11. The file processing method based on a distributed file system according to claim 1, wherein: The homogenizing process of at least one candidate file directory among the plurality of candidate file directories includes: For each candidate file directory in the at least one candidate file directory, merging multiple business files included in each candidate file directory to obtain a business file to be processed; Determining the number of the plurality of target business files after the homogenization processing of each candidate file directory according to the block size of the data block and the sum of the file sizes of the plurality of business files included in each candidate file directory; According to the determined number of the multiple target business files, the to-be-processed business file is split to obtain the multiple target business files, and the difference between the file sizes of any two target business files is within a difference range.

12. A file processing device based on a distributed file system, comprising: A selection module is configured to select multiple candidate file directories from multiple file directories in the distributed file system; A determination module is configured to determine an unevenness ratio among a plurality of business files included in each candidate file directory, wherein the unevenness ratio represents a degree of discreteness among file sizes of the plurality of business files; The uniformization processing module is configured to perform uniformization processing on at least one candidate file directory among the multiple candidate file directories according to the non-uniformity rate corresponding to the multiple candidate file directories, so that the multiple business files included in the at least one candidate file directory are evenly stored in each data block of the distributed file system.

13. A file processing device based on a distributed file system, comprising: Memory; as well as A processor coupled to the memory, the processor being configured to execute the file processing method based on a distributed file system according to any one of claims 1 to 11 based on instructions stored in the memory.

14. A computer storable medium having computer program instructions stored thereon, which, when executed by a processor, implements the file processing method based on a distributed file system as claimed in any one of claims 1 to 11.

Images