Hdfs empty directory positioning method and device, equipment and medium

Abstract

The application provides an HDFS empty directory positioning method, device, equipment and medium, aiming to solve the problems of low efficiency and easy to affect normal data business of HDFS in the HDFS empty directory positioning process, wherein the method comprises: obtaining a binary metadata file of a distributed file system HDFS, and deserializing the metadata file to obtain a first plaintext file; obtaining the number of files corresponding to each directory in the first plaintext file based on a distributed computing engine, and encapsulating the number of files and the first plaintext file into a second plaintext file; loading the second plaintext file in a Hive table to obtain a plaintext file Hive table; and positioning the directories storing empty files in the plaintext file Hive table to obtain positioning information. Through the above method, the empty directory can be quickly and accurately positioned without invasion, the positioning efficiency of the empty directory is improved, support and convenience are provided for the cleaning of the empty directory, and the stability of the HDFS is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the computer technical field, and particularly relates to an HDFS empty directory positioning method, device, equipment and medium. BACKGROUND

[0002] In the actual use process of HDFS (Hadoop Distributed File System), its stability will become worse with the increase of the number of files, especially in the case of a large number of useless empty directories, which will further lead to the instability of the storage system. Therefore, how to quickly locate the empty directory in HDFS is worth studying.

[0003] In the related art, the empty directory in HDFS is mainly located by recursively traversing the directory structure tree of the entire HDFS to obtain all empty directory lists, and then locating the empty directories in all directory lists. This way will cause great data pressure on HDFS, and may affect the normal development of data business, and is very time-consuming. SUMMARY

[0004] In view of the above problems, that is, the problems of the HDFS empty directory positioning process affecting the normal data business of HDFS and low efficiency, the present application provides an HDFS empty directory positioning method, device, equipment and medium.

[0005] In order to achieve the above purpose, the present application provides the following technical solutions:

[0006] The present application provides an HDFS empty directory positioning method of a distributed file system, comprising:

[0007] Obtaining a binary metadata file of a distributed file system HDFS, and deserializing the metadata file to obtain a first plaintext file;

[0008] Obtaining the number of files corresponding to each directory in the first plaintext file based on a distributed computing engine, and encapsulating the number of files and the first plaintext file into a second plaintext file;

[0009] Loading the second plaintext file into a Hive table to obtain a plaintext file Hive table, and positioning the directories storing empty files in the plaintext file Hive table to obtain positioning information.

[0010] In an embodiment, the obtaining of the number of files corresponding to each directory in the first plaintext file based on the distributed computing engine comprises:

[0011] Traversing the parent directories and / or child directories to which all files in the first plaintext file belong respectively based on the distributed computing engine, and obtaining the number of files under each parent directory and / or child directory.

[0012] The number of files under the same parent directory and / or subdirectory is aggregated to obtain the number of files corresponding to each directory.

[0013] In an embodiment, the metadata file of the binary of the distributed file system HDFS is obtained, including:

[0014] The binary metadata file is extracted from the metadata node NameNode of the distributed file system HDFS.

[0015] In an embodiment, the metadata file is deserialized, including:

[0016] The structure information of the metadata file is obtained, and a corresponding deserialization program is obtained based on the structure information;

[0017] The metadata file is deserialized based on the deserialization program.

[0018] In an embodiment, the plaintext file Hive table carries a first field about each directory and a second field about the number of files under the corresponding directory,

[0019] The directory storing the empty file in the plaintext file Hive table is located, including:

[0020] The second field with preset field information in the plaintext file Hive table is queried based on the object-relational mapping framework query language HQL, and the first field corresponding to the second field with preset field information is obtained.

[0021] The target directory is located based on the first field corresponding to the second field with preset field information, and the target directory is the directory storing the empty file.

[0022] In an embodiment, after obtaining the binary metadata file of the distributed file system HDFS, and before deserializing the metadata file, further including:

[0023] The metadata file is pushed to a server cluster irrelevant to the production environment;

[0024] The metadata file is deserialized, including: the metadata file is deserialized in the server cluster.

[0025] In an implementation, the format of the first plaintext file or the second plaintext file comprises: a file path and a directory path HDFS_DIR of a distributed system file HDFS; a replication number REPLICATION; a modification time MODIFICATION_TIME; or an access time ACCESS_TIME.

[0026] According to another aspect of the present application, there is provided a distributed file system HDFS empty directory locating apparatus, comprising:

[0027] a metadata obtaining module configured to obtain a binary metadata file of the distributed file system HDFS and deserialize the metadata file to obtain a first plaintext file;

[0028] a file number obtaining module configured to obtain a file number corresponding to each directory in the first plaintext file based on a distributed computing engine;

[0029] an encapsulating module configured to encapsulate the file number and the first plaintext file into a second plaintext file;

[0030] a loading module configured to load the second plaintext file into a Hive table to obtain a plaintext file Hive table; and a locating module configured to locate a directory storing an empty file in the plaintext file Hive table to obtain locating information.

[0031] According to still another aspect of the present application, there is provided an electronic device, comprising: a memory and a processor;

[0032] the memory stores computer execution instructions;

[0033] the processor executes the computer execution instructions stored in the memory, so that the electronic device executes the distributed file system HDFS empty directory locating method.

[0034] According to yet another aspect of the present application, there is provided a computer readable storage medium, wherein the computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by a processor to implement the distributed file system HDFS empty directory locating method.

[0035] The HDFS empty directory positioning method, device, equipment and medium provided by the application, by obtaining the binary metadata file of the distributed file system HDFS, and deserializing the metadata file to obtain a first plaintext file, then obtaining the number of files corresponding to each directory in the first plaintext file based on the distributed computing engine, and encapsulating the number of files and the first plaintext file into a second plaintext file, loading the second plaintext file in the Hive table to obtain a plaintext file Hive table, and finally positioning the directory storing the empty file in the plaintext file Hive table to obtain positioning information, the empty directory can be quickly and accurately positioned, the positioning efficiency of the empty directory is improved, the cleaning of the empty directory is supported and facilitated, and the stability of the HDFS is effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0036] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0037] Figure 1a A network architecture diagram of the distributed file system HDFS;

[0038] Figure 1b A network architecture diagram of improving the stability of the distributed file system HDFS in the related art;

[0039] Figure 2 A possible scenario diagram provided by the embodiment of the application;

[0040] Figure 3 A flowchart of an HDFS empty directory positioning method provided by the embodiment of the application;

[0041] Figure 4 One of the flowcharts of another HDFS empty directory positioning method provided by the embodiment of the application;

[0042] Figure 5 The second flowchart of another HDFS empty directory positioning method provided by the embodiment of the application;

[0043] Figure 6 A structural diagram of an HDFS empty directory positioning device provided by the embodiment of the application;

[0044] Figure 7 A structural diagram of an electronic device provided by the embodiment of the application.

[0045] The specific embodiments of the application have been shown and described in the foregoing detailed description. These drawings and detailed description are not meant to limit the scope of the application in any way but are to illustrate the principles of the application to one of ordinary skill in the art. DETAILED DESCRIPTION

[0046] HDFS can be deployed on a cluster consisting of multiple machines, including several basic concepts of metadata nodes NameNode, data nodes DataNode and data blocks block, wherein the NameNode is responsible for the metadata management of the entire distributed file system, that is, the file path name, the ID and storage location of the data block, and the information such as which nodes are part of the cluster and how many copies a block has, as shown in Figure 1a .

[0047] It can be seen that the NameNode is a very important component, and its stability and reliability are very important. If the NameNode component hangs, the entire distributed file system will be in an unusable state, and all open source communities have done a lot of work on the high availability of HDFS, as shown in the NameNode high availability principle below Figure 1b , which solves the single point problem by starting two NameNodes to improve the overall stability of HDFS.

[0048] However, in the actual use of HDFS, its stability will deteriorate over time as the number of files increases, especially in the presence of a large number of useless empty directories, further leading to instability of the storage system. The number of files in the HDFS storage system in the big data platform is usually in the tens of millions or even hundreds of millions, and in related technologies, only the number of files in the HDFS file system, the number of files in a certain directory, and the files under a certain small directory can be known, but it is not possible to accurately and quickly locate the empty directories in the entire HDFS storage system. Although the entire HDFS directory structure tree can be recursively traversed to obtain all the file lists to find empty directories, this will cause a lot of pressure on HDFS, and at the same time, it will affect the development of the enterprise's data business, and it takes a very long time, which is almost a solution that is difficult to apply in practice.

[0049] Therefore, the embodiment of the present application provides an HDFS empty directory positioning method, device, equipment and medium. The binary metadata file of the HDFS is obtained, and the metadata file is deserialized to obtain a first plaintext file. Then, the distributed computing engine is used to obtain the number of files corresponding to each directory in the first plaintext file, and the number of files is encapsulated into a second plaintext file and loaded into a Hive table to obtain a plaintext file Hive table. Then, the directory storing the empty file is quickly positioned in the plaintext file Hive table. The method does not invade the HDFS, does not affect the normal data service of the HDFS, and can quickly and accurately position the empty directory, thereby providing support and convenience for subsequent cleaning of the empty directory and effectively improving the stability of the HDFS. In addition, the distributed computing engine is used to obtain the number of files corresponding to each directory in the first plaintext file, and the number of files is encapsulated into the second plaintext file and then loaded into the Hive table. The Hive table can directly display the directories and the number of files under the corresponding directories without further calculation. The terminal device can quickly position the corresponding empty file directory position in the Hive table, thereby positioning the empty file directory and effectively improving the positioning efficiency of the empty file directory.

[0050] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described in more detail below with reference to the drawings in the embodiments of the present application. Identical or similar reference numerals are used to represent identical or similar components or components having identical or similar functions throughout the drawings. The described embodiments are part of the embodiments of the present application, rather than all the embodiments of the present application. The embodiments described below with reference to the drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0051] Figure 2 A possible scenario diagram provided by the embodiment of the present application is shown in FIG. 1. Figure 2 As shown in FIG. 1, the scenario diagram includes a distributed file system HDFS 210 and a terminal device 220. The HDFS 210 is deployed on a server, and the terminal device 220 and the server are connected to each other through a wired or wireless network. In some embodiments, the HDFS 210 is configured to provide a binary metadata file to the terminal device 220. The terminal device 220 is configured to perform deserialization and empty directory positioning based on the metadata file provided by the HDFS 210. Optionally, in the process of deserialization and empty directory positioning, the terminal device 220 undertakes the main computing work or undertakes the computing work alone.

[0052] The terminal device 220 can include, but is not limited to, a computer, a smart phone, a tablet computer, an electronic book reader, a Moving Picture experts group audio layer III (MP3) player, a Moving Picture experts group audio layer IV (MP4) player, a portable computer, a vehicle-mounted computer, a wearable device, a desktop computer, a set-top box, a smart television, and the like.

[0053] The server can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and basic cloud computing services such as big data and artificial intelligence platforms.

[0054] Optionally, the number of the distributed file system HDFS 210 and the terminal device 220 can be more or less, and the embodiments of the present application do not limit this.

[0055] The above briefly describes the scenario schematic diagram of the present application. Next, the terminal device 220 in the application will be taken as an example to illustrate the embodiments of the present application in detail. Figure 2

[0056] Please refer to Figure 3 , Figure 3 The embodiments of the present application provide a distributed file system HDFS empty directory positioning method, including steps S301-S304.

[0057] Step S301, obtain the binary metadata file of the distributed file system HDFS, and deserialize the metadata file to obtain a first plaintext file.

[0058] In this embodiment, for the positioning of the HDFS empty directory, the positioning is not directly performed in the HDFS, but the HDFS metadata file is obtained and analyzed and positioned in the subsequent steps to realize the non-invasive positioning of the HDFS empty directory.

[0059] ​It can be understood that the related technology directly utilizes the API interface of the distributed file system HDFS to scan the empty text. Since the number of files of the HDFS is generally in the order of ten million or even one hundred million, the scanning and positioning process will have performance invasion on the HDFS, and even cause the HDFS to be unavailable. The embodiment obtains and analyzes the HDFS metadata file to non-invasively locate the empty directory, and does not have any influence on the data service of the HDFS.

[0060] In an implementation, the step S201 obtains the binary metadata file of the distributed file system HDFS, specifically: extracts the binary metadata file from the metadata node NameNode of the distributed file system HDFS.

[0061] It can be understood that the NameNode is responsible for the metadata management of the entire distributed file system, including the file path name, the ID and the storage location of the data block, and the like. It can be found by checking the monitoring page of the HDFS itself that the NameNode can obtain how many files and folders, how many blocks and how many objects of the entire file system, and the size of a certain file. The embodiment can accurately locate the empty directory by obtaining the metadata file of the NameNode and analyzing the related metadata file, thereby improving the work efficiency.

[0062] In an implementation, the step 301 deserializes the metadata file, which can include the following steps

[0063] Obtains the structure information of the metadata file, and obtains the corresponding deserialization program based on the structure information; and deserializes the metadata file based on the deserialization program.

[0064] In an implementation, by understanding the structure of the metadata, a Java program deserialization program can be written to deserialize the binary metadata file into a plaintext file, so as to realize efficient analysis of the metadata file.

[0065] The format of the first plaintext file (or the second plaintext file) includes: the file path and the directory path HDFS_DIR of the distributed system file HDFS; the number of copies REPLICATION; the modification time MODIFICATION_TIME; or the access time ACCESS_TIME.

[0066] It can be understood that in addition to the above format information, other information can also be included. For example, the format of the plaintext file in the embodiment is as follows:

[0067] HDFS_DIR: file path and directory path of HDFS; REPLICATION: number of copies; MODIFICATION_TIME: modification time; ACCESS_TIME: access time; PREFERRED_BLOCK_SIZE: preferred block size; BLOCKS_COUNT: number of data blocks; FILE_SIZE: file size; NSQUOTA: name quota; DSQUOTA: target space quota; PERMISSION: permission; USER_NAME: username; GROUP_NAME: group name.

[0068] Further, after obtaining the binary metadata file of the distributed file system HDFS in step S301, and before deserializing the metadata file, the following steps can be further included:

[0069] pushing the metadata file to a server cluster irrelevant to the production environment;

[0070] wherein the binary metadata file is deserialized, specifically, the binary metadata file is deserialized in the server cluster.

[0071] In this embodiment, the empty directory is located non-invasively by parsing the HDFS metadata file on another service cluster irrelevant to HDFS, which can further reduce the degree of business impact on HDFS. In some embodiments, the metadata can also be parsed in a terminal device.

[0072] In step S302, the number of files corresponding to each directory in the first plaintext file is obtained based on a distributed computing engine, and the number of files and the first plaintext file are encapsulated into a second plaintext file.

[0073] It can be understood that the number of first plaintext files obtained by deserializing the metadata file is usually very large, possibly tens of millions or even hundreds of millions. In this embodiment, the distributed computing engine is used for processing, which can effectively ensure the computing efficiency. In one implementation, the first plaintext file can be uploaded to the distributed file system HDFS, and the MapReduce distributed computing engine is used to analyze and process the file to obtain the number of files corresponding to each directory in the first plaintext file.

[0074] It should be noted that the first plaintext file and the second plaintext file in this embodiment are only used to distinguish similar objects, and have no other meanings. They can be the same file or different files. For example, the second plaintext file in this embodiment has the same file content as the first plaintext file, but the number of files corresponding to each directory in the first plaintext file is annotated.

[0075] Step S303, loading the second plaintext file in the Hive table to obtain a plaintext file Hive table.

[0076] In this embodiment, since the second plaintext file carries the number of files corresponding to each directory, the obtained plaintext file Hive table after loading can directly show each directory and the corresponding number of files.

[0077] In one implementation, the table creation statement of the plaintext file Hive table can be as follows:

[0078]

[0079]

[0080] Step S304, positioning the directory storing the empty file in the plaintext file Hive table to obtain positioning information.

[0081] For example, the HQL (Hibernate Query Language, object relationship mapping framework query language) query statement is used to query and position the plaintext file Hive table, and the corresponding empty directory is quickly found to further quickly and accurately obtain the empty directory list.

[0082] Compared with directly loading the deserialized plaintext file (i.e., the first plaintext file) in the Hive table in the related art, the plaintext file Hive table will only list the file data, and if the number of files in each directory needs to be obtained, the query and calculation need to be performed, which undoubtedly consumes a long time. In this embodiment, the number of files corresponding to each directory in the first plaintext file is obtained by using the distributed computing engine, and is encapsulated into the second plaintext file, which is then loaded in the Hive table. The Hive table can directly show the directory and the number of files under the corresponding directory, and does not need to be calculated again. The terminal device can quickly position the corresponding empty file directory position in the Hive table, and then position the empty file directory, which can effectively improve the positioning efficiency of the empty file directory.

[0083] Please refer to Figure 4 , Figure 4The flowchart of another HDFS empty directory positioning method provided by the embodiment of the present application is shown in the figure. Based on the above embodiment, the embodiment further improves the accuracy of empty directory positioning by specifically illustrating the file number acquisition method of each directory. Specifically, steps S301, S303 and S304 in the embodiment are the same as those in the above embodiment, and will not be described again. Different from the above embodiment, the embodiment further divides the step S302 of acquiring the file number corresponding to each directory in the first plaintext file based on the distributed computing engine into steps S302a and S302b.

[0084] In step S302a, the distributed computing engine is used to traverse the parent directories and / or subdirectories to which all files in the first plaintext file belong, and to acquire the file number under each parent directory and / or subdirectory.

[0085] In step S302b, the file numbers under the same parent directory and / or subdirectory are aggregated to obtain the file number corresponding to each directory.

[0086] In the embodiment, the MapReduce distributed computing engine is used to calculate the file number corresponding to each directory from the first plaintext file. For example, there are two files HDFS / a / c / d / 1.txt and / a / c / 2.txt and two folders / d / e and / d / f. The computing engine performs the following operations. First, the file number corresponding to each folder is calculated, for example:

[0087]

[0088] It can be understood that the above is for the parent directories / a and / b, the subdirectories of / a / a / c and / a / c / d, and the subdirectories of / b / d / e and / d / f. The embodiment can obtain the file number under each directory by aggregating the same parent / subdirectories, for example:

[0089]

[0090] In an implementation, after aggregating the file numbers under the same parent directory and / or subdirectory, the empty directory image of the directory can be generated according to the aggregated result, and then encapsulated into a second plaintext file and loaded into the Hive table. The empty directory image can include the file number of each directory and other file information such as file size.

[0091] Further, the plaintext file Hive table in the embodiment carries a first field about each directory and a second field about the file number under the corresponding directory. In step S304, the directory storing the empty file in the plaintext file Hive table is positioned, specifically including the following steps:

[0092] Based on the Object Relational Mapping Framework Query Language (HQL), query the second field in the plaintext file Hive table where the field information is a preset value, and obtain the first field corresponding to the second field where the field information is a preset value;

[0093] The target directory is located based on the first field corresponding to the second field, which is a preset value of the field information. The target directory is a directory where empty files are stored.

[0094] In this embodiment, the preset value can be zero. In some embodiments, those skilled in the art can also adaptively set the preset value according to actual applications. The preset value corresponds to a value where the number of files is zero. Specifically, the second field with a preset value corresponds to a number of files of zero. In this embodiment, by querying the first field corresponding to the second field, the corresponding directory can be found, and the empty file directory (i.e., an empty directory) can be quickly located.

[0095] To facilitate understanding of the embodiments of this application, this embodiment is combined with Figure 5 Further explanation is provided, including the following process:

[0096] Step 1: The terminal device obtains the metadata file non-intrusively.

[0097] Understandably, the latest metadata of the NameNode is stored entirely in memory. Directly reading the NameNode's memory would put a lot of pressure on it and affect data services. This embodiment obtains the NameNode's own serialized metadata file and can push the metadata file to a cluster that is independent of the enterprise's production environment. This demonstrates the non-intrusive nature of HDFS, meaning that copying a single file will not affect the NameNode.

[0098] Step 2: Deserialize the metadata file.

[0099] By understanding the structure of metadata, a Java program is written to deserialize the binary metadata file into a first plaintext file.

[0100] Step 3: Since the first plaintext file after deserialization will be very large, usually in the tens of millions to hundreds of millions, this embodiment uses a distributed computing engine to process it, which can process the first plaintext file on HDFS.

[0101] Step 4: Use the MapReduce distributed computing engine to calculate the number of files corresponding to each directory from the first plaintext file, and generate an empty directory profile for each directory based on the number of files.

[0102] Step 5: encapsulate the empty directory image containing the number of files corresponding to each directory into a second plaintext file, and then load the result into a Hive table.

[0103] Step 6: submit a HQL distributed job to quickly and accurately locate the empty directory, and obtain an empty directory list file, as follows:

[0104] Select hdfsdir

[0105] From hdfs_dir_table

[0106] Where filecount=0

[0107] The embodiment of the application also provides a distributed file system HDFS empty directory positioning device, as shown in Figure 6 , comprising:

[0108] A metadata acquisition module 61 is configured to acquire a binary metadata file of a distributed file system HDFS, and to deserialize the metadata file to obtain a first plaintext file;

[0109] A file number acquisition module 62 is configured to acquire the number of files corresponding to each directory in the first plaintext file based on a distributed computing engine;

[0110] An encapsulation module 63 is configured to encapsulate the number of files and the first plaintext file into a second plaintext file;

[0111] A loading module 64 is configured to load the second plaintext file in a Hive table to obtain a plaintext file Hive table; and a positioning module 65 is configured to locate the directories storing empty files in the plaintext file Hive table to obtain positioning information.

[0112] In an embodiment, the metadata acquisition module 61 comprises an extraction unit configured to extract a binary metadata file from a metadata node NameNode of a distributed file system HDFS.

[0113] In an embodiment, the metadata acquisition module 61 further comprises:

[0114] A program acquisition unit configured to acquire structural information of the metadata file, and to acquire a corresponding deserialization program based on the structural information;

[0115] A deserialization unit configured to deserialize the metadata file based on the deserialization program.

[0116] In an embodiment, the file number acquisition module 62 comprises:

[0117] traversing the first plaintext file based on a distributed computing engine to traverse parent directories and / or subdirectories to which all files in the first plaintext file respectively belong, and to obtain a number of files under each parent directory and / or subdirectory;

[0118] aggregating the number of files under the same parent directory and / or subdirectory to obtain a number of files corresponding to each directory.

[0119] In an embodiment, the plaintext file Hive table carries a first field about each directory and a second field about a number of files under the corresponding directory,

[0120] The positioning module 65 includes:

[0121] a field querying unit configured to query, based on a HQL (Hive Query Language) that is an object-relational mapping framework query language, the second field with preset field information in the plaintext file Hive table, and to obtain the first field corresponding to the second field with the preset field information.

[0122] a positioning unit configured to position a target directory based on the first field corresponding to the second field with the preset field information, the target directory being a directory storing empty files.

[0123] In an embodiment, the apparatus further includes:

[0124] a pushing module configured to push the metadata file to a server cluster irrelevant to a production environment; and the metadata file is deserialized in the server cluster.

[0125] In an embodiment, the format of the first plaintext file or the second plaintext file includes a file path and a directory path HDFS_DIR of a distributed system file HDFS, a number of replicas REPLICATION, a modification time MODIFICATION_TIME, or an access time ACCESS_TIME.

[0126] Embodiments of the present application also provide an electronic device, as shown in the accompanying drawings, including a memory 71 and a processor 72. Figure 7

[0127] The memory 71 stores computer execution instructions.

[0128] The processor 72 executes the computer execution instructions stored in the memory 71, so that the electronic device performs the distributed file system HDFS empty directory positioning method.

[0129] ​Correspondingly, the embodiment of the present application further provides a computer readable storage medium, wherein computer execution instructions are stored in the computer readable storage medium, and the computer execution instructions are used for implementing the HDFS empty directory positioning method of the distributed file system when executed by a processor.

[0130] Those skilled in the art can understand that all or some steps in the method disclosed above, and the function modules / units in the system and the device can be implemented as software, firmware, hardware and appropriate combinations thereof. In the hardware implementation, the division between the function modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component can have multiple functions, or one function or step can be executed by several physical components in cooperation. Some or all of the physical components can be implemented as software executed by a processor, such as a central processor, a digital signal processor or a microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software can be distributed on a computer readable medium, which can include computer storage media (or non-transitory media) and communication media (or transitory media).

[0131] As known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, it is known to those skilled in the art that communication media generally includes computer readable instructions, data structures, program modules or other data in modulated data signals such as carrier waves or other transmission mechanisms, and can include any information delivery medium.

[0132] In the description of the embodiments of the present application, the term "and / or" only represents an association relationship of describing associated objects, which means that there can be three relationships, for example, A and / or B, which can represent three cases of A alone, A and B together, and B alone. In addition, the term "at least one" means any combination of one or at least two of the plurality, for example, including at least one of A, B and C, which can represent any one or more elements selected from the set of A, B and C. In addition, the term "plurality" means two or more, unless otherwise specified.

[0133] In the description of the embodiments of the present application, the terms "first", "second", "third", "fourth" and the like (if any) are used to distinguish similar objects, and do not necessarily have to be described in a particular order or sequential order. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or apparatus including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or apparatuses.

[0134] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for locating empty directories in a distributed file system (HDFS), characterized in that, The method comprises the following steps: obtaining a binary metadata file of a distributed file system HDFS, and deserializing the metadata file to obtain a first plaintext file; obtaining the number of files corresponding to each directory in the first plaintext file based on a distributed computing engine, and encapsulating the number of files and the first plaintext file into a second plaintext file; loading the second plaintext file into a Hive table to obtain a plaintext file Hive table; and positioning the directory storing empty files in the plaintext file Hive table to obtain positioning information.

2. The method of claim 1, wherein, The step of obtaining the number of files corresponding to each directory in the first plaintext file based on a distributed computing engine comprises the following steps: traversing the parent directories and / or child directories to which all files in the first plaintext file belong based on a distributed computing engine, and obtaining the number of files under each parent directory and / or child directory; aggregating the number of files under the same parent directory and / or child directory to obtain the number of files corresponding to each directory.

3. The method according to claim 1 or 2, characterized in that, The step of obtaining the binary metadata file of the distributed file system HDFS comprises the following step: extracting the binary metadata file from a metadata node NameNode of the distributed file system HDFS.

4. The method of claim 1, wherein, The step of deserializing the metadata file comprises the following steps: obtaining structure information of the metadata file, and obtaining a corresponding deserialization program based on the structure information; deserializing the metadata file based on the deserialization program.

5. The method of claim 1, wherein, The plaintext file Hive table carries a first field about each directory and a second field about the number of files under the corresponding directory, The step of positioning the directory storing empty files in the plaintext file Hive table comprises the following steps: querying the second field with preset field information in the plaintext file Hive table based on a HQL, and obtaining the first field corresponding to the second field with preset field information; positioning the target directory based on the first field corresponding to the second field with preset field information, wherein the target directory is the directory storing empty files.

6. The method of claim 1, wherein, After obtaining the binary metadata file of the distributed file system HDFS, and before deserializing the metadata file, the method further comprises the following steps: pushing the metadata file to a server cluster irrelevant to a production environment; The step of deserializing the metadata file comprises the step of deserializing the metadata file in the server cluster.

7. The method of claim 1, wherein, The format of the first plaintext file or the second plaintext file comprises a file path and a directory path HDFS_DIR of a distributed system file HDFS, a replication number REPLICATION, a modification time MODIFICATION_TIME, or an access time ACCESS_TIME.

8. A distributed file system HDFS empty directory locating apparatus, characterized in that, The method comprises the following steps: a metadata obtaining module configured to obtain a binary metadata file of a distributed file system HDFS, and deserialize the metadata file to obtain a first plaintext file; a file number obtaining module configured to obtain the number of files corresponding to each directory in the first plaintext file based on a distributed computing engine; The encapsulation module is configured to encapsulate the file number and the first plaintext file into a second plaintext file; The loading module is configured to load the second plaintext file into a Hive table to obtain a plaintext file Hive table; The positioning module is configured to position a directory storing an empty file in the plaintext file Hive table to obtain positioning information.

9. An electronic device, comprising: The electronic device comprises: a memory and a processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, so that the electronic device executes the HDFS empty directory positioning method in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the HDFS empty directory positioning method in any one of claims 1-7.

Images