Determination method, server and computer storage medium

Abstract

The present invention discloses a determination method, which is applied to a server based on the Hive architecture. The method comprises: parsing an acquired Hive SQL task to obtain output data of table data in the Hive SQL task; selecting a NameNode from a NameNode set according to a preset NameNode selection policy; and determining an accessible NameNode among the selected NameNodes as the NameNode for the output data. Embodiments of the present invention also disclose a server and computer storage medium that prevent NameNode load imbalance when multiple NameNodes store Hive data, thereby achieving balanced storage of Hive data when multiple NameNodes store the data.

Description

Technical Field

[0001] The present invention relates to Hive data storage technology, and in particular to a determination method, a server, and a computer storage medium. Background Art

[0002] Hive is a commonly used data analysis component in the Hadoop big data ecosystem. Analysts use Hive to analyze massive amounts of data by submitting Hive SQL tasks. All data in the open-source Hive component is stored on the same NameNode. Therefore, the scale of data processed by Hive is limited by the storage capacity of the underlying NameNode. Due to architectural limitations, a single NameNode reaches a performance bottleneck after storing more than 150 million files. This not only severely restricts the scale of data processed by Hive but also affects cluster stability.

[0003] To address the storage bottleneck of a single NameNode, the Hadoop open source community proposed the HDFS Federation solution, which involves enabling multiple NameNode services within a Hadoop cluster, each capable of storing files. However, due to the original architectural design of the open source community, upper-layer components such as Hive cannot utilize multiple NameNodes for balanced data storage and analysis. Even with HDFS (Hadoop Distributed System) Federation enabled in a cluster, Hive data is still stored on a single NameNode path, resulting in uneven data storage across the cluster's multiple NameNodes. Hive's data processing is still limited by the performance bottleneck of a single NameNode.

[0004] Currently, all efforts are being made to alleviate the data processing bottleneck of Hive under a single NameNode through tuning. There is no efficient solution for Hive to use multiple NameNodes for balanced data storage. This shows that the existing method of storing Hive data on multiple NameNodes has the technical problem of unbalanced storage. Summary of the Invention

[0005] In view of this, the present invention provides a determination method, a server, and a computer storage medium to solve the technical problem of unbalanced storage of Hive data in multiple NameNodes in the prior art.

[0006] The technical solution of the present invention is achieved as follows:

[0007] In a first aspect, an embodiment of the present invention provides a determination method, which is applied to a server based on the Hive architecture, comprising:

[0008] Parse the acquired Hive SQL task to obtain output data of the table data in the Hive SQL task;

[0009] Select a NameNode from the NameNode set according to a preset NameNode selection strategy, wherein the NameNode selection strategy is a strategy of selecting NameNodes in ascending order of NameNode load.

[0010] An accessible NameNode among the selected NameNodes is determined as the NameNode for outputting the data.

[0011] In the above method, the method further comprises:

[0012] Receive Hive SQL;

[0013] Reconstruct the received Hive SQL task to obtain a reconstructed Hive SQL task;

[0014] The reconstructed Hive SQL task is determined as the obtained Hive SQL task.

[0015] In the above method, reconstructing the received Hive SQL task to obtain the reconstructed Hive SQL task includes:

[0016] When the reconstruction type of the received Hive SQL task is insert type, the storage form of the table data in the received Hive SQL task is a partitioned table, and the received Hive SQL task does not have a statement to create a partition, create and add a partition before the insert statement in the received Hive SQL task, and specify the path of the NameNode where the created partition is used to store the output data, so as to obtain a reconstructed Hive SQL task.

[0017] In the above method, reconstructing the received Hive SQL task to obtain the reconstructed Hive SQL task includes:

[0018] When the reconstruction type of the received Hive SQL task is a selection type, a configured temporary directory is added to the received Hive SQL task, and a path of a NameNode for storing the output data in the temporary directory is specified to obtain a reconstructed Hive SQL task.

[0019] In the above method, reconstructing the received Hive SQL task to obtain the reconstructed Hive SQL task includes:

[0020] When the reconstruction type of the received Hive SQL task is creation and the NameNode path is not specified in the received Hive SQL task, specify the NameNode path of the partition created in the received Hive SQL task for storing table data to obtain the reconstructed Hive SQL task.

[0021] In the above method, selecting a NameNode from a NameNode set according to the NameNode selection policy includes:

[0022] Obtain the spatial information of each NameNode in the NameNode set;

[0023] Calculate the remaining storage space of each NameNode;

[0024] A NameNode is selected from the NameNode set according to the remaining storage space of each NameNode in descending order.

[0025] In the above method, selecting a NameNode from a NameNode set according to the NameNode selection policy includes:

[0026] Obtaining remote call indicator information for each NameNode in the NameNode set;

[0027] Selecting a storage queue length of a remote procedure call of each NameNode from the indicator information of the remote call of each NameNode;

[0028] A NameNode is selected from the NameNode set according to the storage queue length of each NameNode's remote call from short to long.

[0029] In the above method, the method further comprises:

[0030] When querying the table data in the Hive SQL task, and there are at least two NameNode paths for the table data in the Hive SQL task, perform path verification on the at least two NameNode paths for the table data in the Hive SQL task;

[0031] And / or, when querying the table data in the Hive SQL task and path encryption is set for the temporary file of the table data in the Hive SQL task, path encryption is performed on at least two NameNode paths of the temporary file of the table data in the Hive SQL task;

[0032] And / or, when querying the table data in the Hive SQL task, obtaining metadata of the table data stored in the partitioned table in the Hive SQL task and the paths of all partitioned tables in the Hive SQL task in at least two NameNodes, so as to scan the paths of all partitioned tables in the at least two NameNodes.

[0033] In the above method, the method further comprises:

[0034] When storing output data of table data in the Hive SQL task and a partition table exists in the Hive SQL task, the storage path of the output data in the partition table is allowed to be inconsistent with the root directory of the table data in the Hive SQL task;

[0035] And / or, when the output data of the table data in the Hive SQL task is stored and the data in the partition table in the Hive SQL task has been generated, the storage path of the temporary data in the partition table is allowed to be inconsistent with the root directory of the temporary data of the table data in the Hive SQL task.

[0036] In a second aspect, the present invention provides a server, wherein the server is a server based on the Hive architecture, comprising:

[0037] A parsing module is used to parse the acquired Hive SQL task to obtain output data of the table data in the Hive SQL task;

[0038] A selection module is used to select a NameNode from the NameNode set according to a preset NameNode selection strategy; wherein the NameNode selection strategy is a strategy of selecting a NameNode according to the degree of NameNode load from low to high;

[0039] The determination module is used to determine an accessible NameNode among the selected NameNodes as the NameNode for outputting the data.

[0040] In the above server, the above device is further used for:

[0041] Receive Hive SQL;

[0042] Reconstruct the received Hive SQL task to obtain a reconstructed Hive SQL task;

[0043] The reconstructed Hive SQL task is determined as the obtained Hive SQL task.

[0044] In the above device, the above device reconstructs the received Hive SQL task to obtain the reconstructed Hive SQL task, including:

[0045] When the reconstruction type of the received Hive SQL task is insert type, the storage form of the table data in the received Hive SQL task is a partitioned table, and the received Hive SQL task does not have a statement to create a partition, create and add a partition before the insert statement in the received Hive SQL task, and specify the path of the NameNode where the created partition is used to store the output data, so as to obtain a reconstructed Hive SQL task.

[0046] In the server, the server reconstructs the received Hive SQL task to obtain the reconstructed Hive SQL task, including:

[0047] When the reconstruction type of the received Hive SQL task is a selection type, a configured temporary directory is added to the received Hive SQL task, and a path of a NameNode for storing the output data in the temporary directory is specified to obtain a reconstructed Hive SQL task.

[0048] In the server, the server reconstructs the received Hive SQL task to obtain the reconstructed Hive SQL task, including:

[0049] When the reconstruction type of the received Hive SQL task is creation and the NameNode path is not specified in the received Hive SQL task, specify the NameNode path of the partition created in the received Hive SQL task for storing table data to obtain the reconstructed Hive SQL task.

[0050] In the above server, select modules specifically for:

[0051] Obtain the spatial information of each NameNode in the NameNode set;

[0052] Calculate the remaining storage space of each NameNode;

[0053] A NameNode is selected from the NameNode set according to the remaining storage space of each NameNode in descending order.

[0054] In the above server, select modules specifically for:

[0055] Obtaining remote call indicator information for each NameNode in the NameNode set;

[0056] Selecting a storage queue length of a remote procedure call of each NameNode from the indicator information of the remote call of each NameNode;

[0057] A NameNode is selected from the NameNode set according to the storage queue length of each NameNode's remote call from short to long.

[0058] In the above server, the above server is also used to:

[0059] When querying the table data in the Hive SQL task, and there are at least two NameNode paths for the table data in the Hive SQL task, perform path verification on the at least two NameNode paths for the table data in the Hive SQL task;

[0060] And / or, when querying the table data in the Hive SQL task and path encryption is set for the temporary file of the table data in the Hive SQL task, path encryption is performed on at least two NameNode paths of the temporary file of the table data in the Hive SQL task;

[0061] And / or, when querying the table data in the Hive SQL task, obtaining metadata of the table data stored in the partitioned table in the Hive SQL task and the paths of all partitioned tables in the Hive SQL task in at least two NameNodes, so as to scan the paths of all partitioned tables in the at least two NameNodes.

[0062] In the above server, the above server is also used to:

[0063] When storing output data of table data in the Hive SQL task and a partition table exists in the Hive SQL task, the storage path of the output data in the partition table is allowed to be inconsistent with the root directory of the table data in the Hive SQL task;

[0064] And / or, when the output data of the table data in the Hive SQL task is stored and the data in the partition table in the Hive SQL task has been generated, the storage path of the temporary data in the partition table is allowed to be inconsistent with the root directory of the temporary data of the table data in the Hive SQL task.

[0065] In a third aspect, an embodiment of the present invention further provides a server, comprising: a processor and a storage medium storing instructions executable by the processor, wherein the storage medium relies on the processor to perform operations through a communication bus, and when the instructions are executed by the processor, the determination method described in one or more of the above embodiments is executed.

[0066] In a fourth aspect, an embodiment of the present invention provides a computer storage medium storing executable instructions. When the executable instructions are executed by one or more processors, the processors execute the determination method described in one or more of the above embodiments.

[0067] The present invention provides a determination method, server, and computer storage medium. The method is applied to a server based on a Hive architecture and includes: parsing an acquired Hive SQL task to obtain output data of table data in the Hive SQL task, and selecting a NameNode from a NameNode set according to a preset NameNode selection policy. The NameNode selection policy selects NameNodes according to NameNode load levels from low to high, and determines an accessible NameNode among the selected NameNodes as the NameNode for the output data. That is, in an embodiment of the present invention, the output data to be stored is obtained by parsing the acquired Hive SQL task, and after selecting a NameNode according to the preset NameNode selection policy, a NameNode with access rights among the selected NameNodes is determined as the NameNode for the output data. Because the preset NameNode selection policy selects NameNodes according to NameNode load levels from low to high, that is, the load of each NameNode is taken into account, and a NameNode for storing the output data is selected according to the load of each NameNode. This avoids NameNode load imbalance when multiple NameNodes store Hive data, thereby achieving balanced storage when multiple NameNodes store Hive data. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 Schematic diagram of a flow chart of an optional determination method in an embodiment of the present invention;

[0069] Figure 2 A flowchart of semantic parsing of an optional Hive SQL task provided by an embodiment of the present invention;

[0070] Figure 3 A schematic diagram of an optional Hive SQL task processing structure provided by an embodiment of the present invention;

[0071] Figure 4 A schematic diagram of an optional process for reconstructing a Hive SQL task provided by an embodiment of the present invention;

[0072] Figure 5 A schematic diagram of an optional NameNode performance analysis process provided in an embodiment of the present invention;

[0073] Figure 6 A schematic diagram of the structure of an optional server provided in an embodiment of the present invention;

[0074] Figure 7 A schematic structural diagram of another optional server provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0075] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention.

[0076] Example 1

[0077] The embodiment of the present invention provides a determination method. Figure 1 FIG. 1 is a flow chart of an optional determination method in an embodiment of the present invention, such as Figure 1 As shown, the determination method may include:

[0078] S101: Parse the acquired Hive SQL task to obtain output data of the table data in the Hive SQL task;

[0079] Currently, to alleviate the storage imbalance of the NameNode, any of the following solutions can usually be adopted: First, use Hive's large and small file merge function to merge multiple small files on the NameNode into a large file, thereby alleviating the response delay problem caused by the excessive number of files on the NameNode; second, close or shorten the cycle of storing historical deleted data in the NameNode's trash bin to increase the storage capacity of new Hive data; third, when the underlying NameNode storage reaches a bottleneck, create a new Hadoop cluster and store Hive data in the new cluster.

[0080] However, the above three solutions are all tuning solutions centered around a single NameNode storage, and these solutions also bring other negative impacts and cannot fundamentally solve the technical problem of storage imbalance. To solve the problem of data imbalance when multiple NameNodes store data, an embodiment of the present invention provides a determination method. First, the obtained Hive SQL task is parsed to obtain the output data of the table data in the Hive SQL task.

[0081] For Hive SQL tasks, including the path of the NameNode used to store table data, when the Hive SQL task is obtained and the Hive SQL task is parsed, the path of the NameNode of the table data in the Hive SQL task is first queried, that is, the path of the NameNode of the table data in the Hive SQL task is obtained, and then the table data is obtained according to the path of the NameNode of the table data, and the output data is calculated according to the table data. In this embodiment of the present invention, the NameNode for output data is determined.

[0082] In order to utilize multiple NameNodes to store output data, in an optional embodiment, the above method may further include:

[0083] Receive Hive SQL;

[0084] Reconstruct the received Hive SQL task to obtain a reconstructed Hive SQL task;

[0085] The reconstructed Hive SQL task is determined as the obtained Hive SQL task.

[0086] Specifically, since the existing ones all store output data in a single NameNode, it is necessary to reconstruct the Hive SQL task to determine the area of ​​the NameNode path used to store the output data or the area of ​​the NameNode path of the table data. Finally, the reconstructed Hive SQL task is determined as the obtained Hive SQL to implement the parsing of the obtained Hive SQL task and obtain the output data of the table data and the area of ​​the NameNode path for storing the output data.

[0087] Then, after calculating the area of ​​the output data and the path to the NameNode storing the output data, the server can find the NameNode of the output data according to the path to the NameNode of the output data, and store the output data in the NameNode of the output data.

[0088] In order to reconstruct Hive SQL tasks, Hive SQL tasks can generally be divided into three reconstruction types, wherein the three reconstruction types may include insert type (INSERT), select type (SELECT) and create type (CREATE). Different reconstructions are performed on Hive SQL tasks for different types. In an optional embodiment, the received Hive SQL task is reconstructed to obtain a reconstructed Hive SQL task, including:

[0089] When the reconstruction type of the received Hive SQL task is insert, the storage format of the table data in the received Hive SQL task is a partitioned table, and the received Hive SQL task does not have a statement to create partitions, create and add partitions before the insert statement in the received Hive SQL task, and specify the path of the NameNode where the created partition is used to store output data to obtain the reconstructed Hive SQL task.

[0090] Specifically, first determine the reconstruction type of the Hive SQL task. When the reconstruction type of the Hive SQL task is determined to be the insert type, you need to continue to determine whether the storage format of the table data in the Hive SQL task is a partitioned table. When the storage format of the table data in the Hive SQL task is determined to be a partitioned table, continue to determine whether there is a partition creation statement in the Hive SQL task. If there is no partition creation statement in the Hive SQL task, reconstruct the Hive SQL task.

[0091] When refactoring a Hive SQL task, the main step is to add the created partition before the insert statement and specify the path of the NameNode where the created partition is used to store the output data. In other words, before the insert statement, the partition with the path of the NameNode where the output data is stored is added. This allows the Hive SQL task to be refactored, resulting in a refactored Hive SQL task.

[0092] In addition, when the reconstruction type of the Hive SQL task is determined to be the insert type, and the storage format of the table data in the Hive SQL task is determined not to be a partitioned table, the Hive SQL task is not reconstructed. If it is determined that there is a statement to create partitions in the Hive SQL task, the Hive SQL task is not reconstructed either.

[0093] For the case where the reconstruction type of the Hive SQL task is a selection type, in an optional embodiment, the received Hive SQL task is reconstructed to obtain a reconstructed Hive SQL task, including:

[0094] When the reconstruction type of the received Hive SQL task is selection type, add the configured temporary directory to the received Hive SQL task and specify the path of the NameNode where the temporary directory is used to store output data to obtain the reconstructed Hive SQL task.

[0095] Specifically, when it is determined that the reconstruction type of the Hive SQL task is the selection type, the Hive SQL task is reconstructed. The reconstruction of the Hive SQL task includes configuring a temporary directory, which is used to store the path of the NameNode for output data. Finally, the configured temporary directory for the path of the NameNode for storing output data is added to the Hive SQL task to obtain the reconstructed Hive SQL task.

[0096] When the reconstruction type of the Hive SQL task is neither an insert type nor a select type, in an optional embodiment, the received Hive SQL task is reconstructed to obtain a reconstructed Hive SQL task, including:

[0097] When the reconstruction type of the received Hive SQL task is creation and the NameNode path is not specified in the received Hive SQL task, specify the NameNode path of the partition created in the received Hive SQL task for storing table data to obtain the reconstructed Hive SQL task.

[0098] Specifically, when it is determined that the type of Hive SQL task is the creation type, it is further determined whether there is a specified NameNode path in the Hive SQL task. In other words, whether there is a partition with a specified NameNode path for storing table data in the Hive SQL task of this creation type? If not, the Hive SQL task needs to be reconstructed to specify the partition created in the Hive SQL task for the NameNode path for storing table data.

[0099] In addition, if the NameNode path is specified in the Hive SQL task, the Hive SQL task is not rebuilt.

[0100] S102: Select a NameNode from the NameNode set according to a preset NameNode selection policy;

[0101] In order to store the output data, it is necessary to select a suitable NameNode for the output data to achieve balanced storage of the NameNode. Here, a NameNode selection strategy is pre-set, where the NameNode selection strategy is to select the NameNode from low to high according to the NameNode load level; that is, the NameNode is selected for the output data based on the NameNode load level. In this way, the NameNode with the lowest load level is selected as the NameNode for the output data, which can achieve balanced storage when multiple NameNodes store data.

[0102] It should be noted that the load level can be calculated using multiple indicators. For example, the load level of the NameNode can be determined by calculating the remaining storage space of the NameNode, or by the length of the storage queue of the remote call. Here, the embodiment of the present invention does not make specific limitations on this.

[0103] In order to determine the load level of the NameNode by calculating the remaining storage space of the NameNode, in an optional embodiment, S102 may include:

[0104] Get the space information of each NameNode in the NameNode set;

[0105] Calculate the remaining storage space of each NameNode;

[0106] Select a NameNode from the NameNode set based on the remaining storage space of each NameNode, from most to least.

[0107] Specifically, first obtain the space information of each NameNode in the NameNode set, then calculate the remaining storage space of each NameNode based on the space information of each NameNode, and then select the NameNode from the NameNode set in descending order of the calculated remaining storage space of each NameNode.

[0108] That is, the more remaining storage space a NameNode has, the lower its load level is, and the less remaining storage space a NameNode has, the higher its load level is. Here, NameNodes are selected from low to high load levels so that the selected NameNodes have as low a load as possible. This way, balanced storage is achieved when multiple NameNodes store data.

[0109] In order to determine the load level of the NameNode by using the storage queue length of the remote call, in an optional embodiment, S102 may include:

[0110] Get the remote call metrics of each NameNode in the NameNode set;

[0111] From the remote call indicator information of each NameNode, select the storage queue length of each NameNode's remote procedure call;

[0112] Select a NameNode from the NameNode set based on the storage queue length of each NameNode's remote call from short to long.

[0113] Specifically, first obtain the indicator information of the remote procedure call of each NameNode in the NameNode set, then select the storage queue length of the remote procedure call of each NameNode based on the indicator information of the remote procedure call of each NameNode, and then select the NameNode from the NameNode set in the order of the storage queue length of the remote call of each NameNode from short to long.

[0114] That is, the shorter the length of the remote call storage queue, the lower the load level of the NameNode, and the longer the length of the remote call storage queue, the higher the load level of the NameNode. Here, the NameNode is selected from the lowest load level to the highest load level, so that the selected NameNode has the lowest load level possible. In this way, balanced storage is achieved when multiple NameNodes store data.

[0115] S103: Determine an accessible NameNode among the selected NameNodes as the NameNode for outputting data.

[0116] After determining the output data and selecting the NameNode, you need to verify the row access permissions of the selected NameNode. If the selected NameNode is accessible to the user, it can be used as the NameNode for output data. If the selected NameNode is inaccessible to the user, it cannot be used as the NameNode for output data to prevent the inaccessible NameNode from causing storage problems during storage.

[0117] In addition, storing output data in different NameNodes can avoid scenarios where some NameNodes are overloaded with data and the cluster becomes unstable. However, when storing data on multiple NameNodes, semantic parsing anomalies in Hive SQL tasks often occur. In an optional embodiment, the above method may further include:

[0118] When querying table data in a Hive SQL task, if there are at least two NameNode paths for the table data in the Hive SQL task, perform path verification on at least two NameNode paths for the table data in the Hive SQL task.

[0119] And / or, when querying table data in a Hive SQL task and path encryption is set for the temporary files of the table data in the Hive SQL task, encrypt the paths of at least two NameNodes of the temporary files of the table data in the Hive SQL task;

[0120] And / or, when querying table data in a Hive SQL task, metadata of the table data stored in a partitioned table in the Hive SQL task and paths of all partitioned tables in the Hive SQL task on at least two NameNodes are obtained, so as to scan the paths of all partitioned tables on at least two NameNodes.

[0121] Specifically, the abnormal problems that exist in the semantic parsing stage of open source Hive can be divided into abnormal situations that occur during querying and abnormal situations that occur during storage. For abnormal situations that occur during querying, the main focus is to improve the logical execution plan stage of the Hive semantic parsing process.

[0122] When querying table data in a Hive SQL task, if there are at least two NameNode paths for the table data in the Hive SQL task, indicating that the table data is stored in different NameNodes, the existing semantic parsing only supports path verification for a single NameNode. Therefore, path verification is performed on at least two NameNode paths for the table data in the Hive SQL task. That is, Hive first removes the verification of the input path for a single NameNode and adds multi-path verification logic to allow the input path of the Hive table data to come from multiple NameNodes.

[0123] When querying table data in a Hive SQL task, and path encryption is set for the temporary file of the table data in the Hive SQL task, since the existing method only encrypts the path of a single NameNode, here, the paths of at least two NameNodes of the temporary file of the table data in the Hive SQL task are encrypted. That is, the table data is allowed to be output to different encrypted NameNode paths.

[0124] When querying table data in a Hive SQL task, first obtain the metadata of the table data. Then, use the metadata to obtain the paths of all partitioned tables on at least two NameNodes. This allows you to scan the paths of all partitioned tables on at least two NameNodes, facilitating queries on table data on paths to multiple NameNodes.

[0125] Regarding abnormal situations that occur during storage, in an optional embodiment, the above method may further include:

[0126] When storing output data of table data in a Hive SQL task and a partitioned table exists in the task, the storage path of the output data in the partitioned table is allowed to be inconsistent with the root directory of the table data in the Hive SQL task.

[0127] And / or, when storing output data of table data in a Hive SQL task and data in a partitioned table in the Hive SQL task has been generated, the storage path of temporary data in the partitioned table is allowed to be inconsistent with the root directory of temporary data of the table data in the Hive SQL task.

[0128] Specifically, when storing output data, for a single NameNode path, the storage path of the output data in the partition table is not allowed to be inconsistent with the root directory of the table data, nor is the storage path of the temporary data in the partition table allowed to be inconsistent with the root directory of the temporary data of the table data. However, for the paths of multiple NameNodes, abnormal situations may occur.

[0129] To prevent abnormal situations, when storing the output data of table data in a Hive SQL task and a partitioned table exists in the Hive SQL task, the storage path of the output data in the partitioned table is allowed to be inconsistent with the root directory of the table data in the Hive SQL task; and / or, when storing the output data of table data in a Hive SQL task and the data in the partitioned table in the Hive SQL task has been generated, the storage path of the temporary data in the partitioned table is allowed to be inconsistent with the root directory of the temporary data of the table data in the HiveSQL task. In this way, the output data can be ultimately written to different NameNodes.

[0130] For example, Figure 2 A flowchart of semantic parsing of an optional Hive SQL task provided by an embodiment of the present invention is shown as follows: Figure 2 As shown in the figure, Hive SQL tasks are all in the semantic parsing stage, which includes four stages: Abstract Syntax Tree (Abstract Syntax Tree), Logical Plan (Logical Execution Plan), Optimized Logical Plan (Optimized Logical Execution Plan), and Physical Plan (Physical Execution Plan). To address the problems existing in the semantic parsing stage of open source Hive, the SQL parsing source code is optimized in two specific aspects:

[0131] On the one hand, if the table data queried by Hive comes from multiple NameNode paths, optimize the Logical plan stage in the Hive semantic parsing process.

[0132] Hive removes verification of a single NameNode input path and adds multi-path verification logic to allow Hive table data input paths to come from multiple NameNodes. Hive also removes verification of the encryption space for temporary files. That is, if the HDFS path encryption is set for the path of the Hive data temporary file, Hive table data can be output to different encrypted NameNode paths.

[0133] Added multiple data input formats in Hive, such as OrcInputFormat and RcFileInputFormat, to support input acquisition logic for multiple NameNode data sources, allowing Hive tasks to obtain table data from different NameNodes. Added multiple data output formats in Hive, such as OrcOutputFormat and RcFileOutputFormat, to support data writing logic for different NameNodes, allowing temporary data output in Hive tasks to be stored in different NameNodes.

[0134] Optimizes the partition table data scanning logic. By querying the Hive partition table metadata, the specific paths of all partitions on different NameNodes are obtained. This increases the scanning operation of partition paths on different NameNodes when reading data from the Hive partition table.

[0135] On the other hand, if the partitions of the Hive table need to be stored in different NameNode paths, optimize the Optimized Logical plan stage during the Hive semantic parsing process.

[0136] First, determine whether the Hive table in the generated logical plan has partitions. If partitions exist, the HDFS Schema path of the table data can be inconsistent with the parent path (that is, the path of the table data). This allows partitions to be customized to store different NameNode paths.

[0137] When partition data is generated, the temporary data storage directory of the partition path is allowed to be inconsistent with the temporary storage directory of the table data. This optimizes the partition data Move logic and ensures that the partition data is ultimately written to different NameNode paths.

[0138] Through these two aspects of Hive source code optimization, Hive supports a single SQL statement accessing data on different NameNode paths, and allows the same Hive table partitions to be stored on different NameNode paths. This allows Hive to fully utilize multiple NameNode paths for data storage and access, resolving the pain point of excessive storage pressure on a single NameNode.

[0139] The following examples are used to describe the determination method described in one or more of the above embodiments.

[0140] Figure 3 A schematic diagram of an optional Hive SQL task processing structure provided by an embodiment of the present invention is as follows: Figure 3 As shown in the figure, the Hive architecture server can be divided into the following parts when processing Hive SQL tasks:

[0141] Client module (Hive Client, JDBC): Users submit HiveSQL tasks through the Hive Client or JDBC client.

[0142] HiveServer2 Hook plug-in module: parses Hive SQL tasks submitted by users and specifies a NameNode with lower load for storing the output data in Hive SQL.

[0143] Analysis Server NameNode performance analysis module: performs comprehensive analysis on multiple NameNodes of HDFS Federation and returns the NameNode with low load to the HiveServer2 Hook plug-in module.

[0144] Ranger authentication module: performs permission verification on multiple NameNode paths that need to be accessed in Hive SQL.

[0145] The underlying HDFS Federation storage module consists of multiple NameNodes, each of which has an independent NameSpace (NS) storage namespace, such as NS1, NS2, and NSn.

[0146] Tez job computing module: computes SQL tasks and stores output data in the NameNode with low load.

[0147] Figure 4 A schematic diagram of an optional process for reconstructing a Hive SQL task provided by an embodiment of the present invention is as follows: Figure 4 As shown in the figure, the SQL reconstruction plug-in is responsible for assigning the output data of the Hive SQL job submitted by the user to the NameNode with a lower load, so that the Hive data is evenly stored on different NameNode paths and the load of each NameNode is balanced. The above reconstruction may include:

[0148] S401: Hive Client / JDBC receives Hive SQL;

[0149] Specifically, users submit Hive SQL to Hive Client / JDBC, which then submits the Hive SQL to the HiveServer2 Hook service. Table data in Hive SQL comes from multiple NameNode storage paths. The Hive Server2 Hook plug-in intercepts the user's Hive SQL tasks and classifies and reconstructs them.

[0150] There are three types of SQL refactoring: query (SELECT), insert (INSERT), and create (CREATE). The following describes the three types of SQL refactoring strategies in steps.

[0151] S402: Hook intercepts Hive SQL;

[0152] S403: Determine whether the Hive SQL statement is an INSERT statement. If so, execute S404; otherwise, execute S412.

[0153] S404: Determine whether the storage format of the table data in Hive SQL is a partitioned table. If so, execute S405; otherwise, execute S412;

[0154] S405: Determine whether there is a statement to create partitions in Hive SQL. If not, execute S406:; otherwise, execute S412;

[0155] S406: Refactor Hive SQL;

[0156] Specifically, a statement for creating a partition is added before the INSERT statement of Hive SQL, and a NameNode with a lower load is specified in the partition creation. The NameNode with a lower load is returned by the Analysis Server NameNode performance analysis module; S412 is executed;

[0157] S407: Determine whether the Hive SQL statement is SELECT. If so, execute S408; otherwise, execute S409.

[0158] S408: Refactor Hive SQL;

[0159] Specifically, add a configuration to specify that the temporary directory for Hive SQL output results is stored in a NameNode path with a lower load, wherein the NameNode with a lower load is returned by the Analysis Server NameNode performance analysis module; execute S412;

[0160] S409: Determine whether the Hive SQL statement is CREATE. If so, execute S410; otherwise, execute S412.

[0161] S410: Determine whether a specific NameNode storage path has been specified in the Hive SQL submitted by the user. If so, execute S; otherwise, execute S411.

[0162] S411: Refactoring Hive SQL;

[0163] Specifically, the designated table in Hive SQL is stored on a NameNode path with a lower load, wherein the NameNode with a lower load is returned by the Analysis Server NameNode performance analysis module; and S412 is executed.

[0164] S412: Release the Hive SQL and submit the task.

[0165] Figure 5 A schematic diagram of an optional NameNode performance analysis process provided by an embodiment of the present invention is as follows: Figure 5 As shown, the above NameNode performance analysis method may include the following steps:

[0166] S501: The Analysis Server NameNode performance analysis module obtains whether the NameNode selection strategy passed in by the HiveSerer2 Hook is a strategy based on the remaining storage space. If so, execute S502; otherwise, execute S504;

[0167] S502: The Analysis Server NameNode performance analysis module calculates the remaining storage space of the NameNode path;

[0168] Specifically, a request to calculate the remaining storage space of a specified directory is sent to all NameNodes to obtain the remaining storage space information of all NameNodes for the specified path;

[0169] S503: Select NameNode; execute S10;

[0170] Specifically, the storage of all NameNodes in the specified path is compared, and the NameNode with the most remaining storage space in the specified path is selected.

[0171] S504: Determine whether the NameNode selection strategy passed in by HiveSerer2 Hook is the remote procedure call (RPC) request minimum strategy. If so, proceed to S506; otherwise, proceed to S508;

[0172] S506: Statistics of NameNode RPC indicators; execute S507;

[0173] Specifically, the Analysis Server NameNode performance analysis module obtains the RPC metric information of all NameNodes and filters out the storage queue length (CallQueueLength) information of all NameNodes' RPCs;

[0174] S507: Select NameNode; execute S10;

[0175] Specifically, the NameNode with the least RPC requests is selected based on the CallQueueLength size;

[0176] S508: adopt random strategy;

[0177] S509: Randomly return a NameNode;

[0178] Specifically, the random strategy randomly selects a NameNode from the existing NameNodes in the current cluster and returns it; executes S10;

[0179] S510: Outputs the Nameode with the lowest load.

[0180] Through the above examples, this method can simultaneously utilize multiple NameNodes in a cluster for Hive data storage and query, which not only alleviates the performance bottleneck caused by storing data on a single NameNode, but also improves the data processing scale of Hive and balances the Hive data storage of all NameNodes. By storing Hive data on different NameNodes, this method balances the load of all NameNodes, distributes Hive queries to different NameNodes, improves query return speed, and also speeds up other computing engines such as Presto and Spark. The open source Hive component SQL semantic parsing logic is optimized to enable Hive to support data storage on multiple NameNode paths. A pluggable Hive Hook plug-in and NameNode performance analysis microservice are developed to complete SQL reconstruction and NameNode load analysis without intruding into the Hive code.

[0181] The present invention provides a determination method, which is applied to a server based on a Hive architecture, comprising: parsing an acquired Hive SQL task to obtain output data of table data in the Hive SQL task, and selecting a NameNode from a NameNode set according to a preset NameNode selection policy; wherein the NameNode selection policy is a policy of selecting NameNodes according to NameNode load levels from low to high, and determining an accessible NameNode among the selected NameNodes as the NameNode for the output data; that is, in an embodiment of the present invention, the output data to be stored is obtained by parsing the acquired Hive SQL task, and after selecting a NameNode according to the preset NameNode selection policy, a NameNode with access rights among the selected NameNodes is determined as the NameNode for the output data. Since the preset NameNode selection policy is a policy of selecting according to NameNode load levels from low to high, that is, the load of each NameNode is taken into account, and a NameNode for storing the output data is selected according to the load of each NameNode, load imbalance of the NameNodes is avoided when multiple NameNodes store Hive data, thereby achieving balanced storage when multiple NameNodes store Hive data.

[0182] Example 2

[0183] Based on the same inventive concept, an embodiment of the present invention further provides a server, which is a server based on the Hive architecture. Figure 6 A schematic diagram of the structure of an optional server provided in an embodiment of the present invention is shown in FIG. Figure 6 As shown, the server may include:

[0184] The parsing module 61 is used to parse the acquired Hive SQL task to obtain output data of the table data in the Hive SQL task;

[0185] The selection module 62 is configured to select a NameNode from the NameNode set according to a preset NameNode selection strategy, wherein the NameNode selection strategy is a strategy of selecting a NameNode in ascending order of NameNode load.

[0186] The determination module 63 is configured to determine an accessible NameNode among the selected NameNodes as the NameNode for outputting data.

[0187] In an optional embodiment, the server is further configured to:

[0188] Receive Hive SQL;

[0189] Reconstruct the received Hive SQL task to obtain a reconstructed Hive SQL task;

[0190] The reconstructed Hive SQL task is determined as the obtained Hive SQL task.

[0191] In an optional embodiment, the server reconstructs the received Hive SQL task to obtain a reconstructed Hive SQL task including:

[0192] When the reconstruction type of the received Hive SQL task is insert, the storage format of the table data in the received Hive SQL task is a partitioned table, and the received Hive SQL task does not have a statement to create partitions, create and add partitions before the insert statement in the received Hive SQL task, and specify the path of the NameNode where the created partition is used to store output data to obtain the reconstructed Hive SQL task.

[0193] In an optional embodiment, the server reconstructs the received Hive SQL task to obtain a reconstructed Hive SQL task including:

[0194] When the reconstruction type of the received Hive SQL task is a selection type, the configured temporary directory is added to the received Hive SQL task, and the temporary directory is used to specify a path of the NameNode for storing the output data, so as to obtain a reconstructed Hive SQL task.

[0195] In an optional embodiment, the server reconstructs the received Hive SQL task to obtain a reconstructed Hive SQL task including:

[0196] When the reconstruction type of the received Hive SQL task is creation and the NameNode path is not specified in the received Hive SQL task, specify the NameNode path of the partition created in the received Hive SQL task for storing table data to obtain the reconstructed Hive SQL task.

[0197] In an optional embodiment, the selection module 62 is specifically configured to:

[0198] Get the space information of each NameNode in the NameNode set;

[0199] Calculate the remaining storage space of each NameNode;

[0200] Select a NameNode from the NameNode set based on the remaining storage space of each NameNode, from most to least.

[0201] In an optional embodiment, the selection module 62 is specifically configured to:

[0202] Get the remote call metrics of each NameNode in the NameNode set;

[0203] From the remote call indicator information of each NameNode, select the storage queue length of each NameNode's remote procedure call;

[0204] Select a NameNode from the NameNode set based on the storage queue length of each NameNode's remote call from short to long.

[0205] In an optional embodiment, the server is further configured to:

[0206] When querying table data in a Hive SQL task, if there are at least two NameNode paths for the table data in the Hive SQL task, perform path verification on at least two NameNode paths for the table data in the Hive SQL task.

[0207] And / or, when querying table data in a Hive SQL task and path encryption is set for the temporary files of the table data in the Hive SQL task, encrypt the paths of at least two NameNodes of the temporary files of the table data in the Hive SQL task;

[0208] And / or, when querying table data in a Hive SQL task, metadata of the table data stored in a partitioned table in the Hive SQL task and paths of all partitioned tables in the Hive SQL task on at least two NameNodes are obtained, so as to scan the paths of all partitioned tables on at least two NameNodes.

[0209] In an optional embodiment, the server is further configured to:

[0210] When storing output data of table data in a Hive SQL task and a partitioned table exists in the task, the storage path of the output data in the partitioned table is allowed to be inconsistent with the root directory of the table data in the Hive SQL task.

[0211] And / or, when storing output data of table data in a Hive SQL task and data in a partitioned table in the Hive SQL task has been generated, the storage path of temporary data in the partitioned table is allowed to be inconsistent with the root directory of temporary data of the table data in the Hive SQL task.

[0212] In practical applications, the above-mentioned analysis module 61, selection module 62 and determination module 63 can be implemented by a processor located on the server, specifically a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA).

[0213] Figure 7 A schematic diagram of the structure of another optional server provided in an embodiment of the present invention, such as Figure 7 As shown, an embodiment of the present invention provides a server 700, including:

[0214] A processor 71 and a storage medium 72 storing instructions executable by the processor 71, wherein the storage medium 72 relies on the processor 71 to perform operations through a communication bus 73. When the instructions are executed by the processor 71, the determination method described in the above embodiment 1 is executed.

[0215] It should be noted that in actual application, the various components in the terminal are coupled together through the communication bus 73. It is understood that the communication bus 73 is used to realize the connection and communication between these components. In addition to the data bus, the communication bus 73 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 7 Various buses are labeled as communication buses 73.

[0216] An embodiment of the present invention provides a computer storage medium storing executable instructions. When the executable instructions are executed by one or more processors, the processors execute the determination method described in the first embodiment.

[0217] Among them, the computer-readable storage medium can be a magnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory (Flash Memory), a magnetic surface storage device, an optical disc, or a compact disc read-only memory (CD-ROM) and other memories.

[0218] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0219] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0220] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0221] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0222] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention.

Claims

1. A determination method, characterized in that: The method is applied to a server based on the Hive architecture and includes: Parse the acquired Hive SQL task to obtain output data of the table data in the Hive SQL task; Selecting a NameNode from the NameNode set according to a preset NameNode selection strategy; wherein the NameNode selection strategy is a strategy for selecting NameNodes in ascending order of NameNode load; An accessible NameNode among the selected NameNodes is determined as the NameNode for outputting the data.

2. The method according to claim 1, characterized in that The method further comprises: Receive Hive SQL; Reconstruct the received Hive SQL task to obtain a reconstructed Hive SQL task; The reconstructed Hive SQL task is determined as the obtained Hive SQL task.

3. The method according to claim 2, characterized in that The step of reconstructing the received Hive SQL task to obtain the reconstructed Hive SQL task includes: When the reconstruction type of the received Hive SQL task is insert type, the storage form of the table data in the received Hive SQL task is a partitioned table, and the received Hive SQL task does not have a statement to create a partition, create and add a partition before the insert statement in the received Hive SQL task, and specify the path of the NameNode for storing the output data for the created partition to obtain a reconstructed Hive SQL task.

4. The method according to claim 2, characterized in that The step of reconstructing the received Hive SQL task to obtain the reconstructed Hive SQL task includes: When the reconstruction type of the received Hive SQL task is a selection type, a configured temporary directory is added to the received Hive SQL task, and a path of a NameNode for storing the output data in the temporary directory is specified to obtain a reconstructed Hive SQL task.

5. The method according to claim 2, characterized in that The step of reconstructing the received Hive SQL task to obtain the reconstructed Hive SQL task includes: When the reconstruction type of the received Hive SQL task is creation and the NameNode path is not specified in the received Hive SQL task, specify the NameNode path of the partition created in the received Hive SQL task for storing table data to obtain the reconstructed Hive SQL task.

6. The method according to claim 1, characterized in that The step of selecting a NameNode from a NameNode set according to the NameNode selection strategy includes: Obtain the spatial information of each NameNode in the NameNode set; Calculate the remaining storage space of each NameNode; A NameNode is selected from the NameNode set according to the remaining storage space of each NameNode in descending order.

7. The method according to claim 1, characterized in that The step of selecting a NameNode from a NameNode set according to the NameNode selection strategy includes: Obtaining remote call indicator information for each NameNode in the NameNode set; Selecting a storage queue length of a remote procedure call of each NameNode from the indicator information of the remote call of each NameNode; A NameNode is selected from the NameNode set according to the storage queue length of each NameNode's remote call from short to long.

8. The method according to any one of claims 1 to 7, characterized in that The method further comprises: When querying the table data in the Hive SQL task, and there are at least two NameNode paths for the table data in the Hive SQL task, perform path verification on the at least two NameNode paths for the table data in the Hive SQL task; And / or, when querying the table data in the Hive SQL task and path encryption is set for the temporary file of the table data in the Hive SQL task, path encryption is performed on at least two NameNode paths of the temporary file of the table data in the Hive SQL task; And / or, when querying the table data in the Hive SQL task, obtaining metadata of the table data stored in the partitioned table in the Hive SQL task and the paths of all partitioned tables in the Hive SQL task in at least two NameNodes, so as to scan the paths of all partitioned tables in the at least two NameNodes.

9. The method according to any one of claims 1 to 7, characterized in that The method further comprises: When storing output data of table data in the Hive SQL task and a partition table exists in the Hive SQL task, the storage path of the output data in the partition table is allowed to be inconsistent with the root directory of the table data in the Hive SQL task; And / or, when the output data of the table data in the Hive SQL task is stored and the data in the partition table in the Hive SQL task has been generated, the storage path of the temporary data in the partition table is allowed to be inconsistent with the root directory of the temporary data of the table data in the Hive SQL task.

10. A server, characterized in that: The server is a Hive-based server, including: A parsing module is used to parse the acquired Hive SQL task to obtain output data of the table data in the Hive SQL task; A selection module is used to select a NameNode from the NameNode set according to a preset NameNode selection strategy; wherein the NameNode selection strategy is a strategy of selecting a NameNode according to the degree of NameNode load from low to high; The determination module is used to determine an accessible NameNode among the selected NameNodes as the NameNode for outputting the data.

11. A server, characterized in that: include: A processor and a storage medium storing instructions executable by the processor, wherein the storage medium relies on the processor to perform operations through a communication bus, and when the instructions are executed by the processor, the determination method described in any one of claims 1 to 9 is executed.

12. A computer storage medium, characterized in that Executable instructions are stored, and when the executable instructions are executed by one or more processors, the processors execute the determination method according to any one of claims 1 to 9.

Images