A method to solve the storage pressure of HIVE metadata management nodes
By dynamically allocating resources and separating hot and cold data management, the problems of high metadata storage pressure and slow response speed in Hadoop versions are solved, improving Impala's query response speed and metadata management efficiency.
Patent Information
- Application Number
- CN202211323640.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-10-27
- Publication Date
- 2025-11-14
- Estimated Expiration
- 2042-10-27
AI Technical Summary
The native Hadoop version of NameNode's design limits the scalability of HDFS clusters, resulting in high pressure on metadata storage. Existing solutions such as the Apache Ozone ecosystem suffer from high resource consumption and slow response times.
By parsing business query requests, validating parameter validity, dynamically creating table partitions and maintaining hot and cold data partition queues, using Impala as the interactive query engine, and allocating resources on demand, we can achieve separate management of hot and cold data and reduce metadata memory usage.
It improved the response speed of Impala metadata nodes, resolved the query anomaly caused by insufficient metadata memory, and achieved efficient metadata management.
Smart Images
Figure CN115878678B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of interactive big data query, and in particular relates to a method for solving the storage pressure of HIVE metadata management nodes. Background Technology
[0002] The native Hadoop version (including the CDH 5.14 Hadoop distribution used by the CU system) inherently lacks horizontal scaling due to the NameNode design, thus limiting the size of a single big data HDFS cluster. To address metadata storage pressure, most industry solutions employ a fully compute-storage separated cloud-native architecture to achieve horizontal scaling of metadata to support the storage of hundreds of billions of files or objects, such as the next-generation distributed general-purpose data storage open-source framework Apache Ozone and its ecosystem. However, this requires establishing a large number of partitions, consuming significant resources, resulting in slow response times, and even encountering log query anomalies due to insufficient memory on the metadata node. Therefore, a new method is urgently needed to solve these problems. Summary of the Invention
[0003] To address the above problems, this invention provides the following solution: a method for reducing storage pressure on HIVE metadata management nodes, comprising:
[0004] Based on the business query request, the interface request is parsed to obtain the query conditions, and the validity of the parameters is verified; wherein, the query conditions include at least the data center number and the query time;
[0005] Query memory based on the data center number to obtain a cluster list; obtain a target cluster list based on the query time and the cluster list; read the business cluster information of the target cluster list from memory;
[0006] If the query time is determined to be within a fixed time partition, the parameters of the Impala dynamic data source table query input are assembled, and the local dynamic data source interface is called to operate Impala. First, the current table is refreshed, and then the table query is performed to obtain the query result and encapsulate the query result and return it.
[0007] Preferably, verifying the validity of parameters includes sequentially verifying the field type, data center information, and query time span;
[0008] The process of sequentially validating field types, data center information, and query time span includes,
[0009] The system checks the field type and format for errors. If an error is found, the query result is output directly. Otherwise, the system checks the data center information.
[0010] Verify the data center information to determine if the data center exists. If the data center does not exist, output the query results directly; otherwise, perform a query time span verification.
[0011] The query time span is checked to determine whether it falls within the configured parameter range. If it does not, the query result is output directly; otherwise, a memory query is performed based on the query conditions.
[0012] Preferably, the process of obtaining the target cluster list based on the query time and the cluster list includes:
[0013] Based on the query time, the Nacos business cluster configuration is retrieved. After classifying the cluster modes according to the cluster retention time, the data is queried to obtain the target cluster list.
[0014] Preferably, the Nacos service cluster configuration includes historical cluster and disk array information;
[0015] The cluster mode includes preferred clusters and / or historical clusters.
[0016] Preferably, the dynamic data source adopts a service initialization hungry loading method to first create the data source and manage the data source connection. Subsequent query requests will directly obtain the connection from the singleton connection pool. If adding or modifying the cluster connection configuration after the service starts does not trigger the addition or update of the data source, and the connection cannot be found in the connection pool when a query request arrives, a new connection will be created and placed in the connection pool for later use.
[0017] Preferably, the method further includes: when it is determined that the query time is not within a fixed time partition, using the JDBC driver and query condition encapsulation of the current cluster as input parameters for calling the dynamic partition management method;
[0018] After the dynamic partition management method is called based on the input parameters and returns successfully, the impala table refresh operation is performed.
[0019] Preferably, the implementation process of the dynamic partitioning includes:
[0020] After verifying the input parameters, the distributed lock is enabled, the number of newly created dynamic partition lists in this Redis request is obtained, and it is determined whether the number of new additions is within the total queue configuration threshold. If the threshold is exceeded, an error is returned.
[0021] Within the threshold, check if the number of newly added partitions and the original queue length exceed the queue length. If not, delete the Hive table partition element that has not been used for the longest time based on usage time, and add the newly added partition. Update the score of the partition element in the Redis queue and release the distributed lock.
[0022] Preferably, the dynamic partition further includes dynamic partition initialization; the process of dynamic partition initialization includes:
[0023] Load the local configuration file, check if the configuration exists. If it does not exist, the dynamic partition initialization ends; otherwise, load the Nacos configuration file.
[0024] Check if the configuration exists. If it does not exist, the dynamic partition initialization ends. Otherwise, verify the parameter format of the Nacos business cluster configuration and the dynamic partition configuration. If the filter format is incorrect, the dynamic partition initialization ends. Otherwise, traverse the business cluster list to pre-create and manage dynamic data sources.
[0025] Create a distributed scheduled task that executes a fixed-time dynamic partition management task every hour on the hour, and executes a general method for dynamic partitions at fixed times.
[0026] Create a distributed scheduled task to execute a consistency task for Hive table partitioning and dynamic partition cache queues every hour on the hour, execute a general consistency processing method, and complete the dynamic partition initialization.
[0027] Preferably, the dynamic partition further includes dynamic partition maintenance; the dynamic partition maintenance includes a fixed-length queue cache and a fixed-time queue cache.
[0028] The fixed-length queue cache uses a ZSet data structure;
[0029] The fixed-time queue cache uses a set data structure;
[0030] Both the ZSet and set data structures include cluster ID, table name, partition name, and partition directory.
[0031] The present invention discloses the following technical effects:
[0032] This invention provides a method to solve the storage pressure on HIVE metadata management nodes. Starting from the strategy of dynamically allocating resources on demand, it uses Impala, which is deeply integrated with CDH, as an interactive query engine. After a business query request, table partitions are dynamically created according to the business and query time, and cold and hot data partition queue management is maintained. After the partitions are created, Impala refresh is used to synchronize the Hive metadata partition information of a certain controllable scale of the business table to achieve the final interactive query result. This solution does not require major changes to the original component selection architecture.
[0033] This invention implements a hot and cold data separation management mechanism, encapsulating the logic for reading hot and cold data in query requests, thus saving memory resources occupied by Hive and Impala metadata. It also resolves the issue of insufficient memory in Impala metadata nodes causing log query anomalies and improves the response speed of Impala synchronizing Hive metadata. Attached Figure Description
[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0035] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;
[0036] Figure 2 This is a flowchart illustrating the implementation of dynamic partitioning in an embodiment of the present invention.
[0037] Figure 3 This is a flowchart illustrating the dynamic partition initialization process in an embodiment of the present invention. Detailed Implementation
[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0039] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0040] like Figure 1 As shown, this invention provides a method for solving the storage pressure on HIVE metadata management nodes, including the following steps:
[0041] Based on the business query request, the interface request is parsed to obtain the query conditions, and the validity of the parameters is verified; the query conditions include at least the data center number and the query time.
[0042] Further optimization of the scheme includes verifying the validity of parameters by sequentially checking the field type, data center information, and query time span;
[0043] The process of sequentially validating field types, data center information, and query time span includes,
[0044] The system checks the field type and format for errors. If an error is found, the query result is output directly. Otherwise, the system checks the data center information.
[0045] Verify the data center information to determine if the data center exists. If the data center does not exist, output the query results directly; otherwise, perform a query time span verification.
[0046] The query time span is checked to determine if it falls within the configured parameter range. If it does not, the query result is output directly; otherwise, a memory query is performed based on the query conditions.
[0047] Query memory based on data center number to obtain cluster list; obtain target cluster list based on query time and cluster list; read business cluster information of target cluster list from memory;
[0048] Further optimization of the scheme, the process of obtaining the target cluster list based on query time and cluster list includes,
[0049] The system retrieves the Nacos business cluster configuration based on the query time, categorizes the cluster modes according to the cluster retention time, and then queries the data to obtain a list of target clusters.
[0050] Further optimization of the solution includes the configuration of historical clusters and disk arrays in the Nacos business cluster configuration;
[0051] Cluster modes include preferred clusters and / or historical clusters.
[0052] If the query time is within a fixed time partition, assemble the parameters for the Impala dynamic data source table query input, call the local dynamic data source interface to operate Impala, first perform a refresh operation on the current table, then perform the table query, obtain the query results, and encapsulate and return the query results.
[0053] To further optimize the solution, the dynamic data source adopts a service initialization hungry loading approach to first create the data source and manage the data source connection. Subsequent query requests will directly obtain the connection from the singleton connection pool. If adding or modifying the cluster connection configuration after the service starts does not trigger the addition or update of the data source, a new connection will be created and placed in the connection pool before use when a query request arrives and the connection cannot be found.
[0054] Further optimizations include: when the query time is determined to be outside the fixed time partition, the JDBC driver and query condition encapsulation of the current cluster are used as input parameters for calling the dynamic partition management method;
[0055] After calling the dynamic partition management method based on the input parameters and returning a success, the Impala table refresh operation is performed.
[0056] Further optimize the plan, such as Figure 2 As shown, the implementation process of dynamic partitioning includes:
[0057] After validating the input parameters, enable the distributed lock, obtain the number of newly created dynamic partitions in the Redis request, and determine whether the number of new additions is within the total queue configuration threshold. If it exceeds the threshold, return an error.
[0058] Within the threshold, check if the number of newly added partitions and the original queue length exceed the queue length. If not, delete the Hive table partition element that has not been used for the longest time based on usage time, and add the newly added partition. Update the score of the partition element in the Redis queue and release the distributed lock.
[0059] Further optimize the plan, such as Figure 3 As shown, dynamic partitioning also includes dynamic partition initialization; the process of dynamic partition initialization includes:
[0060] Load the local configuration file, check if the configuration exists. If it does not exist, the dynamic partition initialization ends; otherwise, load the Nacos configuration file.
[0061] Check if the configuration exists. If it does not exist, the dynamic partition initialization ends. Otherwise, verify the parameter format of the Nacos business cluster configuration and the dynamic partition configuration. If the filter format is incorrect, the dynamic partition initialization ends. Otherwise, traverse the business cluster list to pre-create and manage dynamic data sources.
[0062] Create a distributed scheduled task that executes a fixed-time dynamic partition management task every hour on the hour, and executes a general method for dynamic partitions at fixed times.
[0063] Create a distributed scheduled task to execute a consistency task for Hive table partitioning and dynamic partition cache queues every hour on the hour, execute a general consistency processing method, and complete the dynamic partition initialization.
[0064] Further optimization of the scheme includes dynamic partitioning maintenance; dynamic partitioning maintenance includes fixed-length queue caching and fixed-time queue caching.
[0065] Redis dynamic partitioned queues are fixed-length queues using a ZSet data structure. The current cluster key is named zClusterQueue: cluster ID, and elements are table name | partition name | partition directory, such as...
[0066] cu_access_log|idc='GD_ATKJ_IDC',dt=20220606,hour=14|idc=GD_ATKJ_IDC / dt=20220606 / hour=14,score value is the timestamp of the last query update in milliseconds;
[0067] Redis dynamic partitioned queues with fixed time intervals use a set data structure named sClusterFixedQueue: cluster ID. Elements are table name | partition name | partition directory, such as cu_access_log | idc = 'GD_ATKJ_IDC', dt = 20220606, hour = 14 | idc = GD_ATKJ_IDC / dt = 20220606 / hour = 14.
[0068] Example 1
[0069] The method of the present invention for solving the storage pressure of HIVE metadata management nodes includes the following steps:
[0070] 1. Parse the interface request to obtain the query conditions (at least including the data center number and time), and verify the validity of the parameters, including field type, data center information, and whether the query time span is within the configured parameter range (time span verification mainly ensures timely response of API queries and also plays a certain role in protecting the computing resources of big data backend).
[0071] 2. Query memory based on data center number to obtain corresponding data center information and cluster relationships (mainly to obtain a cluster list); further search Nacos business cluster configuration (including historical cluster and disk array information) based on time. Determine the cluster retention time by selecting from preferred cluster, historical cluster, preferred cluster + historical cluster (if the query time spans the current and historical), etc., and query the data in the ordered order.
[0072] 3. Based on the list of clusters that need to be queried from the previous step, read the current business cluster information from memory (Nacos maintains configuration and refreshes dynamically). If the query time is within a fixed time partition, directly query and call step 6.
[0073] 4. If the query time is not within the fixed time partition, the query conditions, such as the JDBC driver and URL of the current cluster (e.g., Hive, Impala), are encapsulated into the following parameters: Hive driver, IP address, port, database, table name, query start time, and end time, which are then used as input parameters for calling the dynamic partition management method.
[0074] 5. After calling the dynamic partition management method and returning a successful result, perform the Impala table refresh operation.
[0075] 6. Assemble parameters for Impala dynamic data source table query input (IP, port, database, table name, etc.), call the local dynamic data source interface to operate Impala, first perform a refresh operation on the current table, and then perform a table query. The query results are encapsulated and returned (the query time period is controlled by the caller).
[0076] The technical solution of this embodiment meets the UD1 log detail query of the enterprise CU control layer system, realizes unified access and management of the interface, supports the business layer without distinguishing the data center, supports the situation where data from one data center is in multiple clusters and disk arrays; because there are many small files (far less than 128M each) in the EU reports of various types of files from multiple provinces in the CU system production line, the current CU system directly writes the original EU side reported files to HDFS to facilitate the traceability of the original file information, resulting in a large number of small files in the database. The overall mechanism of the CU system business log query module, such as access log detail query, is based on creating external tables and partitions in Hive and specifying HDFS paths through location (the storage multi-level path includes partition fields such as data center, date, and hour). The big data detail query uses CDH's Impala (Cloudera's self-developed big data interactive query MPP engine). The Impala Catalog metadata management node updates metadata information by synchronizing Hive metadata. However, in many places on the production line, Impala metadata storage nodes have insufficient memory, resulting in insufficient memory storage of catalog node information and slow updating and loading of metadata information, which leads to abnormal business queries for detail data. This invention solves the problems of insufficient memory in Impala metadata nodes and slow updates and loading of metadata information by dynamically maintaining an appropriate amount of partition information, eliminating the need to create a large number of partitions as before.
[0077] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.
Claims
1. A method for solving the storage pressure on HIVE metadata management nodes, characterized in that, include: Based on the business query request, the interface request is parsed to obtain the query conditions, and the validity of the parameters is verified; wherein, the query conditions include at least the data center number and the query time; Query memory based on the data center number to obtain a cluster list; obtain a target cluster list based on the query time and the cluster list; read the business cluster information of the target cluster list from memory; If the query time is determined to be within a fixed time partition, the parameters of the Impala dynamic data source table query input are assembled, the local dynamic data source interface is called to operate Impala, first the current table is refreshed, and then the table query is executed to obtain the query result and encapsulate the query result and return it. Verifying the validity of parameters includes sequentially verifying the field type, data center information, and query time span; The process of sequentially validating field types, data center information, and query time span includes, The system checks the field type and format for errors. If an error is found, the query result is output directly. Otherwise, the system checks the data center information. Verify the data center information to determine if the data center exists. If the data center does not exist, output the query results directly; otherwise, perform a query time span verification. The query time span is checked to determine whether it falls within the configured parameter range. If it does not, the query result is output directly; otherwise, a memory query is performed based on the query conditions. The process of obtaining the target cluster list based on the query time and the cluster list includes: Based on the query time, the Nacos business cluster configuration is retrieved. After classifying the cluster mode according to the cluster storage time, the data is queried to obtain the target cluster list. The dynamic data source adopts a service initialization hungry loading approach to first create the data source and manage the data source connection. Subsequent query requests will directly obtain the connection from the singleton connection pool. If adding or modifying the cluster connection configuration after the service starts does not trigger the addition or update of the data source, a new connection will be created and placed in the connection pool before use when a query request arrives and the connection cannot be found in the connection pool. The method further includes: If it is determined that the query time is not within the fixed time partition, the JDBC driver and query condition encapsulation of the current cluster are used as input parameters for calling the dynamic partition management method. After the dynamic partition management method is called based on the input parameters and returns successfully, the impala table refresh operation is performed.
2. The method for solving the storage pressure of HIVE metadata management nodes according to claim 1, characterized in that: The Nacos service cluster configuration includes historical cluster and disk array information; The cluster mode includes preferred clusters and / or historical clusters.
3. The method for solving the storage pressure of HIVE metadata management nodes according to claim 1, characterized in that, The implementation process of the dynamic partitioning includes: After verifying the input parameters, the distributed lock is enabled, the number of newly created dynamic partition lists in this Redis request is obtained, and it is determined whether the number of new additions is within the total queue configuration threshold. If the threshold is exceeded, an error is returned. Within the threshold, check if the number of newly added partitions and the original queue length exceed the queue length. If not, delete the Hive table partition element that has not been used for the longest time based on usage time, and add the newly added partition. Update the score of the partition element in the Redis queue and release the distributed lock.
4. The method for solving the storage pressure of HIVE metadata management nodes according to claim 1, characterized in that, The dynamic partitioning also includes dynamic partition initialization; the process of dynamic partition initialization includes: Load the local configuration file, check if the configuration exists. If it does not exist, the dynamic partition initialization ends; otherwise, load the Nacos configuration file. Check if the configuration exists. If it does not exist, the dynamic partition initialization ends. Otherwise, verify the parameter format of the Nacos business cluster configuration and the dynamic partition configuration. If the filter format is incorrect, the dynamic partition initialization ends. Otherwise, traverse the business cluster list to pre-create and manage dynamic data sources. Create a distributed scheduled task that executes a fixed-time dynamic partition management task every hour on the hour, and executes a general method for dynamic partitions at fixed times. Create a distributed scheduled task to execute a consistency task for Hive table partitioning and dynamic partition cache queues every hour on the hour, execute a general consistency processing method, and complete the dynamic partition initialization.
5. The method for solving the storage pressure of HIVE metadata management nodes according to claim 1, characterized in that: The dynamic partitioning also includes dynamic partition maintenance; the dynamic partition maintenance includes a fixed-length queue cache and a fixed-time queue cache.
6. The method for solving the storage pressure of HIVE metadata management nodes according to claim 5, characterized in that: The fixed-length queue cache uses a ZSet data structure; The fixed-time queue cache uses a set data structure; Both the ZSet and set data structures include cluster ID, table name, partition name, and partition directory.
Citation Information
Patent Citations
System of hybrid online analysis and disposal and method for data query
CN108875042A