Query node optimization method, device and equipment of Hbase database and storage medium

Abstract

The application relates to a database query technology and provides a query node optimization method, device and equipment of an Hbase database and a storage medium, wherein an abnormal query node set in all query nodes is determined, an optimization configuration template is configured, if real-time query time of each query node in real-time query is acquired, an initial abnormal query node is determined, a target query node matched with the initial abnormal query node in the abnormal query node set is determined, a target optimization configuration template corresponding to the target node is determined, the target optimization configuration template is dynamically configured in the target query node, an optimized target query node is obtained, a test target time is acquired, if the query time is less than a query threshold, the target optimization configuration template is updated, the corresponding optimization configuration template is dynamically configured for different abnormal query nodes, the abnormal query nodes are directly optimized, the node blocking condition in the real-time query process is reduced, and the data real-time query efficiency is improved.

Description

Technical Field

[0001] This invention relates to database query technology, and more particularly to a method, apparatus, device, and storage medium for optimizing query nodes in an HBase database. Background Technology

[0002] Data platform systems provide data storage, offline data transmission and computation, and occasionally offer real-time query services directly to business needs. Clients requesting real-time queries are development teams working on systems or apps that directly serve end users. Their data query functionality prioritizes real-time performance, requiring millisecond-level response times to complete data queries and return results. Therefore, data platform systems handling real-time queries are typically equipped with databases such as HBase and Clickhouse to ensure query efficiency.

[0003] As a distributed database, HBase's query efficiency is constrained by various factors, such as design flaws in the data itself, the expansion of data volume, and network latency in the query chain, resulting in low real-time query efficiency. Therefore, existing technologies based on HBase databases suffer from low real-time query efficiency, making it imperative to improve real-time query efficiency a pressing issue. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, device, and storage medium for optimizing query nodes in HBase databases to address the aforementioned technical problems and solve the issue of low real-time query efficiency.

[0005] Firstly, a method for optimizing query nodes in an HBase database is provided, the method comprising:

[0006] Based on the historical query times of each query node obtained from the preset query system, a set of abnormal query nodes is determined among all query nodes, and an optimized configuration template is set for each abnormal query node in the set of abnormal query nodes; wherein, each abnormal query node corresponds to at least one optimized configuration template.

[0007] If the real-time query time of each query node in the real-time query is obtained, the initial abnormal query node is determined based on the real-time query time of each query node and the preset query threshold.

[0008] The initial abnormal query node is matched one by one with each query node in the abnormal query node set to determine the query node in the abnormal query node set that matches the initial abnormal query node as the target query node.

[0009] Determine the target optimization configuration template from multiple optimization configuration templates corresponding to the target query node, and dynamically configure the target optimization configuration template on the target query node to obtain the optimized target query node;

[0010] Test the query time of the optimized target query node, and if the query time is less than the query threshold, update the target query node using the target optimization configuration template.

[0011] Secondly, an HBase database query node optimization device is provided, the device comprising:

[0012] The optimized configuration template setting module is used to determine the set of abnormal query nodes among all query nodes based on the historical query time of each query node obtained from the preset query system, and to set an optimized configuration template for each abnormal query node in the set of abnormal query nodes; wherein, each abnormal query node corresponds to at least one optimized configuration template.

[0013] The initial abnormal query node determination module is used to determine the initial abnormal query node based on the real-time query time of each query node and the preset query threshold if the real-time query time of each query node is obtained in the real-time query.

[0014] The matching module is used to match the initial abnormal query node with each query node in the abnormal query node set one by one, and determine the query node in the abnormal query node set that matches the initial abnormal query node as the target query node.

[0015] The dynamic configuration module is used to determine the target optimization configuration template from multiple optimization configuration templates corresponding to the target query node, and dynamically configure the target optimization configuration template on the target query node to obtain the optimized target query node;

[0016] The update module is used to test the query time of the optimized target query node, and update the target query node using the target optimization configuration template if the query time is less than the query threshold.

[0017] Thirdly, embodiments of the present invention provide a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the query node optimization method for the HBase database as described in the first aspect.

[0018] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the query node optimization method for the HBase database as described in the first aspect.

[0019] The advantages of this invention compared to the prior art are:

[0020] Based on the historical query times of each query node obtained from the preset query system, an abnormal query node set is determined among all query nodes. An optimized configuration template is set for each abnormal query node in the abnormal query node set, with each abnormal query node corresponding to at least one optimized configuration template. If the real-time query time of each query node in the real-time query is obtained, an initial abnormal query node is determined based on the real-time query time of each query node and a preset query threshold. The initial abnormal query node is matched one-to-one with each query node in the abnormal query node set to determine the query node in the abnormal query node set that matches the initial abnormal query node as the target query node. A target optimized configuration template is selected from multiple optimized configuration templates corresponding to the target query node. The target optimized configuration template is dynamically configured on the target query node to obtain the optimized target query node. The query time of the optimized target query node is tested, and if the query time is less than the query threshold, the target query node is updated using the target optimized configuration template. By dynamically configuring the corresponding optimized configuration template for different abnormal query nodes, the abnormal query nodes are directly optimized, reducing node blocking during the real-time query process and thus improving the efficiency of real-time data query. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of an application environment for an HBase database query node optimization method provided in an embodiment of the present invention;

[0023] Figure 2 This is a flowchart illustrating a query node optimization method for an HBase database according to an embodiment of the present invention.

[0024] Figure 3 This is a flowchart illustrating a query node optimization method for an HBase database according to an embodiment of the present invention.

[0025] Figure 4 This is a flowchart illustrating a query node optimization method for an HBase database according to an embodiment of the present invention.

[0026] Figure 5 This is a flowchart illustrating a query node optimization method for an HBase database according to an embodiment of the present invention.

[0027] Figure 6 This is a schematic diagram of the structure of an HBase database query node optimization device provided in an embodiment of the present invention;

[0028] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0031] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0032] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0033] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0034] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0035] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0036] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0037] An embodiment of the present invention provides a query node optimization method for an HBase database, which can be applied to applications such as... Figure 1 In this application environment, the client communicates with the server. Clients include, but are not limited to, PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). The server can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0038] See Figure 2 This is a flowchart illustrating a query node optimization method for an HBase database according to an embodiment of the present invention. The aforementioned HBase database query node optimization method can be applied to... Figure 1 The server in the example connects to the corresponding client, providing model training services to the client. For example... Figure 2 As shown, the HBase database query node optimization method may include the following steps.

[0039] S201: Based on the historical query time of each query node obtained from the preset query system, determine the set of abnormal query nodes in all query nodes, and set an optimized configuration template for each abnormal query node in the set of abnormal query nodes.

[0040] In step S201, based on the historical query time of each query node during the data query process, a set of abnormal query nodes is determined among all query nodes. An optimized configuration template is set for each abnormal query node in the set of abnormal query nodes. Each abnormal query node corresponds to at least one optimized configuration template. The abnormal query node is the query node with a long response time during the data query process. Based on the obtained multiple abnormal query nodes, an abnormal query node set is obtained. The optimized configuration template is dynamically configured and stored during the real-time query process of the abnormal query node.

[0041] In this embodiment, data querying is performed based on the HBase database. HBase is a distributed storage and query system based on Hadoop, characterized by its open-source, distributed, scalable, and column-oriented storage. It provides random and real-time read and write access for big data. To improve the reliability of stored data and the stability of the system, HDFS is used for data storage, and Zookeeper is used to assist in managing the entire cluster. The data querying process based on the HBase database involves the user sending a query request at the business layer, followed by data read and write operations to the HBase database at the database interaction layer. Microservice calls, such as Dubbo and Feign, are used between the business layer and the database interaction layer. The database interaction layer microservice communicates with the HBase cluster as an hbase-client to complete the data reading. This process involves first obtaining the HRegion containing the query data and its HRegionServer from the Zookeeper cluster, and then sending a request to the HRegionServer to retrieve the data.

[0042] During data querying, various query nodes interact with each other. If one query node has a long response time, it will increase the data query time. Based on the historical query time of each query node, abnormal query nodes are identified. For example, during data querying, if the interaction node between the business layer and the database interaction layer is blocked because the database interaction layer cannot handle a large number of requests from the business layer, then that query node may be an abnormal query node. Similarly, if the HBase-server's ZooKeeper cannot handle a large number of requests from the database interaction layer (acting as an HBase-client), and a blockage occurs between the database interaction layer and the HBase-server's ZooKeeper, then that query node may be an abnormal query node. Likewise, if the HBase-server's HRegionServer cannot handle a large number of requests from the database interaction layer (acting as an HBase-client), and a blockage occurs between the database interaction layer and the HBase-server's HRegionServer, then that query node may be an abnormal query node. By analyzing the historical query time of each query node, abnormal query nodes are identified.

[0043] Once the number of abnormal query nodes is identified, corresponding optimization configuration templates are set for each abnormal query node. When setting the optimization configuration templates, optimization configurations are obtained for the interaction process between each abnormal query node. For example, when the interaction node between the business layer and the database interaction layer is blocked during data query because the database interaction layer cannot handle a large number of requests from the business layer microservices, the business layer and the database interaction layer interact by calling Dubbo and Feign. This can be achieved by optimizing the business layer microservice of Dubbo-consumer and the database interaction layer microservice that acts as Dubbo-provider, or by horizontally scaling Dubbo-provider. When the HRegionServer of HBase-server cannot handle a large number of requests from the database interaction layer that acts as HBase-client, and a blockage occurs between the database interaction layer and the HRegionServer of HBase-server, the interaction process between the HRegionServer of HBase-server and the database interaction layer involves the database interaction layer accessing the HRegionServer of HBase-server, increasing the HBase read cache, increasing the number of threads of the HRegionServer, increasing the number of Regions that the RegionServer can accommodate, and periodically clearing redundant HBase tables, etc.

[0044] It should be noted that after obtaining the configurations that can be optimized for the abnormal query node, the configurations that can be dynamically configured are selected from these configurations as optimization configuration templates. For example, if a node is blocked because the database interaction layer cannot handle a large number of requests from the business layer microservices, a dynamically configured Dubbo template can be set. However, when horizontally scaling the Dubbo provider, manual configuration is required. Therefore, the dynamically configured Dubbo is selected as the optimization configuration template for this abnormal query node. When blocking occurs between the database interaction layer and the HRegionServer of HBase-server, the database connection pool capacity of the database interaction layer microservice can be increased by dynamically adjusting hbase.client.ipc.pool.size. This configuration can be used as an optimization configuration template. Redundant HBase tables can also be dynamically and periodically cleared to reduce the number of Regions in the database. This configuration can also be used as a dynamic configuration template. Therefore, there are two optimization configuration templates for this abnormal query node. The corresponding optimized configuration template is selected for each abnormal query node in turn.

[0045] Optionally, based on the historical query times of each query node obtained from a preset query system, a set of abnormal query nodes among all query nodes is determined, including:

[0046] Based on the historical query times of each query node obtained from the preset query system, the average query time for each query node for each query is calculated.

[0047] Based on the average query time and the preset average query threshold, determine the set of abnormal query nodes in the query nodes.

[0048] In this embodiment, the average query time of each query node is used to determine whether the node is an abnormal node. Based on the historical query time of each query node, the average query time of each query node for each query is calculated. When the average query time of each query node for each query is greater than the preset average query time, the query node is determined to be an abnormal query node, thereby obtaining a set of abnormal query nodes.

[0049] S202: If the real-time query time of each query node in the real-time query is obtained, the initial abnormal query node is determined based on the real-time query time of each query node and the preset query threshold.

[0050] In step S202, when querying data in the database in real time, the real-time query time of each query node is obtained from the preset query system. According to the preset query threshold, when the real-time query time of a query node is greater than the preset query threshold, the query node is an initial abnormal query node.

[0051] In this embodiment, to improve the real-time query efficiency of the database, the database performance monitoring platform monitors the data query process in real time. When abnormal query nodes are detected, the platform optimizes these nodes in real time to improve query efficiency. The monitoring platform monitors the real-time query time of each query node during the data query process and determines the initial abnormal query node based on the real-time query time.

[0052] It's important to note that the query threshold varies depending on the query node. A query threshold can be set for each query node. When the real-time query time is obtained, different query thresholds are selected for different query nodes to obtain a more accurate initial abnormal query node. For example, when the database interaction layer receives a query request sent from the business layer, the time it takes for the HRegionServer of the HBase-server to process the request from the database interaction layer (acting as the HBase-client) to be received may differ, thus requiring different query thresholds to be set.

[0053] It should be noted that the query threshold is dynamic data, and the query threshold may be different in different query systems. For example, the query threshold provided by Java language and the query system provided by Python language may be different on the same query node.

[0054] S203: Match the initial abnormal query node with each query node in the abnormal query node set one by one, and determine the query node in the abnormal query node set that matches the initial abnormal query node as the target query node.

[0055] In step S203, based on the initial abnormal query node obtained during the real-time query process, the target query node corresponding to the initial abnormal query node is determined from the abnormal query node geometry. The target query node is the query node to be optimized.

[0056] In this embodiment, when matching the initial abnormal query node with the query nodes in the abnormal query node set one by one, the matching can be performed based on the keywords of each query node, the keywords of each query node can be extracted, and the cosine value between the feature vector of the keywords in the initial abnormal query node and the feature vector of the keywords in each abnormal query node in the abnormal query node set is calculated. The cosine value is used to represent the similarity between the initial abnormal query node and each abnormal query node in the abnormal query node set, and the abnormal query node with the highest similarity is obtained as the target query node.

[0057] It should be noted that when matching the initial abnormal query node with the abnormal query nodes in the abnormal query node set one by one, a corresponding identifier can be set for each query node. For example, different labels can be set. When matching, the identifier information corresponding to the initial abnormal query node can be directly extracted from the abnormal query node set to obtain the target query node.

[0058] S204: Determine the target optimization configuration template from multiple optimization configuration templates corresponding to the target query node, dynamically configure the target optimization configuration template on the target query node, and obtain the optimized target query node.

[0059] In step S204, when the target query node contains multiple corresponding optimized configuration templates, one of the target optimized configuration templates is selected from the multiple optimized configuration templates, and the target query node is optimized based on the target optimized configuration template to obtain the optimized target query node.

[0060] In this embodiment, the target optimization configuration template is obtained based on the target query node and the corresponding optimization configuration template. When configuring optimization configuration templates for abnormal query nodes, each abnormal query node can often be configured with multiple optimization configuration templates. For example, when the hbase-server's zookeeper cannot handle a large number of requests from the database interaction layer acting as the hbase-client, and a blockage occurs between the database interaction layer and the hbase-server's zookeeper, the abnormal query node can be configured with an optimization configuration template that adjusts hbase.client.ipc.pool.size to expand the connection pool capacity. Alternatively, a data table can be configured as an optimization configuration template for the abnormal query node. The abnormal query node can correspond to two optimization configuration templates.

[0061] Select one of the optimized configuration templates corresponding to the target query node as the target optimized configuration template, and dynamically configure the target optimized configuration template on the target query node to obtain the optimized target query node. When the optimized configuration template value corresponding to the target query node contains an optimized configuration template, directly configure that optimized configuration template dynamically on the target query node to obtain the optimized target query node.

[0062] Optionally, a target optimization configuration template is determined from multiple optimization configuration templates corresponding to the target query node, and the target optimization configuration template is dynamically configured on the target query node to obtain the optimized target query node, including:

[0063] Based on the dynamic configuration time and dynamic configuration memory, multiple optimized configuration templates corresponding to the target query node are scored to obtain a score list corresponding to the optimized configuration templates;

[0064] Based on the score list, the corresponding optimized configuration templates are dynamically configured in descending order of score. If the dynamic configuration is successful, the successfully configured optimized configuration template is determined as the target optimized configuration template.

[0065] In this embodiment, when there are multiple optimized configuration templates corresponding to the target query node, the multiple optimized configuration templates are scored to obtain a score list corresponding to the optimized configuration templates. The corresponding optimized configuration templates are dynamically configured sequentially from high to low according to the score list. If the dynamic configuration is successful, the successfully configured optimized configuration template is determined as the target optimized configuration template. For example, when the hbase-server's zookeeper cannot handle a large number of requests from the database interaction layer acting as the hbase-client, and a blockage occurs between the database interaction layer and the hbase-server's zookeeper, the query node corresponds to multiple optimized configuration templates. Based on the optimized configuration templates, hbase.client.ipc.pool.size can be dynamically adjusted, and the database table area can also be dynamically configured. Dynamically adjusting database parameters and dynamically configuring data tables are both necessary. When dynamically adjusting database parameters, continuous attempts are required until the proportion of slow requests cannot be reduced, resulting in a long dynamic configuration time and continuously increasing the database connection pool capacity, thus increasing dynamic memory. Optimizing the data table area allows for the reasonable division of the data table area based on the data to be queried, reducing the data table area and the corresponding dynamic memory, thus reducing the dynamic configuration time. Therefore, different scores can be assigned to the dynamic configuration time and dynamic configuration content according to the preset scoring rules, and the score list corresponding to the optimized configuration template can be calculated.

[0066] Based on the scores in the score list, the corresponding optimization configuration templates are dynamically configured in descending order of score. During dynamic configuration, if the configuration is successful, the successfully configured optimization configuration template is used as the target optimization configuration template. For example, when dynamic configuration requires increasing dynamic memory, if memory is low, the next score optimization configuration template is selected in sequence.

[0067] S205: Test the query time of the optimized target query node, and if the query time is less than the query threshold, update the target query node using the target optimization configuration template.

[0068] In step S205, the query time of the target query node after test optimization is obtained in real time through the performance monitoring platform. If the query time is less than the query threshold, the target query node is updated using the target optimization configuration template. After the target query node updates the target optimization configuration template, the data query efficiency is improved.

[0069] In this embodiment, after obtaining the query time of the target query node after test optimization from the performance monitoring platform, if the query time is less than the query threshold, the target optimization configuration template is updated. If the query time is not less than the query threshold, the target query node is rolled back and the updated optimized target query node is deleted, so that the query system remains in its original state.

[0070] Based on the historical query times of each query node obtained from the preset update query system, a set of abnormal query nodes is determined among all query nodes, and an optimization configuration template is set for each abnormal query node in the abnormal query node set. Each abnormal query node corresponds to at least one optimization configuration template. If the real-time query times of each query node in the real-time query are obtained, an initial abnormal query node is determined based on the real-time query time of each query node and a preset query threshold. The initial abnormal query node is then matched one-to-one with each query node in the abnormal query node set, and the query node in the abnormal query node set that matches the initial abnormal query node is identified as the target query node. The system dynamically configures the target query node by selecting a target optimization configuration template from multiple optimization configuration templates corresponding to the target query node. This optimized template is then dynamically configured on the target query node to obtain the optimized target query node. The system tests the query time of the optimized target query node, and if the query time is less than a query threshold, the target query node is updated using the optimized configuration template. By dynamically configuring corresponding optimization configuration templates for different abnormal query nodes, abnormal query nodes are directly optimized, reducing node blocking during real-time queries and thus improving the efficiency of real-time data queries.

[0071] See Figure 3 This is a flowchart illustrating a method for optimizing query nodes in a base database according to an embodiment of the present invention. Figure 3 The query node optimization method for this base database may include the following steps:

[0072] S301: Based on the historical query time of each query node obtained from the preset query system, determine the set of abnormal query nodes among all query nodes, and set an optimization configuration template for each abnormal query node in the set of abnormal query nodes; wherein, each abnormal query node corresponds to at least one optimization configuration template.

[0073] The content of step S301 is the same as that of step S201, and can be referred to the description of step S201 above, so it will not be repeated here.

[0074] S302: If the real-time query time of each query node in the real-time query cannot be obtained, calculate the first average query time of each query node in the real-time query.

[0075] S303: If the first average query time is greater than the preset first average query threshold, the optimization configuration template is dynamically configured for each abnormal query node in the abnormal query node set in turn. After an abnormal query node is configured with the optimization configuration template, the second average query time of each query node is obtained.

[0076] S304: If the second average time is less than the first average query time, update the optimized configuration template corresponding to the abnormal query node;

[0077] S305: If the second average time is greater than the first average query time, rollback is performed on the abnormal query node.

[0078] In this embodiment, when the performance monitoring platform cannot obtain the real-time query time of each query node in the real-time query, it monitors the total time from the time the query request for the data to be queried is made until the corresponding data is returned. If the first average query time is greater than the preset first average query threshold, it is considered that there is an abnormal query node in the data query process, but the location of the abnormal query node cannot be determined. Therefore, each abnormal query node in the abnormal query node set is dynamically configured with an optimization configuration template in sequence. When an abnormal query node is configured with an optimization configuration template, the second average query time of each query node is obtained. If the second average time is less than the first average query time, the optimization configuration template corresponding to the abnormal query node is updated. If the second average time is greater than the first average query time, the abnormal query node is rolled back and returned to the query state where the query node is not configured with an optimization configuration template.

[0079] When the real-time query time of each query node cannot be obtained, the average real-time query time of each query node in the entire query system is obtained. The average query time of each query node is used to determine whether there is a situation with low query efficiency. Then, the optimization configuration templates corresponding to the abnormal query nodes in the abnormal query node set are dynamically configured one by one. By traversing the optimization configuration templates, the query nodes are optimized and configured to improve query efficiency.

[0080] See Figure 4 This is a flowchart illustrating a query node optimization method for an HBase database according to an embodiment of the present invention. Figure 4 The query node optimization method for this HBase database may include the following steps:

[0081] S401: Based on the historical query time of each query node obtained from the preset query system, determine the set of abnormal query nodes among all query nodes, and set an optimization configuration template for each abnormal query node in the set of abnormal query nodes; wherein, each abnormal query node corresponds to at least one optimization configuration template.

[0082] S402: If the real-time query time of each query node in the real-time query is obtained, the initial abnormal query node is determined based on the real-time query time of each query node and the preset query threshold.

[0083] S403: Match the initial abnormal query node with each query node in the abnormal query node set one by one, and determine the query node in the abnormal query node set that matches the initial abnormal query node as the target query node;

[0084] S404: Determine the target optimization configuration template from multiple optimization configuration templates corresponding to the target query node, dynamically configure the target optimization configuration template on the target query node, and obtain the optimized target query node;

[0085] S405: Test the query time of the optimized target query node, and update the target query node using the target optimization configuration template if the query time is less than the query threshold.

[0086] The contents of steps S401 to S405 are the same as those of steps S201 to S205, and can be referred to the description of steps S201 to S205, which will not be repeated here.

[0087] S406: Based on the data to be queried in the real-time query, determine the number of historical queries corresponding to the data to be queried. If the number of queries is greater than 0, search for the cached data corresponding to the data to be queried in the preset Redis cluster.

[0088] In this embodiment, to address the issue that configuration templates cannot be dynamically configured and optimized during the interaction service process in query nodes, a Redis cluster is set up for data caching to improve data query efficiency and reduce communication pressure between query nodes. For example, when the same query data needs to be queried within a short period of time, each query requires the database interaction layer to process the query data request from the business layer. Each processing of the query data request requires a significant amount of processing time, increasing the communication pressure between the database service layer and the business layer.

[0089] By setting up a Redis cluster at the business layer, the Redis cluster can cache the returned query data. Once the Redis cluster has cached the query data, it can be directly retrieved from the cached data. This reduces the communication burden between the business layer and the data interaction layer.

[0090] Optionally, in a pre-defined Redis cluster, the cached data corresponding to the data to be queried is searched, including:

[0091] Based on the received query request for the data to be queried, obtain the primary key information from the query request;

[0092] Based on the primary key information, search for the cached data corresponding to the data to be queried in the preset Redis cluster.

[0093] In this embodiment, based on the received query request for the data to be queried, the primary key information in the query request is obtained. The primary key information is a field in a database table, representing data in a specific row or column. When a Redis cluster is set up, the query request for the data to be queried is obtained through Redis. The query is then performed in the Redis cluster according to the query request. During the query, the primary key information in the query request is used as an index to obtain the cached data corresponding to the primary key information.

[0094] It should be noted that Redis cluster is used to cache the corresponding historical query data. When the same data is queried within a short period of time, after the query data is returned for the first time, the corresponding query data is cached in the Redis cluster. The cache is in the format of table name and primary key to facilitate indexing.

[0095] When querying data in a Redis cluster, the application layer nodes can query the cached data. If the corresponding cached data is found in the application layer, the application layer returns the queried data, saving the interaction process between the remaining nodes and improving query efficiency. If the application layer nodes do not have cached data, the corresponding data is queried from the cache database and returned.

[0096] See Figure 5 This is a flowchart illustrating a query node optimization method for an HBase database according to an embodiment of the present invention. Figure 5 The query node optimization method for this HBase database may include the following steps:

[0097] S501: Based on the historical query time of each query node obtained from the preset query system, determine the set of abnormal query nodes among all query nodes, and set an optimization configuration template for each abnormal query node in the set of abnormal query nodes; wherein, each abnormal query node corresponds to at least one optimization configuration template.

[0098] S502: If the real-time query time of each query node in the real-time query is obtained, the initial abnormal query node is determined based on the real-time query time of each query node and the preset query threshold.

[0099] S503: Match the initial abnormal query node with each query node in the abnormal query node set one by one, and determine the query node in the abnormal query node set that matches the initial abnormal query node as the target query node;

[0100] S504: Determine the target optimization configuration template from multiple optimization configuration templates corresponding to the target query node, dynamically configure the target optimization configuration template on the target query node, and obtain the optimized target query node;

[0101] S505: Test the query time of the optimized target query node, and update the target query node using the target optimization configuration template if the query time is less than the query threshold.

[0102] The steps S501 to S505 are the same as the steps S201 to S205, and can be referred to the description of steps S201 to S205, which will not be repeated here.

[0103] S506: Based on the received query requests for the data to be queried, divide the query requests into different request batches and assign batch identifiers to the request batches;

[0104] S507: Based on the batch identifier, merge the query data of the corresponding query requests in the same batch identifier to obtain the merged data of the query data of the corresponding query requests in the same batch identifier, and return the merged data.

[0105] In this embodiment, when a large number of query requests with query data are obtained in a short period of time, the query requests are divided. This division can be based on time or the number of query requests. For example, when dividing requests by time, query requests within a fixed time period can be considered part of the same batch. A time threshold can be set, and query requests obtained within that time threshold are considered part of the same batch. When dividing requests by the number of query requests, the number of requests can be accumulated. When the number of query requests reaches a certain value, the request batches are divided, the accumulated value is recalculated, and the newly obtained query requests are used as the next batch.

[0106] After dividing the request into batches, a different identifier is assigned to each batch of requests so that the business layer can process the query requests according to the corresponding identifier. Query requests in the same batch can reuse the corresponding identifier to facilitate the database interaction layer to read the query requests in one go, thereby reducing the number of communications between the business layer and the database interaction layer.

[0107] Dividing query requests into batches and assigning different batch identifiers to different batch requests greatly reduces the number of identifier allocations. Query requests with the same batch identifier communicate with the database interaction layer based on the same batch identifier, allowing the database interaction layer to read query requests with the same batch number at once, reducing the number of communications with the database interaction layer and thus improving query efficiency.

[0108] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of an HBase database query node optimization device provided in an embodiment of the present invention. In this embodiment, the terminal includes units used for execution... Figures 2 to 5 The steps in the corresponding embodiments. Please refer to the details. Figures 2 to 5 as well as Figures 2 to 5 The relevant descriptions in the corresponding embodiments are shown below. For ease of explanation, only the parts relevant to this embodiment are shown. See also... Figure 6 The query node optimization device 60 of the HBase database includes: an optimization configuration template setting module 61, an initial abnormal query node determination module 62, a matching module 63, a dynamic configuration module 64, and an update module 65.

[0109] The optimization configuration template setting module 61 is used to determine the set of abnormal query nodes in all query nodes based on the historical query time of each query node obtained from the preset query system, and to set an optimization configuration template for each abnormal query node in the set of abnormal query nodes; wherein, each abnormal query node corresponds to at least one optimization configuration template.

[0110] The initial abnormal query node determination module 62 is used to determine the initial abnormal query node based on the real-time query time of each query node and the preset query threshold if the real-time query time of each query node is obtained in the real-time query.

[0111] Matching module 63 is used to match the initial abnormal query node with each query node in the abnormal query node set one by one, and determine the query node in the abnormal query node set that matches the initial abnormal query node as the target query node.

[0112] The dynamic configuration module 64 is used to determine the target optimization configuration template from multiple optimization configuration templates corresponding to the target query node, and dynamically configure the target optimization configuration template on the target query node to obtain the optimized target query node.

[0113] Update module 65 is used to test the query time of the optimized target query node, and update the target query node using the target optimization configuration template if the query time is less than the query threshold.

[0114] Optionally, the above-mentioned optimized configuration template setting module 61 includes:

[0115] The calculation unit is used to calculate the average query time for each query node for each query based on the historical query time of each query node obtained from the preset query system.

[0116] The abnormal query node set determination unit is used to determine the abnormal query node set in the query nodes based on the average query time and the preset average query threshold.

[0117] Optionally, the aforementioned dynamic configuration module 64 includes:

[0118] The scoring unit is used to score multiple optimized configuration templates corresponding to the target query node based on dynamic configuration time and dynamic configuration memory, and to obtain a score list corresponding to the optimized configuration template.

[0119] The target optimization configuration template determination unit is used to dynamically configure the corresponding optimization configuration templates in descending order of the score list. If the dynamic configuration is successful, the successfully configured optimization configuration template is determined as the target optimization configuration template.

[0120] Optionally, the aforementioned optimization device 60 further includes:

[0121] The first average query time determination module is used to calculate the first average query time of each query node in the real-time query if the real-time query time of each query node in the real-time query cannot be obtained.

[0122] The second average query time determination module is used to dynamically configure an optimization configuration template for each abnormal query node in the abnormal query node set if the first average query time is greater than the preset first average query threshold. After an abnormal query node is configured with an optimization configuration template, the second average query time of each query node is obtained.

[0123] The optimized configuration template update module is used to update the optimized configuration template corresponding to the abnormal query node if the second average time is less than the first average query time.

[0124] The rollback module is used to roll back abnormal query nodes if the second average time is greater than the first average query time.

[0125] Optionally, the aforementioned optimization device 60 further includes:

[0126] The caching module is used to determine the number of historical queries corresponding to the data to be queried based on the data to be queried in the real-time query. If the number of queries is greater than 0, the cached data corresponding to the data to be queried is searched in the preset Redis cluster.

[0127] Optionally, the above-mentioned caching module also includes:

[0128] The primary key information acquisition unit is used to acquire the primary key information in the query request based on the received query request for the data to be queried.

[0129] The cache data retrieval unit is used to find the cache data corresponding to the data to be queried in the preset Redis cluster based on the primary key information.

[0130] Optionally, the aforementioned optimization device 60 further includes:

[0131] The batch segmentation module is used to divide query requests into different request batches based on the time of the received query requests for the data to be queried, and to set batch identifiers for the request batches.

[0132] The merging module is used to merge the query data of the corresponding query requests in the same batch according to the batch identifier, and to obtain the merged data of the query requests in the same batch, and then return the merged data.

[0133] It should be noted that the information interaction and execution process between the above-mentioned units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0134] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Figure 7 As shown, the computer device of this embodiment includes: at least one processor ( Figure 7 The diagram shows only one of the following: a memory and a computer program stored in the memory and capable of running on at least one processor. When the processor executes the computer program, it implements the steps in the above embodiments of the query node optimization method for any of the HBase databases.

[0135] This computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 7 The examples of computer devices are merely examples and do not constitute a limitation on computer devices. Computer devices may include more or fewer components than shown in the illustration, or combinations of certain components, or different components, such as network interfaces, displays, and input devices.

[0136] The processor referred to can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0137] Memory includes readable storage media, internal memory, etc., wherein internal memory can be the RAM of a computer device, providing an environment for the operation of the operating system and computer-readable instructions stored in the readable storage media. The readable storage media can be the hard drive of a computer device, or in other embodiments, it can be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, memory can include both internal storage units and external storage devices of a computer device. Memory is used to store the operating system, applications, bootloader, data, and other programs, such as program code for computer programs. Memory can also be used to temporarily store data that has been output or will be output.

[0138] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the functions described above can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention can implement all or part of the processes in the methods of the above embodiments by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0139] The present invention can implement all or part of the processes in the methods of the above embodiments, or it can be accomplished by a computer program product. When the computer program product is run on a computer device, the computer device executes the steps in the above method embodiments.

[0140] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0141] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0142] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0143] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0144] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for optimizing query nodes in an HBase database, characterized in that, The optimization method includes: Based on the historical query times of each query node obtained from the preset query system, a set of abnormal query nodes is determined among all query nodes, and an optimized configuration template is set for each abnormal query node in the set of abnormal query nodes; wherein, each abnormal query node corresponds to at least one optimized configuration template. If the real-time query time of each query node in the real-time query is obtained, the initial abnormal query node is determined based on the real-time query time of each query node and the preset query threshold. The initial abnormal query node is matched one by one with each query node in the abnormal query node set to determine the query node in the abnormal query node set that matches the initial abnormal query node as the target query node. The process involves determining a target optimization configuration template from multiple optimization configuration templates corresponding to the target query node, dynamically configuring the target optimization configuration template on the target query node, and obtaining an optimized target query node. The determination of the target optimization configuration template from the multiple optimization configuration templates corresponding to the target query node includes: Based on the dynamic configuration time and dynamic configuration memory, multiple optimized configuration templates corresponding to the target query node are scored to obtain a score list corresponding to the optimized configuration templates. Based on the score list, the corresponding optimization configuration templates are dynamically configured in descending order of score. If the dynamic configuration is successful, the successfully configured optimization configuration template is determined as the target optimization configuration template. Test the query time of the optimized target query node, and if the query time is less than the query threshold, update the target query node using the target optimization configuration template.

2. The HBase database query node optimization method as described in claim 1, characterized in that, The step of determining the set of abnormal query nodes among all query nodes based on the historical query times of each query node obtained from the preset query system includes: Based on the historical query times of each query node obtained from the preset query system, the average query time for each query node for each query is calculated. Based on the average query time and the preset average query threshold, a set of abnormal query nodes is determined among the query nodes.

3. The HBase database query node optimization method as described in claim 1, characterized in that, After determining the set of abnormal query nodes among all query nodes based on the historical query times of each query node obtained from the preset query system, and setting an optimized configuration template for each abnormal query node in the set of abnormal query nodes, the process further includes: If the real-time query time of each query node in the real-time query cannot be obtained, the first average query time of each query node in the real-time query is calculated. If the first average query time is greater than the preset first average query threshold, the optimized configuration template is dynamically configured for each abnormal query node in the abnormal query node set in sequence. After the optimized configuration template is configured for an abnormal query node in the abnormal query node set, the second average query time of each query node is obtained. If the second average query time is less than the first average query time, update the optimized configuration template corresponding to the abnormal query node; If the second average query time is greater than the first average query time, the abnormal query node is rolled back.

4. The HBase database query node optimization method as described in claim 1, characterized in that, The step of testing the query time of the optimized target query node, and updating the target query node using the target optimization configuration template if the query time is less than the query threshold, further includes: Based on the data to be queried in the real-time query, determine the number of historical queries corresponding to the data to be queried. If the number of queries is greater than 0, search for the cached data corresponding to the data to be queried in the preset Redis cluster.

5. The HBase database query node optimization method as described in claim 4, characterized in that, The step of searching for the cached data corresponding to the data to be queried in the preset Redis cluster includes: Based on the received query request for the data to be queried, obtain the primary key information in the query request; Based on the primary key information, the cached data corresponding to the data to be queried is searched in the preset Redis cluster.

6. The HBase database query node optimization method as described in claim 1, characterized in that, The step of testing the query time of the optimized target query node, and updating the target query node using the target optimization configuration template if the query time is less than the query threshold, further includes: Based on the time of the received query requests for the data to be queried, the query requests are divided into different request batches, and batch identifiers are set for the request batches. Based on the batch identifier, the query data of the corresponding query requests in the same batch identifier are merged to obtain the merged data of the query data of the corresponding query requests in the same batch identifier, and the merged data is returned.

7. A query node optimization device for an HBase database, characterized in that, The device includes: The optimized configuration template setting module is used to determine the set of abnormal query nodes among all query nodes based on the historical query time of each query node obtained from the preset query system, and to set an optimized configuration template for each abnormal query node in the set of abnormal query nodes; wherein, each abnormal query node corresponds to at least one optimized configuration template. The initial abnormal query node determination module is used to determine the initial abnormal query node based on the real-time query time of each query node and the preset query threshold if the real-time query time of each query node is obtained in the real-time query. The matching module is used to match the initial abnormal query node with each query node in the abnormal query node set one by one, and determine the query node in the abnormal query node set that matches the initial abnormal query node as the target query node. A dynamic configuration module is used to determine a target optimized configuration template from multiple optimized configuration templates corresponding to the target query node, and dynamically configure the target optimized configuration template on the target query node to obtain an optimized target query node; the step of determining the target optimized configuration template from multiple optimized configuration templates corresponding to the target query node includes: Based on the dynamic configuration time and dynamic configuration memory, multiple optimized configuration templates corresponding to the target query node are scored to obtain a score list corresponding to the optimized configuration templates. Based on the score list, the corresponding optimization configuration templates are dynamically configured in descending order of score. If the dynamic configuration is successful, the successfully configured optimization configuration template is determined as the target optimization configuration template. The update module is used to test the query time of the optimized target query node, and update the target query node using the target optimization configuration template if the query time is less than the query threshold.

8. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the query node optimization method for the HBase database as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the query node optimization method for the HBase database as described in any one of claims 1 to 6.

Images