Method and device for processing hotspots in database, and computer readable storage medium

Abstract

The application discloses a hotspot processing method and device in a database and a computer readable storage medium. The method comprises the following steps: when it is determined that a hotspot region exists in an HBase open source database, acquiring all primary key data corresponding to the hotspot region in the HBase open source database; processing all the primary key data by using a predetermined processing mode to process rowkey values, and obtaining new rowkey values; simulating access to the HBase open source database based on the new rowkey values; when it is determined that no hotspot region exists in the HBase open source database during the simulation access, determining that the predetermined processing mode is a target optimization mode; and optimizing the rowkey values by using the target optimization mode to process the hotspot region existing in the HBase open source database.

Description

Technical Field

[0001] This invention relates to the field of data processing, and more specifically, to a method and apparatus for processing hotspots in a database, as well as a computer-readable storage medium. Background Technology

[0002] With the rapid development of emerging technologies such as big data and cloud computing, the scale of data is constantly increasing. The financial industry, especially banks, possesses a large amount of customer data, and the starting unit of measurement for the data used for daily queries is at least petabytes (P) or exabytes (E). For the storage and querying of petabyte-scale massive data, distributed and scalable NoSQL databases, such as key-value databases, are commonly used. However, due to the uneven distribution of data, performance is often low under high-concurrency transactions due to hotspot issues.

[0003] In the design of key-value database tables, the key is the most crucial component, determining the evenness of data distribution. An improperly designed key can lead to a concentration of data keys between the start and end keys of one or a few regions. This can result in extremely high request volumes for these regions, potentially creating a hotspot problem. A hotspot problem refers to the phenomenon where the number of requests to a region in a database table exceeds a certain threshold. The main causes of region hotspot problems are uneven load on region servers or uneven data request volumes within those servers. Excessive requests to a region can negatively impact database performance, leading to performance degradation or even unavailability for services related to the key-value database table.

[0004] The techniques provided in related technologies to solve the region hotspot problem involve pre-partitioning the table when it is created. For example, the number of cluster regions is determined in advance based on the number of machines in the cluster, and the start key and end key of the distributed keys are specified on each region. This method can effectively avoid the hotspot problem when the amount of data is not too large. However, when the amount of data continues to increase and the number of regions exceeds the initially determined number of regions, it still cannot avoid uneven data distribution and the occurrence of hotspot problems.

[0005] There is currently no effective solution to the above problems. Summary of the Invention

[0006] This invention provides a method and apparatus for handling hotspots in a database, as well as a computer-readable storage medium, to at least solve the technical problem of region hotspots caused by excessive data in the use of key-value databases in related technologies.

[0007] According to one aspect of the present invention, a method for handling hotspots in a database is provided, comprising: when it is determined that a hotspot region exists in an HBase open-source database, obtaining all primary key data corresponding to the hotspot region in the HBase open-source database; processing all primary key data using a predetermined processing method to process rowkey values ​​to obtain new rowkey values, wherein the rowkey values ​​are values ​​generated when the HBase open-source database is accessed; simulating access to the HBase open-source database based on the new rowkey values; when it is determined that the hotspot region does not exist in the HBase open-source database during the simulated access process, determining the predetermined processing method as a target optimization method; and optimizing the rowkey values ​​using the target optimization method to handle the presence of the hotspot region in the HBase open-source database.

[0008] Optionally, determining that there are hot regions in the HBase open-source database includes: obtaining the number of access requests for each region in the HBase open-source database based on the rowkey value; determining the access request ratio for each region based on the number of access requests; and determining that there are hot regions in the HBase open-source database when the access request ratio exceeds a predetermined threshold.

[0009] Optionally, processing all primary key data using a predetermined processing method to process the rowkey value and obtain a new rowkey value includes: performing discrete processing on all primary key data using the predetermined processing method to obtain a first deformed value; determining the absolute value of the remainder obtained by dividing the first deformed value by the total number of regions corresponding to each region in the HBase open-source database as a second deformed value; and concatenating the second deformed value with the primary keys corresponding to all primary key data to obtain the new rowkey value.

[0010] Optionally, simulating access to the HBase open-source database based on the new rowkey value includes: counting the number of rows of primary key data distributed across each region in the HBase open-source database; determining the dispersion value of the new rowkey value based on the number of rows of primary key data across each region; selecting the rowkey optimization method that minimizes the dispersion value to optimize the rowkey value; and using the optimized rowkey value to simulate access to the HBase open-source database.

[0011] Optionally, determining that the HBase open-source database does not have the hotspot region during the simulated access process includes: obtaining the number of simulated access requests for each region in the HBase open-source database during the simulated access period; determining the proportion of simulated access requests for each region based on the number of simulated access requests; and determining that the HBase open-source database does not have the hotspot region when the proportion of simulated access requests does not exceed a predetermined threshold.

[0012] Optionally, the method further includes: when the proportion of simulated access requests still exceeds a predetermined threshold, reprocessing the rowkey value to continue simulating access to the HBase open-source database until it is determined that the HBase open-source database does not contain the hotspot region.

[0013] According to another aspect of the present invention, a hotspot processing apparatus for a database is also provided, comprising: an acquisition module, configured to acquire all primary key data corresponding to the hotspot region in the HBase open-source database when it is determined that a hotspot region exists in the HBase open-source database; a processing module, configured to process all primary key data using a predetermined processing method to process rowkey values ​​to obtain new rowkey values, wherein the rowkey values ​​are values ​​generated when the HBase open-source database is accessed; a simulated access module, configured to perform simulated access to the HBase open-source database based on the new rowkey value; a determination module, configured to determine the predetermined processing method as a target optimization method when it is determined that the HBase open-source database does not have the hotspot region during the simulated access process; and an optimization module, configured to optimize the rowkey value using the target optimization method to handle the presence of the hotspot region in the HBase open-source database.

[0014] Optionally, the acquisition module includes: a first acquisition unit, configured to acquire the number of access requests for each region in the HBase open-source database based on the rowkey value; a first determination unit, configured to determine the access request ratio of each region based on the number of access requests; and a second determination unit, configured to determine that there is a hot region in the HBase open-source database when there is an access request ratio exceeding a predetermined threshold in the access request ratio.

[0015] Optionally, the processing module includes: a discrete processing unit, used to perform discrete processing on all primary key data using the predetermined processing method to obtain a first deformed value; a third determining unit, used to determine that the absolute value of the remainder obtained by dividing the first deformed value by the total number of each region in the HBase open-source database is a second deformed value; and a concatenation unit, used to concatenate the second deformed value with the primary keys corresponding to all primary key data to obtain the new rowkey value.

[0016] Optionally, the simulated access module includes: a statistics unit, used to count the number of rows of primary key data distributed in each region of the HBase open-source database; a fourth determination unit, used to determine the dispersion value of the new rowkey value based on the number of rows of primary key data in each region; and an optimization unit, used to select the rowkey optimization method that minimizes the dispersion value to optimize the rowkey value, and use the optimized rowkey value to simulate accessing the HBase open-source database.

[0017] Optionally, the determining module includes: a second obtaining unit, configured to obtain the number of simulated access requests for each region in the HBase open-source database during the simulated access period; a fifth determining unit, configured to determine the proportion of simulated access requests for each region based on the number of simulated access requests; and a sixth determining unit, configured to determine that the HBase open-source database does not have the hotspot region when the proportion of simulated access requests does not exceed a predetermined threshold.

[0018] Optionally, the apparatus further includes a reprocessing unit, configured to reprocess the rowkey value when the simulated access request ratio still exceeds a predetermined threshold, so as to continue simulating access to the HBase open-source database until it is determined that the HBase open-source database does not contain the hotspot region.

[0019] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed by a processor, it controls the device where the computer-readable storage medium is located to perform the database hotspot processing method described in any one of the above descriptions.

[0020] According to another aspect of the present invention, a processor is also provided, the processor being configured to run a computer program, wherein the computer program, when running, performs the database hotspot processing method described in any one of the above embodiments.

[0021] In this embodiment of the invention, when it is determined that a hot region exists in the HBase open-source database, all primary key data corresponding to the hot region in the HBase open-source database are obtained; all primary key data are processed using a predetermined processing method to process the rowkey value, resulting in a new rowkey value, wherein the rowkey value is the value generated when the HBase open-source database is accessed; a simulated access to the HBase open-source database is performed based on the new rowkey value; if it is determined that no hot region exists in the HBase open-source database during the simulated access process, the predetermined processing method is determined to be the target optimization method; the rowkey value is optimized using the target optimization method to handle the hot region in the HBase open-source database. Through the hot spot handling method in the database provided by this embodiment of the invention, the purpose of processing the database based on the judgment result of whether a hot region exists in the HBase open-source database is achieved, thereby realizing the technical effect of making the data distribution uniform and reasonable, and thus solving the technical problem of region hotspots often caused by excessive data in the use of key-value databases in related technologies. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0023] Figure 1 This is a flowchart of a database hotspot processing method according to an embodiment of the present invention;

[0024] Figure 2 This is a flowchart of hotspot detection according to an embodiment of the present invention;

[0025] Figure 3 This is a flowchart of the rowkey design according to an embodiment of the present invention;

[0026] Figure 4 This is a flowchart of the discreteness sorting according to an embodiment of the present invention;

[0027] Figure 5 This is a flowchart illustrating the simulation of access to determine the proportion of region requests according to an embodiment of the present invention;

[0028] Figure 6 This is a flowchart of a preferred method for handling hotspots in a database according to an embodiment of the present invention;

[0029] Figure 7 This is a schematic diagram of a database hotspot processing device according to an embodiment of the present invention. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] Example 1

[0033] According to an embodiment of the present invention, a method embodiment for processing hotspots in a database is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0034] Figure 1 This is a flowchart of a database hotspot processing method according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0035] Step S102: When it is determined that there is a hot region in the HBase open source database, obtain all primary key data corresponding to the hot region in the HBase open source database;

[0036] Optionally, in the above steps, when it is determined that there is a hot region in the HBase open-source database, all primary key data corresponding to the hot region in the HBase open-source database are obtained. It should be noted that HBase is a distributed, column-oriented storage system. When real-time read and write and random access to ultra-large-scale datasets are required, HBase database can be used.

[0037] Step S104: Process all primary key data using a predefined processing method to process the rowkey value and obtain a new rowkey value, wherein the rowkey value is the value generated when the HBase open source database is accessed;

[0038] Optionally, in the above steps, different preset hash algorithms (including but not limited to: md4, md5, sha-1, tiger, adler32, crc32, BKDRHash, APHash, DJBHash, ELFHash) are used to hash the primary key data of the HBase table to obtain a new rowkey value.

[0039] Step S106: Simulate access to the HBase open-source database based on the new rowkey value;

[0040] Optionally, in the above steps, the rowkey values ​​accessed in the last two hours can be selected, and the rowkey can be designed according to the hash algorithm with the smallest Datahash value. This dynamically simulates the client's access to the HBase cluster, that is, the distribution of the designed rowkey on each region can be counted based on the start key and end key of each region, and then the number of accesses to each region can be counted.

[0041] Step S108: When it is determined that there are no hot regions in the HBase open-source database during the simulated access process, the predetermined processing method is determined to be the target optimization method;

[0042] Optionally, if the threshold is still exceeded in the above steps, the Datahash value of the rowkey design method is calculated and sorted to simulate the next rowkey design method until a suitable rowkey design method is selected.

[0043] Step S110: Optimize the rowkey value using the target optimization method to handle hot regions in the HBase open-source database.

[0044] As can be seen from the above, in this embodiment of the invention, firstly, when it is determined that there is a hot region in the HBase open-source database, all primary key data corresponding to the hot region in the HBase open-source database can be obtained; then, all primary key data can be processed using a predetermined processing method to process the rowkey value, obtaining a new rowkey value, wherein the rowkey value is the value generated when the HBase open-source database is accessed; then, the HBase open-source database can be simulated based on the new rowkey value; then, when it is determined that there is no hot region in the HBase open-source database during the simulated access process, the predetermined processing method is determined as the target optimization method; finally, the rowkey value can be optimized using the target optimization method to handle the hot region in the HBase open-source database. Through the hot spot processing method in the database provided by this embodiment of the invention, the purpose of processing the database based on the judgment result of whether there is a hot region in the HBase open-source database is achieved, thereby realizing the technical effect of making the data distribution uniform and reasonable, and thus solving the technical problem of region hotspots often caused by excessive data in the use of key-value databases in related technologies.

[0045] As an optional implementation, determining that there are hot regions in the HBase open-source database includes: obtaining the number of access requests for each region in the HBase open-source database based on the rowkey value; determining the access request ratio for each region based on the number of access requests; and determining that there are hot regions in the HBase open-source database when the access request ratio exceeds a predetermined threshold.

[0046] Figure 2 This is a flowchart of hotspot detection according to an embodiment of the present invention, such as... Figure 2 As shown, the request ratio of each region in the HBase table is checked at preset time intervals, where the preset time interval can be 24 hours. The request ratio of each region is... Where, r j This is the number of requests from each region within a preset time interval. This is the average number of requests per region over a preset time interval. The system detects the region request ratio; if it exceeds a set upper threshold, it is considered a region hotspot issue.

[0047] It should be noted that when the access request sent by the client is obtained, the rowkey value of the client request is recorded, with "timestamp-rowkey" as the key and visits of the key in a certain minute as the value. The timestamp is stored in the in-memory database with minute-level precision, such as 2021-06-08-10-12-${rowkey},${visits}.

[0048] When a new access record is received, based on the key value, if the key value does not exist, a new record is created and the visits value is set to 1; if the key value already exists, the visits value of the existing record is incremented by 1. The storage period is a preset storage time, where the in-memory database can be Redis, and the preset storage time can be 24 hours.

[0049] As an optional embodiment, all primary key data is processed using a predetermined processing method to process the rowkey value and obtain a new rowkey value, including: performing discrete processing on all primary key data using the predetermined processing method to obtain a first deformed value; determining the absolute value of the remainder obtained by dividing the first deformed value by the total value corresponding to each region in the HBase open source database as a second deformed value; and concatenating the second deformed value with the primary keys corresponding to all primary key data to obtain a new rowkey value.

[0050] Figure 3 This is a flowchart of the rowkey design according to an embodiment of the present invention, such as... Figure 3 As shown, firstly, the primary key data of the HBase table experiencing hotspot issues is retrieved, and this data is used as the basis for the rowkey design. Next, different preset hash algorithms (md4, md5, sha-1, tiger, adler32, crc32, BKDRHash, APHash, DJBHash, ELFHash) are used to hash the primary key data of this HBase table. The main steps include:

[0051] (1) The primary key data corresponding to each primary key content is transformed into the corresponding first transformed value through a hash algorithm;

[0052] (2) Modulate the first deformation value by the number of regions in the cluster and take the absolute value to obtain the second deformation value;

[0053] (3) Concatenate the second deformed value with the primary key to form a new rowkey.

[0054] As an optional implementation, simulated access to the HBase open-source database based on the new rowkey value includes: counting the number of rows of primary key data distributed across each region of the HBase open-source database; determining the dispersion value of the new rowkey value based on the number of rows of primary key data in each region; selecting the rowkey optimization method that minimizes the dispersion value to optimize the rowkey value, and using the optimized rowkey value to simulate access to the HBase open-source database.

[0055] Figure 4 This is a flowchart of the discrete sorting according to an embodiment of the present invention, such as... Figure 4 As shown, the distribution of the designed rowkey across existing regions is statistically analyzed using the calculation engine, i.e., the number of primary key data rows distributed across each region; among them, the dispersion of the designed rowkey is calculated using a preset calculation method, with the formula as follows: Where n is the total number of partitions, r j It is the number of rows of data in partition j. It is the average number of data rows across the n partitions. It should be noted that for various preset hash algorithms, the Datahash values ​​are sorted in ascending order.

[0056] As an optional implementation, determining that there are no hot regions in the HBase open-source database during the simulated access process includes: obtaining the number of simulated access requests for each region in the HBase open-source database during the simulated access period; determining the proportion of simulated access requests for each region based on the number of simulated access requests; and determining that there are no hot regions in the HBase open-source database when the proportion of simulated access requests does not exceed a predetermined threshold.

[0057] Figure 5 This is a flowchart illustrating the simulation of access to determine the proportion of region requests according to an embodiment of the present invention, such as... Figure 5As shown, the rowkey values ​​accessed at preset time intervals are retrieved from the in-memory database. For example, the rowkey values ​​accessed within the last two hours are selected. The rowkey is designed according to the hash algorithm with the smallest Datahash value. The system dynamically simulates client access to the HBase cluster, that is, it counts the distribution of the designed rowkey across each region based on the start and end keys of each region, and then counts the number of accesses to each region. Next, the request ratio of the region during the simulated access period is calculated. If it does not exceed the threshold, the rowkey design method is selected to optimize the rowkey design of the HBase table. If it still exceeds the threshold, the Datahash value of the rowkey design method is calculated and sorted to select the next ranked rowkey design method for simulation, until a suitable rowkey design method is selected. After the rowkey design method is selected, the HBase hotspot problem is eliminated.

[0058] As an optional embodiment, the method further includes: when the proportion of simulated access requests still exceeds a predetermined threshold, reprocessing the rowkey value to continue simulating access to the HBase open-source database until it is determined that there are no hot regions in the HBase open-source database.

[0059] Figure 6 This is a flowchart of a preferred method for handling hotspots in a database according to an embodiment of the present invention, such as... Figure 6 As shown, the process first obtains the client's access request and records the rowkey value of the client's request. Then, it detects hotspot issues and determines whether a hotspot issue has occurred. Based on the determination result, it selects the algorithm involved in rowkey, calculates and sorts the Datahash value of the rowkey design method, and finally optimizes the rowkey value.

[0060] The hotspot handling method in the database provided in this embodiment of the invention is applied to a system for eliminating hotspots in a key-value database. This system may include the following modules:

[0061] The storage module is used to store the key values ​​of key-value database access requests within a certain preset time period.

[0062] The detection module is used to detect the number of requests for each region in the table and calculate the region request ratio.

[0063] The key design module is used to generate a first transformed value based on different hash algorithms, and then generate a second transformed value.

[0064] The calculation module designs different methods for different keys and calculates the datahash value.

[0065] The sorting module sorts data based on its datahash value.

[0066] Problem scenario reproduction module: Based on the rowkey values ​​stored in the storage module at a certain preset time interval, dynamically simulate client access to the key-value database to reproduce the problem scenario, count the number of requests for each region and calculate the request ratio, and determine whether the hotspot problem of the database table has been resolved.

[0067] As can be seen from the above, the hotspot handling method in the database provided by this invention can store access records of the key-value database within a certain period of time in an in-memory database as an access log. After a hotspot problem occurs, the scenario of accessing the key-value database is dynamically simulated to determine whether the designed method has eliminated the hotspot. Next, the primary key data of the hotspot problem is obtained, and the second transformed value obtained by hashing and modulo operations on the primary key data is concatenated with the original primary key data to form the final key. Then, the Datahash value of each key design method is calculated and sorted. Finally, the key design method with the smallest Datahash value is selected to design the table of the hotspot, and the access to the key-value database is simulated based on data stored in the database at a certain preset time interval. The number of region requests is recorded and the region request ratio is calculated to determine whether the hotspot problem is solved. If it is not solved, the key design method with the next highest Datahash value is selected until the best hotspot solution is found. Furthermore, it has the following advantages:

[0068] (1) A method for statically storing key-value database access records is provided: an in-memory database is used to store access records of key-value database tables, with the key being "timestamp-key" and the value being "visits". The granularity of the timestamp is down to the minute level, and the visits are the number of times a certain key is accessed in a certain minute. The storage period is also set.

[0069] (2) A method for detecting region hotspot issues is provided: the given formula calculates the region request ratio of each region, sets a threshold, and if the threshold is exceeded, it is considered that a region hotspot issue has occurred.

[0070] (3) A method for designing a key is provided: the transformed value obtained by hashing the primary key data and performing modulo operations is concatenated with the original primary key data to form the final rowkey.

[0071] (4) A method for evaluating the dispersion of key design is provided: the dispersion of the key distribution on the region after design is calculated by a given formula, i.e., the Datahash value. Then, the design algorithms of all keys can be sorted in order of Datahash value from small to large. The smaller the Datahash value, the better the key design algorithm.

[0072] (5) A method is provided to dynamically simulate access to the key-value database, reproduce the problem scenario, and select the optimal hotspot elimination scheme: Based on the access records of the key in the key-value database stored in the memory database within a certain time period, such as the access records within two hours, the key is transformed according to the selected design method, the distribution of the transformed key in each region is statistically analyzed, and then the number of accesses to each region within the selected time period is obtained. The region request ratio is calculated. If it does not exceed the set threshold, the method is considered feasible.

[0073] Furthermore, compared to other existing methods for eliminating hotspots in key-value database regions, the hotspot handling method provided in this embodiment of the invention is more effective. It can resolve hotspot issues arising during actual table access based on actual business usage, significantly improving the database's capacity for processing and storing large amounts of data. This allows for full utilization and leverage of database performance, and automatically resolves region hotspot issues caused by continuous data writing, changes over time, or changes in business requirements. It provides multiple key design methods based on hash algorithms and offers an optimal selection function. By considering the dispersion of key distribution and the reproducibility of problem scenarios, the best key design method is selected, ensuring that the hotspot problem in the key-value database table is truly resolved.

[0074] Example 2

[0075] According to another aspect of the present invention, a hotspot processing apparatus for a database is also provided. Figure 7 This is a schematic diagram of a database hotspot processing apparatus according to an embodiment of the present invention, such as... Figure 7 As shown, it includes: an acquisition module 71, a processing module 73, a simulated access module 75, a determination module 77, and an optimization module 79. The following describes the hotspot processing device in this database.

[0076] The acquisition module 71 is used to acquire all primary key data corresponding to the hot regions in the HBase open source database when it is determined that there are hot regions in the HBase open source database;

[0077] Processing module 73 is used to process all primary key data using a predetermined processing method to process the rowkey value and obtain a new rowkey value, wherein the rowkey value is the value generated when the HBase open source database is accessed;

[0078] The simulated access module 75 is used to simulate access to the HBase open-source database based on the new rowkey value;

[0079] Module 77 is used to determine the target optimization method when it is determined that there are no hot regions in the HBase open source database during the simulated access process.

[0080] Optimization module 79 is used to optimize rowkey values ​​using a target optimization method to handle hot regions in the HBase open-source database.

[0081] It should be noted that the acquisition module 71, processing module 73, simulated access module 75, determination module 77, and optimization module 79 mentioned above correspond to steps S102 to S110 in Embodiment 1. The instances and application scenarios implemented by the above modules and their corresponding steps are the same, but they are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules, as part of the device, can be executed in a computer system such as a set of computer-executable instructions.

[0082] As can be seen from the above, in this embodiment of the invention, firstly, the acquisition module 71 can acquire all primary key data corresponding to the hot regions in the HBase open-source database when it is determined that there are hot regions; then, the processing module 73 can process all primary key data using a predetermined processing method to process the rowkey value, obtaining a new rowkey value, wherein the rowkey value is the value generated when the HBase open-source database is accessed; next, the simulated access module 75 can simulate access to the HBase open-source database based on the new rowkey value; then, the determination module 77 can determine that there are no hot regions in the HBase open-source database during the simulated access process, and determine the predetermined processing method as the target optimization method; finally, the optimization module 79 can optimize the rowkey value using the target optimization method to handle the hot regions in the HBase open-source database. Through the hotspot processing device in the database provided by this embodiment of the invention, the purpose of processing the database based on the judgment result of whether there are hot regions in the HBase open-source database is achieved, thereby realizing the technical effect of making the data distribution uniform and reasonable, and thus solving the technical problem of region hotspots often caused by excessive data in the use of key-value databases in related technologies.

[0083] Optionally, the acquisition module includes: a first acquisition unit, used to acquire the number of access requests for each region in the HBase open-source database based on the rowkey value; a first determination unit, used to determine the access request ratio of each region based on the number of access requests; and a second determination unit, used to determine that there are hot regions in the HBase open-source database when the access request ratio exceeds a predetermined threshold.

[0084] Optionally, the processing module includes: a discrete processing unit, used to perform discrete processing on all primary key data using a predetermined processing method to obtain a first deformed value; a third determining unit, used to determine that the absolute value of the remainder obtained by dividing the first deformed value by the total value corresponding to each region in the HBase open-source database is a second deformed value; and a concatenation unit, used to concatenate the second deformed value with the primary keys corresponding to all primary key data to obtain a new rowkey value.

[0085] Optionally, the simulated access module includes: a statistics unit for counting the number of rows of primary key data distributed across each region in the HBase open-source database; a fourth determination unit for determining the dispersion value of the new rowkey value based on the number of rows of primary key data across each region; and an optimization unit for selecting the rowkey optimization method that minimizes the dispersion value to optimize the rowkey value, and using the optimized rowkey value to simulate accessing the HBase open-source database.

[0086] Optionally, the determining module includes: a second obtaining unit, used to obtain the number of simulated access requests for each region in the HBase open-source database during the simulated access period; a fifth determining unit, used to determine the proportion of simulated access requests for each region based on the number of simulated access requests; and a sixth determining unit, used to determine that there are no hot regions in the HBase open-source database when the proportion of simulated access requests does not exceed a predetermined threshold.

[0087] Optionally, the apparatus further includes a reprocessing unit, used to reprocess the rowkey value when the proportion of simulated access requests still exceeds a predetermined threshold, so as to continue simulating access to the HBase open-source database until it is determined that there are no hot regions in the HBase open-source database.

[0088] Example 3

[0089] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is run by a processor, it controls the device where the computer-readable storage medium is located to perform the database hotspot processing method described above.

[0090] Example 4

[0091] According to another aspect of the present invention, a processor is also provided, which is used to run a computer program, wherein the computer program executes the database hotspot processing method described above.

[0092] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0093] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0094] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0095] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0096] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0097] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0098] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for handling hotspots in a database, characterized in that, include: When it is determined that there is a hot region in the HBase open source database, obtain all primary key data corresponding to the hot region in the HBase open source database; All primary key data are processed using a predetermined processing method to process the rowkey value and obtain a new rowkey value, wherein the rowkey value is the value generated when the HBase open source database is accessed; Simulated access to the HBase open-source database is performed based on the new rowkey value; When it is determined that the HBase open-source database does not have the hotspot region during the simulated access process, the predetermined processing method is determined to be the target optimization method; The rowkey value is optimized using the target optimization method to address the presence of hot regions in the HBase open-source database. Determining that there are hot regions in the HBase open-source database includes: obtaining the number of access requests for each region in the HBase open-source database based on the rowkey value; determining the access request ratio for each region based on the number of access requests; and determining that there are hot regions in the HBase open-source database when the access request ratio exceeds a predetermined threshold. Simulated access to the HBase open-source database based on the new rowkey value includes: counting the number of rows of primary key data distributed across each region of the HBase open-source database; determining the dispersion value of the new rowkey value based on the number of rows of primary key data across each region; optimizing the rowkey value by selecting the rowkey optimization method that minimizes the dispersion value; and simulating access to the HBase open-source database using the optimized rowkey value.

2. The method according to claim 1, characterized in that, All primary key data are processed using a predetermined processing method to process the rowkey values ​​and obtain new rowkey values, including: The first deformed value is obtained by discretizing all the primary key data using the predetermined processing method. The absolute value of the remainder obtained by dividing the first deformed value by the total number of each region in the HBase open-source database is the second deformed value. The second deformed value is concatenated with the primary key corresponding to all the primary key data to obtain the new rowkey value.

3. The method according to claim 1 or 2, characterized in that, Determining that the HBase open-source database does not contain the hotspot region during the simulated access process includes: Obtain the number of simulated access requests for each region in the HBase open-source database during the simulated access period; The proportion of simulated access requests for each region is determined based on the number of simulated access requests. If the simulated access request ratio does not exceed a predetermined threshold, it is determined that the HBase open-source database does not contain the hotspot region.

4. The method according to claim 3, characterized in that, The method further includes: If the proportion of simulated access requests still exceeds a predetermined threshold, the rowkey value is reprocessed to continue simulating access to the HBase open-source database until it is determined that the HBase open-source database does not contain the hotspot region.

5. A hotspot processing device for a database, characterized in that, include: The acquisition module is used to acquire all primary key data corresponding to the hot region in the HBase open-source database when it is determined that there is a hot region in the HBase open-source database; The processing module is used to process all the primary key data using a predetermined processing method to process the rowkey value and obtain a new rowkey value, wherein the rowkey value is the value generated when the HBase open source database is accessed; The simulated access module is used to simulate access to the HBase open-source database based on the new rowkey value; The determination module is used to determine the predetermined processing method as the target optimization method when it is determined that the HBase open source database does not have the hot region during the simulated access process; An optimization module is used to optimize the rowkey value using the target optimization method to handle the existence of hot regions in the HBase open-source database; The acquisition module includes: a first acquisition unit, configured to acquire the number of access requests for each region in the HBase open-source database based on the rowkey value; a first determination unit, configured to determine the access request ratio of each region based on the number of access requests; and a second determination unit, configured to determine that there is a hot region in the HBase open-source database when there is an access request ratio exceeding a predetermined threshold in the access request ratio. The simulated access module includes: a statistics unit, used to count the number of rows of primary key data distributed across each region of the HBase open-source database; a fourth determination unit, used to determine the dispersion value of the new rowkey value based on the number of rows of primary key data across each region; and an optimization unit, used to select the rowkey optimization method that minimizes the dispersion value to optimize the rowkey value, and use the optimized rowkey value to simulate accessing the HBase open-source database.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed by a processor, it controls the device where the computer-readable storage medium is located to perform the database hotspot processing method according to any one of claims 1 to 4.

7. A processor, characterized in that, The processor is used to run a computer program, wherein the computer program executes the method for processing hotspots in the database according to any one of claims 1 to 4.

Images