Data query method, device, equipment and storage medium based on row key salt value
By adding salt values to the data row keys in the HBase and Presto systems, dispersing them to multiple data partitions and forming a salt value data row key encapsulation list, data hotspot problems are solved and query performance is improved.
Patent Information
- Application Number
- CN202010727785.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-07-23
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2040-07-23
AI Technical Summary
In HBase and Presto systems, when the data reaches 100 million, hot issues arise, resulting in high resource occupancy, low computing efficiency, and degradation of query performance.
By adding salt values to the data row keys, dispersing them into multiple data partitions, and determining the start and terminate keys in the data partition, forming a salt value data row key encapsulation list, optimizing the processing of data query requests.
Reduce resource occupancy, improve computing efficiency, improve data query performance, and solve hot issues in data partitions.
Smart Images

Figure CN111858659B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of big data technology, and in particular to a data query method, device, equipment and storage medium based on row key salt value. Background Art
[0002] Hbase is a distributed, column-oriented open source database and a distributed and scalable big data warehouse. HBase can integrate the key / value storage mode to bring real-time query capabilities, as well as offline processing or batch processing capabilities through MapReduce. Hbase can quickly query records in large amounts of data and obtain comprehensive analysis reports from them. Presto is a distributed query engine that performs calculations based on memory. It does not store data itself, but can access multiple data sources and supports cascade query data query engines across data sources.
[0003] In the prior art, data is usually queried based on Hbase and Presto. However, when the data reaches the billion level or above, hot spots will appear in the data, the resource occupancy rate is high and the computing efficiency is low, resulting in a decrease in the query performance of the open source Presto HBase. Summary of the invention
[0004] The main purpose of the present invention is to solve the problem of poor query performance caused by low computing efficiency and high resource occupancy rate when querying data in the prior art.
[0005] The first aspect of the present invention provides a data query method based on row key salt value, comprising: obtaining table data in a database, and reading multiple data row keys from the table data, wherein the table data includes multiple data partitions; counting the number of the multiple data partitions to obtain the number of data partitions, obtaining multiple target salt values according to the number of data partitions, and obtaining multiple salt value data row keys according to the multiple target salt values and the multiple data row keys; determining multiple target data partitions according to the multiple salt value data row keys and the multiple data partitions; obtaining a data query request from a target terminal, and in the multiple target data partitions, obtaining a data query request from a target terminal according to the number of data partitions; obtaining a data query request from a target terminal, and .... According to the query request and the multiple salt value data row keys, determine the data start key corresponding to the start salt value data row key and the data end key corresponding to the end salt value data row key; encapsulate the multiple salt value data row keys corresponding to the data start key to the data end key to obtain a salt value data row key encapsulation list, the salt value data row key encapsulation list includes the start salt value data row key corresponding to the data start key and the end salt value data row key corresponding to the data end key; determine the target salt value data row key in the salt value data row key encapsulation list according to the data query request of the target terminal, and transmit the target salt value data row key to the target terminal.
[0006] Optionally, in a first implementation method of the first aspect of the present invention, counting the number of the multiple data partitions to obtain the number of data partitions, obtaining multiple target salt values based on the number of data partitions, and obtaining multiple salt value data row keys based on the multiple target salt values and the multiple data row keys includes: counting the number of the multiple data partitions to obtain the number of data partitions; setting a salt value range according to the number of data partitions to obtain multiple salt value selections, the number of salt value selections is the same as the number of data partitions, and each salt value selection corresponds to multiple initial salt values; setting separators for the multiple initial salt values to obtain multiple target salt values; and using a random number algorithm to randomly set the multiple target salt values to the prefix position of the multiple data row keys to obtain multiple salt value data row keys.
[0007] Optionally, in a second implementation manner of the first aspect of the present invention, determining multiple target data partitions based on the multiple salt value data row keys and the multiple data partitions includes: reading corresponding target salt values in the multiple salt value data row keys in sequence to obtain multiple target row key salt values; classifying the multiple salt value data row keys based on the multiple target row key salt values to obtain multiple salt value data row key categories; and assigning the multiple salt value data row key categories to corresponding data partitions to obtain multiple target data partitions.
[0008] Optionally, in a third implementation method of the first aspect of the present invention, obtaining the data query request of the target terminal, and determining, in the multiple target data partitions, according to the data query request and the multiple salt value data row keys, the data start key corresponding to the start salt value data row key and the data end key corresponding to the end salt value data row key, includes: obtaining the data query request of the target terminal, and reading multiple row key header data respectively from the multiple salt value data row keys, the row key header data being the first digit or letter after the target salt value; determining the data start key and the data end key based on the data query request, the ASK coding sequence and the multiple row key header data.
[0009] Optionally, in a fourth implementation manner of the first aspect of the present invention, determining the target salt value data row key in the salt value data row key encapsulation list according to the data query request of the target terminal, and transmitting the target salt value data row key to the target terminal includes: reading the encapsulation row key identifier in the salt value data row key encapsulation list; judging whether the encapsulation row key identifier matches a preset row key identifier; if the encapsulation row key identifier matches the preset row key identifier, determining a target query mode; determining the target salt value data row key according to the data query request of the target terminal, the target query mode and the salt value data row key encapsulation list, and transmitting the target salt value data row key to the target terminal.
[0010] Optionally, in a fifth implementation manner of the first aspect of the present invention, determining the target salt value data row key according to the data query request of the target terminal, the target query mode and the salt value data row key encapsulation list, and transmitting the target salt value data row key to the target terminal includes: performing predicate parsing on the data query request of the target terminal through the target query mode to obtain a query field name; reading multiple preset data row key field names from the salt value data row key encapsulation list; matching the query field name with the multiple preset data row key field names to obtain a target field name, determining the target salt value data row key according to the target field name, and transmitting the target salt value data row key to the target terminal.
[0011] Optionally, in a sixth implementation of the first aspect of the present invention, before obtaining table data in the database and reading multiple data row keys from the table data, wherein the table data includes multiple data partitions, the data query method based on row key salt value also includes: setting a monitoring file and a configuration file in the server, wherein, when it is detected through the monitoring file that new data is added, data is deleted, or data is modified in the database or server, the changed data is configured to the server cluster in real time through the configuration file.
[0012] The second aspect of the present invention provides a data query device based on row key salt value, including: a data row key acquisition module, used to acquire table data in a database, and read multiple data row keys from the table data, wherein the table data includes multiple data partitions; a salt value row key setting module, used to count the number of the multiple data partitions, obtain the number of data partitions, obtain multiple target salt values according to the number of data partitions, and obtain multiple salt value data row keys according to the multiple target salt values and the multiple data row keys; a data partition setting module, used to determine multiple target data partitions according to the multiple salt value data row keys and the multiple data partitions; a data row key selection module, used to acquire a data query request from a target terminal, and select the data row key from the multiple target terminals. In the target data partition, according to the data query request and the multiple salt value data row keys, the data start key corresponding to the start salt value data row key and the data end key corresponding to the end salt value data row key are determined; a row key encapsulation module is used to encapsulate the corresponding multiple salt value data row keys from the data start key to the data end key to obtain a salt value data row key encapsulation list, the salt value data row key encapsulation list includes the corresponding start salt value data row key of the data start key and the end salt value data row key corresponding to the data end key; a target data query module is used to determine the target salt value data row key in the salt value data row key encapsulation list according to the data query request of the target terminal, and transmit the target salt value data row key to the target terminal.
[0013] Optionally, in a first implementation method of the second aspect of the present invention, the salt value row key setting module is specifically used to: count the number of the multiple data partitions to obtain the number of data partitions; set the salt value range according to the number of data partitions to obtain multiple salt value selections, the number of salt value selections is the same as the number of data partitions, and each salt value selection corresponds to multiple initial salt values; set separators for the multiple initial salt values to obtain multiple target salt values; use a random number algorithm to randomly set the multiple target salt values to the prefix position of the multiple data row keys to obtain multiple salt value data row keys.
[0014] Optionally, in a second implementation method of the second aspect of the present invention, the data partition setting module is specifically used to: read the corresponding target salt values in the multiple salt value data row keys in sequence to obtain multiple target row key salt values; classify the multiple salt value data row keys based on the multiple target salt values to obtain multiple salt value data row key categories; assign the multiple salt value data row key categories to corresponding data partitions to obtain multiple target data partitions.
[0015] Optionally, in a third implementation method of the second aspect of the present invention, the data row key selection module is specifically used to: obtain a data query request from the target terminal, and read multiple row key header data from the multiple salt value data row keys respectively, the row key header data being the first digit or letter after the target salt value; determine the data start key and the data end key based on the data query request, the ASKS coding sequence and the multiple row key header data.
[0016] Optionally, in a fourth implementation manner of the second aspect of the present invention, the target data query module specifically includes: an identifier reading unit, used to read the encapsulation row key identifier in the salt value data row key encapsulation list; a judgment unit, used to judge whether the encapsulation row key identifier matches the preset row key identifier; a query mode determination unit, used to determine the target query mode if the encapsulation row key identifier matches the preset row key identifier; a target data selection unit; determine the target salt value data row key according to the data query request of the target terminal, the target query mode and the salt value data row key encapsulation list, and transmit the target salt value data row key to the target terminal.
[0017] Optionally, in a fifth implementation method of the second aspect of the present invention, the target data selection unit is specifically used to: perform predicate parsing on the data query request of the target terminal through the target query pattern to obtain a query field name; read multiple preset data row key field names from the salt value data row key encapsulation list; match the query field name with the multiple preset data row key field names to obtain a target field name, determine a target salt value data row key according to the target field name, and transmit the target salt value data row key to the target terminal.
[0018] Optionally, in a sixth implementation method of the second aspect of the present invention, the data query device based on row key salt value also includes: a monitoring and configuration module, which is used to set a monitoring file and a configuration file in the server, wherein when it is detected through the monitoring file that new data is added, data is deleted, or data is modified in the database or server, the configuration file is used to configure the changed data to the server cluster in real time through the configuration file.
[0019] The third aspect of the present invention provides a data query device based on row key salt value, comprising: a memory and at least one processor, the memory storing instructions, the memory and the at least one processor being interconnected via lines; the at least one processor calling the instructions in the memory so that the data query device based on row key salt value executes the above-mentioned data query method based on row key salt value.
[0020] A fourth aspect of the present invention provides a computer-readable storage medium, in which instructions are stored, which, when executed on a computer, enable the computer to execute the above-mentioned data query method based on row key salt value.
[0021] In the technical solution provided by the present invention, table data in a database is obtained, and multiple data row keys are read from the table data, wherein the table data includes multiple data partitions; the number of the multiple data partitions is counted to obtain the number of data partitions, multiple target salt values are obtained according to the number of data partitions, and multiple salt value data row keys are obtained according to the multiple target salt values and the multiple data row keys; multiple target data partitions are determined according to the multiple salt value data row keys and the multiple data partitions; a data query request of a target terminal is obtained, and in the multiple target data partitions, a query request is generated according to the data query request and the multiple data row keys. Multiple salt value data row keys, determine the data starting key corresponding to the starting salt value data row key and the data ending key corresponding to the ending salt value data row key; encapsulate the multiple salt value data row keys corresponding to the data starting key to the data ending key to obtain a salt value data row key encapsulation list, the salt value data row key encapsulation list includes the starting salt value data row key corresponding to the data starting key and the ending salt value data row key corresponding to the data ending key; determine the target salt value data row key in the salt value data row key encapsulation list according to the data query request of the target terminal, and transmit the target salt value data row key to the target terminal. In an embodiment of the present invention, by adding salt values to the data row keys, the data is dispersed to each partition, and then all the salt value data row keys are packaged, and finally the target salt value data row key is queried in the salt value data row key encapsulation list, which solves the problem of data hotspots in data partitions, reduces resource occupancy, improves computing efficiency, and thus improves data query performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 A schematic diagram of an embodiment of a data query method based on row key salt value in an embodiment of the present invention;
[0023] Figure 2 It is a schematic diagram of another embodiment of the data query method based on row key salt value in an embodiment of the present invention;
[0024] Figure 3 A schematic diagram of an embodiment of a data query device based on row key salt value in an embodiment of the present invention;
[0025] Figure 4 It is a schematic diagram of another embodiment of a data query device based on row key salt value in an embodiment of the present invention;
[0026] Figure 5 The figure is a schematic diagram of an embodiment of a data query device based on row key salt value in an embodiment of the present invention. DETAILED DESCRIPTION
[0027] The embodiments of the present invention provide a data query method, apparatus, device and storage medium based on row key salt value, which is used to disperse data to each partition by adding salt to the data row key, obtain multiple salt value data row keys and data start key and data end key, package the corresponding salt value data row keys from the data start key to the data end key, obtain the salt value data row key encapsulation list, and then query the target salt value data row key in the salt value data row key encapsulation list according to the target query mode, thereby solving the problem of data hot spots in data partitions, reducing resource occupancy, improving computing efficiency, and thus improving data query performance.
[0028] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0029] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , an embodiment of a data query method based on a row key salt value in an embodiment of the present invention includes:
[0030] 101. Obtain table data in a database, and read multiple data row keys from the table data, where the table data includes multiple data partitions;
[0031] The server reads table data including a plurality of data partitions in a database, and the server reads a plurality of data row keys from the table data including the plurality of data partitions.
[0032] In this embodiment, the database uses the HBase database, which is a distributed, column-oriented open source database. The data in the HBase database is table data. When the table data becomes larger as the number of records increases, it will gradually split into multiple data partition regions. Different regions will be assigned to the corresponding RegionServer for management. The HBase database uses KeyValue to store data, and the data row key Rowkey is the Key in KeyValue. Rowkey is also a binary code stream with a maximum length of 64KB, and the content can be customized by the user. When loading data, it is generally carried out from small to large according to the binary order of Rowkey. After reading the table data in the database, the server reads multiple data row keys from the table data.
[0033] It is understandable that the execution subject of the present invention may be a data query device based on row key salt value, or a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking the server as the execution subject as an example.
[0034] 102. Count the number of multiple data partitions to obtain the number of data partitions, obtain multiple target salt values according to the number of data partitions, and obtain multiple salt value data row keys according to the multiple target salt values and the multiple data row keys;
[0035] The server obtains the number of data partitions through statistics, and then determines multiple target salt values for obtaining multiple salt value data row keys according to the number of data partitions.
[0036] Table data is distributed to different data partitions according to different data row keys. Unreasonable data row key distribution will lead to data hotspot problems. Data hotspot problems can be understood as: multiple data row keys are distributed to the same data partition, and a large number of data requests directly access this data partition, causing the single server where this data partition is located to exceed its own capacity, causing performance degradation or even making the data partition unavailable.
[0037] In this embodiment, the server randomly sets different salt values as prefixes for multiple data row keys to obtain multiple data row keys with different prefixes, that is, multiple salt value data row keys, to solve the data hot spot problem. The specific solution is: the server assigns a random prefix, that is, a salt value, to multiple data row keys. The number of salt values included in the salt value range is the same as the number of data partitions. For example, if the initial number of partitions is 5, then the salt value can be set to a-, b-, c-, d- or e-, and a-, b-, c-, d-, or e- is randomly set at the front of multiple data row keys as prefixes to obtain multiple salt value data row keys, so that the multiple data row keys are sorted differently from before. The server assigns multiple salt value data row keys to corresponding data partitions according to their salt values.
[0038] 103. Determine multiple target data partitions according to multiple salt value data row keys and multiple data partitions;
[0039] After obtaining multiple salt value data row keys, the server can break up the original data partitions, and distribute the multiple salt value data row keys to the corresponding data partitions according to the prefixes of the salt value data row keys to obtain multiple target data partitions.
[0040] It should be noted that the prefix of the salt value data row key can be either letters or numbers.
[0041] For example, there are five data partitions, and the prefixes corresponding to the multiple salt value data row keys range from "a-" to "e-". The server assigns the multiple salt value data row keys with the prefix of "a-" to the data partition ranked first in sequence, the multiple salt value data row keys with the prefix of "b-" to the data partition ranked second in sequence, the multiple salt value data row keys with the prefix of "c-" to the data partition ranked third in sequence, the multiple salt value data row keys with the prefix of "d-" to the data partition ranked fourth in sequence, and the multiple salt value data row keys with the prefix of "e-" to the data partition ranked fifth in sequence, thereby obtaining multiple target data partitions.
[0042] 104. Obtain a data query request from a target terminal, and determine, in multiple target data partitions, a data start key corresponding to a start salt value data row key and a data end key corresponding to an end salt value data row key according to the data query request and multiple salt value data row keys;
[0043] The server obtains a data query request from a target terminal, and determines a data start key and a data end key in multiple target data partitions according to the data query request and multiple salt value data row keys, wherein the data start key is a data key corresponding to a start salt value data row key, and the data end key is a data key corresponding to an end salt value data row key.
[0044] It should be noted that the target terminal may issue multiple data query requests at the same time.
[0045] For example, suppose the data query request of the target terminal is "query all employees of Company A", and the multiple salt value row key data are "a-100XX", "a-200XX" ... "a-N00XX". The server reads the header data of multiple salt value data row keys in multiple target partitions, and the header data of multiple salt value data row keys are "1", "2" ... "N" respectively. According to the data query request of "query all employees of Company A", the server determines the data start key "a-100XX" and the data end key "a-900XX" in the multiple salt value row key data according to the order of the Asker code and the order of the multiple row key headers.
[0046] 105. Encapsulate multiple salt value data row keys corresponding to the data start key to the data end key to obtain a salt value data row key encapsulation list, wherein the salt value data row key encapsulation list includes a start salt value data row key corresponding to the data start key and an end salt value data row key corresponding to the data end key;
[0047] The server encapsulates the starting salt value data row key corresponding to the data starting key, the ending salt value data row key corresponding to the data ending key, and the salt value data row keys corresponding to the data starting key to the data ending key to obtain a salt value data row key encapsulation list.
[0048] The server encapsulates multiple salt value data row keys into a salt value data row key list, which can reduce the number of interactions between the server and the database, and one query can return multiple results.
[0049] For example, the data query request of the target terminal is "query the administrative staff of Company A" and "query the test staff with two years of service in Company A". At this time, the data start key can be regarded as the employee with the earliest time of joining Company A, and the data end key can be regarded as the employee with the latest time of joining Company A. In this embodiment, the data start key can also be regarded as the employee with the oldest age in Company A, and the data end key can be regarded as the employee with the youngest age in Company A. The embodiment of the present invention does not make specific restrictions on the data start key and the data end key. The server encapsulates the employee with the earliest time of joining Company A corresponding to the data start key to the employee with the latest time of joining Company A corresponding to the data end key into a salt value data row key list A. The server can obtain the query results of "administrative staff of Company A" and "test staff with two years of service in Company A" in the salt value data row key list A. It is not necessary to first obtain the query result of "administrative staff of Company A" in the database, return the result to the target terminal, and then obtain the query result of "test staff with two years of service in Company A" from the database, and return the result to the target terminal again.
[0050] 106. Determine a target salt value data row key in the salt value data row key encapsulation list according to the data query request of the target terminal, and transmit the target salt value data row key to the target terminal.
[0051] The server obtains the target salt value data row key from the salt value data row key encapsulation list according to the data query request of the target terminal, and transmits the target salt value data row key to the target terminal.
[0052] For example, the data query requests of the target terminal are "Query the administrative staff of Company A" and "Query the testing staff with two years of service at Company A", the server directly obtains the target salt value data row keys of "all employees of Company A" and "testing staff with two years of service at Company A" from the salt value data row key list through the query requests of "Query the administrative staff of Company A" and "Query the testing staff with two years of service at Company A", and transmits the target salt value data row keys of "all employees of Company A" and "testing staff with two years of service at Company A" to the target terminal.
[0053] In an embodiment of the present invention, by adding salt values to data row keys, the data is dispersed to various partitions, and then all salt value data row keys are packaged, and finally the target salt value data row key is queried in the salt value data row key encapsulation list, which solves the problem of data hotspots in data partitions, reduces resource occupancy, improves computing efficiency, and thus improves data query performance.
[0054] See also Figure 2 Another embodiment of the data query method based on row key salt value in the embodiment of the present invention includes:
[0055] 201. Set a monitoring file and a configuration file in the server, wherein when it is detected through the monitoring file that data is added, deleted or modified in the database or the server, the changed data is configured to the server cluster in real time through the configuration file.
[0056] The server configures the monitoring file and configuration file for itself. The server can monitor the changes in the data in the database or server through the monitoring file, and the server can configure the changed data to the server cluster through the configuration file. When the server monitors the addition, deletion or modification of data in the database or server through the monitoring file, the server configures the changed data to the server cluster in real time through the configuration file.
[0057] 202. Obtain table data in a database, and read multiple data row keys from the table data, where the table data includes multiple data partitions;
[0058] The server reads table data including a plurality of data partitions in a database, and the server reads a plurality of data row keys from the table data including the plurality of data partitions.
[0059] In this embodiment, the database adopts HBase database, which is a distributed, column-oriented open source database. The data in the HBase database is table data. When the table data becomes larger as the number of records continues to increase, it will gradually split into multiple data partition regions. Different regions will be assigned to the corresponding RegionServer for management. The HBase database uses KeyValue to store data, so the data row key Rowkey is the Key in KeyValue. Rowkey is also a binary code stream with a maximum length of 64KB, and the content can be customized by the user. When loading data, it is generally carried out from small to large according to the binary order of Rowkey. After reading the table data in the database, the server reads multiple data row keys from the table data.
[0060] 203. Count the number of multiple data partitions to obtain the number of data partitions, obtain multiple target salt values according to the number of data partitions, and obtain multiple salt value data row keys according to the multiple target salt values and the multiple data row keys;
[0061] The server obtains the number of data partitions through statistics, and then determines multiple target salt values for obtaining multiple salt value data row keys according to the number of data partitions.
[0062] Table data is distributed to different data partitions according to different data row keys. Unreasonable data row key distribution will lead to data hotspot problems. Data hotspot problems can be understood as: multiple data row keys are distributed to the same data partition, and a large number of data requests directly access this data partition, causing the single server where this data partition is located to exceed its own capacity, causing performance degradation or even making the data partition unavailable.
[0063] In this embodiment, the server randomly sets different salt values as prefixes for multiple data row keys to obtain multiple data row keys with different prefixes, that is, multiple salt value data row keys, to solve the data hot spot problem. The specific solution is: the server assigns a random prefix, that is, a salt value, to multiple data row keys. The number of salt values included in the salt value range is the same as the number of data partitions. For example, if the initial number of partitions is 5, then the salt value can be set to a-, b-, c-, d- or e-, and a-, b-, c-, d-, or e- is randomly set at the front of multiple data row keys as prefixes to obtain multiple salt value data row keys, so that the multiple data row keys are sorted differently from before. The server assigns multiple salt value data row keys to corresponding data partitions according to their salt values.
[0064] Specifically, the server first counts the number of multiple data partitions to obtain the number of data partitions. After obtaining the number of data partitions, the server sets the value range of the salt value and determines the salt value selection corresponding to the multiple initial salt values according to the value range of the salt value. If multiple initial salt values are to be effective, a separator needs to be set for the initial salt value. The server sets separators for multiple initial salt values to obtain multiple target salt values. Finally, the server uses a random number algorithm to randomly assign multiple target salt values to the prefix positions of multiple data row keys, thereby obtaining multiple salt value data row keys.
[0065] It should be noted that the use of a random number algorithm can evenly distribute multiple target salt values, so that in the later target data partitions, multiple salt value data row keys can be evenly distributed to multiple target data partitions. The number of selected salt values is the same as the number of data partitions. For example, if the number of data partitions is 5, the number of selected salt values is also 5, and each selected salt value corresponds to multiple initial salt values. For example, when the salt value is 1, the corresponding initial salt value can be 1000 or 20000.
[0066] For example, assuming that the number of data partitions obtained by the server statistics is 5, the server sets the salt value range to 1-5 according to the number of data partitions 5, the number of salt value selections is 5, and each salt value selection corresponds to multiple initial salt values, then there are multiple initial salt values a, multiple initial salt values b, multiple initial salt values c, multiple initial salt values d, and multiple initial salt values e. The server sets a separator "-" for the multiple initial salt values to obtain multiple target salt values a-, multiple target salt values b-, multiple target salt values c-, multiple target salt values d-, and multiple target salt values e-. The multiple target salt values a-, multiple target salt values b-, multiple target salt values c-, multiple target salt values d-, and multiple target salt values e- are randomly assigned to the head positions of multiple data row keys as prefixes, thereby obtaining multiple salt value data row keys.
[0067] 204. Determine multiple target data partitions according to multiple salt value data row keys and multiple data partitions;
[0068] After obtaining multiple salt value data row keys, the server can break up the original data partitions, and distribute the multiple salt value data row keys to the corresponding data partitions according to the prefixes of the salt value data row keys to obtain multiple target data partitions.
[0069] It should be noted that the prefix of the salt value data row key can be either letters or numbers.
[0070] For example, there are five data partitions, and the prefixes corresponding to the multiple salt value data row keys range from "a-" to "e-". The server assigns the multiple salt value data row keys with the prefix of "a-" to the data partition ranked first in sequence, the multiple salt value data row keys with the prefix of "b-" to the data partition ranked second in sequence, the multiple salt value data row keys with the prefix of "c-" to the data partition ranked third in sequence, the multiple salt value data row keys with the prefix of "d-" to the data partition ranked fourth in sequence, and the multiple salt value data row keys with the prefix of "e-" to the data partition ranked fifth in sequence, thereby obtaining multiple target data partitions.
[0071] Specifically, the server first reads the target salt values corresponding to the multiple salt value data row keys in sequence to obtain multiple target row key salt values; then the server classifies the multiple salt value data row keys into different salt value data row key categories that are the same as the data partition category according to the multiple target row key salt values; finally, the server assigns the salt value data row key categories to the corresponding data partitions, thereby obtaining multiple target data partitions.
[0072] For example, the server reads five target row key salt values, namely, target row key salt values a-, b-, c-, d-, and e-, and classifies and counts multiple salt value data with the target row key salt value a- to obtain the a-salt value data row key category, classifies and counts multiple salt value data with the target row key salt value b- to obtain the b-salt value data row key category, classifies and counts multiple salt value data with the target row key salt value c- to obtain the c-salt value data row key category, classifies and counts multiple salt value data with the target row key salt value d- to obtain the d-salt value data row key category, and classifies and counts multiple salt value data with the target row key salt value e- The server classifies and counts multiple salt value data to obtain an e-salt value data row key category; the server assigns the a-salt value data row key category to the data partition sorted first, the b-salt value data row key category to the data partition sorted second, the c-salt value data row key category to the data partition sorted third, the d-salt value data row key category to the data partition sorted fourth, and the e-salt value data row key category to the data partition sorted fifth, to obtain target data partition a, target data partition b, target data partition c, target data partition d and target data partition e.
[0073] 205. Obtain a data query request from a target terminal, and determine, in multiple target data partitions, a data start key corresponding to a start salt value data row key and a data end key corresponding to an end salt value data row key according to the data query request and multiple salt value data row keys;
[0074] The server obtains a data query request from a target terminal, and determines a data start key and a data end key in multiple target data partitions according to the data query request and multiple salt value data row keys, wherein the data start key is a data key corresponding to a start salt value data row key, and the data end key is a data key corresponding to an end salt value data row key.
[0075] It should be noted that the target terminal may issue multiple data query requests at the same time.
[0076] For example, suppose the data query request of the target terminal is "query all employees of Company A", and the multiple salt value row key data are "a-100XX", "a-200XX" and "a-N00XX". The server reads the header data of multiple salt value data row keys in multiple target partitions, and the header data of multiple salt value data row keys are "1", "2" and "N" respectively. According to the data query request of "query all employees of Company A", the server determines the data start key as "a-100XX" and the data end key as "a-900XX" in the multiple salt value row key data according to the order of the Asker code and the order of the multiple row key headers.
[0077] Specifically, the server first obtains the data query request of the target terminal, and then reads multiple row key header data from multiple salt value row keys respectively. The row key header data can be numbers or letters. For example, the format of a salt value row key data is "a-100XXX", and the row key header data is the first digit "1" after the salt value "a-". Based on the data query request of the target terminal, the server determines the first-ranked header row key data and the last-ranked row key data from multiple header row key data in the order of the Askers code. The salt value data row key corresponding to the first-ranked header row key data is the data start key, and the salt value data row key corresponding to the last-ranked header row key data is the data end key.
[0078] 206. Encapsulate multiple salt value data row keys corresponding to the data start key to the data end key to obtain a salt value data row key encapsulation list, wherein the salt value data row key encapsulation list includes a start salt value data row key corresponding to the data start key and an end salt value data row key corresponding to the data end key;
[0079] The server encapsulates the starting salt value data row key corresponding to the data starting key, the ending data row key corresponding to the data ending key, and the salt value data row keys corresponding to the data starting key to the data ending key to obtain a salt value data row key encapsulation list.
[0080] The server encapsulates multiple data row keys into a salt value data row key list, which can reduce the number of interactions between the server and the database, and one query can return multiple results.
[0081] For example, the data query request of the target terminal is "query the administrative staff of Company A" and "query the test staff with two years of service in Company A". At this time, the data start key can be regarded as the employee with the earliest time of joining Company A, and the data end key can be regarded as the employee with the latest time of joining Company A. In this embodiment, the data start key can also be regarded as the employee with the oldest age in Company A, and the data end key can be regarded as the employee with the youngest age in Company A. The embodiment of the present invention does not make specific restrictions on the data start key and the data end key. The server encapsulates the employee with the earliest time of joining Company A corresponding to the data start key to the employee with the latest time of joining Company A corresponding to the data end key into a salt value data row key list A. The server can obtain the query results of "administrative staff of Company A" and "test staff with two years of service in Company A" in the salt value data row key list A. It is not necessary to first obtain the query result of "administrative staff of Company A" in the database, return the result to the target terminal, and then obtain the query result of "test staff with two years of service in Company A" from the database, and return the result to the target terminal again.
[0082] 207. Determine a target salt value data row key in the salt value data row key encapsulation list according to the data query request of the target terminal, and transmit the target salt value data row key to the target terminal.
[0083] The server obtains the target salt value data row key from the salt value data row key encapsulation list according to the data query request of the target terminal, and transmits the target salt value data row key to the target terminal.
[0084] For example, the data query requests of the target terminal are "Query the administrative staff of Company A" and "Query the testing staff with two years of service at Company A", the server directly obtains the target salt value data row keys of "all employees of Company A" and "testing staff with two years of service at Company A" from the salt value data row key list through the query requests of "Query the administrative staff of Company A" and "Query the testing staff with two years of service at Company A", and transmits the target salt value data row keys of "all employees of Company A" and "testing staff with two years of service at Company A" to the target terminal.
[0085] Specifically, the server reads the encapsulated row key identifier in the salt value data row key encapsulation list; then the server determines whether the encapsulated row key identifier matches the preset row key identifier; if the encapsulated row key identifier matches the preset row key identifier, the target query mode is determined; finally, the server queries the salt value data row key encapsulation list according to the data query request and the target query mode of the target terminal, thereby obtaining the target salt value data row key, and transmitting the target salt value data row key to the target terminal.
[0086] The server queries the salt value data row key encapsulation list according to the data query request of the target terminal and the target query mode, and the specific process of obtaining the target salt value data row key is as follows:
[0087] The server performs dimension predicate analysis on the data query request of the target terminal through the target query mode to obtain the query field name. The server then reads multiple preset data row key field names through the salt value data row key encapsulation list. The server matches the query field name with the multiple preset data row key names to obtain the target field name. The salt value data row key corresponding to the target field name is the target salt value data row key. The server transmits the target salt value data row key to the target terminal.
[0088] For example, the query mode obtained by the server according to the identifier of the salt value data encapsulation row key encapsulation list is the SQL data query mode, and the SQL query mode is: select * from t_event where rk in('rk1','rk2'); assuming that the data query request of the target terminal is "query for test staff with a length of service of two years in Company A", and the salt value row key encapsulation list is "all employees of Company A", the server uses the SQL data query mode to determine that the target data row key is "test staff with a length of service of two years" in the salt value row key encapsulation list of "all employees of Company A", and the server finally transmits the target data row key of "test staff with a length of service of two years" to the target terminal.
[0089] In an embodiment of the present invention, by adding salt values to data row keys, the data is dispersed to various partitions, and then all salt value data row keys are packaged, and finally the target salt value data row key is queried in the salt value data row key encapsulation list, which solves the problem of data hotspots in data partitions, reduces resource occupancy, improves computing efficiency, and thus improves data query performance.
[0090] The above describes the data query method based on the row key salt value in the embodiment of the present invention. The following describes the data query device based on the row key salt value in the embodiment of the present invention. Figure 3 , an embodiment of a data query device based on row key salt value in an embodiment of the present invention includes:
[0091] A data row key acquisition module 301 is used to acquire table data in a database and read multiple data row keys from the table data, wherein the table data includes multiple data partitions;
[0092] The salt value row key setting module 302 is used to count the number of the multiple data partitions to obtain the number of data partitions, obtain multiple target salt values according to the number of data partitions, and obtain multiple salt value data row keys according to the multiple target salt values and the multiple data row keys;
[0093] A data partition setting module 303, configured to determine a plurality of target data partitions according to the plurality of salt value data row keys and the plurality of data partitions;
[0094] The data row key selection module 304 is used to obtain a data query request from a target terminal, and determine a data start key corresponding to a start salt value data row key and a data end key corresponding to an end salt value data row key in the multiple target data partitions according to the data query request and the multiple salt value data row keys;
[0095] A row key encapsulation module 305 is used to encapsulate multiple salt value data row keys corresponding to the data start key to the data end key to obtain a salt value data row key encapsulation list, wherein the salt value data row key encapsulation list includes a start salt value data row key corresponding to the data start key and an end salt value data row key corresponding to the data end key;
[0096] The target data query module 306 is used to determine the target salt value data row key in the salt value data row key encapsulation list according to the data query request of the target terminal, and transmit the target salt value data row key to the target terminal.
[0097] In an embodiment of the present invention, by adding salt values to data row keys, the data is dispersed to various partitions, and then all salt value data row keys are packaged, and finally the target salt value data row key is queried in the salt value data row key encapsulation list, which solves the problem of data hotspots in data partitions, reduces resource occupancy, improves computing efficiency, and thus improves data query performance.
[0098] See also Figure 4 Another embodiment of the data query device based on row key salt value in the embodiment of the present invention includes:
[0099] A data row key acquisition module 301 is used to acquire table data in a database and read multiple data row keys from the table data, wherein the table data includes multiple data partitions;
[0100] The salt value row key setting module 302 is used to count the number of the multiple data partitions to obtain the number of data partitions, obtain multiple target salt values according to the number of data partitions, and obtain multiple salt value data row keys according to the multiple target salt values and the multiple data row keys;
[0101] A data partition setting module 303, configured to determine a plurality of target data partitions according to the plurality of salt value data row keys and the plurality of data partitions;
[0102] The data row key selection module 304 is used to obtain a data query request from a target terminal, and determine a data start key corresponding to a start salt value data row key and a data end key corresponding to an end salt value data row key in the multiple target data partitions according to the data query request and the multiple salt value data row keys;
[0103] A row key encapsulation module 305 is used to encapsulate multiple salt value data row keys corresponding to the data start key to the data end key to obtain a salt value data row key encapsulation list, wherein the salt value data row key encapsulation list includes a start salt value data row key corresponding to the data start key and an end salt value data row key corresponding to the data end key;
[0104] The target data query module 306 is used to determine the target salt value data row key in the salt value data row key encapsulation list according to the data query request of the target terminal, and transmit the target salt value data row key to the target terminal.
[0105] Optionally, the salt value row key setting module 302 may also be specifically used for:
[0106] Counting the number of the multiple data partitions to obtain the number of data partitions;
[0107] The salt value range is set according to the number of data partitions to obtain multiple salt value selections, the number of salt value selections is the same as the number of data partitions, and each salt value selection corresponds to multiple initial salt values;
[0108] Set separators for multiple initial salt values to obtain multiple target salt values;
[0109] The multiple target salt values are randomly set to prefix positions of the multiple data row keys using a random number algorithm to obtain multiple salt value data row keys.
[0110] Optionally, the data partition setting module 303 may also be specifically used for:
[0111] Read the target salt values corresponding to the multiple salt value data row keys in sequence to obtain multiple target row key salt values;
[0112] Classifying the multiple salt value data row keys based on the multiple target row key salt values to obtain multiple salt value data row key categories;
[0113] The multiple salt value data row key categories are allocated to corresponding data partitions to obtain multiple target data partitions.
[0114] Optionally, the data row key selection module 304 may also be specifically used for:
[0115] Obtain a data query request from a target terminal, and read a plurality of row key header data from the plurality of salt value data row keys, respectively, wherein the row key header data is the first digit or letter after the target salt value;
[0116] Based on the data query request, the ASX encoding sequence and the multiple row key header data, a data start key and a data end key are determined.
[0117] Optionally, the target data query module 306 includes:
[0118] An identifier reading unit 3061 is used to read an encapsulation row key identifier in the salt value data row key encapsulation list;
[0119] A judging unit 3062, configured to judge whether the encapsulated row key identifier matches a preset row key identifier;
[0120] A query mode determining unit 3063 is used to determine a target query mode if the encapsulated row key identifier matches the preset row key identifier;
[0121] The target data selection unit 3064 determines the target salt value data row key according to the data query request of the target terminal, the target query mode and the salt value data row key encapsulation list, and transmits the target salt value data row key to the target terminal.
[0122] Optionally, the target data selection unit 3064 may also be specifically used for:
[0123] Performing predicate parsing on the data query request of the target terminal through the target query mode to obtain a query field name;
[0124] Read multiple preset data row key field names from the salt value data row key encapsulation list;
[0125] The query field name is matched with the multiple preset data row key field names to obtain a target field name, a target salt value data row key is determined according to the target field name, and the target salt value data row key is transmitted to a target terminal.
[0126] Optionally, the data query device based on the row key salt value also includes:
[0127] The monitoring and configuration module 307 is used to set the monitoring file and the configuration file in the server, wherein when the monitoring file detects that data is added, deleted or modified in the database or server, it is used to configure the changed data to the server cluster in real time through the configuration file.
[0128] In an embodiment of the present invention, by adding salt values to data row keys, the data is dispersed to various partitions, and then all salt value data row keys are packaged, and finally the target salt value data row key is queried in the salt value data row key encapsulation list, which solves the problem of data hotspots in data partitions, reduces resource occupancy, improves computing efficiency, and thus improves data query performance.
[0129] above Figure 3 and Figure 4The data query device based on row key salt value in the embodiment of the present invention is described in detail from the perspective of modular functional entity. The data query device based on row key salt value in the embodiment of the present invention is described in detail from the perspective of hardware processing.
[0130] Figure 5 1 is a schematic diagram of the structure of a data query device based on row key salt value provided by an embodiment of the present invention. The data query device 500 based on row key salt value may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 510 (for example, one or more processors) and a memory 520, and one or more storage media 530 (for example, one or more massive storage devices) storing application programs 533 or data 532. Among them, the memory 520 and the storage medium 530 can be short-term storage or permanent storage. The program stored in the storage medium 530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the data query device 500 based on row key salt value. Furthermore, the processor 510 may be configured to communicate with the storage medium 530, and execute a series of instruction operations in the storage medium 530 on the data query device 500 based on row key salt value to execute the steps of the data query method based on row key salt value.
[0131] The row key salt value-based data query device 500 may further include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input and output interfaces 560, and / or one or more operating systems 531, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. It will be appreciated by those skilled in the art that Figure 5 The structure of the data query device based on row key salt value shown does not constitute a limitation on the data query device based on row key salt value, and may include more or less components than shown in the figure, or a combination of certain components, or a different arrangement of components.
[0132] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the data query method based on row key salt values.
[0133] Furthermore, the computer-usable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.
[0134] Technical personnel in the relevant field can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0135] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.
[0136] The blockchain referred to in this invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.
[0137] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, ordinary technical personnel in the field should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A data query method based on row key salt value, characterized in that: The data query method based on row key salt value includes: Acquire table data in a database, and read multiple data row keys from the table data, wherein the table data includes multiple data partitions; Counting the number of the multiple data partitions to obtain the number of data partitions; setting a salt value range according to the number of data partitions to obtain multiple salt value selections, the number of salt value selections is the same as the number of data partitions, and each salt value selection corresponds to multiple initial salt values; setting separators for the multiple initial salt values to obtain multiple target salt values; using a random number algorithm to randomly set the multiple target salt values to the prefix positions of the multiple data row keys to obtain multiple salt value data row keys; Read the target salt values corresponding to the multiple salt value data row keys in sequence to obtain multiple target row key salt values; classify the multiple salt value data row keys based on the multiple target row key salt values to obtain multiple salt value data row key categories; assign the multiple salt value data row key categories to corresponding data partitions to obtain multiple target data partitions; Obtain a data query request from a target terminal, and read a plurality of row key header data from the plurality of salt value data row keys, wherein the row key header data is the first digit or letter after the target salt value; determine a data start key and a data end key based on the data query request, the Asker coding sequence and the plurality of row key header data; Encapsulate multiple salt value data row keys corresponding to the data start key to the data end key to obtain a salt value data row key encapsulation list, wherein the salt value data row key encapsulation list includes a start salt value data row key corresponding to the data start key and an end salt value data row key corresponding to the data end key; The target salt value data row key is determined in the salt value data row key encapsulation list according to the data query request of the target terminal, and the target salt value data row key is transmitted to the target terminal.
2. The data query method based on row key salt value according to claim 1 is characterized in that: The determining the target salt value data row key in the salt value data row key encapsulation list according to the data query request of the target terminal, and transmitting the target salt value data row key to the target terminal comprises: Reading an encapsulation row key identifier in the salt value data row key encapsulation list; Determining whether the encapsulated row key identifier matches a preset row key identifier; If the encapsulated row key identifier matches the preset row key identifier, determining a target query mode; According to the data query request of the target terminal, the target query mode and the salt value data row key encapsulation list, a target salt value data row key is determined, and the target salt value data row key is transmitted to the target terminal.
3. The data query method based on row key salt value according to claim 2 is characterized in that: The step of determining a target salt value data row key according to the data query request of the target terminal, the target query mode, and the salt value data row key encapsulation list, and transmitting the target salt value data row key to the target terminal comprises: Performing predicate parsing on the data query request of the target terminal through the target query mode to obtain a query field name; Read multiple preset data row key field names from the salt value data row key encapsulation list; The query field name is matched with the multiple preset data row key field names to obtain a target field name, a target salt value data row key is determined according to the target field name, and the target salt value data row key is transmitted to a target terminal.
4. The data query method based on row key salt value according to any one of claims 1 to 3, characterized in that: Before obtaining the table data in the database and reading a plurality of data row keys from the table data, wherein the table data includes a plurality of data partitions, the data query method based on the row key salt value further includes: A monitoring file and a configuration file are set in the server, wherein when the monitoring file detects that data is added, deleted or modified in the database or the server, the configuration file configures the changed data to the server cluster in real time.
5. A data query device based on row key salt value, characterized in that: The data query device based on row key salt value includes: A data row key acquisition module, used to acquire table data in a database and read multiple data row keys from the table data, wherein the table data includes multiple data partitions; A salt value row key setting module is used to count the number of the multiple data partitions to obtain the number of data partitions; set a salt value range according to the number of data partitions to obtain multiple salt value selections, the number of salt value selections is the same as the number of data partitions, and each salt value selection corresponds to multiple initial salt values; set separators for the multiple initial salt values to obtain multiple target salt values; use a random number algorithm to randomly set the multiple target salt values to the prefix positions of the multiple data row keys to obtain multiple salt value data row keys; A data partition setting module is used to read the target salt values corresponding to the multiple salt value data row keys in sequence to obtain multiple target row key salt values; classify the multiple salt value data row keys based on the multiple target row key salt values to obtain multiple salt value data row key categories; and assign the multiple salt value data row key categories to corresponding data partitions to obtain multiple target data partitions; A data row key selection module is used to obtain a data query request from a target terminal, and read a plurality of row key header data from the plurality of salt value data row keys, wherein the row key header data is the first digit or letter after the target salt value; based on the data query request, the Asker coding sequence and the plurality of row key header data, determine a data start key and a data end key; A row key encapsulation module is used to encapsulate multiple salt value data row keys corresponding to the data start key to the data end key to obtain a salt value data row key encapsulation list, wherein the salt value data row key encapsulation list includes a start salt value data row key corresponding to the data start key and an end salt value data row key corresponding to the data end key; The target data query module is used to determine the target salt value data row key in the salt value data row key encapsulation list according to the data query request of the target terminal, and transmit the target salt value data row key to the target terminal.
6. A data query device based on row key salt value, characterized in that: The data query device based on row key salt value includes: a memory and at least one processor, wherein the memory stores instructions; The at least one processor calls the instruction in the memory so that the row key salt value-based data query device executes the row key salt value-based data query method as described in any one of claims 1-4.
7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data query method based on row key salt value as described in any one of claims 1 to 4 is implemented.
