A data processing method, device and equipment based on an HBase data table and a storage medium

Abstract

The application discloses a data processing method and device based on an HBase data table, equipment and a storage medium, and relates to the technical field of big data processing. The method comprises the following steps: obtaining the number of pre-partitions of the HBase data table according to the space occupation information of the relational data table; counting the distribution frequency of at least one candidate character of a data field in the relational data table, and performing hash mapping on the candidate character to obtain the hash value of the candidate character; wherein the candidate character is a non-ASCII code character; constructing a pre-partition mapping table according to the at least one candidate character in the relational data table and the distribution frequency and hash value of the candidate character; determining the partition interval of the pre-partition according to the number of partitions, the pre-partition mapping table and the distribution frequency of the at least one candidate character in the relational data table, and creating an HBase partition table; and importing the data in the relational data table into the HBase partition table based on the pre-partition mapping table and the partition interval. The HBase data table is pre-partitioned based on the non-ASCII code character.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more particularly to the field of big data technology, specifically to a data processing method, apparatus, device, and storage medium for HBase data tables. Background Technology

[0002] With the rapid development of information technology, Hadoop has been widely used in various industries, providing big data solutions. Hadoop is a distributed system infrastructure that stores and processes data in a reliable, efficient, and scalable manner, meeting the massive data query needs of various industries.

[0003] Hadoop data is primarily stored in HBase databases. HBase tables handle large amounts of data; for example, when the data volume exceeds 10GB (Gigabytes), they automatically partition, but this has a significant impact on performance. HBase provides a pre-partitioning feature, allowing users to partition tables according to certain rules during table creation. HBase only supports pre-partitioning using ASCII characters (occupying 1 byte) and does not support pre-partitioning using non-ASCII (American Standard Code for Information Interchange) characters (occupying multiple bytes). Summary of the Invention

[0004] This invention provides a data processing method, apparatus, device, and storage medium based on HBase data tables to achieve pre-partitioning of data tables based on non-ASCII codes.

[0005] According to one aspect of the present invention, a data processing method based on an HBase data table is provided, comprising:

[0006] Based on the space usage information of the relational data table, the number of partitions for the pre-partitioning of the HBase data table is obtained;

[0007] The distribution frequency of at least one candidate character in a data field of a statistical relational data table is calculated, and the candidate characters are hashed to obtain their hash values; wherein, the candidate characters are non-ASCII characters.

[0008] Construct a pre-partitioned mapping table based on at least one candidate character in the relational data table, as well as the distribution frequency and hash value of the candidate character;

[0009] Based on the number of partitions, the pre-partition mapping table, and the distribution frequency of at least one candidate character in the relational data table, determine the partition range of the pre-partition and create an HBase partition table.

[0010] Based on the pre-partitioned mapping table and partition range, data from the relational data table is imported into the HBase partitioned table.

[0011] According to another aspect of the present invention, a data processing apparatus based on an HBase data table is provided, comprising:

[0012] The partition number determination module is used to determine the number of partitions for pre-partitioning an HBase data table based on the space usage information of the relational data table.

[0013] The hash value acquisition module is used to statistically analyze the distribution frequency of at least one candidate character in a data field of a relational data table, and to perform hash mapping on the candidate characters to obtain their hash values; wherein, the candidate characters are non-ASCII characters;

[0014] The mapping table construction module is used to construct a pre-partitioned mapping table based on at least one candidate character in the relational data table, as well as the distribution frequency and hash value of the candidate character;

[0015] The partitioning interval determination module is used to determine the partitioning interval of the pre-partition based on the number of partitions, the pre-partition mapping table, and the distribution frequency of at least one candidate character in the relational data table, and to create an HBase partition table.

[0016] The data import module is used to import data from relational data tables into HBase partitioned tables based on pre-partitioned mapping tables and partition ranges.

[0017] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0018] At least one processor; and

[0019] A memory that is communicatively connected to at least one processor; wherein,

[0020] The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to perform the data processing method based on the HBase data table according to any embodiment of the present invention.

[0021] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions for causing a processor to execute and implement the data processing method based on an HBase data table according to any embodiment of the present invention.

[0022] The technical solution of this invention involves: obtaining the number of partitions for pre-partitioning an HBase data table based on the space occupancy information of the relational data table; statistically analyzing the distribution frequency of at least one candidate character in the data field of the relational data table and performing hash mapping on the candidate characters to obtain their hash values; wherein the candidate characters are non-ASCII characters; constructing a pre-partitioning mapping table based on at least one candidate character in the relational data table, its distribution frequency, and hash values; determining the partition intervals for pre-partitioning based on the number of partitions, the pre-partitioning mapping table, and the distribution frequency of at least one candidate character in the relational data table, and creating an HBase partition table; and importing data from the relational data table into the HBase partition table based on the pre-partitioning mapping table and the partition intervals. This technical solution allows for pre-partitioning of HBase data tables based on non-ASCII characters, improving the flexibility of pre-partitioning HBase data tables.

[0023] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1A This is a flowchart of a data processing method based on an HBase data table according to Embodiment 1 of the present invention;

[0026] Figure 1B This is a correlation diagram of candidate characters, candidate character occurrence frequency, and candidate character hash value provided in Embodiment 1 of the present invention;

[0027] Figure 2A This is a flowchart of a data processing method based on an HBase data table according to Embodiment 2 of the present invention;

[0028] Figure 2B This is a candidate digit occurrence frequency diagram for a 1-bit target data bit provided according to Embodiment 2 of the present invention;

[0029] Figure 2C This is a candidate digit occurrence frequency diagram for a 2-bit target data bit provided according to Embodiment 2 of the present invention;

[0030] Figure 3This is a flowchart of a data processing method based on an HBase data table according to Embodiment 3 of the present invention;

[0031] Figure 4 This is a structural diagram of a data processing device based on an HBase data table according to Embodiment 4 of the present invention;

[0032] Figure 5 This is a structural diagram of a data processing device based on an HBase data table according to Embodiment 5 of the present invention;

[0033] Figure 6 This is a structural diagram of an electronic device based on a data processing method using an HBase data table, as provided in Embodiment Six of the present invention. Detailed Implementation

[0034] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0035] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0036] Furthermore, it should be noted that the collection, storage, use, processing, transmission, provision, and disclosure of data tables and other data involved in the technical solution of this invention all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0037] Example 1

[0038] Figure 1AThis is a flowchart of a data processing method based on an HBase data table provided in Embodiment 1 of the present invention. This embodiment is applicable to the case of pre-partitioning an HBase data table based on non-ASCII characters. The method can be executed by a data processing device based on an HBase data table. The device can be implemented in hardware and / or software and can be configured in an electronic device.

[0039] like Figure 1A As shown, the method includes:

[0040] S101. Based on the space usage information of the relational data table, obtain the number of partitions for the pre-partitioning of the HBase data table.

[0041] In this embodiment, the space occupancy information can be the data value of the space occupied by the relational data table in the relational database; the HBase data table is the data table in the HBase database; pre-partitioning can be dividing the HBase data table into at least two partition tables; the partition tables can be more granular sub-data tables.

[0042] Specifically, if the space occupancy information of the relational data table is less than or equal to a first preset value, then the HBase data table is not pre-partitioned, and there is no need to obtain the number of partitions for pre-partitioning the HBase data table. If the space occupancy information of the relational data table is greater than the first preset value, then space of the size of the space occupancy information is reserved in the HBase data table. The data value to be stored in each partition is used as a second preset value. The ratio of the space occupancy information of a set multiple to the second preset value is determined, and this ratio is rounded up to obtain the number of partitions for pre-partitioning the HBase data table. The first and second preset values, and the set multiple, can be determined by technical personnel based on actual needs or practical experience; this invention does not impose any limitations on this.

[0043] For example, the space usage information of the relational data table is 3GB, the first preset value is 5GB, the space usage information is less than the first preset value, and the HBase data table is not pre-partitioned.

[0044] For example, the space usage of a relational data table is 22GB, the first preset value is 5GB, and the second preset value is 10GB. The space usage is greater than the first preset value. If 22GB of space is reserved in the HBase data table, the ratio of twice the space usage to the second preset value is 4.4. Rounding up to 5, the number of partitions in the pre-partitioned HBase data table is 5.

[0045] Understandably, by adopting the above scheme, when determining the number of partitions for pre-partitioning of an HBase data table, reserving space for this space occupancy information in the HBase data table allows for adjustments to the HBase data table based on the actual data volume in the relational data table, thus improving the flexibility in determining the number of partitions for pre-partitioning of the HBase data table.

[0046] In one specific embodiment, the space usage information of a relational data table can be automatically determined based on the relational database connection address, the relational database name, and the relational data table name.

[0047] S102. Calculate the distribution frequency of at least one candidate character in the data field of the statistical relational data table, and perform hash mapping on the candidate character to obtain the hash value of the candidate character.

[0048] In this embodiment, candidate characters can be characters associated with data in a relational data table and recorded in the corresponding data field. Candidate characters are non-ASCII characters, such as Chinese characters, Arabic characters, and Cyrillic letters. Distribution frequency can be the frequency of the corresponding candidate character appearing in the relational data table.

[0049] Specifically, based on the frequency of each candidate character in the relational data table and the total number of data records in the relational data table, the distribution frequency of each candidate character is calculated. A hash function is then used to perform a hash mapping on each candidate character to obtain its hash value. It should be noted that any existing technology can be used for the hash function, and this invention does not limit it. For example, the hash function can be a direct addressing hash function, a number analysis hash function, or a mid-square hash function, etc. Preferably, the hash function is one that results in a low probability of collision among the obtained hash values.

[0050] It should be noted that the hash value has multiple data bits, preferably 5 bits.

[0051] like Figure 1B The graph shows the relationship between candidate characters, the frequency of occurrence of candidate characters, and the hash value of candidate characters. The first column represents candidate characters; the second column represents the frequency of occurrence of candidate characters (in %); and the third column represents the hash value of candidate characters.

[0052] S103. Construct a pre-partitioned mapping table based on at least one candidate character in the relational data table, as well as the distribution frequency and hash value of the candidate character.

[0053] The pre-partition mapping table can be used to record the association between each candidate character and its partition.

[0054] Specifically, based on each candidate character in the relational data table, the distribution frequency of each candidate character, and the hash value of each candidate character, the truncation length of the hash value is determined, and a pre-partition mapping table is constructed by matching the candidate characters with the corresponding hash value truncation lengths.

[0055] S104. Based on the number of partitions, the pre-partition mapping table, and the distribution frequency of at least one candidate character in the relational data table, determine the partition interval of the pre-partition and create an HBase partition table.

[0056] The sub-section interval can be the interval occupied by each partition in the pre-partition table in the HBase data table; the HBase partition table can be a data table composed of the partition tables obtained from the pre-partition operation.

[0057] Specifically, based on the number of partitions, the distribution frequency of each candidate character, and the pre-partition mapping table, the partition intervals in which the data of each partition table can be evenly distributed are determined. Based on the partition intervals of each partition table, the HBase data table is pre-partitioned to obtain the HBase partition table.

[0058] S105. Based on the pre-partitioned mapping table and partition range, import the data from the relational data table into the HBase partitioned table.

[0059] In this embodiment, based on the corresponding partitions of the candidate characters of the data fields in the relational data table in the pre-partition mapping table, the data in the relational data table is imported into the partition range of the corresponding partition table in the HBase partition table.

[0060] This invention, in its embodiments, determines the number of partitions for pre-partitioning an HBase data table based on the space occupancy information of the relational data table; it then statistically analyzes the distribution frequency of at least one candidate character in the data fields of the relational data table and performs hash mapping on these candidate characters to obtain their hash values; wherein, the candidate characters are non-ASCII characters; a pre-partition mapping table is constructed based on at least one candidate character in the relational data table, its distribution frequency, and hash values; the partition intervals for pre-partitioning are determined based on the number of partitions, the pre-partition mapping table, and the distribution frequency of at least one candidate character in the relational data table, and an HBase partition table is created; finally, data from the relational data table is imported into the HBase partition table based on the pre-partition mapping table and the partition intervals. By employing this technical solution, pre-partitioning of HBase data tables based on non-ASCII characters can be performed, improving the flexibility of pre-partitioning HBase data tables.

[0061] Example 2

[0062] Figure 2AThis is a flowchart of a data processing method based on an HBase data table provided in Embodiment 2 of the present invention. Based on the above embodiments, this embodiment optimizes and improves the construction operation of the pre-partitioned mapping table.

[0063] Furthermore, the step of "constructing a pre-partition mapping table based on at least one candidate character in a relational data table, as well as the distribution frequency and hash value of the candidate character" is refined to "determining the target hash value truncation bit length based on the distribution frequency and hash value of at least one candidate character in a relational data table; trunculating the hash value corresponding to the candidate character based on the target hash value truncation bit length to obtain the target truncation number; and concatenating the candidate character, hash value truncation bit length, and target truncation number based on preset data concatenation rules to obtain the pre-partition mapping table," thereby improving the pre-partition mapping table acquisition operation.

[0064] It should be noted that for parts not described in detail in the embodiments of the present invention, please refer to the relevant descriptions in other embodiments, which will not be repeated here.

[0065] like Figure 2A As shown, the method includes:

[0066] S201. Based on the space usage information of the relational data table, obtain the number of partitions for the pre-partitioning of the HBase data table.

[0067] S202. Calculate the distribution frequency of at least one candidate character in a data field of a statistical relational data table, and perform a hash mapping on the candidate character to obtain its hash value.

[0068] Among them, the candidate characters are non-ASCII characters.

[0069] S203. Determine the number of bits to be truncated in the target hash value based on the distribution frequency and hash value of at least one candidate character in the relational data table.

[0070] The target hash value truncation length can be the number of bits to truncate the hash value of the candidate characters, such as the first bit of the hash value or the first two bits of the hash value.

[0071] Optionally, the numbers in the target data bits of the hash values ​​of candidate characters in the relational data table are statistically analyzed to obtain at least one set of candidate numbers for the target data bits; wherein the target data bits include at least one data bit in the hash value; the frequency of occurrence of candidate numbers is determined based on the distribution frequency of candidate characters; the maximum frequency of occurrence is determined from the frequency of occurrence of at least one set of candidate numbers; and the number of bits to be truncated in the target hash value is determined based on the maximum frequency of occurrence and the storable frequency.

[0072] In this system, the target data bits are the data bits in the hash values ​​of candidate characters that need to be counted; the candidate numbers are the numbers in the target data bits of the hash values ​​of each candidate character; and the frequency of occurrence is the frequency with which the candidate numbers appear in the hash values ​​of each candidate character. The storable frequency can be the frequency with which each partition can store data corresponding to candidate characters with the same candidate number, used to characterize the ability to evenly store the same candidate characters. It can be determined by taking the reciprocal of the number of partitions. For example, if the number of partitions is 5, then the storable frequency is 0.2.

[0073] Specifically, following the data order from left to right, the numbers in the first data position of the hash value of the candidate characters in the relational data table are counted to obtain at least one set of candidate numbers. The frequency of each different candidate number in the hash value of each candidate character is determined from this at least one set of candidate numbers, and the maximum frequency is determined from among them.

[0074] In one optional embodiment, if the maximum occurrence frequency is less than the storable frequency, the number of bits of the data bit corresponding to the target data bit is determined as the target hash value truncation bit; if the maximum occurrence frequency is greater than the storable frequency, the number of data bits of the target data bit is increased to update the target data bit, and the target hash value truncation bit is re-determined based on at least one set of candidate numbers of the new target data bit.

[0075] For example, such as Figure 2B As shown, the number of partitions is 5. The first column is the candidate numbers, the second column indicates that the target data bits are 1 (the first two bits of the hash value), and the third column is the frequency of occurrence of the corresponding candidate numbers, in percentage. The maximum frequency of occurrence is 53.05%, and the storable frequency is 0.2. Since the maximum frequency of occurrence exceeds the storable frequency, pre-partitioning cannot be completed. The target data bits are increased to 2 (the first two bits of the hash value), as shown below. Figure 2C As shown, the first column represents candidate numbers, the second column indicates that the target data is 2 bits, and the third column represents the frequency of occurrence of the corresponding candidate numbers in percentage. The maximum frequency of occurrence is 15.10%, which is less than the storable frequency, so pre-partitioning can be performed, and the target hash value will be truncated to 2 bits.

[0076] Understandably, by adopting the above technical solution, the number of bits to be truncated in the target hash value can be determined based on the relationship between the maximum occurrence frequency and the storable frequency, or the number of data bits in the target data bits can be increased to redetermine the number of bits to be truncated in the target hash value, thereby improving the flexibility in determining the number of bits to be truncated in the target hash value.

[0077] In another alternative embodiment, if the number of partitions is too large, resulting in a low storage frequency of the partition table, and pre-partitioning cannot be performed based on the five-bit hash value of a candidate character, the target hash value truncation bits can be re-determined by concatenating the hash value of the candidate character with the hash value of the next character.

[0078] Understandably, by adopting the above technical solution, candidate numbers are obtained by statistically analyzing the numbers in the target data bits of the hash value of candidate characters; the frequency of occurrence of candidate characters is determined based on their distribution frequency, and the maximum frequency of occurrence is determined from them; the number of bits to be truncated in the target hash value is determined based on the maximum frequency of occurrence and the storable frequency, which can ensure that the data corresponding to the candidate characters of the maximum frequency of occurrence can be stored in each partition, thereby improving the stability of data import based on the pre-partition mapping table.

[0079] S204. Based on the target hash value, truncate the hash value corresponding to the candidate character to obtain the target truncated number.

[0080] Specifically, based on the target hash value truncation number, the hash value corresponding to each candidate character is truncated to obtain the target truncation number corresponding to each candidate character.

[0081] S205. Based on the preset data concatenation rules, the candidate characters, the number of bits to be truncated in the hash value, and the target truncated number are concatenated to obtain the pre-partition mapping table.

[0082] Specifically, a candidate character, the number of bits to be truncated in the hash value corresponding to the candidate character, and the target truncated number corresponding to the candidate character are concatenated into a row, and the table formed by the concatenated rows corresponding to each candidate character is used as the pre-partition mapping table.

[0083] S206. Based on the number of partitions, the pre-partition mapping table, and the distribution frequency of at least one candidate character in the relational data table, determine the partition interval of the pre-partition and create an HBase partition table.

[0084] S207. Based on the pre-partitioned mapping table and partition range, import the data from the relational data table into the HBase partitioned table.

[0085] This invention determines the target hash value truncation length based on the distribution frequency and hash value of at least one candidate character in a relational data table; truncates the hash value corresponding to the candidate character according to the target hash value truncation length to obtain the target truncation number; and concatenates the candidate character, the hash value truncation length, and the target truncation number based on a preset data concatenation rule to obtain a pre-partitioned mapping table. By employing this technical solution, determining the target hash value truncation length and the target truncation number, and concatenating the candidate character, hash value truncation length, and target truncation number to obtain the pre-partitioned mapping table, the accuracy of the partition corresponding to the candidate character in the pre-partitioned mapping table is improved, thereby enhancing the accuracy of data import.

[0086] Example 3

[0087] Figure 3 This is a flowchart of a data processing method based on an HBase data table provided in Embodiment 3 of the present invention. Based on the above embodiments, this embodiment has optimized and improved the data import operation.

[0088] Furthermore, the process of "importing data from a relational data table into an HBase partition table based on a pre-partition mapping table and a pre-partition interval" is refined to "determining the target character of the field to be stored in the relational data table; querying the pre-partition mapping table to determine the number of bits to be truncated from the first hash value corresponding to the target character; determining the first truncation number to truncate the hash value of the target character based on the first truncation number, and determining the target partition table from the HBase partition table based on the first truncation number; concatenating the first truncation number and the field to be stored according to the preset concatenation rules, and storing the concatenation result in the corresponding target partition table in the HBase partition table," to improve the data import operation.

[0089] It should be noted that for parts not described in detail in the embodiments of the present invention, please refer to the relevant descriptions in other embodiments, which will not be repeated here.

[0090] like Figure 3 As shown, the method includes:

[0091] S301. Based on the space usage information of the relational data table, obtain the number of partitions for the pre-partitioning of the HBase data table.

[0092] S302. Calculate the distribution frequency of at least one candidate character in a data field of a statistical relational data table, and perform a hash mapping on the candidate character to obtain the hash value of the candidate character.

[0093] Among them, the candidate characters are non-ASCII characters.

[0094] S303. Construct a pre-partition mapping table based on at least one candidate character in the relational data table, as well as the distribution frequency and hash value of the candidate character.

[0095] S304. Determine the partition interval of the pre-partition according to the number of partitions, the pre-partition mapping table, and the distribution frequency of at least one candidate character in the relational data table, and create a HBase partition table.

[0096] S305. Determine the target character of the field to be stored in the relational data table.

[0097] Among them, the field to be stored can be the data field of the data to be stored in the relational data table, such as the Rowkey field. Specifically, the first character in the characters of the field to be stored is used as the target character. Exemplarily, if the field to be stored is "Xu XX", then "Xu" is used as the target character.

[0098] S306. Query the number of digits to intercept the first hash value corresponding to the target character from the pre-partition mapping table.

[0099] S307. Determine the first interception number for intercepting the hash value of the target character according to the number of digits to intercept the first hash value, and determine the target partition table from the HBase partition table according to the first interception number.

[0100] Among them, the target partition table can be the partition table storing the data corresponding to the field to be stored. Specifically, according to the first interception number, the partition table corresponding to the interval where the first interception number is located is selected from the HBase partition table as the target partition table.

[0101] Exemplarily, the number of digits to intercept the first hash value of "Xu" is 2, the hash value is 35768, then the first interception number is 35, and the intervals of each partition table are 0 to 23, 23 to 26, 26 to 29, 29 to 37, and 37 to 99 in sequence, then the partition table with the interval of 29 to 37 is used as the target partition table.

[0102] S308. According to the preset splicing rule, splice the first interception number and the field to be stored, and store the splicing result and the data corresponding to the field to be stored into the corresponding target partition table in the HBase partition table.

[0103] Specifically, splice the first interception number in front of the target character of the field to be stored as the splicing result, and store the splicing result and the data corresponding to the field to be stored into the corresponding target partition table in the HBase partition table.

[0104] This invention, in its embodiments, determines the target character of the field to be stored in a relational data table; queries the pre-partitioning mapping table for the first hash value truncation bit length corresponding to the target character; determines the first truncation number to truncate the hash value of the target character based on the first hash value truncation bit length, and determines the target partition table from the HBase partition table based on the first truncation number; and concatenates the first truncation number and the field to be stored according to a preset concatenation rule, storing the concatenation result and the data corresponding to the field to be stored in the target partition table corresponding to the HBase partition table. Using this technical solution, the first truncation number of the hash value of the target character can be concatenated with the character to be stored in the field to be stored, and the concatenated field to be stored can be stored in the target partition table. This allows for data import into HBase partition tables based on non-ASCII characters, improving the flexibility of data import.

[0105] Example 4

[0106] Figure 4 This is a flowchart of a data processing method based on an HBase data table provided in Embodiment 4 of the present invention. This embodiment has been further optimized based on the above embodiments.

[0107] Furthermore, the following steps are added: "Obtain the field to be queried; based on the pre-partitioned mapping table and the HBase partitioned table, query the data associated with the field to be queried from the HBase partitioned table according to the field to be queried," to improve the data processing method based on HBase data tables.

[0108] It should be noted that for parts not described in detail in the embodiments of the present invention, please refer to the relevant descriptions in other embodiments, which will not be repeated here.

[0109] like Figure 4 As shown, the method includes:

[0110] S401. Based on the space usage information of the relational data table, obtain the number of partitions for the pre-partitioning of the HBase data table.

[0111] S402. Calculate the distribution frequency of at least one candidate character in a data field of a statistical relational data table, and perform a hash mapping on the candidate character to obtain the hash value of the candidate character.

[0112] Among them, the candidate characters are non-ASCII characters.

[0113] S403. Construct a pre-partitioned mapping table based on at least one candidate character in the relational data table, as well as the distribution frequency and hash value of the candidate character.

[0114] S404. Determine the partition range of the pre-partition according to at least one of the number of partitions, the pre-partition mapping table, and the distribution frequency of candidate characters in the relational data table, and create an HBase partition table.

[0115] S405. Import the data in the relational data table into the HBase partition table based on the pre-partition mapping table and the partition range.

[0116] S406. Obtain the field to be queried.

[0117] Among them, the field to be queried can be the field for recording the character to be queried. Among them, the character to be queried can be a non-ASCII character. It should be noted that any method in the prior art can be used to obtain the field to be queried, and the present invention does not limit this.

[0118] S407. Query the data associated with the field to be queried from the HBase partition table based on the pre-partition mapping table and the HBase partition table according to the field to be queried.

[0119] Optionally, based on the pre-partition mapping table, determine the number of digits to intercept the second hash value according to the field to be queried; determine the second intercepted number of the hash value of the field to be queried according to the number of digits to intercept the second hash value; splice the field to be queried and the second intercepted number according to the preset splicing rule; query the data associated with the field to be queried from the HBase partition table according to the splicing result.

[0120] Among them, the splicing rule for splicing the field to be queried and the second intercepted number is the same as the splicing rule for splicing the field to be stored and the first intercepted number in the data import operation.

[0121] Specifically, query the number of digits to intercept the hash value of the character to be queried in the field to be queried in the pre-partition mapping table as the number of digits to intercept the second hash value. The number obtained by intercepting the hash value of the character to be queried according to the number of digits to intercept the second hash value is used as the second intercepted number. Splice the character to be queried in the field to be queried and the second intercepted number according to the preset splicing rule, and use the obtained splicing result as the updated field to be queried, and query the associated data of the updated field to be queried from the HBase partition table. Among them, the splicing rule can be independently determined by those skilled in the art according to actual needs or practical experience, and the present invention does not limit this.

[0122] Exemplarily, the character to be queried in the field to be queried is "Xu". Query the pre-partition mapping table to obtain the corresponding number of digits to intercept the second hash value as 2. According to the hash value of "Xu" being 35768, intercept the first 2 digits and splice them before the character to be queried in the field to be queried, that is, the splicing result is "35-Xu", and perform a fuzzy query in the corresponding partition of the HBase partition table.

[0123] Understandably, by adopting the above technical solution, the second truncation number of the field to be queried is determined, and the field to be queried and the second truncation number are concatenated according to the preset concatenation rules. Data is then queried from the HBase partition table based on the concatenation result. Data can be queried from the HBase partition table by recording the field to be queried with non-ASCII characters, which improves the flexibility of data query.

[0124] This invention provides an embodiment that obtains the field to be queried; based on a pre-partitioned mapping table and an HBase partitioned table, it queries the data associated with the field to be queried from the HBase partitioned table. By employing this technical solution, after data is imported into the HBase database, the associated data of the query field in the HBase partitioned table can be queried using non-ASCII characters in the field to be queried, thus improving the flexibility of data querying.

[0125] Example 5

[0126] Figure 5 This is a structural diagram of a data processing device based on an HBase data table, provided in Embodiment 5 of the present invention. This embodiment is applicable to situations where a data table is pre-partitioned based on non-ASCII characters. The device can be implemented in hardware and / or software and can be configured in an electronic device. Figure 5 As shown, the device includes: a partition number determination module 501, a hash value acquisition module 502, a mapping table construction module 503, a partition interval determination module 504, and a data import module 505, wherein...

[0127] The partition number determination module 501 is used to determine the number of partitions for the pre-partitioning of the HBase data table based on the space usage information of the relational data table;

[0128] The hash value acquisition module 502 is used to statistically analyze the distribution frequency of at least one candidate character in a data field of a relational data table, and to perform hash mapping on the candidate character to obtain the hash value of the candidate character; wherein, the candidate character is a non-ASCII character;

[0129] The mapping table construction module 503 is used to construct a pre-partitioned mapping table based on at least one candidate character in the relational data table, as well as the distribution frequency and hash value of the candidate character;

[0130] The partition interval determination module 504 is used to determine the partition interval of the pre-partition based on the number of partitions, the pre-partition mapping table, and the distribution frequency of at least one candidate character in the relational data table, and to create an HBase partition table.

[0131] The data import module 505 is used to import data from relational data tables into HBase partitioned tables based on pre-partitioned mapping tables and partition ranges.

[0132] This invention employs a partition number determination module to determine the number of partitions in the HBase partition table based on the space occupancy information of the relational data table; a hash value acquisition module to statistically analyze the distribution frequency of at least one candidate character in the data field of the relational data table and perform hash mapping on the candidate characters to obtain their hash values; wherein, the candidate characters are non-ASCII characters; a mapping table construction module to construct a pre-partition mapping table based on at least one candidate character in the relational data table, its distribution frequency, and hash value; a partition interval determination module to determine the partition intervals of the pre-partition based on the number of partitions, the pre-partition mapping table, and the distribution frequency of at least one candidate character in the relational data table, and to create the HBase partition table; and a data import module to import data from the relational data table into the HBase partition table based on the pre-partition mapping table and the partition intervals. By adopting the above technical solution, HBase data tables can be pre-partitioned based on non-ASCII characters, improving the flexibility of pre-partitioning HBase data tables.

[0133] Optionally, the mapping table construction module 503 includes:

[0134] The truncation bit determination unit is used to determine the truncation bit of the target hash value based on the distribution frequency and hash value of at least one candidate character in the relational data table.

[0135] The truncation number determination unit is used to truncate the hash value corresponding to the candidate character according to the target hash value to obtain the target truncation number;

[0136] The mapping table acquisition unit is used to pre-partition the mapping table by concatenating candidate characters, hash value truncation length, and target truncation number based on preset data concatenation rules.

[0137] Optionally, the truncation bit determination unit includes:

[0138] The candidate number acquisition subunit is used to count the numbers in the target data bits of the hash values ​​of candidate characters in a relational data table to obtain at least one set of candidate numbers in the target data bits; wherein, the target data bits include at least one data bit in the hash value;

[0139] The frequency determination subunit is used to determine the frequency of occurrence of candidate numbers based on the distribution frequency of candidate characters;

[0140] The maximum frequency determination subunit is used to determine the maximum occurrence frequency from the occurrence frequencies of at least one set of candidate numbers;

[0141] The truncation bit determination sub-unit is used to determine the truncation bit of the target hash value based on the maximum occurrence frequency and the storable frequency.

[0142] Optionally, the number of bits is truncated to determine the sub-unit, specifically used for:

[0143] If the maximum occurrence frequency is less than the storable frequency, then determine the number of bits corresponding to the target data bit, and use it as the number of bits to truncate the target hash value.

[0144] If the maximum occurrence frequency is greater than the storable frequency, the number of data bits in the target data bit is increased to update the target data bit, and the number of bits to be truncated in the target hash value is re-determined based on at least one set of candidate numbers in the new target data bit.

[0145] Optional, the data import module 505 includes:

[0146] The target character determination module is used to determine the target character of the field to be stored in a relational data table;

[0147] The first truncation bit determination unit is used to query the first hash value corresponding to the target character from the pre-partition mapping table to determine the truncation bit.

[0148] The partition table determination unit is used to determine the first truncation number to be truncated from the hash value of the target character based on the first hash value and to determine the target partition table from the HBase partition table based on the first truncation number.

[0149] The concatenation result storage unit is used to concatenate the first extracted number and the field to be stored according to the preset concatenation rules, and store the concatenation result and the data corresponding to the field to be stored in the HBase partition table.

[0150] Optionally, the device may also include:

[0151] The field query unit is used to retrieve the field to be queried;

[0152] The data query unit is used to query data associated with the fields to be queried from the HBase partition table based on the pre-partitioned mapping table and the HBase partition table.

[0153] Optional, data query unit, including:

[0154] The second truncation bit determination subunit is used to determine the second hash value truncation bit based on the pre-partitioned mapping table and the field to be queried.

[0155] The second truncation number determination subunit is used to determine the second truncation number of the hash value of the field to be queried based on the number of truncation bits of the second hash value.

[0156] The splicing sub-unit is used to splice the field to be queried and the second extracted number according to the preset splicing rules;

[0157] The data query sub-unit is used to query the data associated with the fields to be queried from the HBase partition table based on the concatenation result.

[0158] The data processing device based on HBase data tables provided in the embodiments of the present invention can execute the data processing method based on HBase data tables provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the data processing method based on HBase data tables.

[0159] Example 6

[0160] Figure 6 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0161] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0162] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0163] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as data processing methods based on HBase data tables.

[0164] In some embodiments, the HBase-based data processing method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the HBase-based data processing method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the HBase-based data processing method by any other suitable means (e.g., by means of firmware).

[0165] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0166] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0167] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0168] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0169] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0170] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0171] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0172] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data processing method based on HBase data tables, characterized in that, include: Based on the space usage information of the relational data table, the number of partitions for the pre-partitioning of the HBase data table is obtained; The frequency distribution of at least one candidate character in the data field of the relational data table is statistically analyzed, and the candidate character is hashed to obtain the hash value of the candidate character; wherein, the candidate character is a non-ASCII character; Based on at least one candidate character in the relational data table, and the distribution frequency and hash value of the candidate character, a pre-partitioned mapping table is constructed; Based on the number of partitions, the pre-partition mapping table, and the distribution frequency of at least one candidate character in the relational data table, the partition interval of the pre-partition is determined, and an HBase partition table is created. Based on the pre-partition mapping table and the partition range, the data in the relational data table is imported into the HBase partition table.

2. The method according to claim 1, characterized in that, The step of constructing a pre-partitioned mapping table based on at least one candidate character in the relational data table, and the distribution frequency and hash value of the candidate character, includes: The number of bits to be truncated in the target hash value is determined based on the distribution frequency and hash value of at least one candidate character in the relational data table. Based on the target hash value, the hash value corresponding to the candidate character is truncated to obtain the target truncated number; Based on preset data concatenation rules, the candidate characters, the number of bits to be truncated in the hash value, and the target truncated number are concatenated to obtain a pre-partitioned mapping table.

3. The method according to claim 2, characterized in that, The step of determining the target hash value truncation bits based on the distribution frequency and hash value of at least one candidate character in the relational data table includes: The numbers in the target data bits of the hash values ​​of candidate characters in the relational data table are counted to obtain at least one set of candidate numbers for the target data bits; wherein, the target data bits include at least one data bit in the hash value; The frequency of occurrence of the candidate numbers is determined based on the distribution frequency of the candidate characters; Determine the maximum frequency of occurrence from the frequency of occurrence of the at least one set of candidate numbers; The number of bits to be truncated in the target hash value is determined based on the maximum occurrence frequency and the storable frequency.

4. The method according to claim 3, characterized in that, Determining the target hash value truncation bits based on the maximum occurrence frequency and the storable frequency includes: If the maximum occurrence frequency is less than the storable frequency, then the number of bits corresponding to the target data bit is determined as the number of bits to be truncated in the target hash value. If the maximum occurrence frequency is greater than the storable frequency, then the number of data bits of the target data bit is increased to update the target data bit, and the number of bits to be truncated in the target hash value is re-determined based on at least one set of candidate numbers of the new target data bit.

5. The method according to claim 1, characterized in that, The step of importing data from the relational data table into the HBase partition table based on the pre-partition mapping table and the pre-partition interval includes: Determine the target character of the field to be stored in the relational data table; Query the first hash value corresponding to the target character from the pre-partition mapping table and extract the number of bits; Based on the first hash value, determine the first truncation number to truncate the hash value of the target character, and based on the first truncation number, determine the target partition table from the HBase partition table; According to the preset concatenation rules, the first truncated number and the field to be stored are concatenated, and the concatenation result and the data corresponding to the field to be stored are stored in the target partition table corresponding to the HBase partition table.

6. The method according to claim 1, characterized in that, Also includes: Get the field to be queried; Based on the pre-partition mapping table and the HBase partition table, the data associated with the field to be queried is retrieved from the HBase partition table according to the field to be queried.

7. The method according to claim 6, characterized in that, The step of querying data associated with the field to be queried from the HBase partition table based on the pre-partition mapping table and the pre-partition table, according to the field to be queried, includes: Based on the pre-partition mapping table, the number of bits to be truncated in the second hash value is determined according to the field to be queried; Based on the number of bits truncated from the second hash value, determine the second truncated number of the hash value of the field to be queried; According to the preset splicing rules, the field to be queried and the second extracted number are spliced ​​together; Based on the concatenation result, query the data associated with the field to be queried from the HBase partition table.

8. A data processing device based on HBase data tables, characterized in that, include: The partition number determination module is used to determine the number of partitions for pre-partitioning an HBase data table based on the space usage information of the relational data table. The hash value acquisition module is used to statistically analyze the distribution frequency of at least one candidate character in a data field of a relational data table, and to perform hash mapping on the candidate characters to obtain their hash values; wherein, the candidate characters are non-ASCII characters; The mapping table construction module is used to construct a pre-partitioned mapping table based on at least one candidate character in the relational data table, as well as the distribution frequency and hash value of the candidate character; The partitioning interval determination module is used to determine the partitioning interval of the pre-partition based on the number of partitions, the pre-partition mapping table, and the distribution frequency of at least one candidate character in the relational data table, and to create an HBase partition table. The data import module is used to import data from relational data tables into HBase partitioned tables based on pre-partitioned mapping tables and partition ranges.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data processing method based on an HBase data table as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the data processing method based on an HBase data table as described in any one of claims 1-7.

Images