An HBase data query method, apparatus, and computer-readable storage medium

Abstract

This invention provides an HBase data query method, apparatus, and computer-readable storage medium. The method includes: designing a rowkey for the HBase main table based on an auxiliary table, wherein the rowkey includes at least one primary key field and at least one non-primary key field that meets data query requirements; receiving a data query request; and obtaining data corresponding to the rowkey according to the data query request. This method, apparatus, and computer-readable storage medium can solve the problem of high HBase data query overhead caused by the limited rowkey design of the HBase main table in Oracle+HBase read-write separation scenarios in existing technologies.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to an HBase data query method, apparatus, and computer-readable storage medium. Background Technology

[0002] In the early days, centralized systems in the telecommunications industry mostly used Oracle as the core database to carry core interactive and computing services. However, as market business needs developed, centralized systems based on Oracle became significantly limited in terms of scalability. In addition, users and the market have increasingly higher demands for real-time data applications, leading to the widespread application of various distributed database technologies, real-time incremental synchronization, and read-write separation technologies.

[0003] At present, leveraging the excellent features of HBase and establishing real-time incremental data synchronization between Oracle and HBase through OGG (Oracle Golden Gate) + Kafka, allowing businesses with large data volumes and high query concurrency to be handled by HBase, is a good choice for implementing read-write separation in Oracle.

[0004] However, in Oracle+HBase read-write separation scenarios, the existing technology suffers from high HBase data query overhead due to the limited rowkey design of the HBase master table. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to address the above-mentioned shortcomings of the prior art by providing an HBase data query method, apparatus and computer-readable storage medium to solve the problem of high HBase data query overhead caused by the limited rowkey design of the HBase main table in the Oracle+HBase read-write separation scenario.

[0006] In a first aspect, the present invention provides an HBase data query method, comprising:

[0007] The row key of the HBase main table is designed based on the auxiliary table, wherein the row key includes at least one primary key field and at least one non-primary key field that meets the data query requirements;

[0008] Receive data query requests;

[0009] The data corresponding to the rowkey is obtained according to the data query request.

[0010] Furthermore, the design of the HBase main table's rowkey based on the auxiliary table specifically includes:

[0011] An auxiliary table is introduced during the data synchronization process based on incremental messages. The auxiliary table is used to synchronize the field information corresponding to the rowkey. The auxiliary table includes at least one primary key field and at least one non-primary key field.

[0012] Furthermore, the incremental message is an Insert message, and the step of introducing an auxiliary table during data synchronization based on the incremental message, and synchronizing the data on the field information corresponding to the rowkey through the auxiliary table, specifically includes:

[0013] The rowkey of the auxiliary table and the HBase main table is constructed based on the at least one primary key field and at least one non-primary key field corresponding to the Insert message;

[0014] Write the field information of at least one primary key field and at least one non-primary key field in the Insert message into the auxiliary table;

[0015] Write all field information corresponding to the Insert message into the HBase main table.

[0016] Furthermore, the incremental message is an Update or key message. The step of introducing an auxiliary table during data synchronization based on the incremental message, and synchronizing the data for the field information corresponding to the rowkey through the auxiliary table, specifically includes:

[0017] Based on the at least one primary key field corresponding to the Update or key message, the rowkey of the HBase main table is concatenated through the auxiliary table;

[0018] Update the field information corresponding to the rowkey in the HBase main table according to the Update or key message;

[0019] Based on the updated field information in the HBase main table, determine whether the auxiliary table needs to be updated;

[0020] In response to the need to update the auxiliary table, update the field information of at least one primary key field and / or at least one non-primary key field in the auxiliary table.

[0021] Furthermore, determining whether the auxiliary table needs to be updated based on the updated field information in the HBase main table specifically includes:

[0022] If the updated field information involves an update operation of at least one primary key field or at least one non-primary key field, then it is determined that the auxiliary table needs to be updated.

[0023] Furthermore, the incremental message is a Delete message, and the step of introducing an auxiliary table during data synchronization based on the incremental message, and synchronizing the data on the field information corresponding to the rowkey through the auxiliary table, specifically includes:

[0024] Based on the at least one primary key field corresponding to the Delete message, the rowkey of the HBase main table is concatenated through the auxiliary table;

[0025] Delete the field information of at least one primary key field and at least one non-primary key field in the auxiliary table, as well as all field information corresponding to the rowkey in the HBase main table.

[0026] Further, obtaining the data corresponding to the rowkey according to the data query request specifically includes:

[0027] The Scan method is used to retrieve the data corresponding to the rowkey based on the query field corresponding to the data query request.

[0028] Secondly, the present invention provides an HBase data query device, comprising:

[0029] The design module is used to design the rowkey of the HBase main table based on the auxiliary table. The rowkey includes at least one primary key field and at least one non-primary key field that meets the data query requirements.

[0030] A receiving module, connected to the design module, is used to receive data query requests;

[0031] The acquisition module, connected to the receiving module, is used to acquire the data corresponding to the rowkey according to the data query request.

[0032] Thirdly, the present invention provides an HBase data query apparatus, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to implement the HBase data query method described in the first aspect above.

[0033] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the HBase data query method described in the first aspect.

[0034] The HBase data query method, apparatus, and computer-readable storage medium provided by this invention first design the rowkey of the HBase main table based on an auxiliary table. The rowkey includes at least one primary key field and at least one non-primary key field that meets the data query requirements. Then, a data query request is received. Finally, the data corresponding to the rowkey is obtained according to the data query request. This invention flexibly designs the rowkey of the HBase main table to meet query needs through an auxiliary table, avoiding the limitation that the HBase main table's rowkey can only consist of a primary key. This allows the data query party to satisfy the data query requirements by querying the HBase main table only once, solving the problem of high HBase data query overhead caused by the limited rowkey design of the HBase main table in Oracle+HBase read-write separation scenarios in existing technologies. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the existing Oracle and HBase data structures;

[0036] Figure 2 This is a schematic diagram of the query process of existing technology in a specific order scenario;

[0037] Figure 3 This is a flowchart of an HBase data query method according to Embodiment 1 of the present invention;

[0038] Figure 4 This is a schematic diagram of the query process of an HBase data query method in a specific order scenario according to an embodiment of the present invention;

[0039] Figure 5 This is a schematic diagram of the structure of an HBase data query device according to Embodiment 2 of the present invention;

[0040] Figure 6 This is a schematic diagram of the structure of an HBase data query device according to Embodiment 3 of the present invention. Detailed Implementation

[0041] To enable those skilled in the art to better understand the technical solution of the present invention, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0042] It is understood that the specific embodiments and accompanying drawings described herein are merely for explaining the invention and are not intended to limit the invention.

[0043] It is understood that, without conflict, the various embodiments and features in the embodiments of the present invention can be combined with each other.

[0044] It is understood that, for ease of description, only the parts related to the present invention are shown in the accompanying drawings, while the parts unrelated to the present invention are not shown in the drawings.

[0045] It is understood that each unit or module involved in the embodiments of the present invention may correspond to only one entity structure, or may be composed of multiple entity structures, or multiple units or modules may be integrated into one entity structure.

[0046] It is understood that, without conflict, the functions and steps marked in the flowcharts and block diagrams of this invention may occur in a different order than that marked in the accompanying drawings.

[0047] It is understood that the flowcharts and block diagrams of this invention illustrate the possible architecture, functions, and operations of systems, apparatuses, devices, and methods according to various embodiments of this invention. Each block in the flowchart or block diagram may represent a unit, module, program segment, or code, containing executable instructions for implementing the specified function. Furthermore, each block or combination of blocks in the block diagram and flowchart can be implemented using a hardware-based system to achieve the specified function, or using a combination of hardware and computer instructions.

[0048] It is understood that the units and modules involved in the embodiments of the present invention can be implemented by software or by hardware. For example, the units and modules can be located in a processor.

[0049] Application Overview

[0050] In the early days, centralized systems in the telecommunications industry mostly used Oracle as the core database to carry core interactive and computing services. However, as market business needs developed, centralized systems based on Oracle became significantly limited in terms of scalability. In addition, users and the market have increasingly higher demands for real-time data applications, leading to the widespread application of various distributed database technologies, real-time incremental synchronization, and read-write separation technologies.

[0051] The incremental real-time streaming data offloading solution developed using Oracle's OGG technology has become the mainstream solution for data transmission during the transition from traditional AOE architecture to distributed microservice architecture. Combined with the high-throughput Kafka middleware, it can effectively achieve data sharing between headquarters and subsidiaries as well as across projects.

[0052] HBase, as a column-based distributed database, possesses excellent characteristics such as large-capacity storage, high query performance, low cost, and scalability. It plays an important role in large-scale distributed architectures and is often used for large-scale data storage and querying, such as historical orders and detailed account statements.

[0053] At present, leveraging the excellent features of HBase and establishing real-time incremental data synchronization between Oracle and HBase through Ogg+Kafka, allowing businesses with large data volumes and high query concurrency to be handled by HBase, is a good choice for implementing read-write separation in Oracle.

[0054] Existing HBase data querying methods have the following drawbacks:

[0055] 1. Limitations in HBase table rowkey design

[0056] like Figure 1 As shown, in order for the original Oracle table data to be restored in HBase using incremental OGG messages, the four types of incremental messages must be able to uniquely identify a piece of data in HBase using the given field information. Therefore, the rowkey must only contain the primary key field. For example... Figure 2 To better illustrate the problem, a specific order scenario is introduced:

[0057] The order table contains a serial number (orderId), a user ID (userId), a transaction time (time), and other data fields. The serial number is used as the primary key field of this table. In order to achieve incremental synchronization, the row key of the HBase table can only use the serial number. However, in actual business query scenarios, apart from querying a small number of serial numbers, the main query scenario is to query the order information of a certain user within a time range.

[0058] 2. Additional query overhead caused by index tables

[0059] By introducing additional index tables for HBase data queries, the double-query approach becomes a bottleneck for the HBase cluster as data size and query pressure increase. This leads to a higher rate of query timeouts, impacting online services and user experience. Figure 2 As shown, for a single order record query by a user, the client will engage in n+1 request interactions with HBase, where n is the number of orders that meet the criteria. Therefore, for high-concurrency, low-latency queries targeting end users, this interaction pattern becomes a bottleneck affecting the overall interface return speed, frequently causing timeouts during peak business periods, and even leading to prolonged garbage collection (GC) on the HBase regionServer, potentially causing regionServer downtime.

[0060] For existing HBase data queries, there are limitations in the rowkey design, which prevents the rowkey design from being oriented towards the query design, resulting in additional overhead for HBase queries.

[0061] Unlike relational databases, especially HBase, non-relational databases can only meet business query requirements through flexible and reasonable rowkey design. Therefore, designing rowkeys for the final business query scenario is the key to improving query performance, reducing timeout rate, and enhancing user experience.

[0062] In summary, existing technologies in Oracle+HBase read-write separation scenarios suffer from high HBase data query overhead due to the limited rowkey design of the HBase master table.

[0063] To address the aforementioned technical issues, this application provides an HBase data query method, apparatus, and computer-readable storage medium. By flexibly designing the rowkey of the HBase main table for query needs through an auxiliary table, it avoids the limitation that the rowkey of the HBase main table can only be composed of the primary key. This allows the data query party to satisfy the data query requirements by querying the HBase main table only once, thereby at least solving the problem of high HBase data query overhead caused by the limited rowkey design of the HBase main table in the Oracle+HBase read-write separation scenario in the prior art.

[0064] After introducing the basic principles of this application, various non-limiting embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0065] Example 1:

[0066] This embodiment provides an HBase data query method, such as Figure 3 As shown, the method includes:

[0067] Step S101: Design the rowkey of the HBase main table based on the auxiliary table, wherein the rowkey includes at least one primary key field and at least one non-primary key field that meets the data query requirements.

[0068] It should be noted that the rowkey of the HBase main table, which is flexibly designed through auxiliary tables, consists of the primary key of the corresponding data and at least one non-primary key field. The primary key can include one or more primary key fields.

[0069] In one optional embodiment, the step of designing the rowkey of the HBase main table based on the auxiliary table specifically includes:

[0070] An auxiliary table is introduced during the data synchronization process based on incremental messages. The auxiliary table is used to synchronize the field information corresponding to the rowkey. The auxiliary table includes at least one primary key field and at least one non-primary key field.

[0071] Specifically, an auxiliary table is introduced during the data synchronization process of incremental messages. Based on the field information given in the incremental message, the auxiliary table is used to synchronize the field information corresponding to the rowkey of the HBase main table.

[0072] In one optional embodiment, the incremental message is an Insert message, and the step of introducing an auxiliary table during data synchronization based on the incremental message, and synchronizing the data on the field information corresponding to the rowkey through the auxiliary table, specifically includes:

[0073] The rowkey of the auxiliary table and the HBase main table is constructed based on the at least one primary key field and at least one non-primary key field corresponding to the Insert message;

[0074] Write the field information of at least one primary key field and at least one non-primary key field in the Insert message into the auxiliary table;

[0075] Write all field information corresponding to the Insert message into the HBase main table.

[0076] It should be noted that the Insert message is used to perform Insert operations on the HBase main table, that is, to insert the data corresponding to the Insert message into the HBase main table. Therefore, the Insert message will carry all field information corresponding to the data.

[0077] Specifically, based on at least one primary key field and at least one non-primary key field corresponding to the Insert message, rowkey_assist (i.e., the rowkey of the auxiliary table) and rowkey_main (i.e., the rowkey of the HBase main table) are constructed. The rowkey of the auxiliary table consists of at least one primary key field. At the same time, all field information given by the Insert message is written to the HBase main table, and field information (i.e., at least one primary key field and at least one non-primary key field) for assisting in concatenating the corresponding fields of rowkey_main is written to the auxiliary table.

[0078] In one optional embodiment, the incremental message is an Update or key message, and the step of introducing an auxiliary table during data synchronization based on the incremental message, and synchronizing the data on the field information corresponding to the rowkey through the auxiliary table, specifically includes:

[0079] Based on the at least one primary key field corresponding to the Update or key message, the rowkey of the HBase main table is concatenated through the auxiliary table;

[0080] Update the field information corresponding to the rowkey in the HBase main table according to the Update or key message;

[0081] Based on the updated field information in the HBase main table, determine whether the auxiliary table needs to be updated;

[0082] In response to the need to update the auxiliary table, update the field information of at least one primary key field and / or at least one non-primary key field in the auxiliary table.

[0083] It should be noted that Update or key messages are used to perform Update or key operations on the HBase main table. That is, to update the field information that needs to be updated in the HBase main table corresponding to the data in the Update or key message (if the field information to be updated contains any one of the primary key fields, the incremental message is a key message; otherwise, the incremental message is an Update message). Therefore, the Update or key message will carry the field information that needs to be updated for the corresponding data and the field information of at least one primary key field used to uniquely identify the data in the HBase main table.

[0084] Specifically, the steps to update the field information that needs to be updated in the data corresponding to the Update or key message are as follows:

[0085] Step 1: Based on the field information of at least one primary key field given in the Update or key message, retrieve the corresponding rowkey in the auxiliary table;

[0086] Step 2: Based on the field information of at least one primary key field and at least one non-primary key field corresponding to the rowkey in the auxiliary table, concatenate the corresponding rowkey in the main table;

[0087] Step 3: Based on the field information to be updated given in the Update or key message, update the field information corresponding to the rowkey in the HBase main table. The specific method is as follows:

[0088] Determine if the rowkey in the HBase master table contains the field that needs to be updated. If the rowkey does not contain the field that needs to be updated, directly replace the value of the field that needs to be updated corresponding to the rowkey in the HBase master table. If the rowkey contains the field that needs to be updated, delete all field information corresponding to the rowkey in the HBase master table, replace the value of the field that needs to be updated in all field information, and rewrite all the replaced field information into the HBase master table (that is, replace the value of the field that needs to be updated in the rowkey of the HBase master table).

[0089] Step 4: Based on the field information to be updated given in the Update or key message (i.e., the field information to be updated in the HBase main table), determine whether the auxiliary table needs to be updated;

[0090] Step 5: In response to the need to update the auxiliary table, update the field information of at least one primary key field and at least one non-primary key field corresponding to the rowkey in the auxiliary table. The specific method is as follows:

[0091] Determine if the rowkey in the auxiliary table contains the field that needs to be updated. If the rowkey in the auxiliary table does not contain the field that needs to be updated, directly replace the value of the field that needs to be updated corresponding to the rowkey in the auxiliary table. If the rowkey in the auxiliary table contains the field that needs to be updated, delete the field information of at least one primary key field and at least one non-primary key field corresponding to the rowkey in the auxiliary table, replace the value of the field that needs to be updated in the field information of at least one primary key field and at least one non-primary key field, and rewrite the replaced field information of at least one primary key field and at least one non-primary key field into the auxiliary table (that is, change the value of the field that needs to be updated in the rowkey in the auxiliary table).

[0092] In one optional embodiment, determining whether the auxiliary table needs to be updated based on the updated field information in the HBase main table specifically includes:

[0093] If the updated field information involves an update operation of at least one primary key field or at least one non-primary key field, then it is determined that the auxiliary table needs to be updated.

[0094] Specifically, if the field information to be updated given in the Update or key message involves an update operation of any one of the primary key fields and at least one non-primary key fields corresponding to the rowkey in the auxiliary table, then it is determined that the auxiliary table needs to be updated; otherwise, it is determined that the auxiliary table does not need to be updated.

[0095] In one optional embodiment, the incremental message is a Delete message, and the step of introducing an auxiliary table during data synchronization based on the incremental message, and synchronizing the data on the field information corresponding to the rowkey through the auxiliary table, specifically includes:

[0096] Based on the at least one primary key field corresponding to the Delete message, the rowkey of the HBase main table is concatenated through the auxiliary table;

[0097] Delete the field information of at least one primary key field and at least one non-primary key field in the auxiliary table, as well as all field information corresponding to the rowkey in the HBase main table.

[0098] It should be noted that the Delete message is used to perform a Delete operation on the HBase main table, that is, to delete the data corresponding to the Delete message in the HBase main table. Therefore, the Delete message will carry field information of at least one primary key field used to uniquely identify the data in the HBase main table.

[0099] Specifically, first, based on the field information of at least one primary key field given in the Delete message, the corresponding rowkey in the auxiliary table is retrieved. Then, based on the field information of at least one primary key field and at least one non-primary key field corresponding to the rowkey in the auxiliary table, the corresponding rowkey in the main table is concatenated. Finally, the field information of at least one primary key field and at least one non-primary key field corresponding to the rowkey in the auxiliary table, as well as all field information corresponding to the rowkey in the HBase main table, are deleted.

[0100] Step S102: Receive data query request.

[0101] Specifically, users send data query requests through the client, and HBase receives the data query requests sent by users.

[0102] Step S103: Obtain the data corresponding to the rowkey according to the data query request.

[0103] Specifically, the Scan method is used to retrieve the data corresponding to the rowkey (i.e., all corresponding field information) based on the query fields corresponding to the data query request. The query fields can be one or more fields selected from at least one primary key field and at least one non-primary key field. For example... Figure 4 As shown, taking a specific order scenario as an example, the order table (i.e., the HBase main table) contains a serial number - orderId, a user ID - userId, a transaction time - time, and other data fields - data. The rowkey of this order table consists of orderId, userId, and time. orderId is the primary key field, while userId and time are non-primary key fields. When querying HBase data, you can directly use the HBase Scan method to query based on the user-inputted userId and time conditions.

[0104] It is worth mentioning that this invention is based on the design concept of a query-oriented non-relational database. By preprocessing the business data during the data synchronization process, introducing and building auxiliary tables, and supplementing the HBase main table with the required fields for the rowkey concatenation as needed, this invention flexibly designs the HBase main table rowkey for query needs. This solves the problem of additional query overhead caused by the limited rowkey design. A user's order record query will only send one query request to HBase. Compared with the existing HBase data query methods, the number of queries is reduced by n. For the overall user query, the number of queries is reduced by m*n, where m is the number of order queries on the user side, and n is the number of orders returned in a single query.

[0105] In one specific embodiment, the data synchronization process of this HBase data query method may include the following steps:

[0106] Step 1: Based on the data operation type of the incremental message (i.e., Insert, Update, Delete, Key), determine whether it is necessary to concatenate auxiliary table data with the HBase main table's rowkey. If yes, proceed to Step 2; otherwise, proceed to Step 3. The specific method for determining whether auxiliary table data needs to be concatenated with the HBase main table's rowkey is as follows:

[0107] If any field in the rowkey of the HBase main table is not included in the field information given in the incremental message, it is determined that auxiliary table data needs to be concatenated with the rowkey of the HBase main table.

[0108] Step 2: Query the auxiliary table to obtain the fields required to concatenate the rowkey of the HBase main table;

[0109] Step 3: Based on the existing data, construct a data writing object, which includes the rowkey and data fields of the HBase main table;

[0110] Step 4: Determine whether the data entry needs to be ignored based on the SCN (System Change Number). If yes, end the process; otherwise, proceed to Step 5. The specific method for determining whether the data entry needs to be ignored based on the SCN is as follows:

[0111] If the SCN number of this data is less than the current SCN number, then this data needs to be ignored.

[0112] Step 5: Perform the data write operation;

[0113] Step 6: Determine if the write operation was successful and if the auxiliary table information needs to be updated. If yes, proceed to Step 7; otherwise, end the process.

[0114] Step 7: Construct a data update object for the auxiliary table, including the rowkey and update field of the auxiliary table, and perform the data write operation;

[0115] Step 8: Determine whether the auxiliary table data update operation was successful. If successful, end the entire process; otherwise, write all data to the failure queue.

[0116] The HBase data query method provided in this invention first designs the rowkey of the HBase main table based on an auxiliary table. The rowkey includes at least one primary key field and at least one non-primary key field that meets the data query requirements. Then, it receives a data query request. Finally, it retrieves the data corresponding to the rowkey according to the data query request. This invention flexibly designs the rowkey of the HBase main table to meet query needs through an auxiliary table, avoiding the limitation that the HBase main table's rowkey can only consist of a primary key. This allows the data queryer to satisfy the data query requirements by querying the HBase main table only once, solving the problem of high HBase data query overhead caused by the limited rowkey design of the HBase main table in Oracle+HBase read-write separation scenarios in existing technologies.

[0117] Example 2:

[0118] like Figure 5 As shown, this embodiment provides an HBase data query device for executing the above-described HBase data query method, including:

[0119] Design module 11 is used to design the row key of the HBase main table based on the auxiliary table, wherein the row key includes at least one primary key field and at least one non-primary key field that meets the data query requirements;

[0120] The receiving module 12, connected to the design module 11, is used to receive data query requests;

[0121] The acquisition module 13 is connected to the receiving module 12 and is used to acquire the data corresponding to the rowkey according to the data query request.

[0122] Furthermore, the design module 11 specifically includes:

[0123] The data synchronization unit is used to introduce an auxiliary table during the data synchronization process based on incremental messages, and to synchronize the field information corresponding to the rowkey through the auxiliary table. The auxiliary table includes at least one primary key field and at least one non-primary key field.

[0124] Furthermore, the incremental message is an Insert message, and the data synchronization unit specifically includes:

[0125] A construction unit is configured to construct the rowkey of the auxiliary table and the HBase main table based on the at least one primary key field and at least one non-primary key field corresponding to the Insert message;

[0126] The first writing unit is used to write the field information of at least one primary key field and at least one non-primary key field in the Insert message into the auxiliary table;

[0127] The second write unit is used to write all field information corresponding to the Insert message into the HBase main table.

[0128] Furthermore, the incremental message is an Update or key message, and the data synchronization unit specifically includes:

[0129] The first concatenation unit is used to concatenate the rowkey of the HBase main table through the auxiliary table based on the at least one primary key field corresponding to the Update or key message;

[0130] The first update unit is used to update the field information corresponding to the rowkey in the HBase main table according to the Update or key message;

[0131] The judgment unit is used to determine whether the auxiliary table needs to be updated based on the updated field information in the HBase main table;

[0132] The second update unit is used to update the field information of at least one primary key field and / or at least one non-primary key field in the auxiliary table in response to the need to update the auxiliary table.

[0133] Furthermore, the determination unit is specifically used for:

[0134] If the updated field information involves an update operation of at least one primary key field or at least one non-primary key field, then it is determined that the auxiliary table needs to be updated.

[0135] Furthermore, the incremental message is a Delete message, and the data synchronization unit specifically includes:

[0136] The second concatenation unit is used to concatenate the rowkey of the HBase main table through the auxiliary table based on the at least one primary key field corresponding to the Delete message;

[0137] The deletion unit is used to delete the field information of at least one primary key field and at least one non-primary key field in the auxiliary table, as well as all field information corresponding to the rowkey in the HBase main table.

[0138] Furthermore, the acquisition module 13 specifically includes:

[0139] The acquisition unit is used to obtain the data corresponding to the rowkey by using the Scan method according to the query field corresponding to the data query request.

[0140] Example 3:

[0141] refer to Figure 6 This embodiment provides an HBase data query device, including a memory 21 and a processor 22. The memory 21 stores a computer program, and the processor 22 is configured to run the computer program to execute the HBase data query method in Embodiment 1.

[0142] The memory 21 is connected to the processor 22. The memory 21 can be a flash memory, a read-only memory or other memory, and the processor 22 can be a central processing unit or a microcontroller.

[0143] Example 4:

[0144] This embodiment provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the HBase data query method in Embodiment 1 above.

[0145] The computer-readable storage medium includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, computer program modules, or other data). Computer-readable storage media include, but are not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other memory technologies, CD-ROM (Compact Disc Read-Only Memory), DVD or other optical disc storage, cartridges, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer.

[0146] In summary, the HBase data query method, apparatus, and computer-readable storage medium provided in this invention first design the rowkey of the HBase main table based on an auxiliary table. The rowkey includes at least one primary key field and at least one non-primary key field that meets the data query requirements. Then, a data query request is received. Finally, the data corresponding to the rowkey is obtained according to the data query request. This invention flexibly designs the rowkey of the HBase main table to meet query needs through an auxiliary table, avoiding the limitation that the HBase main table's rowkey can only consist of a primary key. This allows the data queryer to satisfy the data query requirements by querying the HBase main table only once, solving the problem of high HBase data query overhead caused by the limited rowkey design of the HBase main table in Oracle+HBase read-write separation scenarios in existing technologies.

[0147] It is understood that the above embodiments are merely exemplary implementations used to illustrate the principles of the present invention, and the present invention is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. An HBase data query method, characterized in that, The method includes: The row key of the HBase main table is designed based on the auxiliary table, wherein the row key includes at least one primary key field and at least one non-primary key field that meets the data query requirements; Receive data query requests; Retrieve the data corresponding to the rowkey according to the data query request; The design of the HBase main table's row key based on the auxiliary table specifically includes: An auxiliary table is introduced during the data synchronization process based on incremental messages. The auxiliary table is used to synchronize the field information corresponding to the rowkey. The auxiliary table includes at least one primary key field and at least one non-primary key field.

2. The method according to claim 1, characterized in that, The incremental message is an Insert message. The process of introducing an auxiliary table during data synchronization based on the incremental message, and synchronizing data on the field information corresponding to the rowkey through the auxiliary table, specifically includes: The rowkey of the auxiliary table and the HBase main table is constructed based on the at least one primary key field and at least one non-primary key field corresponding to the Insert message; Write the field information of at least one primary key field and at least one non-primary key field in the Insert message into the auxiliary table; Write all field information corresponding to the Insert message into the HBase main table.

3. The method according to claim 1, characterized in that, The incremental message is either an Update or a key message. The step of introducing an auxiliary table during data synchronization based on the incremental message, and synchronizing the data for the field information corresponding to the rowkey through the auxiliary table, specifically includes: Based on the at least one primary key field corresponding to the Update or key message, the rowkey of the HBase main table is concatenated through the auxiliary table; Update the field information corresponding to the rowkey in the HBase main table according to the Update or key message; Based on the updated field information in the HBase main table, determine whether the auxiliary table needs to be updated; In response to the need to update the auxiliary table, update the field information of at least one primary key field and / or at least one non-primary key field in the auxiliary table.

4. The method according to claim 3, characterized in that, The step of determining whether the auxiliary table needs to be updated based on the updated field information in the HBase main table specifically includes: If the updated field information involves an update operation of at least one primary key field or at least one non-primary key field, then it is determined that the auxiliary table needs to be updated.

5. The method according to claim 1, characterized in that, The incremental message is a Delete message. The process of introducing an auxiliary table during data synchronization based on the incremental message, and synchronizing the data for the field information corresponding to the rowkey through the auxiliary table, specifically includes: Based on the at least one primary key field corresponding to the Delete message, the rowkey of the HBase main table is concatenated through the auxiliary table; Delete the field information of at least one primary key field and at least one non-primary key field in the auxiliary table, as well as all field information corresponding to the rowkey in the HBase main table.

6. The method according to claim 1, characterized in that, The step of obtaining the data corresponding to the rowkey according to the data query request specifically includes: The Scan method is used to retrieve the data corresponding to the rowkey based on the query field corresponding to the data query request.

7. An HBase data query device, characterized in that, include: The design module is used to design the rowkey of the HBase main table based on the auxiliary table. The rowkey includes at least one primary key field and at least one non-primary key field that meets the data query requirements. A receiving module, connected to the design module, is used to receive data query requests; An acquisition module, connected to the receiving module, is used to acquire the data corresponding to the rowkey according to the data query request; The design module specifically includes: The data synchronization unit is used to introduce an auxiliary table during the data synchronization process based on incremental messages, and to synchronize the field information corresponding to the rowkey through the auxiliary table. The auxiliary table includes at least one primary key field and at least one non-primary key field.

8. An HBase data query device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to implement the HBase data query method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the HBase data query method as described in any one of claims 1-6.

Images