Data processing method and device, storage medium and electronic equipment

Through the OLAP analysis engine, query requests are processed from the data lake, reducing the resource usage of HBase cluster, solving the problem of excessive load on the Phoenix server during complex queries, and improving the stability of the HBase cluster.

CN118939702APending Publication Date: 2024-11-12CHINA MOBILE (SUZHOU) SOFTWARE TECH CO LTD +1
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202410990263.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

When processing complex association queries and analytical queries, the Phoenix server occupies a large amount of CPU and memory resources in the HBase cluster, resulting in excessive load and affecting stability.

Method used

The OLAP analysis engine obtains the corresponding data of the query plan from the data lake for processing, reduces the CPU and memory resources of the HBase cluster, and synchronizes the data processing results into the data lake to avoid directly obtaining data from the HBase cluster.

Benefits of technology

It effectively reduces the load and network jitter of the HBase cluster, improves the stability of the cluster, and supports complex association query and analytical query scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939702A_ABST
    Figure CN118939702A_ABST
Patent Text Reader

Abstract

The invention discloses a data processing method and device, a storage medium and electronic equipment, and relates to the technical field of cloud computing, and the method comprises the steps that firstly, a query request, sent by a client, for data in an HBase cluster is received, and the query request carries a data query statement; then obtaining a query plan corresponding to the data query statement; data corresponding to the query plan is obtained from a data lake through the OLAP analysis engine to be processed, a data processing result is obtained, and the data lake synchronously stores the data subjected to write-in processing in the HBase cluster; and finally, sending a data processing result to the client. Compared with the prior art, the method and the device have the advantages that application scenes involving complex association query and analysis type query can be supported, occupation of CPU and memory resources of the HBase cluster in the query process is effectively reduced, the load of the HBase cluster is reduced, and the stability of the HBase cluster is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of cloud computing technology, and in particular to a data processing method, device, storage medium and electronic device. Background Art

[0002] The HBase cluster is a highly reliable, high-performance, column-oriented, scalable distributed storage system that supports fast writing, modification, deletion, and simple query in high-concurrency scenarios, and can quickly obtain a specified row or multiple rows of data.

[0003] At present, in the related technology, Phoenix server is used to manage HBase cluster, and can use standard Structured Query Language (SQL) to query and update data in HBase cluster, thereby realizing the management and analysis of large-scale distributed data.

[0004] However, the Phoenix server in the related technology can only support simple data query scenarios. For application scenarios involving complex association queries and analytical queries, it will occupy a large amount of central processing unit (CPU) and memory resources of the HBase cluster, resulting in excessive load on the HBase cluster and affecting the stability of the HBase cluster. Summary of the invention

[0005] In view of this, the present application provides a data processing method, device, storage medium and electronic device, the main purpose of which is to improve the current Phoenix server that can only support simple data query scenarios. For application scenarios involving complex associated queries and analytical queries, it will occupy a large amount of CPU and memory resources of the HBase cluster, resulting in excessive load on the HBase cluster, affecting the technical problem of the stability of the HBase cluster.

[0006] In a first aspect, the present application provides a data processing method, comprising:

[0007] Receive a query request for data in an HBase cluster sent by a client, wherein the query request carries a data query statement;

[0008] Obtaining a query plan corresponding to the data query statement;

[0009] The OLAP analysis engine obtains data corresponding to the query plan from the data lake for processing to obtain data processing results, and the data lake synchronously stores the data that has completed the write processing in the HBase cluster;

[0010] The data processing result is sent to the client.

[0011] In a second aspect, the present application provides a data processing device, comprising:

[0012] A receiving module is configured to receive a query request for data in an HBase cluster sent by a client, wherein the query request carries a data query statement;

[0013] An acquisition module is configured to acquire a query plan corresponding to the data query statement;

[0014] An acquisition module is configured to acquire data corresponding to the query plan from the data lake through the OLAP analysis engine for processing to obtain data processing results, wherein the data lake synchronously stores the data that has completed the write processing in the HBase cluster;

[0015] The sending module is configured to send the data processing result to the client.

[0016] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the data processing method described in the first aspect.

[0017] In a fourth aspect, the present application provides an electronic device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the data processing method described in the first aspect when executing the computer program.

[0018] In a fifth aspect, the present application provides a computer program product having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the data processing method described in the first aspect.

[0019] By means of the above technical scheme, the present application provides a data processing method, device, storage medium and electronic device, which first receive a query request for data in an HBase cluster sent by a client, wherein the query request carries a data query statement; then obtain a query plan corresponding to the data query statement; then obtain the data corresponding to the query plan from the data lake through the OLAP analysis engine for processing to obtain a data processing result, wherein the data lake synchronously stores the data that has completed the write processing in the HBase cluster; finally, send the data processing result to the client. Compared with the current existing technology, the present application can receive a query request for data in an HBase cluster sent by a client, then obtain a query plan corresponding to the data query statement in the query request, and synchronously store the data that has completed the write processing in the HBase cluster in the data lake, and then obtain the data corresponding to the query plan from the data lake through the OLAP analysis engine for processing. The OLAP analysis engine can quickly analyze and query large-scale data sets, and support application scenarios involving complex associated queries and analytical queries. Since the data lake stores the data that has completed the write processing in the HBase cluster, the OLAP analysis engine can directly obtain the data that is synchronously stored in the HBase cluster from the data lake without obtaining data from the HBase cluster, thereby effectively reducing the CPU and memory resources occupied by the HBase cluster during the query process, reducing the load on the HBase cluster, reducing network jitter, and thereby improving the stability of the HBase cluster.

[0020] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0023] Figure 1 A schematic diagram of a data processing method provided in an embodiment of the present application is shown;

[0024] Figure 2 A schematic diagram of a data processing method provided in an embodiment of the present application is shown;

[0025] Figure 3 A schematic diagram of an example process provided by an embodiment of the present application is shown;

[0026] Figure 4 A schematic diagram of an example process provided by an embodiment of the present application is shown;

[0027] Figure 5 A schematic diagram of an example process provided by an embodiment of the present application is shown;

[0028] Figure 6 A schematic diagram of an example process provided by an embodiment of the present application is shown;

[0029] Figure 7 A structural schematic diagram of a data processing device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0030] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0031] In order to improve the current Phoenix server that allows users to access data in the HBase cluster through the SQL query language, but the analytical query performance in application scenarios involving complex associated queries and analytical queries is poor and time-consuming, and can only support simple data query scenarios. For application scenarios involving complex associated queries and analytical queries, the data in the HBase cluster is directly accessed through an external OLAP engine or the Hfile file of the HBase cluster is directly read. When query and analysis operations are frequent or the amount of data is large, a large amount of access to data in the HBase cluster or pulling of Hfile files will occupy a large amount of CPU and memory resources of the HBase cluster, resulting in an excessively high load on the HBase cluster, which is prone to network jitter and affects the stability of the HBase cluster. This embodiment provides a data processing method, such as Figure 1 As shown, the method includes:

[0032] Step 101: Receive a query request for data in an HBase cluster sent by a client, where the query request carries a data query statement.

[0033] In this embodiment, a client (e.g., Phoenix client) can be used to send a query request for data in an HBase cluster to a server (e.g., Phoenix server, HBase server, Apache Calcite). The query request can carry a data query statement written in SQL, and the Phoenix server can serve as a unified SQL service entry, supporting data query language (DQL, Data Query Language), data manipulation language (DML, Data Manipulation Language), data definition language (DDL, Data Definition Language), etc.

[0034] Step 102: Obtain a query plan corresponding to the data query statement.

[0035] For example, the application server can parse the data query statement to obtain the query plan, where the query plan may include a series of steps or a set of operation instructions corresponding to the data query statement, which helps to find a method for efficiently executing the query request, thereby reducing resource consumption and improving the query speed. Correspondingly, the query plan can be represented in the form of a query plan tree, describing the logical execution steps and data processing flow of the query. The query plan tree may include the tables, fields, functions, operator types, etc. involved in the data query statement.

[0036] Step 103: The data corresponding to the query plan is obtained from the data lake through the OLAP analysis engine for processing to obtain data processing results. The data lake synchronously stores the data that has completed the write processing in the HBase cluster.

[0037] In this embodiment, the online analytical processing (OLAP) analysis engine can quickly analyze and query large-scale data sets, support application scenarios involving complex relational queries and analytical queries, execute SQL statements sent by the client, obtain data corresponding to the query plan from the data lake for processing, obtain data processing results, and send the data processing results to the server. Among them, the OLAP analysis engine may include but is not limited to the Spark engine, the Presto engine, etc. The Spark engine is a big data computing engine, and its advantage lies in the ability to process and analyze large quantities of data. The Spark engine can perform complex OLAP analysis operations on the data in the HBase cluster, so that the server supports application scenarios of complex relational queries and analytical queries.

[0038] Correspondingly, the data lake can provide low-cost data storage services and support the OLAP analysis engine to update the processed data to the storage layer. It can store various types of data, including but not limited to text files, audio, video, database export files, etc. The data that has completed the write processing in the HBase cluster is synchronously stored in the data lake, and then the OLAP analysis engine obtains the data corresponding to the query plan from the data lake for processing, which can effectively reduce the CPU and memory resources occupied by the HBase cluster during the query process, reduce the load of the HBase cluster, and improve the stability of the HBase cluster.

[0039] Step 104: Send the data processing result to the client.

[0040] Exemplarily, the server may send the data processing results to the client so that the client can perform tasks such as report analysis based on the data processing results to improve data analysis efficiency.

[0041] Compared with the current existing technology, this embodiment can receive a query request for data in an HBase cluster sent by a client, then obtain a query plan corresponding to the data query statement in the query request, and synchronously store the data that has completed the write processing in the HBase cluster in the data lake, and then obtain the data corresponding to the query plan from the data lake through the OLAP analysis engine for processing. The OLAP analysis engine can quickly analyze and query large-scale data sets, and support application scenarios involving complex associated queries and analytical queries. Since the data lake stores the data that has completed the write processing in the HBase cluster, the OLAP analysis engine can directly obtain the data that is synchronously stored in the HBase cluster from the data lake without obtaining data from the HBase cluster, thereby effectively reducing the CPU and memory resources occupied by the HBase cluster during the query process, reducing the HBase cluster load, reducing network jitter, and thereby improving the stability of the HBase cluster.

[0042] In order to further illustrate the specific implementation process of the method of this embodiment, this embodiment provides the following Figure 2 The specific method shown includes:

[0043] Step 201: Receive an SQL statement sent by a client for operating data in an HBase cluster.

[0044] The SQL statements may include query, write, update, delete and other operation statements for data in the HBase cluster.

[0045] For example, Figure 3The figure shows the overall architecture of a data processing method. By combining the advantages of Phoenix server, HBase cluster, Kafka cluster, Spark engine and data lake, the HBase cluster supports complete and high-performance OLAP functions, while ensuring the data writing capability of the HBase cluster itself and the stability of the cluster. The details are as follows:

[0046] Phoenix server: Based on the native Phoenix server, a query routing module and an SQL dialect module are added, and the native query optimizer is extended. The extended query optimizer can query multiple databases and data sources. Among them, the query routing module can be used to determine the object for processing query requests and route the query requests to the object. For example, when the SQL request is determined to be a query request, the query routing module can be used to further analyze the query scenario involved in the query request. If it is determined that the query request involves a query scenario of complex associated query or analytical query, the object can be determined as an OLAP analysis engine, and the query request can be routed to the OLAP analysis engine, and then the OLAP analysis engine can be called to process the query request; if it is determined that the query request involves a simple query scenario, the object can be determined as a Phoenix server, and the Phoenix server continues to process the query request.

[0047] HBase cluster: As the service foundation, it stores the most basic data, supports DML and DDL of Phoenix server, and can store metadata of Phoenix tables, etc. The data written in the HBase cluster can be synchronized to the Kafka cluster in real time through a custom replication plug-in.

[0048] Kafka cluster: As a distributed message queue, it is mainly used to temporarily store data synchronized from the HBase cluster. When the Spark engine fails, it can temporarily store data or re-consume data when the failure is recovered, thereby ensuring the final consistency of the Spark engine storage end data and the HBase cluster. At the same time, the HBase cluster and the Spark engine are decoupled, so that when the traffic surges on the production end of the HBase cluster, the Spark engine will not be unable to handle it and data will not be lost.

[0049] Spark engine: As a distributed computing engine for OLAP, it has two main functions. The first function is to consume data in the Kafka cluster, that is, to consume the add, delete, and modify related data produced by the HBase cluster, parse each line of JSON data according to the corresponding table structure, and synchronize the corresponding add, delete, and modify data to the data lake, thereby ensuring that the data in the data lake and the data in the HBase cluster maintain eventual consistency; the second function is to serve as an OLAP analysis engine, execute SQL statements sent by the Phoenix server, and return the results to the Phoenix server.

[0050] Data Lake: As a data storage layer, it mainly provides cheap data storage services and supports the Spark engine to update the added, deleted, and modified data to the storage layer.

[0051] Step 202: Parse the SQL statement.

[0052] In this embodiment, parsing the SQL statement can convert the SQL statement into an internal structure or instruction that can be executed by the database, and determine the type of data request sent by the client based on the parsing result, where the data request may include a data write request, a data query request, etc.

[0053] For example, Figure 4 As shown, after the client sends the SQL request, the Phoenix server can be used to receive the SQL request and parse it, and determine whether the SQL request is a query request based on the parsing result. If the SQL request is a query request, the SQL request can be routed to the lakehouse through the query routing module, and then the lakehouse receives the SQL request and performs corresponding operations. If the SQL request is not a query request, or the SQL request cannot be routed to the lakehouse, the Phoenix server can continue to process the SQL request.

[0054] Optionally, when it is determined based on the analysis result that a data write request sent by the client is received, the data that has completed the write processing in the HBase cluster is synchronously stored in the data lake through the OLAP analysis engine.

[0055] For example, Figure 5As shown in the figure, the data writing and synchronization process is shown. The HBase cluster can receive data writing requests (for example, data writing requests, data updating requests, and data deleting requests) sent by the client or Phoenix server, and then use the replication plug-in to parse the Write-Ahead Logging (WAL) data and write it to the Kafka cluster. Then, the Spark engine is used to consume the data stored synchronously in the Kafka cluster, parse the data and write it to the data lake, providing high-performance OLAP analysis and writing capabilities.

[0056] Optionally, the data written in the HBase cluster is synchronously stored in the data lake through the OLAP analysis engine, which may include: using the replication plug-in to write the data corresponding to the data write request in the HBase cluster to the Kafka cluster, and the Kafka cluster is used to temporarily store the data in the HBase cluster; consuming the data in the Kafka cluster through the OLAP analysis engine, and writing the consumed data in the Kafka cluster into the data lake.

[0057] In this embodiment, the Kafka cluster is used to temporarily store the data in the HBase cluster. When the Spark engine fails, the Kafka cluster can temporarily store the data in the HBase cluster, and re-consume the data when the failure is recovered to prevent data loss. Accordingly, the data in the Kafka cluster after consumption is written to the data lake, which enables the OLAP analysis engine to directly obtain data from the data lake when querying and analyzing data, reducing the load of the HBase cluster and improving the stability of the HBase cluster. Among them, the replication plug-in can provide flexible configuration items, and users can configure Kafka connection information and fields that need to be synchronized.

[0058] Optionally, a replication plug-in is used to write the data corresponding to the data write request in the HBase cluster to the Kafka cluster, which may specifically include: writing the data corresponding to the data write request to the write-ahead log, and parsing the row key corresponding to the log data in the write-ahead log; using the row key as the partition field of the Kafka cluster, and writing the log data to the Kafka cluster according to the partition field.

[0059] In this embodiment, you can first install the replication plug-in, create the Phoenix table that needs to be synchronized, and configure the HBase cluster-related configuration corresponding to the Phoenix table that needs to be synchronized. The HBase cluster then writes the data to the WAL, and then actually performs write-related operations. Specifically, the replication plug-in can be used to parse the encoded log data into corresponding fields and corresponding data types according to the table structure information defined in the Phoenix metadata, and then merge the data with the same row key to reduce the number of data synchronization requests. At the same time, it supports Kafka partitioning based on rowkey to ensure that the data with the same rowkey is produced in order to Kafka, and at the same time ensure the orderliness and consistency of the data when the consumer consumes the data, thereby ensuring the consistency of the data in the HBase cluster and the data lake.

[0060] Optionally, before taking the row key as the partition field of the Kafka cluster and writing the log data to the Kafka cluster according to the partition field, the method may further include: dividing the log data into byte arrays according to the joint primary key information of the data corresponding to the data write request; converting the byte array into data of the corresponding data type according to the joint primary key field type information of the data corresponding to the data write request; and merging the data with the same row key to obtain the target data.

[0061] Optionally, the row key is used as a partition field of the Kafka cluster, and the log data is written into the Kafka cluster according to the partition field. Specifically, the row key may be used as a partition field of the Kafka cluster, and the target data is written into the Kafka cluster according to the partition field.

[0062] For example, Figure 6 As shown, the replication and parsing process is shown, and the specific process is as follows:

[0063] (1) Use the replication plug-in to automatically read a batch of data entries from the WAL for processing;

[0064] (2) Filter out cells that do not need to be synchronized according to the configuration to reduce unnecessary resource waste. For example, if the source HBase table has 10 fields, but the user only synchronizes 5 of them through configuration instructions, the data of the remaining 5 fields that do not need to be synchronized will be filtered out;

[0065] (3) According to the Phoenix table structure metadata, first parse the rowkey of each cell, calculate the number of bytes occupied by each field according to the table's joint primary key information and the joint primary key field type information, and use delimiters to split the fields of variable length. In this way, the fields contained in the rowkey are divided into multiple byte arrays one by one, and each byte array corresponds to one field.

[0066] (4) converting the segmented multiple byte arrays into data of corresponding data types according to the data type information and sorting information of each field;

[0067] (5) Parse the value of each Cell. Each Cell contains a field, and parse it into data of corresponding data type according to the data type;

[0068] (6) Each time a piece of data is processed, it is necessary to determine whether the data operation type (including modification and deletion) of the current data is consistent with the previous one. If it is consistent, step (8) can be executed; if it is inconsistent, step (7) can be executed;

[0069] (7) Since each row of data contains only one cell, that is, only one field, after parsing, the data with the same row key needs to be merged to form a complete row of data including all fields to obtain the target data;

[0070] (8) Determine whether the batch of data read has been processed. If so, execute step (9); if not, execute step (3);

[0071] (9) Convert each row of parsed data into JSON format, use the rowkey as the partition field of the Kafka cluster, and use the Kafka producer client to write the target data into the Kafka cluster;

[0072] (10) Start a new cycle from step (1).

[0073] Step 203: When it is determined that the SQL statement is a data query statement according to the analysis result of the SQL statement, it is determined to receive a query request, and the query request carries the data query statement.

[0074] For example, after parsing the SQL statement, if it is determined that the SQL statement is a data query statement based on the parsing result of the SQL statement, the Phoenix server can use the query routing module to send the query request to the OLAP analysis engine, and the OLAP analysis engine executes the specific query task.

[0075] Step 204: Obtain a query plan corresponding to the data query statement.

[0076] In this embodiment, the query routing module in the Phoenix server can determine the object of processing the query request by parsing the query plan corresponding to the data query statement. For example, if the query scenario corresponding to the query request is relatively simple, the Phoenix server can continue to process it. If the query scenario corresponding to the query request is relatively complex, the query request can be forwarded to the OLAP analysis engine, and the data corresponding to the query plan is obtained from the data lake by the OLAP analysis engine for processing, thereby reducing the CPU and memory resources occupied by the HBase cluster during the query process, reducing the load of the HBase cluster, and improving the stability of the HBase cluster.

[0077] Step 205: The data corresponding to the query plan is obtained from the data lake through the OLAP analysis engine for processing to obtain data processing results. The data lake synchronously stores the data that has completed the write processing in the HBase cluster.

[0078] In this way, the network jitter caused by the OLAP analysis engine directly accessing HBase data or directly reading HBase's Hfile files is reduced, and the stability of the HBase cluster itself is improved.

[0079] Optionally, step 205 may specifically include: converting the query plan into a dialect SQL statement supported by the OLAP analysis engine; using the dialect SQL statement to obtain data corresponding to the query plan from the data lake through the OLAP analysis engine for processing.

[0080] Specifically, the server can parse the SQL statement to obtain the query plan, and then add the SQL dialect module, expand multiple SQL dialects through syntax configuration, convert the query plan into SQL statements, and obtain the dialect SQL statements supported by the OLAP analysis engine, so that the server can process data as a supported multiple SQL statements, add more syntax support for SQL parsing, unify the SQL syntax and access entry, and facilitate querying multiple databases and data sources. At the same time, it can be compatible with different OLAP analysis engines, so that data writing and OLAP analysis both use the server as the entry. Such a unified access entry eliminates the need for users to connect to multiple systems at the same time, making it simpler to use. The unified SQL syntax eliminates the need for users to use multiple incompatible SQL statements, reducing learning costs, improving syntax compatibility, improving ease of use, and eliminating network jitter and the impact on cluster stability caused by large-scale OLAP analysis.

[0081] For example, in order to be compatible with the SQL statements of the SparkSQL module in the Spark engine, the query plan can be converted into the dialect SQL statements supported by the Spark engine, and then the SparkSQL module is used to obtain the data corresponding to the query plan from the data lake for processing.

[0082] Optionally, before step 205, it can be determined whether the query data involved in the query plan has been synchronized to the data lake; and, it can be determined whether the functions involved in the query plan are all supported for calling in the OLAP analysis engine; and, it can be determined whether the operator type involved in the query plan is a preset operator type corresponding to the OLAP analysis engine.

[0083] Optionally, when the query data involved in the query plan has been synchronized to the data lake, the functions involved in the query plan are all supported for calling in the OLAP analysis engine, and the operator type involved in the query plan is a preset operator type, the data corresponding to the query plan is obtained from the data lake through the OLAP analysis engine for processing.

[0084] Specifically, when the SQL statement is determined to be a data query statement based on the analysis result of the SQL statement, it can be further determined whether to send the data query request to the OLAP analysis engine based on the query plan corresponding to the data query statement. In the query optimization logic, it can be determined whether to forward the query request to the OLAP analysis engine based on the tables, fields, function operator types, etc. involved in the query plan tree corresponding to the query plan.

[0085] For example, it can be determined whether all tables involved in the query plan have been synchronized to the data lake, and whether the functions involved are supported in SparkSQL, and whether the operator contains join, etc. If the above conditions are met, the query plan can be converted into the SparkSQL dialect and forwarded to SparkSQL, and the corresponding query statement can be continued to be executed, and the result can be returned to the client. Otherwise, the Phoenix server can continue to execute the corresponding query statement and return the execution result to the client. After receiving the query request, SparkSQL will consume the data in the Kafka cluster in real time when executing the Spark service, and synchronize the added, deleted and modified data to the data lake to ensure the consistency of the data in the data lake and the data in the HBase cluster.

[0086] Step 206: Send the data processing result to the client.

[0087] Compared with the current existing technology, this embodiment can parse the SQL statement sent by the client, determine the object to execute the SQL statement according to the parsing result, and if the SQL statement is a data query statement, obtain the data corresponding to the query plan from the data lake through the OLAP analysis engine for processing; if the SQL statement is a data write statement, the OLAP analysis engine synchronously stores the data completed in the HBase cluster in the data lake through the OLAP analysis engine, thereby providing a unified service entrance for supporting queries and writes, improving ease of use, and eliminating network jitter caused by large-scale OLAP analysis and the impact on the stability of the HBase cluster.

[0088] Further, as Figure 1 The specific implementation of the method shown in this embodiment provides a data processing device, such as Figure 7 As shown, the device includes: a receiving module 31, an acquiring module 32, and a sending module 33.

[0089] The receiving module 31 is configured to receive a query request for data in the HBase cluster sent by a client, wherein the query request carries a data query statement;

[0090] An acquisition module 32 is configured to acquire a query plan corresponding to a data query statement;

[0091] The acquisition module 32 is configured to acquire data corresponding to the query plan from the data lake through the OLAP analysis engine for processing to obtain data processing results. The data lake synchronously stores the data that has completed the write processing in the HBase cluster;

[0092] The sending module 33 is configured to send the data processing result to the client.

[0093] In some examples of this embodiment, the acquisition module 32 is specifically configured to convert the query plan into a dialect SQL statement supported by the OLAP analysis engine; and use the dialect SQL statement to obtain data corresponding to the query plan from the data lake through the OLAP analysis engine for processing.

[0094] In some examples of the present embodiment, the acquisition module 32 is further specifically configured to determine whether the query data involved in the query plan has been synchronized to the data lake; and, determine whether the functions involved in the query plan all support calling in the OLAP analysis engine; and, determine whether the operator type involved in the query plan is a preset operator type corresponding to the OLAP analysis engine; when the query data involved in the query plan has been synchronized to the data lake, and the functions involved in the query plan all support calling in the OLAP analysis engine, and the operator type involved in the query plan is a preset operator type, the data corresponding to the query plan is obtained from the data lake through the OLAP analysis engine for processing.

[0095] In some examples of this embodiment, the acquisition module 32 is specifically configured to receive SQL statements sent by the client for data operations in the HBase cluster; parse the SQL statements; when it is determined according to the parsing result that a data write request sent by the client is received, synchronously store the data written in the HBase cluster in the data lake through the OLAP analysis engine; when it is determined according to the parsing result that the SQL statement is a data query statement, determine to receive the query request.

[0096] In some examples of this embodiment, the acquisition module 32 is specifically configured to use the replication plug-in to write the data corresponding to the data write request in the HBase cluster into the Kafka cluster, and the Kafka cluster is used to temporarily store the data in the HBase cluster; consume the data in the Kafka cluster through the OLAP analysis engine, and write the consumed data in the Kafka cluster into the data lake.

[0097] In some examples of this embodiment, the acquisition module 32 is specifically configured to write the data corresponding to the data write request into the pre-write log, and parse the row key corresponding to the log data in the pre-write log; use the row key as the partition field of the Kafka cluster, and write the log data into the Kafka cluster according to the partition field.

[0098] In some examples of the present embodiment, the acquisition module 32 is further specifically configured to divide the log data into byte arrays according to the joint primary key information of the data corresponding to the data write request; convert the byte array into data of the corresponding data type according to the joint primary key field type information of the data corresponding to the data write request; merge the data with the same row key to obtain the target data; use the row key as the partition field of the Kafka cluster, and write the target data into the Kafka cluster according to the partition field.

[0099] It should be noted that for other corresponding descriptions of the functional units involved in the data processing device provided in this embodiment, reference can be made to Figure 1 and Figure 2 The corresponding description in will not be repeated here.

[0100] Based on the above Figure 1 and Figure 2 The method shown in the embodiment also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned Figure 1 and Figure 2 The method shown.

[0101] Based on the above Figure 1 and Figure 2The method shown in the embodiment also provides a computer program product, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned Figure 1 and Figure 2 The method shown.

[0102] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of the present application.

[0103] Based on the above Figure 1 and Figure 2 The method shown, and Figure 7 In order to achieve the above-mentioned purpose, the embodiment of the present application also provides an electronic device, such as a personal computer, a server, a laptop computer, a smart phone, an intelligent robot or other intelligent terminal, which includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to achieve the above-mentioned Figure 1 and Figure 2 The method shown.

[0104] Optionally, the above-mentioned physical device may also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.

[0105] Those skilled in the art will appreciate that the above-mentioned physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or a combination of certain components, or different arrangements of components.

[0106] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the above-mentioned physical device, and supports the operation of the information processing program and other software and / or programs. The network communication module is used to realize the communication between the components inside the storage medium, and the communication with other hardware and software in the information processing physical device.

[0107] Through the description of the above implementation methods, the technical personnel in this field can clearly understand that the present application can be implemented by means of software plus the necessary general hardware platform, or it can be implemented by hardware. By applying the technical solution of this embodiment, this embodiment can first receive a query request for data in the HBase cluster sent by the client, and the query request carries a data query statement; then obtain the query plan corresponding to the data query statement; then obtain the data corresponding to the query plan from the data lake through the OLAP analysis engine for processing to obtain the data processing result, and the data lake synchronously stores the data that has completed the write processing in the HBase cluster; finally, send the data processing result to the client. In this way, it can support application scenarios involving complex associated queries and analytical queries, effectively reduce the CPU and memory resources occupied by the HBase cluster during the query process, reduce the load of the HBase cluster, and improve the stability of the HBase cluster.

[0108] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0109] The above is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features applied for herein.

Claims

1. A data processing method, characterized in that: include: Receive a query request for data in an HBase cluster sent by a client, wherein the query request carries a data query statement; Obtaining a query plan corresponding to the data query statement; The OLAP analysis engine obtains data corresponding to the query plan from the data lake for processing to obtain data processing results, and the data lake synchronously stores the data that has completed the write processing in the HBase cluster; The data processing result is sent to the client.

2. The method according to claim 1, characterized in that: The obtaining data corresponding to the query plan from the data lake by the OLAP analysis engine for processing includes: Converting the query plan into a dialect SQL statement supported by the OLAP analysis engine; The dialect SQL statement is used to obtain data corresponding to the query plan from the data lake through the OLAP analysis engine for processing.

3. The method according to claim 1, characterized in that Before obtaining the data corresponding to the query plan from the data lake by the OLAP analysis engine for processing, the method further includes: Determining whether the query data involved in the query plan has been synchronized to the data lake; and, Determining whether all functions involved in the query plan are supported for calling in the OLAP analysis engine; and, Determining whether the operator type involved in the query plan is a preset operator type corresponding to the OLAP analysis engine; Wherein, obtaining data corresponding to the query plan from the data lake through the OLAP analysis engine for processing includes: When the query data involved in the query plan has been synchronized to the data lake, the functions involved in the query plan are all supported for calling in the OLAP analysis engine, and the operator type involved in the query plan is the preset operator type, the data corresponding to the query plan is obtained from the data lake by the OLAP analysis engine for processing.

4. The method according to claim 1, characterized in that: Before receiving the query request for data in the HBase cluster sent by the client, the method further includes: Receive SQL statements sent by the client for operating data in the HBase cluster; Parsing the SQL statement; When it is determined according to the analysis result that a data write request sent by the client is received, the data written in the HBase cluster is synchronously stored in the data lake by the OLAP analysis engine; The receiving of a query request for data in the HBase cluster sent by a client includes: In the case where the SQL statement is determined to be a data query statement according to the parsing result of the SQL statement, it is determined to receive the query request.

5. The method according to claim 4, characterized in that The synchronously storing the data written in the HBase cluster in the data lake by the OLAP analysis engine includes: Using a replication plug-in, the data corresponding to the data write request in the HBase cluster is written into a Kafka cluster, where the Kafka cluster is used to temporarily store the data in the HBase cluster; The data in the Kafka cluster is consumed by the OLAP analysis engine, and the consumed data in the Kafka cluster is written into the data lake.

6. The method according to claim 5, characterized in that The using the replication plug-in to write the data corresponding to the data write request in the HBase cluster into the Kafka cluster includes: Writing the data corresponding to the data write request into a write-ahead log, and parsing the row key corresponding to the log data in the write-ahead log; The row key is used as a partition field of a Kafka cluster, and the log data is written into the Kafka cluster according to the partition field.

7. The method according to claim 6, characterized in that Before using the row key as a partition field of the Kafka cluster and writing the log data into the Kafka cluster according to the partition field, the method further includes: According to the joint primary key information of the data corresponding to the data write request, the log data is divided into a byte array; According to the joint primary key field type information of the data corresponding to the data write request, converting the byte array into data of the corresponding data type; Merge the data with the same row key to obtain target data; The step of using the row key as a partition field of a Kafka cluster and writing the log data into the Kafka cluster according to the partition field includes: The row key is used as a partition field of the Kafka cluster, and the target data is written into the Kafka cluster according to the partition field.

8. A data processing device, characterized in that: include: A receiving module is configured to receive a query request for data in an HBase cluster sent by a client, wherein the query request carries a data query statement; An acquisition module is configured to acquire a query plan corresponding to the data query statement; An acquisition module is configured to acquire data corresponding to the query plan from the data lake through the OLAP analysis engine for processing to obtain data processing results, wherein the data lake synchronously stores the data that has completed the write processing in the HBase cluster; The sending module is configured to send the data processing result to the client.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product having a computer program stored thereon, characterized in that: When the computer program product is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Data processing method and system and non-instantaneous computer readable storage medium

    CN113760983A

  • Data lake data processing method and system

    CN115185955A

  • Data processing method and device and storage medium

    CN116186053A

  • Processing system for lightweight big data

    CN116860746A

  • Data storage and processing method based on heterogeneous technology

    CN117251414A