Database system, data processing method, device and computer storage medium

Abstract

An embodiment of the present invention provides a database system, a data processing method, an apparatus, and a computer storage medium. The database system includes a database cluster and a computing service cluster; the database cluster includes a distributed SQL engine and a database node storing a data source; the distributed SQL engine is used to receive a first data analysis request, wherein the first data analysis request includes a request to perform data analysis on a data source in the database cluster; and convert the first data analysis into a second data analysis request executable by the computing service cluster, so that the computing service cluster directly performs the operation requested by the second data analysis request on the data source of the database cluster. Through the embodiment of the present invention, efficient big data analysis can be performed based on a relational database system.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer technology, and in particular to a database system, a data processing method, a device, and a computer storage medium. Background Art

[0002] The data generated by internet businesses is characterized by high concurrency and rapid growth in data size. To reliably store this data and meet the online transaction requirements based on this data, developers typically choose a sharding solution based on relational databases (for example, MySQL sharding) to meet the data storage and usage needs of internet businesses.

[0003] With the advancement of the times and the rise of big data, internet businesses are increasingly demanding large volumes of production data with high timeliness. In addition to the aforementioned data storage and usage requirements, there is also a need for big data analysis involving large amounts of data. However, traditional relational databases with sharded databases and tables are often single-machine databases or use single-threaded execution mechanisms, making them unable to support big data analysis needs.

[0004] Therefore, how to achieve efficient and real-time big data analysis based on a relational database with sharded libraries and tables has become an urgent problem that needs to be solved. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a database system, a data processing method, an apparatus, and a computer storage medium to solve some or all of the above problems.

[0006] According to a first aspect of an embodiment of the present invention, a database system is provided, which includes a database cluster and a computing service cluster; the database cluster includes a distributed SQL engine and database nodes storing data sources, and the distributed SQL engine is used to receive a first data analysis request, wherein the first data analysis request includes a request to perform data analysis on the data source in the database cluster; the first data analysis is converted into a second data analysis request that can be executed by the computing service cluster, so that the computing service cluster directly performs the operation requested by the second data analysis request on the data source of the database cluster.

[0007] According to a second aspect of an embodiment of the present invention, a data processing method is provided, which is used for the aforementioned database system, and the method comprises: receiving a first data analysis request, wherein the first data analysis request comprises a request for performing data analysis on a data source in the database cluster; converting the first data analysis request into a second data analysis request that can be executed by a computing service cluster of the database system, so that the computing service cluster directly performs the operation requested by the second data analysis request on the data source of the database cluster; wherein the computing service cluster and the distributed database cluster are incompatible data sources.

[0008] According to a third aspect of an embodiment of the present invention, a data processing device is provided, comprising: a receiving module for receiving a first data analysis request, wherein the first data analysis request is used to request data analysis of a data source in the database cluster; a conversion module for converting the first data analysis request into a second data analysis request that can be executed by a computing service cluster of the database system, so that the computing service cluster performs the operation requested by the second data analysis request on the data source of the database cluster.

[0009] According to a fourth aspect of an embodiment of the present invention, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the data processing method according to the second aspect is implemented.

[0010] According to the database system provided by an embodiment of the present invention, a distributed SQL engine is used to convert data analysis requests, so that the computing service cluster does not need to pay attention to the details of the database cluster's sharding and table sharding, and can directly perform the requested operation on the data source of the database cluster according to the second data analysis request, thereby making the computing service cluster compatible with database clusters with sharding and table sharding data sources without the need to establish its own storage layer to store data, and thus the process of synchronizing the data source of the database cluster can be omitted, thereby ensuring the real-time and high efficiency of big data analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0012] Figure 1 This is a structural block diagram of a database system according to the first embodiment of the present invention;

[0013] Figure 2This is a flowchart of a data processing method according to the second embodiment of the present invention;

[0014] Figure 3 This is a flowchart of a data processing method according to Embodiment 3 of the present invention;

[0015] Figure 4 A flowchart of the steps of a data processing method according to a usage scenario of the present invention;

[0016] Figure 5 This is a structural diagram of a data processing device according to a fourth embodiment of the present invention;

[0017] Figure 6 2 is a structural diagram of a data processing device according to embodiment 5 of the present invention. DETAILED DESCRIPTION

[0018] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in the embodiments of the present invention should fall within the scope of protection of the embodiments of the present invention.

[0019] The specific implementation of the embodiment of the present invention is further described below with reference to the accompanying drawings of the embodiment of the present invention.

[0020] Example 1

[0021] Reference Figure 1 , shows a structural block diagram of a database system according to embodiment 1 of the present invention.

[0022] The database system of this embodiment includes a database cluster 100, a distributed SQL engine 101, and a computing service cluster 300.

[0023] The database cluster 100 includes a distributed SQL engine 101 and a database node 102 storing a data source. The distributed SQL engine 101 is used to receive a first data analysis request, wherein the first data analysis request includes a request to perform data analysis on the data source in the database cluster 100.

[0024] The distributed SQL engine 101 is also used to convert the first data analysis into a second data analysis request that can be executed by the computing service cluster 300, so that the computing service cluster 300 directly performs the operation requested by the second data analysis request on the data source of the database cluster 100.

[0025] In this embodiment, database cluster 100 may be a relational database that employs a sharding algorithm to perform database and table sharding operations. Distributed database cluster 100 includes multiple database instances. The sharding algorithm divides a data table into multiple shards according to a specific rule, thereby distributing and storing the data table across one or more database instances.

[0026] The distributed SQL engine 101 can be independent of the database nodes 102, or it can be deployed on the same server as one or more database nodes 102. Optionally, the distributed SQL engine 101 can be implemented as a middleware service. The middleware service is used to add a service process between the user and the distributed database nodes 102 to provide routing capabilities for the distributed database nodes 102. Requests sent by users are routed to database instances on one or more database nodes 102 based on the sharding algorithm of the middleware service.

[0027] The computing service cluster 300 can be a cluster of devices deployed with a general-purpose parallel computing framework (such as Spark or Flink). Both Spark and Flink are general-purpose distributed computing frameworks that can provide distributed computing capabilities. They can support distributed computing, thereby meeting the data analysis needs in big data scenarios.

[0028] Since the computing service cluster 300 and the database cluster 100 are incompatible data sources, that is, because the computing service cluster 300 and the database cluster 100 adopt different architectures, the two are not compatible with each other, so the computing service cluster 300 cannot directly access the sharded database and table data sources in the database cluster 100 to obtain data, nor can it correctly parse SQL statements, and cannot be fully compatible with and implement the functions of the MySQL database.

[0029] In order to solve the problem that the database system cannot meet the data analysis requirements due to the incompatible data sources between the computing service cluster 300 and the database cluster 100, in this embodiment, the computing service cluster 300 is integrated with the distributed SQL engine 101, so that the database system not only has the data storage and processing capabilities of a traditional database, but also has data analysis capabilities, which can meet the data analysis needs in big data scenarios.

[0030] Specifically, when the distributed SQL engine 101 receives a first data analysis request requesting to perform data analysis on a data source in the database cluster 100 , the distributed SQL engine 101 converts the first data analysis request into a second data analysis request executable by the computing service cluster 300 .

[0031] Since the distributed SQL engine 101 is used to convert data analysis requests, the computing service cluster 300 does not need to pay attention to the details of the sharding of the database cluster 100. It can directly perform the requested operation on the data source of the database cluster 100 according to the second data analysis request, so that the computing service cluster 300 can be compatible with the database cluster 100 with sharding of the data source without having to establish its own storage layer to store the data. As a result, the process of synchronizing the data source of the database cluster 100 can be omitted, thereby ensuring the real-time and high efficiency of big data analysis.

[0032] Optionally, to improve the compatibility of the computing service cluster 300 with the database cluster 100 using MySQL, the following is an example of a feasible implementation method for the distributed SQL engine 101 to convert the first data analysis request into a second data analysis request that can be executed by the computing service cluster 300. The implementation method includes the following steps:

[0033] Step A: Generate a logical execution plan according to the first data analysis request.

[0034] In a specific implementation, the first data analysis request includes an SQL statement, and the distributed SQL engine 101 can parse the SQL statement, generate an abstract syntax tree, and generate a logical execution plan based on the abstract syntax tree.

[0035] In the database system of this embodiment, the distributed SQL engine 101 processes the SQL statement in the first data analysis request, so that the database system can be perfectly compatible with MySQL, allowing users to use the database system for big data analysis without any learning cost, thereby improving convenience and ease of use.

[0036] Optionally, in this embodiment, the database cluster 100 includes a primary database and a backup database with consistent data. In order to improve adaptability and realize dynamic management of data sources, when generating a logical execution plan based on the abstract syntax tree, the distributed SQL engine 101 generates the first data analysis request into a logical execution plan containing information for accessing the primary database or accessing the backup database based on the load information of the primary database and the load information of the backup database.

[0037] This enables dynamic management of data sources and can dynamically switch traffic between the primary and standby databases to meet various needs.

[0038] Step B: Convert the logical execution plan into multiple distributed execution plans executable by the computing service cluster 300.

[0039] The computing service cluster 300 may include multiple computing nodes, which may perform parallel computing to improve computing capabilities and enhance the scalability of the computing service cluster 300 .

[0040] By generating a distributed execution plan and sending it to the computing service cluster 300, the computing service cluster 300 can obtain the data requested by the second data analysis from the data source of the database cluster 100 by executing the multiple distributed execution plans, and perform the operation requested by the second data analysis request.

[0041] In a specific implementation, step B includes the following sub-steps:

[0042] Sub-step B1: Generate a physical layer execution plan based on the physical layer of the database cluster 100 according to the logical execution plan, wherein the physical layer execution plan includes a target sub-table to be executed on the physical layer.

[0043] The target shard table to be executed may be one or more shard tables in the shard database and shard table data source.

[0044] For example, the logical execution plan is optimized by the optimizer in the distributed SQL engine 101, and the physical layer table routing calculation is performed based on the relevant key values ​​in the SQL statement (such as table name, field name, etc.) and the metadata in each database node 102 (for example, the metadata of the database instance), and the target table to be executed involved in the first data analysis request is determined, so as to generate a physical layer execution plan including the target table to be executed at the physical layer based on the physical layer of the database cluster 100.

[0045] By pushing the logical execution plan down to the physical layer and generating a physical layer execution plan, the distributed execution plans subsequently generated based on the physical layer execution plan are all execution plans pushed down to the physical layer, thereby optimizing the efficiency of pulling data when the subsequent computing service cluster 300 executes the distributed execution plan.

[0046] Of course, for a database cluster 100 that adopts different database and table sharding algorithms, the manner of generating a physical layer execution plan may be different, and this embodiment does not limit this.

[0047] Sub-step B2: converting the physical layer execution plan into multiple physical layer distributed execution plans executable by the computing service cluster 300.

[0048] Because the computing service cluster 300 and the database cluster 100 use different architectures and have incompatible data sources, the computing service cluster 300 cannot directly process the data source of the database cluster 100. Therefore, in order to make the computing service cluster 300 compatible with the functions of the database cluster 100 and directly obtain data sources from the database cluster 100, the computing service cluster 300 can be modified in advance. This modification includes the following two parts:

[0049] The first part: Encapsulate the logical table of the database cluster 100 as a data set (i.e., Dataframe, which is a distributed data set) in the computing service cluster 300. Each sub-table in the logical table serves as a partition (i.e., Partition) of the Dataframe to implement the mapping of the logical table to the data set to establish a table association relationship. This opens up the data channel between the database cluster 100 and the computing service cluster 300, allowing the computing service cluster 300 to directly use the data source of the database cluster 100 as data without the need to establish its own data storage or perform data synchronization, thereby ensuring the real-time nature of the data during data analysis.

[0050] The second part: Custom functions, customized aggregate functions, and executor modifications are performed in the Compute Service Cluster 300 to fully support functions, aggregate functions, and subqueries in MySQL. This makes the Compute Service Cluster 300 highly compatible with MySQL functions, making it extremely easy to use without requiring additional learning or business transformation.

[0051] In view of the different computing frameworks used by the computing service cluster 300, the custom functions, custom aggregation functions and executor transformations may be performed in a manner that is compatible with the computing framework, which is not limited in this embodiment.

[0052] Of course, depending on different functional requirements, the computing service cluster 300 may not be modified, as long as it is ensured that the computing service cluster 300 has the required functions.

[0053] For the transformed computing service cluster 300 , the physical layer execution plan is converted into a physical layer distributed execution plan executable by the computing service cluster 300 , so that the computing service cluster 300 can subsequently perform distributed computing.

[0054] Specifically, sub-step B2 is implemented as follows: determining the corresponding first operation to be executed and / or the target sub-table corresponding to the first operation to be executed according to the physical layer execution plan; correspondingly determining the second operation to be executed that can be executed by the computing service cluster 300 corresponding to the first operation to be executed, and / or determining the target partition corresponding to the target sub-table in the partition of the data set of the computing service cluster 300; generating the multiple physical layer distributed execution plans according to the second operation to be executed and / or the target partition.

[0055] When determining the second operation to be executed corresponding to the first operation to be executed, the operation association relationship may be used to indicate the correspondence between the operation in the database cluster 100 (eg, a database cluster using MySQL) and the operation in the computing service cluster 300 .

[0056] For example, a first execution function corresponding to the first execution operation is determined, and the second execution operation is generated according to a second execution function corresponding to the first execution function in the computing service cluster 300 .

[0057] The first execution function may be any function in MySQL, such as select (query operation), sum (sum operation), order by (sort operation), join (join operation), etc.

[0058] Each first execution function can be determined by syntax analysis or other methods, and then a second execution function corresponding to the first execution function can be determined. Then, a second to-be-executed operation can be generated based on the second execution function and the syntax logic between the functions. The second execution function can be a native function in the computing service cluster 300, or a custom function or a custom aggregation function.

[0059] When determining the target partition corresponding to the target shard table, the target partition may be determined based on a table association relationship indicating a mapping relationship between each shard table in the database cluster 100 and each partition in the data set of the computing service cluster 300 .

[0060] The target sub-table corresponding to the first operation to be performed may be any sub-table in the database cluster 100. The target sub-table may be determined by syntax analysis or other methods.

[0061] After determining the second operation to be executed and the target partition, multiple physical layer distributed execution plans can be generated based on the second operation to be executed, the target partition, and the grammatical rules used by the computing service cluster 300, so that the computing service cluster 300 can subsequently execute the distributed execution plans to complete data analysis.

[0062] A feasible process for the computing service cluster 300 to execute the distributed execution plan is as follows:

[0063] Step C: Acquire data required by the multiple distributed execution plans from the multiple sub-databases of the database cluster 100 and generate a distributed elastic data set.

[0064] One or more sub-libraries may be deployed in a database node 102 , and each sub-library may include one or more sub-tables.

[0065] The computing service cluster 300 receives the distributed execution plan submitted by the distributed SQL engine 101. It executes the SQL statements in the distributed execution plan to request and retrieve the required data from each shard in the distributed database cluster 100. Based on the retrieved data, it generates a distributed resilient distributed dataset (RDD) for subsequent data analysis. A distributed resilient dataset is a partitioned collection of read-only records.

[0066] Step D: Execute the multiple distributed execution plans on the distributed elastic data set to perform the operation requested by the second data analysis request, obtain the request result and feed it back to the distributed SQL engine 101.

[0067] The computing service cluster 300 performs distributed computing on the distributed elastic data set according to the distributed execution plan, collects the calculation results of each execution plan, and returns the collected calculation results to the distributed SQL engine 101, so that the distributed SQL engine 101 can subsequently package the calculation results into a result set and return it to the user side.

[0068] Through this embodiment, the computing service cluster 300 with big data processing technology is integrated with the database cluster 100 with relational database technology, that is, the computing service cluster 300 is integrated with the distributed SQL engine 101 of the distributed database cluster 100, so that the database system can achieve efficient and high-real-time big data calculations, and can be a relational database with separate libraries and tables.

[0069] In addition, the computing service cluster 300 has been transformed to make it functionally unified with and compatible with the MySQL database.

[0070] For distributed computing scenarios, a physical layer execution plan is generated at the execution plan level based on the partitioned tables of the physical layer of the distributed database, and the complete conversion to the distributed execution plan of the computing service cluster 300 is completed based on the physical layer execution plan, so that user-initiated access to the logical table can be completed through the computing service cluster 300 in a multi-machine parallel manner, achieving the effect of using big data technology to accelerate the calculation of relational databases.

[0071] In addition, the optimizer of the distributed SQL engine 101 has the ability to push down any calculation (such as joins, aggregations, subqueries, etc.) to the underlying MySQL, ensuring the optimal efficiency of data pulling.

[0072] In addition, through custom functions, custom aggregate functions, executor transformation and other methods, complete support for functions, aggregate functions and subqueries in MySQL is achieved, ensuring high compatibility with MySQL functions and extremely easy use, without the need for users to learn or transform their business.

[0073] For example, for the group_concat aggregation function in MySQL, a customized aggregation function groupconcat is created in Spark to achieve the same functionality as the group_concat aggregation function in MySQL.

[0074] When defining functions, customizing aggregate functions, and modifying executors, you can use any appropriate existing methods or tools to implement them based on specific needs. For example, you can use Spark's basic capability interfaces for secondary development to implement functions, aggregate functions, or executors with the same functionality as MySQL.

[0075] This database system can perform distributed computing directly based on the original data in the database, without the need for data synchronization for offline computing. It solves the problem that some database systems in the existing technology (such as TiDB) have their own storage layer and need to migrate business production data before data analysis. It overcomes the shortcomings of huge transformation costs and poor usability for businesses that already use traditional relational database (sharding / no sharding) solutions.

[0076] The optimizer in the distributed SQL engine 101 can automatically perform routing calculations and identification for sharded databases and tables. Users and the computing service cluster 300 do not need to worry about the logic of sharding databases and tables. The optimizer can push down various calculations (aggregation, connection, subquery, etc.) to the physical layer of the distributed database, thereby optimizing the data pulling speed of the computing service cluster 300.

[0077] Through the compatible processing of the distributed SQL engine 101, the functional differences between the computing service cluster 300 and the relational database are shielded, so that the usage experience can be consistent with the native MySQL, and users do not need to perform additional learning.

[0078] This database system enables real-time big data analysis directly on raw data in relational databases, without having to worry about sharding logic or rewriting SQL. It is both user-friendly and highly real-time. For example, users can efficiently perform complex analytical queries such as associations, aggregations, and sorting between logical tables directly on large amounts of data in the database using SQL, just as with native MySQL. The response speed is far superior to that of common relational databases. By introducing a parallel computing service framework, SQL statements can be accelerated in big data scenarios, providing a user experience consistent with MySQL, without the need for additional learning.

[0079] Example 2

[0080] Reference Figure 2 , shows a step flow chart of a data processing method according to embodiment 2 of the present invention.

[0081] The data processing method of this embodiment is used in the database system of the aforementioned embodiment 1, and the method includes the following steps:

[0082] Step S102: Receive a first data analysis request.

[0083] Since the computing service cluster and the database cluster are incompatible data sources, that is, because the computing service cluster and the database cluster adopt different architectures, the two are not compatible with each other, so the computing service cluster cannot directly access the database cluster's sub-library and sub-table data sources to obtain data, nor can it correctly parse SQL statements, and cannot be fully compatible with and implement the functions of the MySQL database.

[0084] In order to solve the problem that the database system cannot meet data analysis requirements due to incompatible data sources between the computing service cluster and the database cluster, in this embodiment, a database system that integrates the computing service cluster with a distributed SQL engine is used to process the first data analysis request. The first data analysis request includes a request to perform data analysis on the data source in the database cluster. For example, the first data analysis request indicates that based on the commodity sales data in the database cluster, the age distribution of people who purchased down jackets in Zhejiang Province from November to December 2019 should be analyzed, etc.

[0085] Step S104: converting the first data analysis request into a second data analysis request executable by the computing service cluster of the database system, so that the computing service cluster directly executes the operation requested by the second data analysis request on the data source of the database cluster.

[0086] The second data analysis request is used to instruct the computing service cluster to directly obtain data from the sharded database and table data sources of the database cluster and operate on the data to complete the analysis.

[0087] The second data analysis request may include a distributed execution plan to facilitate the computing service cluster to perform distributed computing, thereby improving computing efficiency.

[0088] Those skilled in the art may use any appropriate method to convert the first data analysis request into a second data analysis request that can be executed by the computing service cluster.

[0089] For example, the distributed SQL engine rewrites the first data analysis request according to the conversion rule to generate the second data analysis request. The conversion rule can be defined as needed.

[0090] Through this embodiment, a distributed SQL engine is used to convert data analysis requests, so that the computing service cluster does not need to pay attention to the details of the database cluster's sharding and table sharding. It can directly perform the requested operation on the data source of the database cluster according to the second data analysis request, so that the computing service cluster can be compatible with database clusters with sharded and table sharded data sources without having to establish its own storage layer to store data. As a result, the process of synchronizing the data source of the database cluster can be omitted, solving the problem of high latency in the existing technology that data analysis must wait until data synchronization is completed, thereby ensuring the real-time and high efficiency of big data analysis.

[0091] The data processing method of this embodiment can be executed by any appropriate electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as tablet computers, mobile phones, etc.) and PCs, etc.

[0092] Example 3

[0093] Reference Figure 3 , shows a step flow chart of a data processing method according to embodiment three of the present invention.

[0094] The data processing method of this embodiment is used in the database system in the aforementioned embodiment 1, and the method includes the aforementioned steps S102 to S104.

[0095] In this embodiment, step S104 includes the following sub-steps:

[0096] Sub-step S1041: Generate a logical execution plan according to the first data analysis request.

[0097] In a specific implementation, the first data analysis request includes an SQL statement, and the distributed SQL engine can parse the SQL statement, generate an abstract syntax tree, and generate a logical execution plan based on the abstract syntax tree.

[0098] Optionally, in this embodiment, the distributed SQL engine can be implemented as a middleware service to facilitate user operation and management of each database node.

[0099] In the database system of this embodiment, the SQL statement in the first data analysis request is processed by a distributed SQL engine, so that the database system can be perfectly compatible with MySQL, allowing users to use the database system for big data analysis without any learning cost, thereby improving convenience and ease of use.

[0100] Optionally, in this embodiment, the database cluster includes a primary database and a backup database with consistent data. In order to improve adaptability and realize dynamic management of data sources, when generating a logical execution plan based on the abstract syntax tree, the distributed SQL engine generates the first data analysis request into a logical execution plan containing information for accessing the primary database or accessing the backup database based on the load information of the primary database and the load information of the backup database.

[0101] This enables dynamic management of data sources and can dynamically switch traffic between the primary and standby databases to meet various needs.

[0102] Sub-step S1042: Convert the logical execution plan into multiple distributed execution plans executable by the computing service cluster, so that the computing service cluster obtains the data of the second data analysis request from the data source of the distributed database cluster by executing the multiple distributed execution plans, and performs the operation requested by the second data analysis request.

[0103] By generating a distributed execution plan and sending it to the computing service cluster, the computing service cluster can obtain the data requested by the second data analysis from the data source of the database cluster and perform the operation requested by the second data analysis request by executing the multiple distributed execution plans.

[0104] The sub-step S1042 includes the following sub-steps:

[0105] Sub-step S1042a: Generate a physical layer execution plan based on the physical layer of the database cluster according to the logical execution plan, wherein the physical layer execution plan includes a target sub-table to be executed on the physical layer.

[0106] For example, the logical execution plan is optimized by the optimizer in the distributed SQL engine, and the physical layer table routing calculation is performed based on the relevant key values ​​in the SQL statement (such as table name, field name, etc.) and the metadata of each database instance to determine the target table to be executed involved in the first data analysis request, so as to generate a physical layer execution plan including the target table to be executed at the physical layer based on the physical layer of the database cluster.

[0107] By pushing the logical execution plan down to the physical layer and generating a physical layer execution plan, the distributed execution plans subsequently generated based on the physical layer execution plan are all execution plans pushed down to the physical layer. This optimizes the efficiency of pulling data when the subsequent computing service cluster executes the distributed execution plan.

[0108] Of course, for distributed database clusters that use different database and table sharding algorithms, the manner of generating the physical layer execution plan may be different, and this embodiment does not limit this.

[0109] Sub-step S1042b: converting the physical layer execution plan into multiple physical layer distributed execution plans executable by the computing service cluster.

[0110] Because the Compute Service cluster and the database cluster use different architectures, the Compute Service cluster cannot directly process MySQL's physical layer execution plan. Therefore, to make the Compute Service cluster compatible with the database cluster's functions and directly obtain data sources from the distributed database's sharded database tables, you can pre-modify the Compute Service cluster. This modification includes the following two parts:

[0111] The first part: Encapsulate the logical table of the distributed database cluster as a data set (i.e., Dataframe, which is a distributed data set) in the computing service cluster. Each shard table in the logical table serves as a partition (i.e., Partition) of the Dataframe to implement the mapping from the logical table to the data set and establish table associations. This opens up the data channel between the database cluster and the computing service cluster, allowing the computing service cluster to directly use the sharded database and sharded table data sources of the database cluster as data without the need for additional independent data storage or data synchronization, ensuring real-time data analysis.

[0112] The second part: Custom functions, custom aggregate functions, and executor modifications are performed in the Compute service cluster to fully support functions, aggregate functions, and subqueries in MySQL. This makes the Compute service cluster highly compatible with MySQL functions, making it extremely easy to use without requiring additional learning or business transformation.

[0113] Of course, depending on the functional requirements, the computing service cluster does not need to be modified, as long as it is ensured that the computing service cluster has the required functions.

[0114] For the transformed computing service cluster, the physical layer execution plan is converted into a physical layer distributed execution plan that can be executed by the computing service cluster, so that the computing service cluster can perform distributed computing in the future.

[0115] Sub-step S1042b can be implemented as follows: determining the corresponding first operation to be executed and / or the target sub-table corresponding to the first operation to be executed according to the physical layer execution plan; correspondingly determining the second operation to be executed that can be executed by the computing service cluster corresponding to the first operation to be executed, and / or determining the target partition corresponding to the target sub-table in the partition of the data set of the computing service cluster; generating the multiple physical layer distributed execution plans according to the second operation to be executed and / or the target partition.

[0116] When determining the second operation to be executed corresponding to the first operation to be executed, the operation association relationship may be used to indicate the correspondence between operations in a database cluster (eg, a database cluster using MySQL) and operations in a computing service cluster.

[0117] For example, a first execution function corresponding to the first to-be-executed operation is determined, and the second to-be-executed operation is generated according to a second execution function corresponding to the first execution function in the computing service cluster.

[0118] The first execution function may be any function in MySQL, such as select (query operation), sum (sum operation), order by (sort operation), update (update operation), join (join operation), etc.

[0119] Syntax analysis or other methods can be used to determine each first execution function, and then determine the second execution function corresponding to the first execution function. The second execution function can then be generated based on the second execution function and the syntactic logic between the functions. The second execution function can be a native function in the computing service cluster, a custom function, or a custom aggregation function.

[0120] When determining the target partition corresponding to the target shard table, the target partition may be determined based on a table association relationship indicating a mapping relationship between each shard table in the database cluster and each partition in the data set of the computing service cluster.

[0121] The target sub-table corresponding to the first operation to be executed may be any sub-table in the database cluster, and the target sub-table may be determined by syntax analysis or other methods.

[0122] After determining the second operation to be executed and the target partition, multiple physical layer distributed execution plans can be generated based on the second operation to be executed, the target partition, and the grammatical rules used by the computing service cluster, so that the computing service cluster can subsequently execute the distributed execution plans to complete data analysis.

[0123] Through this embodiment, a distributed SQL engine is used to convert data analysis requests, so that the computing service cluster does not need to pay attention to the details of the database cluster's sharding and table sharding. It can directly perform the requested operation on the data source of the database cluster according to the second data analysis request, thereby making the computing service cluster compatible with database clusters with sharding and table sharding data sources without the need to establish its own storage layer to store data. In addition, the process of synchronizing the data source of the distributed database cluster can be omitted, thereby ensuring the real-time and high efficiency of big data analysis.

[0124] The data processing method of this embodiment can be executed by any appropriate electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as tablet computers, mobile phones, etc.) and PCs, etc.

[0125] Usage scenarios:

[0126] Reference Figure 4 , which shows a flow chart of data analysis using a database system.

[0127] S1: The user sends SQL to a distributed SQL engine (for example, the distributed SQL engine can be implemented as a middleware service, namely DRDS).

[0128] S2: The distributed SQL engine parses the SQL to generate an abstract syntax tree.

[0129] S3: Generate a logical execution plan based on the abstract syntax tree.

[0130] S4: Optimize the logical execution plan, perform table sharding routing calculations based on the relevant key values ​​in the SQL, and determine the target table shards involved in the calculations.

[0131] S5: Convert the optimized execution plan into a distributed execution plan for the corresponding computing service cluster (such as Spark).

[0132] S6: Submit the distributed execution plan to the computing service cluster.

[0133] S7: The computing service cluster executes the corresponding SQL to request the required data from the sub-databases of each database node, and generates a distributed elastic data set with all the required data pulled.

[0134] S8: The computing service cluster performs distributed computing on the distributed elastic dataset according to the execution plan.

[0135] S9: The computing service cluster collects and returns the calculation results to the distributed SQL engine.

[0136] S10: The distributed SQL engine packages the received calculation results into a result set and sends it back to the user side.

[0137] Example 4

[0138] Reference Figure 5 , shows a structural diagram of a data processing device according to a fourth embodiment of the present invention.

[0139] like Figure 5 As shown, the data processing device includes: a receiving module 502, used to receive a first data analysis request, wherein the first data analysis request is used to request data analysis on the data source in the database cluster; a conversion module 504, used to convert the first data analysis request into a second data analysis request that can be executed by the computing service cluster of the database system, so that the computing service cluster performs the operation requested by the second data analysis request on the data source of the database cluster.

[0140] Through this embodiment, the first data analysis request is converted to obtain the second data analysis request, so that the computing service cluster does not need to pay attention to the details of the database cluster's sharding and table sharding, and can directly perform the requested operation on the data source of the database cluster according to the second data analysis request, so that the computing service cluster can be compatible with database clusters with sharding and table sharding data sources without the need to establish its own storage layer to store data, and thus the process of synchronizing the data source of the distributed database cluster can be omitted, thereby ensuring the real-time and high efficiency of big data analysis.

[0141] Example 5

[0142] Reference Figure 6 , shows a structural diagram of a data processing device according to a fifth embodiment of the present invention.

[0143] like Figure 6 As shown, the data processing device includes: a receiving module 602, configured to receive a first data analysis request, wherein the first data analysis request is used to request data analysis on a data source in the database cluster;

[0144] The conversion module 604 is used to convert the first data analysis request into a second data analysis request that can be executed by the computing service cluster of the database system, so that the computing service cluster performs the operation requested by the second data analysis request on the data source of the database cluster.

[0145] Optionally, the conversion module 604 includes: a first generation module 6041, used to generate a logical execution plan according to the first data analysis request; a distribution plan generation module 6042, used to convert the logical execution plan into multiple distributed execution plans executable by the computing service cluster, so that the computing service cluster obtains the data of the second data analysis request from the data source of the database cluster by executing the multiple distributed execution plans, and performs the operation requested by the second data analysis request.

[0146] Optionally, the distribution plan generating module 6042 includes:

[0147] A second generating module 6042a is configured to generate a physical layer execution plan based on the physical layer of the database cluster according to the logical execution plan, wherein the physical layer execution plan includes a target sub-table to be executed on the physical layer;

[0148] The third generating module 6042b is configured to convert the physical layer execution plan into multiple physical layer distributed execution plans executable by the computing service cluster.

[0149] Optionally, the third generating module 6042b includes:

[0150] A first determining module is configured to determine, according to the physical layer execution plan, a corresponding first operation to be executed and / or a target sub-table corresponding to the first operation to be executed;

[0151] A second determining module is configured to determine a second operation to be executed that is executable by the computing service cluster corresponding to the first operation to be executed, and / or determine a target partition in a partition of a data set of the computing service cluster that corresponds to the target shard table;

[0152] A fourth generating module is configured to generate the plurality of physical layer distributed execution plans according to the second operation to be executed and / or the target partition.

[0153] Optionally, when determining the target partition corresponding to the target sub-table in the partition of the data set of the computing service cluster, the second determination module determines the target partition based on a table association relationship and the target sub-table, wherein the table association relationship indicates a mapping relationship between each sub-table in the database cluster and each partition in the data set of the computing service cluster.

[0154] Optionally, when determining the second to-be-executed operation that can be executed by the computing service cluster corresponding to the first to-be-executed operation, the second determination module determines a first execution function corresponding to the first execution operation, and generates the second execution operation according to a second execution function corresponding to the first execution function in the computing service cluster.

[0155] Optionally, the database cluster includes a primary database and a standby database with consistent data; the first generation module 6041 is used to generate the first data analysis request into a logical execution plan containing information for accessing the primary database or accessing the standby database based on the load information of the primary database and the load information of the standby database.

[0156] Optionally, the data processing device can be used as a distributed SQL engine of a database system.

[0157] The data processing device of this embodiment can implement the method and effects of the aforementioned method embodiment, so it will not be described in detail here.

[0158] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present invention can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present invention.

[0159] The method according to the embodiment of the present invention described above can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or as computer code that is originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded via a network and will be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor or programmable or dedicated hardware (such as an ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the data processing method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the data processing method shown here, the execution of the code converts the general-purpose computer into a dedicated computer for executing the data processing method shown here.

[0160] Those skilled in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present invention.

[0161] The above implementation methods are only used to illustrate the embodiments of the present invention, and are not intended to limit the embodiments of the present invention. Ordinary technicians in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the embodiments of the present invention. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of the present invention, and the scope of patent protection of the embodiments of the present invention should be defined by the claims.

Claims

1. A database system, characterized in that: It includes a database cluster and a computing service cluster, wherein the database cluster is a relational database and the computing service cluster is a device cluster deployed with a general parallel computing framework and includes multiple computing nodes; The database cluster includes a distributed SQL engine and a database node storing a data source, wherein the distributed SQL engine is used to receive a first data analysis request, wherein the first data analysis request includes a request to perform data analysis on the data source in the database cluster; The distributed SQL engine is used to: generating a logical execution plan according to the first data analysis request; Generate a physical layer execution plan based on the physical layer of the database cluster according to the logical execution plan, wherein the physical layer execution plan includes a target table to be executed at the physical layer; Converting the physical layer execution plan into multiple physical layer distributed execution plans executable by the computing service cluster; The computing service cluster is used to: Acquire data required by the multiple physical layer distributed execution plans from multiple sub-databases of the database cluster, and generate a distributed elastic data set; Distributed computing is performed on the distributed elastic data set according to the multiple physical layer distributed execution plans, computing results of each execution plan are collected, and the collected computing results are fed back to the distributed SQL engine.

2. The system according to claim 1, wherein: The distributed SQL engine is further used to: Determine, according to the physical layer execution plan, a corresponding first operation to be executed and / or a target sub-table corresponding to the first operation to be executed; Correspondingly determining a second operation to be executed that can be executed by the computing service cluster corresponding to the first operation to be executed, and / or determining a target partition in a partition of a data set of the computing service cluster that corresponds to the target shard table; The multiple physical layer distributed execution plans are generated according to the second operation to be executed and / or the target partition.

3. The system according to claim 2, characterized in that The distributed SQL engine is used to determine the target partition according to a table association relationship and the target shard table, wherein the table association relationship indicates a mapping relationship between each shard table in the database cluster and each partition in the data set of the computing service cluster.

4. The system according to claim 2, wherein: The distributed SQL engine is configured to determine a first execution function corresponding to the first execution operation, and generate a second execution operation according to a second execution function corresponding to the first execution function in the computing service cluster.

5. The system according to claim 1, wherein: The database cluster includes a primary database and a standby database with consistent data; The distributed SQL engine is further configured to generate the first data analysis request into a logical execution plan containing information on accessing the primary database or accessing the standby database based on the load information of the primary database and the load information of the standby database.

6. A data processing method, characterized in that: The method is used in the database system according to any one of claims 1 to 5, and the method comprises: Receive a first data analysis request, wherein the first data analysis request is used to request data analysis on a data source in the database cluster; generating a logical execution plan according to the first data analysis request; Generate a physical layer execution plan based on the physical layer of the database cluster according to the logical execution plan, wherein the physical layer execution plan includes a target table to be executed at the physical layer; The physical layer execution plan is converted into multiple physical layer distributed execution plans executable by the computing service cluster; the computing service cluster is used to obtain the data required for the multiple physical layer distributed execution plans from the multiple sub-databases of the database cluster and generate a distributed elastic data set; distributed computing is performed on the distributed elastic data set according to the multiple physical layer distributed execution plans, the computing results of each execution plan are collected, and the collected computing results are fed back to the distributed SQL engine of the database cluster.

7. The method according to claim 6, characterized in that The converting the physical layer execution plan into a plurality of physical layer distributed execution plans executable by the computing service cluster includes: Determine, according to the physical layer execution plan, a corresponding first operation to be executed and / or a target sub-table corresponding to the first operation to be executed; Correspondingly determining a second operation to be executed that can be executed by the computing service cluster corresponding to the first operation to be executed, and / or determining a target partition in a partition of a data set of the computing service cluster that corresponds to the target shard table; The multiple physical layer distributed execution plans are generated according to the second operation to be executed and / or the target partition.

8. The method according to claim 7, characterized in that The determining of a target partition corresponding to the target sub-table in the partition of the data set of the computing service cluster includes: The target partition is determined according to a table association relationship and the target sub-table, wherein the table association relationship indicates a mapping relationship between each sub-table in the database cluster and each partition in the data set of the computing service cluster.

9. The method according to claim 7, characterized in that The determining a second operation to be performed that is executable by the computing service cluster and corresponds to the first operation to be performed includes: A first execution function corresponding to the first execution operation is determined, and a second execution operation is generated according to a second execution function corresponding to the first execution function in the computing service cluster.

10. The method according to claim 6, characterized in that The database cluster includes a primary database and a standby database with consistent data; Generating a logical execution plan according to the first data analysis request includes: The first data analysis request is generated into a logic execution plan containing information for accessing the primary database or the standby database according to the load information of the primary database and the load information of the standby database.

11. A data processing device, characterized in that: include: A receiving module, configured to receive a first data analysis request, wherein the first data analysis request is used to request data analysis on a data source in a database cluster; a conversion module, configured to generate a logical execution plan according to the first data analysis request; Generate a physical layer execution plan based on the physical layer of the database cluster according to the logical execution plan, wherein the physical layer execution plan includes a target table to be executed at the physical layer; The physical layer execution plan is converted into multiple physical layer distributed execution plans that can be executed by the computing service cluster; the computing service cluster is used to obtain the data required for the multiple physical layer distributed execution plans from the multiple sub-databases of the database cluster and generate a distributed elastic data set; distributed computing is performed on the distributed elastic data set according to the multiple physical layer distributed execution plans, the computing results of each execution plan are collected, and the collected computing results are fed back to the distributed SQL engine of the database cluster.

12. A computer storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the data processing method according to any one of claims 6 to 10 is implemented.

Images