Data volume statistics method of distributed database, storage medium and equipment

Abstract

The application provides a data volume statistical method of a distributed database, a storage medium and equipment. The method comprises the following steps: a plurality of computing nodes obtain a shard query plan for calculating the data volume of a table to be calculated, which is issued by a master node; the plurality of computing nodes execute the respective shard query plans to complete a query process and an aggregation process; wherein the query process comprises: the plurality of computing nodes querying the table identifier and the data volume of each table to be calculated stored in the computing nodes, and sending the table identifier and the data volume of the table to be calculated to a corresponding summation node according to the table identifier of the table to be calculated; the aggregation process comprises: the summation node aggregating the data volume obtained from the plurality of computing nodes, and sending an aggregation result to the master node, wherein the aggregation result comprises the table identifier and the data volume sum of the table to be calculated corresponding to the summation node. The scheme of the application utilizes a plurality of summation nodes to calculate the data volume of a plurality of tables to be calculated in parallel, thereby improving the data volume calculation speed.

Description

TECHNICAL FIELD

[0001] The present application relates to database technology, in particular to a data volume statistics method of a distributed database, a storage medium and an apparatus. BACKGROUND

[0002] With the development of the information age, data is growing explosively, and applications of using a shared nothing distributed database to store data are increasing. However, if the number of cluster nodes is not enough for storing huge data in the early planning, online expansion is needed to increase nodes to store more data. Database expansion needs to ensure that online expansion does not interrupt business, and the expansion time should be as short as possible. In order to be easy to use, it is also necessary to be able to see the expansion status in real time, such as being able to view the current expansion progress and estimate the completion time of expansion. However, when the data volume of the cluster is particularly large and the number of tables is particularly large, it is very time-consuming to calculate the expansion progress, because the total data volume and the data volume of each table need to be calculated, so that in the expansion process, the current expansion progress can be calculated according to the total data volume of the expanded table divided by the total data volume that needs to be expanded.

[0003] At present, the conventional method of calculating the total data volume and the data volume of each table is to calculate each table one by one. However, the data of a distributed cluster is distributed to each node. That is, the data volume of a table is usually calculated by sending a calculation command from the master node to each calculation node, each calculation node calculates the data volume of the local table, and then each calculation node sends the calculation result to the master node, and the master node accumulates the total data volume of each table after receiving the calculation result, so that the data volume of each table can be calculated, and then the expansion process accumulates the data volume of all tables to obtain the data volume of the entire cluster.

[0004] Therefore, the distributed database needs to collect the table data volume of each node to obtain the global table data volume. For a database with a large number of tables, such as a database with more than 1000 tables or even more than 10000 tables, the above-mentioned table-by-table calculation method will take at least several minutes to complete the calculation, resulting in a prolonged statistical time and poor user experience. SUMMARY

[0005] An object of the present application is to provide a method capable of improving the data volume statistics speed of tables to be counted in a distributed database.

[0006] A further object of the present application is to reduce the resource consumption of data volume statistics.

[0007] In particular, the present application provides a data volume statistics method of a distributed database, the distributed database comprising a master node and a plurality of calculation nodes, and the data volume statistics method comprising:

[0008] The plurality of computing nodes obtain a sharding query plan for counting data volume of the to-be-counted tables issued by the master node; the sharding query plan specifies a summation node for each to-be-counted table, and the summation node is one of the plurality of computing nodes;

[0009] The plurality of computing nodes execute respective sharding query plans to complete a query process and an aggregation process; wherein

[0010] The query process includes that the plurality of computing nodes query table identifiers and data volume of each to-be-counted table stored in the plurality of computing nodes, and send the queried table identifiers and data volume of the to-be-counted tables to corresponding summation nodes according to the table identifiers of the to-be-counted tables.

[0011] The aggregation process includes that the summation nodes aggregate data volume obtained from the plurality of computing nodes, and send aggregation results to the master node, the aggregation results including table identifiers and total data volume of to-be-counted tables corresponding to the summation nodes.

[0012] Optionally, the step of the summation nodes aggregating data volume obtained from the plurality of computing nodes includes:

[0013] The summation nodes aggregate data volume obtained from the plurality of computing nodes according to the table identifiers, and calculate total data volume of to-be-counted tables corresponding to each table identifier respectively.

[0014] Optionally, the step of the plurality of computing nodes querying table identifiers and data volume of each to-be-counted table stored in the plurality of computing nodes includes:

[0015] The plurality of computing nodes sequentially scan respective metadata tables to obtain pre-stored table identifiers and data volume of to-be-counted tables in the metadata tables.

[0016] Optionally, the step of sending the queried table identifiers and data volume of the to-be-counted tables to corresponding summation nodes according to the table identifiers of the to-be-counted tables includes:

[0017] According to the table identifiers of the to-be-counted tables, a preset algorithm is used to determine the summation nodes corresponding to the to-be-counted tables, and the queried table identifiers and data volume of the to-be-counted tables are sent to the corresponding summation nodes.

[0018] Optionally, before the step of the plurality of computing nodes obtaining a sharding query plan for counting data volume of the to-be-counted tables issued by the master node, the method further includes:

[0019] After the master node obtains the data volume query instruction, the master node generates a distributed query plan according to the data volume query instruction, splits the distributed query plan into a plurality of sharding query plans, and issues the plurality of sharding query plans to the plurality of computing nodes.

[0020] Optionally, the data volume query instruction is an SQL statement; the step of generating a distributed query plan according to the data volume query instruction includes:

[0021] The data volume query instruction is subjected to SQL analysis and query optimization to generate a distributed query plan.

[0022] Optionally, the tables to be counted include all data tables stored in the distributed database.

[0023] Optionally, after the step of sending the summary result to the master node, the method further comprises:

[0024] The master node receives the summary results sent by the summation nodes, and sends the summary results to an expansion tool;

[0025] The expansion tool receives the summary results integrated by the master node, and calculates the expansion progress of the distributed database according to the total data volume of the distributed database.

[0026] According to another aspect of the present application, there is also provided a machine readable storage medium having stored thereon a machine executable program which, when executed by a processor, implements any of the above-mentioned data volume counting methods for a distributed database.

[0027] According to still another aspect of the present application, there is also provided a computer device comprising a memory, a processor, and a machine executable program stored in the memory and running on the processor, and the processor implements any of the above-mentioned data volume counting methods for a distributed database when executing the machine executable program.

[0028] The data volume counting method for a distributed database of the present application assigns a summation node to each table to be counted, and causes multiple computing nodes storing partial data of the same table to send data volume information to the corresponding summation node, and then causes the summation node to summarize and sum the received data volume information to obtain the total data volume of the corresponding table to be counted. The present application improves the summation calculation of data volume in the master node into parallel summation calculation of data volume in each lower computing node, fully utilizes the parallel computing capability of each node, and improves the data volume counting speed.

[0029] Further, the data volume counting method for a distributed database of the present application improves the table-by-table calculation size into one-time batch calculation of all table sizes, greatly reducing the consumption caused by multiple distributed queries.

[0030] The above and other objects, advantages and features of the present application will become more apparent from the following detailed description of specific embodiments thereof, when taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0031] Some specific embodiments of the present application will be described in detail with reference to the accompanying drawings, which are by way of illustration, and thus are not limitative of the present application. Wherever possible, the same reference numbers are used in different drawings. It should be understood that the drawings are not necessarily to scale. In the drawings:

[0032] Figure 1 is a schematic diagram of a system architecture of a data volume statistics method of a distributed database according to an embodiment of the present application;

[0033] Figure 2 is a schematic diagram of a flow of a data volume statistics method of a distributed database according to Figure 1

[0034] Figure 3 is a schematic diagram of a flow of a query process in a data volume statistics method of a distributed database according to Figure 1

[0035] Figure 4 is a schematic diagram of a flow of a summary process in a data volume statistics method of a distributed database according to Figure 1

[0036] Figure 5 is a schematic diagram of a flow of a sharding query plan in a data volume statistics method of a distributed database according to an embodiment of the present application;

[0037] Figure 6 is a schematic diagram of a flow of a data volume statistics method of a distributed database according to another embodiment of the present application;

[0038] Figure 7 is a schematic diagram of a flow of a data volume statistics method of a distributed database according to yet another embodiment of the present application;

[0039] Figure 8 is a schematic diagram of a flow of a data volume statistics method of a distributed database according to Figure 7

[0040] Figure 9 is a schematic diagram of a machine-readable storage medium according to an embodiment of the present application;

[0041] Figure 10 is a schematic diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in a clear and complete manner with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. ​​​​

[0043] Those skilled in the art should understand that these embodiments are merely for explaining the technical principles of this application and are not intended to limit the scope of protection of this application. Those skilled in the art can make adjustments as needed to adapt to specific application scenarios.

[0044] Figure 1 This is a schematic diagram of the system architecture for a data volume statistics method for a distributed database according to an embodiment of the present invention. The distributed database generally includes a master node 10 and multiple computing nodes 20 corresponding to the master node 10. In a distributed database, the data of a data table is often distributed across multiple computing nodes 20. That is, to count the data volume of a specific data table (i.e., the table to be counted), it is necessary to obtain the data volume of each part of the table to be counted from the multiple computing nodes 20 storing the table to be counted. When expanding the above-mentioned distributed database, the expansion tool 30 adopts smooth expansion, which requires first obtaining the data volume of each table. To improve the speed of data volume statistics, this embodiment provides the following data volume statistics method.

[0045] The aforementioned distributed database can have one or more computing instances. The following mainly describes the case with one computing instance. Those skilled in the art will understand that when there are multiple computing instances, the data volume statistics method of this embodiment can be applied to each computing instance separately.

[0046] Figure 2 This is a flowchart illustrating a data volume statistics method for a distributed database according to an embodiment of the present invention.

[0047] refer to Figure 2 As shown, methods for statistical analysis of data volume in distributed databases generally include:

[0048] Step S102: Multiple computing nodes 20 obtain the sharding query plan issued by the master node 10 for calculating the amount of data in the table to be counted;

[0049] In step S104, multiple computing nodes 20 execute their respective sharded query plans to complete the query and aggregation processes.

[0050] Figure 3 It is based on Figure 1 The diagram shows the query process for a distributed database data volume statistics method. Figure 4 It is based on Figure 1 The diagram shows the summary process of the data volume statistics method for distributed databases.

[0051] refer to Figure 3 As shown, in step S104, the query process includes:

[0052] Step S202: Multiple computing nodes 20 query the table identifier and data volume of each table to be statistically analyzed that they store;

[0053] Step S204: Based on the table identifier of the table to be counted, send the table identifier and data volume of the retrieved table to the corresponding summation node.

[0054] refer to Figure 4 As shown, in step S104, the summary process includes:

[0055] Step S302: The summation node summarizes the amount of data obtained from multiple computing nodes 20;

[0056] Step S304: Send the summary results to master node 10.

[0057] In step S102, the sharded query plan assigns a summation node to each table to be counted. That is, the data size of each table to be counted is calculated by the corresponding summation node. However, in particular, a summation node may summarize one table to be counted, or it may summarize multiple tables to be counted simultaneously.

[0058] The summation node is one of multiple compute nodes 20. It can be understood that all or only some of the compute nodes 20 corresponding to the master node 10 may be summation nodes; that is, some or all of the compute nodes 20 may be used for table count aggregation. Preferably, when the number of tables to be aggregated exceeds the number of compute nodes 20, each compute node 20 is simultaneously a summation node. In other words, when the number of tables to be aggregated exceeds the number of compute nodes 20, the sharded query plan assigns a table to be aggregated to each compute node 20, thereby fully utilizing the computing power of the multiple compute nodes 20.

[0059] In step S104, multiple computing nodes 20 execute their respective sharded query plans to complete the query and aggregation processes. In this step, the summing nodes among the computing nodes 20 obtain the table identifiers and data volumes of the tables to be aggregated sent by the multiple computing nodes 20. However, in some cases, some computing nodes 20 are not summing nodes or do not correspond to tables to be aggregated that require aggregation. These computing nodes 20 will directly end step S302 (aggregate the data volumes sent by 0 computing nodes 20) and will not send the aggregation results to the master node 10 to complete the aggregation process, or they will send specified preset content as the aggregation result to complete the aggregation process.

[0060] In step S202, the computing node 20 can obtain the table identifier and the data volume of the table to be counted in the metadata table by sequentially scanning the metadata table. The distributed database stores the data tables in multiple computing nodes 20, and the same data table in the multiple computing nodes 20 has a unique table identifier, so that the multiple computing nodes 20 can send the data volume of the table to be counted to the same specified summation node according to the table identifier. Since the computing node 20 in the distributed database usually counts and stores the data volume of the data table stored in itself in the metadata table, the computing node 20 can quickly obtain the table identifier and the data volume of the table to be counted stored in itself by sequentially scanning the metadata table.

[0061] In step S204, the summation node corresponding to the table to be counted can be determined according to the table identifier of the table to be counted by using a preset algorithm, and the table identifier and the data volume of the table to be counted obtained by the query are sent to the corresponding summation node. The preset algorithm can be a hash algorithm, and the summation node corresponding to the table identifier can be determined by calculating the table identifier of the table to be counted according to a preset hash function. However, in some other optional embodiments, other algorithms can also be used to determine the summation node corresponding to the table to be counted.

[0062] In step S302, the summation node can respectively aggregate the data volumes obtained from the multiple computing nodes 20 according to the table identifier, and calculate the total data volume of the table to be counted corresponding to each table identifier. In some cases, one summation node corresponds to multiple tables to be counted, that is, the summation node needs to aggregate the data volumes of multiple tables to be counted. According to the table identifier, the data volumes of the multiple tables to be counted obtained can be respectively aggregated, so that the summation node can accurately complete the aggregation even if it corresponds to multiple tables to be counted.

[0063] It should be noted that one computing node 20 can include one or more computing instances. The above embodiment corresponds to the case that one computing node 20 includes one computing instance, and in other embodiments, one computing node 20 includes two or more computing instances. In this case, each computing instance can independently execute the above steps S102 and S104, and complete the query process and the aggregation process. Those skilled in the art can directly deduce the case that one computing node 20 includes two or more computing instances based on the case that one computing node 20 includes one computing instance, and thus the following will not be described in detail.

[0064] Figure 5 FIG. 1 is a flow diagram of a sharding query plan in a data volume counting method of a distributed database according to an embodiment of the present application.

[0065] Figure 5With an embodiment including two computing nodes 20, the data transmission direction between the multiple computing nodes 20 and between the computing nodes 20 and the master node 10 during the execution of the sharded query plan is shown by arrows. Referring to Figure 5 As shown, in one specific embodiment of the present application, the distributed database includes one master node 10 and two computing nodes 20: a first computing node 201 and a second computing node 202. And the present data volume statistics needs to count four tables to be counted: TA, TB, TC, and TD, wherein the table identifier of the TA table is 1, the table identifier of the TB table is 2, the table identifier of the TC table is 3, and the table identifier of the TD table is 4.

[0066] During the execution of the sharded query plan by the computing nodes 20, the query process is executed by the first worker process 21, and the summary process is executed by the second worker process 22. The first computing node 201 and the second computing node 202 first execute their own query processes in the respective first worker processes 21, and query the table identifier and data volume of each table to be counted stored in themselves. The first computing node 201 queries the following data: the table identifier of the TA table is 1, the data volume is 100 MB, which can be recorded as (1, 100 MB); the table identifier of the TB table is 2, the data volume is 300 MB, which can be recorded as (2, 300 MB); the table identifier of the TC table is 3, the data volume is 500 MB, which can be recorded as (3, 500 MB); and the table identifier of the TD table is 4, the data volume is 700 MB, which can be recorded as (4, 700 MB). The second computing node 202 queries the following data: the table identifier of the TA table is 1, the data volume is 200 MB, which can be recorded as (1, 200 MB); the table identifier of the TB table is 2, the data volume is 400 MB, which can be recorded as (2, 400 MB); the table identifier of the TC table is 3, the data volume is 600 MB, which can be recorded as (3, 600 MB); and the table identifier of the TD table is 4, the data volume is 800 MB, which can be recorded as (4, 800 MB).

[0067] According to the above query results, the first worker processes 21 of the first computing node 201 and the second computing node 202 respectively send the data queried by themselves to the corresponding summation nodes. Among them, the sharded query plan specifies that the tables to be counted TA and TC correspond to the first summation node (i.e. the first computing node 201), and the tables to be counted TB and TD correspond to the second summation node (i.e. the second computing node 202). Thus, the first worker process 21 of the first computing node 201 sends data (1, 100 MB) and (3, 500 MB) to the first summation node, and sends data (2, 300 MB) and (4, 700 MB) to the second summation node; and the first worker process 21 of the second computing node 202 sends data (1, 200 MB) and (3, 600 MB) to the first summation node, and sends data (2, 400 MB) and (4, 800 MB) to the second summation node.

[0068] The second worker process 22 of the first summation node executes the summarization process, obtains the data from the first computing node 201 and the second computing node 202, and respectively sums up the data amounts according to the table identifiers, so as to obtain the total data amounts of the tables TA and TC in a summation manner: TA table 100+200=300MB, which can be recorded as (1, 300MB), and TC table 500+600=1100MB, which can be recorded as (3, 1100MB); the second worker process 22 of the second summation node executes the summarization process, obtains the data from the first computing node 201 and the second computing node 202, and respectively sums up the data amounts according to the table identifiers, so as to obtain the total data amounts of the tables TB and TD in a summation manner: TB table 300+400=700MB, which can be recorded as (2, 700MB), and TD table 700+800=1500MB, which can be recorded as (4, 1500MB).

[0069] According to the total data amounts of the tables to be counted and the table identifiers corresponding to the tables to be counted, the second worker process 22 of the first summation node sends the summarized data (1, 300MB) and (3, 1100MB) to the master node 10, and the second worker process 22 of the second summation node sends the summarized data (2, 700MB) and (4, 1500MB) to the master node 10, so that the master node 10 obtains the total data amounts of the tables TA, TB, TC and TD respectively.

[0070] It can be understood that the data (1, 100MB) is intended to indicate that the transmitted data includes the table identifier and the data amount.

[0071] From Figure 5 It can be seen that the second worker processes 22 of the plurality of summation nodes can respectively summarize the tables to be counted corresponding thereto when executing the summarization process, and the data amounts of the plurality of tables to be counted are calculated in parallel through the plurality of summation nodes, so that the computing capacity of the plurality of computing nodes 20 is fully utilized, and the speed of counting the data amount is greatly improved. In addition, the master node 10 only needs to send the sharding query plan to each computing node 20 once, and compared with the method that the master node 10 sends a data amount query request to each table to be counted in each computing node 20, the consumption caused by multiple queries is greatly reduced.

[0072] In particular, although the second worker process 22 needs to aggregate the data volume obtained from the plurality of computing nodes 20 first when performing the aggregation procedure, the second worker process 22 is not limited to start the aggregation procedure only after obtaining the entire data volume of the same table to be counted sent by the plurality of computing nodes 20. For example, when a computing node 20 sends data later than other computing nodes 20, the second worker process 22 can start the aggregation procedure and aggregate the data already obtained before obtaining the data sent by the computing node 20. In some preferred embodiments, the second worker process 22 starts the aggregation procedure as soon as obtaining the data sent by any computing node 20, thereby shortening the overall time consumed by the computing nodes 20 to execute the sharded query plan.

[0073] In some optional embodiments, a computing node 20 can include a plurality of computing instances, and each computing instance can include a set of first worker processes 21 and second worker processes 22 respectively. In this embodiment, the first worker process 21 of each computing instance of each computing node 20 can independently execute the query procedure and send the table identifier and data volume of the table to be counted stored in the computing instance to the second worker process 22 of the corresponding computing instance of the corresponding computing node 20.

[0074] Figure 6 is a flowchart of a data volume counting method of a distributed database according to another embodiment of the present application.

[0075] Referring to Figure 6 In some further embodiments, the data volume counting method of the distributed database includes:

[0076] Step S402, the master node 10 obtains a data volume query instruction;

[0077] Step S404, generating a distributed query plan according to the data volume query instruction;

[0078] Step S406, splitting the distributed query plan into a plurality of sharded query plans;

[0079] Step S408, distributing the sharded query plans to the plurality of computing nodes 20;

[0080] Step S410, the plurality of computing nodes 20 obtain the sharded query plans for counting the data volume of the table to be counted distributed by the master node 10;

[0081] Step S412, the plurality of computing nodes 20 respectively execute the respective sharded query plans to complete the query procedure and the aggregation procedure.

[0082] In some optional implementations, the data volume query instruction in step S402 can be an SQL statement. Based on this, in step S404, the data volume query instruction can be parsed and optimized to generate a distributed query plan.

[0083] Figure 7 This is a flowchart illustrating a data volume statistics method for a distributed database according to another embodiment of the present invention.

[0084] In some alternative implementations, the summary results received by master node 10 can be further applied to the expansion of the distributed database. The table to be summarized may include all data tables stored in the distributed database, and refer to... Figure 7 As shown, the procedure following step S412 further includes:

[0085] In step S502, the master node 10 receives the summary results sent by each summing node and integrates the summary results to send to the expansion tool 30;

[0086] Step S504: The expansion tool 30 receives the summary results from the master node 10 and calculates the total amount of data in the distributed database.

[0087] Step S506: The expansion tool 30 calculates the expansion progress. The expansion tool 30 can quickly collect the data volume of each table and calculate the total data volume of all tables, thus obtaining the expansion progress. This expansion progress can be further used for visualization to understand the expansion process of the distributed database. The expansion progress mentioned above represents the ratio of the data volume of the tables that have been redistributed to the total data volume.

[0088] Figure 8 It is based on Figure 7 The diagram shows a flowchart of a distributed database data volume statistics method. Figure 8 The arrows indicate the content and direction of data transmission between the master node 10, compute node 20, and expansion tool 30. (Reference) Figure 8 As shown, the data volume statistics method in this embodiment can be mainly applied to the expansion process of distributed databases, enabling efficient and rapid understanding of the expansion progress. The master node 10 receives a data volume query command sent by the expansion tool 30 and distributes a sharding query plan to multiple computing nodes 20. After receiving the summary results sent by multiple computing nodes 20, the master node 10 sends the integrated summary results back to the expansion tool 30. Based on the integrated summary results, the expansion tool 30 obtains the total data volume of each data table stored in multiple computing nodes 20 and applies it to the expansion process. The expansion process can then execute further steps and calculate the total data volume including all data tables.

[0089] In the expansion process of the distributed database, the online expansion needs to be ensured without interrupting the business, the expansion time needs to be as short as possible, and in order to be easy to use, the expansion state needs to be seen in real time, such as checking the current expansion progress and estimating the completion time of the expansion. In order to calculate the expansion progress, the total data amount of the data table and the data amount of each table need to be calculated, so that in the expansion process, the current expansion progress is calculated according to the total data amount of the expanded table divided by the total data amount of the tables to be expanded.

[0090] According to the above embodiment of the present application, the data amount of each data table and the total data amount can be quickly calculated, and the calculation time is shortened from several minutes to several seconds, so that the expansion progress can be displayed in real time.

[0091] According to the data amount calculation method of the distributed database of the present application, the plurality of computing nodes 20 are allocated the tables to be calculated and calculate the data amount in parallel, so that the computing capacity of each computing node 20 is fully utilized and the data amount calculation speed is improved. In addition, according to the data amount calculation method of the present application, the master node 10 only needs to send the shard query plan to each computing node 20 once, which greatly reduces the consumption caused by multiple queries compared with the method of sending multiple query requests by the master node 10 for each table to be calculated.

[0092] Further, the data amount calculation method of the distributed database of the present application enables the computing node 20 to respectively aggregate the data amount corresponding to different tables to be calculated according to the table identifier, so that each computing node 20 can calculate the total data amount of a plurality of tables to be calculated. Even if the number of tables to be calculated is greater than the number of computing nodes 20, or the tables to be calculated include all the data tables stored in the distributed database, the computing nodes 20 can be reasonably allocated and the data amount calculation speed of all the data tables in the distributed database can be improved. Moreover, the data amount is respectively aggregated according to the table identifier, which can avoid mixing the data amount of multiple tables to be calculated obtained by one computing node 20, so that the computing node 20 can accurately complete the aggregation even if it corresponds to multiple tables to be calculated.

[0093] The data amount calculation method of the present application will be further described below through a specific embodiment of the present application.

[0094] In the first step, the expansion tool 30 sends a data amount query instruction for calculating the size of all tables once to the master node 10 of the distributed database server in order to obtain the data amount of each table.

[0095] Second step, the distributed database cluster is completed by the master node 10 and multiple computing nodes 20 distributed query, the master node 10 receives the data volume query instruction sent by the expansion tool 30, carries out SQL analysis and query optimization to generate a distributed query plan, then sharding the distributed query plan (based on data transmission node (corresponding to the computing node 20) the distributed query plan is split into multiple plan shards, that is, shard query plan, so that the query plan can be executed in parallel in each computing node 20), and the shard query plan is issued to each computing node 20.

[0096] Third step, the computing node 20 receives the shard query plan, executes and gets the summary result, and then returns the result to the master node 10. The master node receives the summary results of all computing nodes 20 and integrates them, and then sends the data to the expansion tool 30 for expansion process. The steps of the computing node 20 executing the shard query plan are as follows (taking two computing nodes 20 as an example, which can be referred to Figure 5 ):

[0097] (1) After each computing node 20 receives the distributed execution plan shard, the first working process 21 of each computing node 20 scans the metadata table (the local table data size is usually stored in the metadata table) in sequence, gets the OID (unique ID of each data table, that is, table identifier) and corresponding table size data of all tables, and then sends the data to the second working process 22 of each computing node 20 according to the OID.

[0098] For example, the distributed database has four tables, TA, TB, TC and TD.

[0099] The data scanned by the first working process 21 of the first computing node 201 in sequence is assumed to be: TA table OID is 1, data size is 100MB; TB table OID is 2, data size is 300MB; TC table OID is 3, data size is 500MB; TD table OID is 4, data size is 700MB;

[0100] The data scanned by the first working process 21 of the second computing node 202 in sequence is assumed to be: TA table OID is 1, data size is 200MB; TB table OID is 2, data size is 400MB; TC table OID is 3, data size is 600MB; TD table OID is 4, data size is 800MB;

[0101] The data is re-hashed according to the OID and sent to the second worker process 22 of each computing node 20, that is, the first worker process 21 of each computing node 20 sends the table data of tables TA and TC to the second worker process 22 of the first computing node 201 respectively, and sends the table data of tables TB and TD to the second worker process 22 of the second computing node 202 respectively.

[0102] (2) The second worker process 22 of each computing node 20 receives the data re-distributed by the first worker process 21 of each computing node 20 through the hash method, groups and aggregates according to the OID, sums up the table size, and then sends the summary result to the master node 10.

[0103] For example, the second worker process 22 of the first computing node 201 receives the TA table OID 1 with a data size of 100MB and the TA table OID 1 with a data size of 200MB, and then sums up 300MB, which is the size of the table to be counted in the entire distributed database. Similarly, the first computing node 20 also receives two pieces of data of the TC table, which are summed up to 1100MB (= 500+600MB). At the same time, the second worker process 22 of the second computing node 202 receives two pieces of data of the TB table and the TD table respectively, which are summed up to 700MB (= 300+400MB) and 1500MB (= 700+800MB) respectively. In this way, the size of each table in the distributed database is calculated.

[0104] (3) The master node 10 receives the summary result sent by the second worker process 22 of each computing node 20 and integrates the data.

[0105] The master node 10 receives (TA, 300MB) and (TC, 1100MB) sent by the first computing node 201 and (TB, 700MB) and (TD, 1500MB) sent by the second computing node 202, and integrates them.

[0106] In the fourth step, the expansion process of the expansion tool 30 receives the data, obtains the size of each table, calculates the total size of the data, and obtains the basic data for displaying the expansion progress during the expansion. Therefore, when the expansion is performed, the progress can be displayed according to the processing situation.

[0107] The application also provides a machine-readable storage medium and a computer device. Figure 9 Fig. 4 is a schematic diagram of a machine-readable storage medium 40 according to an embodiment of the application, Figure 10 Fig. 5 is a schematic diagram of a computer device 50 according to an embodiment of the application.

[0108] The machine readable storage medium 40 has stored thereon machine executable programs 41 which, when executed by the processor, implement the method of data volume statistics of the distributed database of any of the above embodiments.

[0109] The computer device 50 can include a memory 520, a processor 510, and a machine executable program 41 stored on the memory 520 and running on the processor 510, and the processor 510 implements the method of data volume statistics of the distributed database of any of the above embodiments when executing the machine executable program 41.

[0110] It should be noted that the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a list of executable instructions for implementing logic functions, and can be specifically embodied in any machine readable storage medium for use by or in connection with an instruction execution system, apparatus or device, such as a computer-based system, a system including a processor or other system that can fetch the instructions from the instruction execution system, apparatus or device and execute the instructions, or in conjunction with such instruction execution system, apparatus or device.

[0111] For the description of the present embodiment, the machine readable storage medium 40 can be any device that can contain, store, communicate, propagate or transport programs for use by or in connection with an instruction execution system, apparatus or device, or in conjunction with such instruction execution system, apparatus or device. More specific examples (non-exhaustive list) of computer readable medium include the following: electrical connections having one or more wires (electronic devices), portable computer diskette (magnetic devices), random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), fiber optic devices, and portable compact disk read only memory (CDROM). In addition, the computer readable medium 40 can even be paper or other suitable medium on which the program is printed, as the program can be electronically obtained, for example, by optical scanning of the paper or other medium, followed by electronic conversion of the optical scanning into an electronic format, and then editing, interpreting or otherwise processing the program into a usable format for the computer.

[0112] It should be understood that parts of the present application can be implemented in hardware, software, firmware or a combination thereof. In the above described embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system.

[0113] The computer device 50 can be, for example, a server, a desktop computer, a notebook computer, a tablet computer, or a smartphone. In some examples, the computer device 50 can be a cloud computing node 20. The computer device 50 can be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules can include routines, programs, objects, components, logic, data structures, and so on that perform particular tasks or implement particular abstract data types. The computer device 50 can be practiced in a distributed cloud computing environment with remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules can be located in both local and remote computer system storage media including memory storage devices.

[0114] The computer device 50 can include a processor 510 adapted to execute instructions stored in memory 520, which in operation provides temporary storage for the instructions during execution by the processor 510. The processor 510 can be a single core processor, multi-core processor, computing cluster, or any number of other configurations. The memory 520 can include random access memory (RAM), read only memory (ROM), flash memory, or any other suitable memory systems.

[0115] The processor 510 can be connected through a system interconnect (e.g., a PCI, PCI-Express, or other interconnect) to an I / O interface adapted to connect the computer device 50 to one or more I / O devices (input / output devices). The I / O devices can include, for example, a keyboard and a pointing device, where the pointing device can include a touchpad or touchscreen, among others. The I / O devices can be built-in components of the computer device 50, or can be devices externally connected to the computer device.

[0116] The processor 510 can also be linked through the system interconnect to a display interface adapted to connect the computer device 50 to a display device. The display device can include a display screen that is a built-in component of the computer device 50. The display device can also include a computer monitor, television, or projector, among others, that is externally connected to the computer device 50. Further, a network interface controller (NIC) can be adapted to connect the computer device 50 to a network through the system interconnect. In some embodiments, the NIC can use any suitable interface or protocol, such as an Internet Small Computer Systems Interface, among others, to transfer data. The network can be a cellular network, a radio network, a wide area network (WAN), a local area network (LAN), or the Internet, among others. Remote devices can be connected to the computer device through the network.

[0117] The flowcharts provided by the embodiments are not intended to indicate that the operations of the methods will be executed in any particular order, or that all of the operations of the methods are included in every case. In addition, the methods can include additional operations. Additional changes can be made to the above-described methods within the scope of the technical ideas provided by the embodiments.

[0118] Thus, those skilled in the art will recognize that the present application has been well- described and clearly demonstrated by way of example, and that numerous modifications, substitutions and changes can be made thereto without departing from the spirit and scope of the present application as set forth above. It is intended that the scope of the present application be limited solely by the scope of the appended claims.

Claims

1. A data volume statistics method of a distributed database, the distributed database comprising a master node and a plurality of computing nodes, and being configured to store one or more tables to be counted. And the data volume statistics method comprises: The plurality of computing nodes obtain a shard query plan for statistics of data volume of a to-be-counted table issued by the master node; the shard query plan specifies a summation node for each to-be-counted table, and the summation node is one of the plurality of computing nodes; the to-be-counted table comprises all data tables stored in the distributed database; The plurality of computing nodes execute respective shard query plans to complete a query process and an aggregation process; wherein The query process comprises: the plurality of computing nodes querying a table identifier and a data volume of each to-be-counted table stored in the plurality of computing nodes, and sending the table identifier and the data volume of the to-be-counted table to the corresponding summation node according to the table identifier of the to-be-counted table; The aggregation process comprises: the summation node aggregating the data volume obtained from the plurality of computing nodes, and sending an aggregation result to the master node, wherein the aggregation result comprises a table identifier and a total data volume of the to-be-counted table corresponding to the summation node; The step of sending the table identifier and the data volume of the to-be-counted table to the corresponding summation node according to the table identifier of the to-be-counted table comprises: According to the table identifier of the to-be-counted table, a preset algorithm is used to determine the summation node corresponding to the to-be-counted table, and the table identifier and the data volume of the to-be-counted table are sent to the corresponding summation node.

2. The data volume statistics method of the distributed database according to claim 1, wherein The step of the summation node aggregating the data volume obtained from the plurality of computing nodes comprises: The summation node aggregates the data volume obtained from the plurality of computing nodes according to the table identifier, and calculates the total data volume of the to-be-counted table corresponding to each table identifier.

3. The data volume statistics method of the distributed database according to claim 1, wherein The step of the plurality of computing nodes querying a table identifier and a data volume of each to-be-counted table stored in the plurality of computing nodes comprises: The plurality of computing nodes sequentially scan respective metadata tables to obtain a pre-stored table identifier and a data volume of the to-be-counted table in the metadata table.

4. The data volume statistics method of the distributed database according to claim 1, wherein Before the step of the plurality of computing nodes obtaining a shard query plan for statistics of data volume of a to-be-counted table issued by the master node, the method further comprises: After the master node obtains a data volume query instruction, the master node generates a distributed query plan according to the data volume query instruction, splits the distributed query plan into a plurality of shard query plans, and issues the plurality of shard query plans to the plurality of computing nodes.

5. The data volume statistics method of the distributed database according to claim 4, wherein The data volume query instruction is an SQL statement; The step of generating a distributed query plan according to the data volume query instruction comprises: SQL parsing and query optimization are performed on the data volume query instruction to generate the distributed query plan.

6. The data volume statistics method of the distributed database according to claim 4, wherein After the step of sending an aggregation result to the master node, the method further comprises: The master node receives the summary results sent by each of the sum nodes, and sends the summary results to a capacity expansion tool; The capacity expansion tool receives the summary results integrated by the master node, and calculates the capacity expansion progress of the distributed database.

7. A machine readable storage medium having stored thereon a machine executable program which, when executed by a processor, implements the data volume statistics method of the distributed database according to any one of claims 1 to 6.

8. A computer device comprising a memory, a processor, and a machine executable program stored on the memory and running on the processor, and the processor implements the data volume statistics method of the distributed database according to any one of claims 1 to 6 when executing the machine executable program.

Images