A data statistics method, apparatus, device and storage medium

Abstract

This application discloses a data statistics method, apparatus, device, and storage medium. It performs data statistics on Hive based on the Spark platform without requiring the development of a new program. Spark generates a query execution plan based on the data volume query command for Hive, and executes the query execution plan to read table data information from Hive for statistical analysis. During the statistical process, an Rdd corresponding to each table to be analyzed is created, and the data volume of one or more tasks corresponding to the partition of each Rdd is read to determine the total data volume for each Rdd. Since the names of the Rdds created in this application correspond to the names of the tables to be analyzed that store the data, the total table data volume of each table to be analyzed can be determined. Based on this, highly efficient and near real-time response to data volume query requests can be achieved, and the accuracy of table data statistics can be improved through Spark's automatic reading of task data.

Description

Technical Field

[0001] This application relates to the field of data statistics technology, and more specifically, to a data statistics method, apparatus, device, and storage medium. Background Technology

[0002] In Hive (or a data warehouse tool), tables are the logical units of data organization. Data is stored in tables, and each table contains a certain amount of data, which can be structured, semi-structured, or unstructured. When using Hive to store and manage data, it is necessary to determine all stored tables and the amount of data in each table. By querying the amount of data in each table, a comprehensive understanding of the dataset stored in Hive can be obtained, leading to better data management and analysis.

[0003] However, currently, if we want to count the amount of data in a table, we need to develop a program specifically for that task and execute it to calculate the amount of data in each table. This does not allow for timely statistics on the tables, and thus, it is impossible to promptly identify data storage or management problems. Summary of the Invention

[0004] In view of this, this application provides a data statistics method, apparatus, device and storage medium to solve the problem that existing data statistics methods cannot perform near real-time and high-efficiency statistics of table data volume.

[0005] To achieve the above objectives, the following solution is proposed:

[0006] A data statistics method applied to the Spark computing engine platform includes:

[0007] Obtain the data volume query command for the Hive data storage tool and generate a query execution plan corresponding to the data volume query command;

[0008] Execute the query execution plan to read the table data information in the Hive data storage tool corresponding to the query execution plan. The table data information includes: at least one table to be counted, and the table dataset and table name corresponding to the table to be counted.

[0009] Create at least one distributed dataset Rdd corresponding to each of the tables to be counted, and determine the name and partition of each Rdd. The name of the Rdd corresponds to the table name of the table to be counted corresponding to the Rdd, and each partition of the Rdd includes at least one subtask.

[0010] Read the amount of data for each task;

[0011] The partition corresponding to each task is matched with the partition of each Rdd to obtain the total amount of data for all tasks corresponding to each Rdd;

[0012] Based on the correspondence between the name of each Rdd and the table name of the table to be counted, and the total amount of data of all tasks corresponding to each Rdd, determine the total amount of table data of each table to be counted;

[0013] The total amount of data in each of the tables to be statistically analyzed is determined as the data volume statistical result.

[0014] Optionally, the step of matching the partition corresponding to each task with the partition of each Rdd to determine the total amount of data for all tasks corresponding to each Rdd includes:

[0015] Based on the partition and data volume corresponding to each task, a first relational structure representing the correspondence between the data volume and the partition of the task is generated;

[0016] Based on the name and partition of each Rdd, a second relation structure is generated that represents the correspondence between the name and partition of the Rdd;

[0017] Traverse the first relation structure and the second relation structure to determine the amount of data belonging to the same partition and the name of the Rdd;

[0018] Based on the data volume with the same name as the corresponding Rdd, determine the total data volume of all tasks corresponding to each Rdd.

[0019] Optional, also includes:

[0020] Obtain the average total amount of data in each of the tables to be statistically analyzed within a preset time period;

[0021] Determine the difference between the total amount of data in each table to be statistically analyzed and the average total amount of data in the table, and determine whether the difference for each table to be statistically analyzed satisfies the threshold condition;

[0022] When there is a difference in the difference corresponding to each of the tables to be counted that does not meet the threshold condition, it is determined that there is a data statistical anomaly in the total amount of data in the table, and an alarm signal is sent to the user terminal.

[0023] Optionally, the step of creating at least one distributed dataset Rdd corresponding to each of the tables to be statistically analyzed, and determining the name and partition of each Rdd, includes:

[0024] The execution mode for executing the query execution plan is determined, and the execution mode includes: a preset deployment mode and a local execution mode;

[0025] When the execution mode is the preset deployment mode, an Rdd is created at the first execution entry point, and the name and partition of the Rdd are determined.

[0026] When the execution mode is the local execution mode, an Rdd is created at the second execution entry point, and the name and partition of the Rdd are determined.

[0027] Optionally, obtaining the data volume query command for the data warehouse tool Hive and generating a query execution plan corresponding to the data volume query command includes:

[0028] Obtain a data volume query instruction from Hive, a data storage tool for statistical data. The data volume query instruction includes at least one of a data volume query statement or a data volume query task instruction.

[0029] The data volume query command is parsed into an abstract syntax tree;

[0030] The abstract syntax tree is transformed into a first logical plan by a preset analyzer. The first logical plan includes table data information to be analyzed.

[0031] The logical plan is optimized by a preset optimizer to generate a second logical plan, which includes an execution strategy for querying the amount of data in the table to be statistically analyzed.

[0032] Based on the preset planner, the second logical plan is transformed into a Spark execution plan, and the Spark execution plan is confirmed as a query execution plan.

[0033] Optional, also includes:

[0034] The statistical results of the data volume are fed back to the data management terminal.

[0035] A data statistics device, applied to the Spark computing engine platform, includes:

[0036] The plan generation unit is used to obtain the data volume query instruction of the Hive data storage tool and generate a query execution plan corresponding to the data volume query instruction;

[0037] The data reading unit is used to execute the query execution plan to read the table data information in the Hive data storage tool corresponding to the query execution plan. The table data information includes at least one table to be counted, as well as the table dataset and table name corresponding to the table to be counted.

[0038] The Rdd creation unit is used to create at least one distributed dataset Rdd corresponding to each table to be counted, and to determine the name and partition of each Rdd. The name of the Rdd corresponds to the table name of the table to be counted corresponding to the Rdd, and each partition of the Rdd includes at least one subtask.

[0039] A data volume reading unit is used to read the data volume of each task.

[0040] A partition matching unit is used to match the partition corresponding to each task with the partition of each Rdd respectively, and determine the total amount of data of all tasks corresponding to each Rdd;

[0041] The table data total quantity determination unit is used to determine the total table data quantity of each table to be counted based on the correspondence between the name of each Rdd and the table name of the table to be counted, and the total data quantity of all tasks corresponding to each Rdd.

[0042] The statistical result determination unit is used to determine the total amount of data in each of the tables to be statistically analyzed as the data volume statistical result.

[0043] Optionally, the partition matching unit includes:

[0044] The first relation structure generation subunit is used to generate a first relation structure representing the correspondence between the data volume of a task and the partition based on the partition and the data volume corresponding to each task;

[0045] The second relation structure generation subunit is used to generate a second relation structure representing the correspondence between the name and partition of each Rdd based on the name and partition of each Rdd;

[0046] The relational structure traversal sub-unit is used to traverse the first relational structure and the second relational structure to determine the amount of data belonging to the same partition and the name of the Rdd;

[0047] The total data volume determination subunit is used to determine the total data volume of all tasks corresponding to each Rdd based on the data volume with the same name as the corresponding Rdd.

[0048] A data statistics device, including a memory and a processor;

[0049] The memory is used to store programs;

[0050] The processor is used to execute the program to implement each step of any of the data statistics methods described.

[0051] A storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the data statistics methods described above.

[0052] This application performs data statistics on Hive data based on the Spark platform without developing a new program. It generates a query execution plan in near real-time based on the data volume query command obtained from the Hive data storage tool, directly reading table data information from Hive for statistical analysis. During the statistical process, an Rdd corresponding to each table to be analyzed is created, and the data volume of one or more tasks corresponding to the partition of each Rdd is read to determine the total data volume for each Rdd. Since the names of the Rdds created in this application correspond to the names of the tables to be analyzed, the total table data volume for each table can be determined.

[0053] This application utilizes the Spark platform to respond to data volume query commands and count the total amount of data in tables in Hive without the need to develop a separate calculation program. It achieves timely response to query requests with high efficiency and near real-time. Furthermore, by automatically reading task data through Spark, the accuracy of table data statistics can be improved. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0055] Figure 1 A schematic diagram of an optional process for implementing the data statistics method provided in an embodiment of this application;

[0056] Figure 2 An optional example diagram of the task generation method provided in the embodiments of this application;

[0057] Figure 3 An example diagram of the Spark execution layer structure provided in the embodiments of this application;

[0058] Figure 4 A schematic diagram of an optional structure of the data statistics device provided in the embodiments of this application;

[0059] Figure 5 This is a schematic diagram of an optional structure of the data statistics device provided in an embodiment of this application. Detailed Implementation

[0060] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0061] Hive is a data warehouse tool on Hadoop. It maps structured data files to database tables, allowing you to track the data volume of each table and gain a holistic understanding of the datasets stored in Hive, thus enabling better data management and analysis. However, current methods for calculating the data volume of Hive tables have drawbacks, including the need to develop separate statistical tasks, additional computational resource consumption, and the inability to perform near real-time statistics and timely detection of data problems.

[0062] Based on this, the data statistics method provided in this application embodiment can be applied to the Spark computing engine platform, responding in near real-time to data volume query commands obtained in any way from the Hive data storage tool, eliminating the need to develop a separate statistical task to count the total amount of data in the Hive table. Furthermore, by automatically reading task data through Spark, the accuracy of table data statistics can be improved.

[0063] Reference Figure 1 This application provides an optional flowchart illustrating a data statistics method implemented using the Spark platform. Specifically, the statistics process may include:

[0064] Step S110: Obtain the data volume query instruction for the statistical data warehouse tool Hive, and generate a query execution plan corresponding to the data volume query instruction.

[0065] It is understandable that the data volume query command is a data volume computation task for the Spark platform. Spark offers multiple options for obtaining computation tasks, such as command line submission, SparkShell, Spark API, scheduling tools, and Spark UI.

[0066] The following example illustrates how Spark can retrieve data volume query commands. For instance, Spark platform data volume query commands can be obtained via Spark SQL queries. Users can input Spark SQL through the query interface of Spark's integrated visual interface to query data volume. Alternatively, Spark can be used in conjunction with ETL (or ExtractTransform Load) tools. It can extract data from the data storage tool Hive in real time via ETL, clean, transform, and integrate the data, and generate data volume query commands to load the processed data into Spark for data volume statistics.

[0067] Furthermore, the Spark platform needs to generate an executable Spark plan, or query execution plan, based on the query requirements expressed by the data volume query command, including the query object, query intent, and requirements for the query results. The Spark plan represents the specific steps or execution strategy for performing the query operation. Spark implements the corresponding operations according to the Spark plan to obtain the final query results that meet the query requirements.

[0068] Understandably, there are multiple ways to generate a SparkPlan, such as static compilation, dynamic compilation, interpreted execution, and compiled execution. In practical applications, users or the Spark platform can choose the appropriate way to generate a SparkPlan based on the application scenario and requirements.

[0069] Optionally, in this embodiment of the application, a feasible method for generating a query execution plan is employed, comprising: obtaining a data volume query instruction from the Hive data storage tool to be statistically analyzed, wherein the data volume query instruction includes at least one of a data volume query statement or a data volume query task instruction; parsing the data volume query instruction into an abstract syntax tree; converting the abstract syntax tree into a first logical plan through a preset analyzer, wherein the first logical plan includes table data information to be statistically analyzed; optimizing the logical plan according to a preset optimizer to generate a second logical plan, wherein the second logical plan includes an execution strategy for querying the data volume of the table data information to be statistically analyzed; and converting the second logical plan into a Spark execution plan based on a preset planner, and confirming the Spark execution plan as the query execution plan.

[0070] The data volume query instruction may include a data volume query statement or a data volume query task instruction. The steps performed for both data volume query statements and data volume query task instructions are the same; therefore, this application embodiment will describe the processing procedure for a data volume query statement. (Refer to...) Figure 2The following is an example diagram of an optional task generation method provided in this application embodiment, which illustrates the process by which a Spark platform transforms a data volume query statement SparkSql into an executable query execution plan Spark plan.

[0071] Spark SQL uses the ANTLR parser component to parse the SQL string, generating an unanalyzed logical plan (Logical Plan). This unanalyzed logical plan is a tree structure representing the various components of the query and their relationships. Then, Spark's pre-defined analyzer performs a series of analyses and transformations on the unanalyzed logical plan, such as type checking and alias resolution. Finally, the unanalyzed logical plan is transformed into an analyzed logical plan (the first logical plan), which determines the query intent, such as the table name or table data to be analyzed.

[0072] Furthermore, the Logical Optimizer performs rule-based optimization on the first logical plan, such as constant folding and predicate pushdown, to make the final optimized LogicalPlan (or second logical plan) more efficient. The optimized LogicalPlan includes strategies for one or more physical execution plans, which consider data distribution, access patterns, and specific query requirements to achieve more efficient query execution. Therefore, in this embodiment, the second logical plan may include an execution strategy for querying the amount of data in the table to be statistically analyzed, such as how to obtain the table data from Hive and how to partition the created Rdd. Next, the optimized LogicalPlan is transformed into a physical execution plan SparkPlan using the Spark platform's planner component, and the physical execution plan is confirmed as a Spark executable SparkPlan, i.e., a query execution plan.

[0073] Step S120: Execute the query execution plan to read the table data information in the Hive statistical data warehouse tool corresponding to the query execution plan.

[0074] The table data information includes: at least one table to be statistically analyzed, as well as the table dataset and table name corresponding to the table to be statistically analyzed.

[0075] Hive includes several tables. The query execution plan can determine whether the query is for the total amount of data in all tables or for one or a few tables. Based on this, the table dataset and table name corresponding to the table to be queried can be obtained in a targeted manner.

[0076] Step S130: Create at least one distributed dataset Rdd corresponding to each table to be statistically analyzed, and determine the name and partition of each Rdd. Each partition of the Rdd includes at least one subtask.

[0077] Rdd (or Resilient Distributed Dataset) is the foundation of distributed datasets in Spark, enabling distributed storage and computation across multiple nodes in a cluster. By partitioning data into multiple partitions and distributing computation across different nodes, the processing speed of large-scale data can be significantly improved. Therefore, it is necessary to create Rdd corresponding to the table data information to be analyzed, in order to partition the Rdd and achieve distributed computation. In this embodiment, an Rdd corresponding to the table dataset of each table to be analyzed is created to achieve the statistical analysis of the data volume of each table.

[0078] Reference Figure 3 The example diagram of the Spark execution structure provided in this application embodiment illustrates that, for each query execution plan, the Spark scheduler will create a [structure / structure] corresponding to [the previous one]. Figure 3 The execution structure is the same as the execution structure used to execute the query execution plan. Figure 3 This includes: Spark application (or distributed application), job, stage (or partition), and task (or task / subtask).

[0079] A Spark application is a Spark distributed application that contains the aforementioned data query statements (or Spark SQL). When Spark receives a Spark application, it allocates data resources to the Spark application for computation, that is, it assigns the read table data information to the Spark application and generates an executable Spark application.

[0080] Spark creates Rdds (Resource Execution Modules) based on SparkPlans to generate the aforementioned execution structure. Then, one or more jobs are generated from these Rdds. A job is a Spark operational unit that contains multiple Rdds and various operations performed on them. Spark automatically splits each job into multiple task groups, each consisting of a stage. These tasks can then perform parallel computations to complete the distributed calculations of the table data to be analyzed.

[0081] Based on the above, the created Rdd will be split into multiple execution task units. Since neither the Rdd nor the split Rdd has source information, even if Spark can read the data volume of each task, it cannot associate the task with the table to be analyzed, and the data volume of each table remains undetermined. Therefore, in this embodiment, during the conversion of the optimized LogicalPlan into a SparkPlan, Spark explicitly sets the Rddname to the read tableName when returning the Rdd, without modifying the RDD class, thus achieving association between the Rdd and the table to be analyzed.

[0082] Optionally, Rdds can be named. Operators can customize the naming of each created Rdd through Spark's visual interface, and Spark will respond to the custom naming operation to name each Rdd. Alternatively, Spark can automatically name the corresponding Rdd based on the table name of each table to be analyzed.

[0083] Furthermore, this embodiment of the application takes into account that Spark has different execution modes. Therefore, when creating and naming Rdd, the naming method is adaptively adjusted according to different modes. Optionally, creating at least one distributed dataset Rdd corresponding to each table to be statistically analyzed, and determining the name and partition of each Rdd, includes: determining the execution mode for executing the query execution plan, wherein the execution mode includes: a preset deployment mode and a local execution mode; when the execution mode is the preset deployment mode, creating an Rdd at a first execution entry point and determining the name and partition of the Rdd; when the execution mode is the local execution mode, creating an Rdd at a second execution entry point and determining the name and partition of the Rdd.

[0084] Understandably, different execution modes result in different task entry points, requiring modifications to different Java classes. The following example illustrates that Spark's execution modes can include: local (local execution mode) and Spark on Yarn (Yarn cluster mode).

[0085] When Spark is running in local execution mode, Hadoop Rdds can be created in the Spark DataSourceScanExec class, and each Rdd can be named, corresponding to the name of the table to be analyzed. When Spark is running in Spark on Yarn execution mode, Hadoop Rdds need to be created in the spark TableReader class, and each Rdd can be named.

[0086] As can be seen above, different execution modes in Spark result in different entry points for creating Rdd classes, or different Rdd classes being created. In this embodiment, no specific limitations are made on the first and second execution modes, or the first and second execution entry points. Spark's execution modes may also include Standalone mode, or other executable modes. Therefore, this embodiment aims to illustrate the idea that different execution modes may result in different execution entry points and different Rdd classes being created.

[0087] Step S140: Read the data volume of each task.

[0088] Step S150: Match the partition corresponding to each task with the partition of each Rdd to obtain the total data of all tasks corresponding to each Rdd.

[0089] After each task is computed, retrieve the recordsRead (or counter) of taskMetrics during the Spark computation process, and obtain the amount of data for each task computed by Spark.

[0090] Tasks within the same stage share a common parent Rdd, and this parent Rdd (the Rdd created above) corresponds to one or more stages. Based on this, the data volume of all tasks corresponding to each stage is determined, and then the data volume of all stages belonging to the same Rdd is determined. Each Rdd corresponds to a table to be statistically analyzed, and based on this, the table data volume of each table to be statistically analyzed can be determined.

[0091] When matching the data volume of a task with an Rdd, it is necessary to consider the partitions of the task and the partitions of the Rdd. In this embodiment, the process of matching the partition corresponding to each task with the partition of each Rdd to determine the total data volume of all tasks corresponding to each Rdd may include: generating a first relation structure representing the correspondence between the data volume of a task and the partition based on the partition and the data volume corresponding to each task; generating a second relation structure representing the correspondence between the name of an Rdd and the partition based on the name and partition of each Rdd; traversing the first relation structure and the second relation structure to determine the data volume and the name of the Rdd belonging to the same partition; and determining the total data volume of all tasks corresponding to each Rdd based on the data volume with the same name of the corresponding Rdd.

[0092] Understandably, when the Job starts, the names of the Rdd created above can be filtered out, and the relationship between the Rdd names and the corresponding partition stageIds can be saved. The corresponding relational function or relational result can then be established, such as creating a map function.<Rdd_name,stageId> .

[0093] After each task is completed, save the stageId of each task and the data volume (task_count) of each task obtained from the counter. Establish a relationship function or structure between task_count and stageId, such as a map function.<task_count,stageId> .

[0094] Based on map<task_count,stageId> and map<Rdd_name,stageId> This allows matching of task_count and Rdd_name belonging to the same stageId. Then, task_counts with the same Rdd_name are merged to obtain the total data volume for a complete Rdd corresponding to that Rdd_name.

[0095] Step S160: Determine the total amount of table data for each table to be counted based on the correspondence between the name of each Rdd and the table name of the table to be counted, and the total amount of data for all tasks corresponding to each Rdd.

[0096] Step S170: The total amount of data in each of the tables to be counted is determined as the data volume statistics result.

[0097] As described above, each created Rdd corresponds to a table to be statistically analyzed. Therefore, the total amount of data corresponding to each Rdd obtained above can be confirmed as the total amount of table data in the table to be statistically analyzed corresponding to each Rdd. The correspondence between Rdd and the table to be statistically analyzed can be determined based on the correspondence between Rdd_name and the table name. Assuming that an Rdd is created corresponding to the table to be statistically analyzed, Hive1, and the name of the Rdd is set to Hive1', then Rdd_name represents the total amount of data corresponding to Hive1', which is the total amount of table data in the table to be statistically analyzed, Hive1.

[0098] After the statistics are completed, the table names, total amount of data in all the tables to be counted, and the corresponding Rdd_name can be collected to generate a statistical result table or statistical result report. The statistical result table or statistical result report described above can be used as the statistical result of the table data information in the data storage tool Hive.

[0099] In summary, this application embodiment performs data statistics on Hive data based on the Spark platform without developing a new program. It generates a query execution plan corresponding to the data volume query command obtained from the Hive data storage tool, directly reading table data information from Hive for statistical analysis. During the statistical process, each table to be analyzed is represented by an Rdd, and the data volume of one or more tasks corresponding to the partition of each Rdd is read to determine the total data volume for each Rdd. Since the names of the Rdds created in this application correspond to the names of the tables to be analyzed, the total table data volume for each table can be determined.

[0100] This application embodiment utilizes the Spark platform to respond to the data volume query command and count the total amount of data in the Hive table. There is no need to develop a separate calculation program. It can achieve timely response to query requests with high efficiency and near real-time. Furthermore, by automatically reading task data through Spark, the accuracy of table data statistics can be improved.

[0101] In addition, after completing the data statistics for Hive, the data volume statistics results can be fed back to the data management terminal so that the data management method for Hive can be adjusted according to the data statistics results.

[0102] The data management endpoint can be a Hive data management module, enabling Hive to automatically manage data scientifically based on statistical results. Alternatively, the data management endpoint can be a user endpoint, where Spark displays statistical results to users through a visual interface, allowing users to understand Hive's storage status and manually intervene in Hive's data management.

[0103] Inevitably, any execution program or platform may have the possibility of calculation errors. Therefore, in order to avoid large errors in data statistics, this application proposes a method for alerting users of abnormal statistical results. This method may include: obtaining the average total amount of table data for each table to be statistically analyzed within a preset time period; determining the difference between the total amount of table data for each table to be statistically analyzed and the average total amount of table data; determining whether the difference for each table to be statistically analyzed meets a threshold condition; and when there is a difference for each table to be statistically analyzed that does not meet the threshold condition, determining that there is an abnormality in the total amount of table data, and sending an alarm signal to the user terminal.

[0104] It is understandable that if Spark performs near real-time statistics on table data in Hive, the difference between the statistical results obtained each time will be minimal if no large amount of data is stored at once. If a large difference occurs, it can be determined whether a large amount of data was stored at once. If no large amount of data was stored, then there is a high probability that a statistical anomaly has occurred.

[0105] Based on this, this application embodiment checks for statistical anomalies on a table-by-table basis. Taking the anomaly detection process for the statistical results of a table to be statistically analyzed as an example, firstly, the total amount of table data within a preset time period is obtained, and the average value of the total amount of table data is determined. It can be understood that the preset time period is the time before the current statistical time when data statistics are performed. The average value of the total amount of table data can be any form of average, such as a data volume waveform with a certain regularity, or an average value of a single data point.

[0106] Furthermore, it is determined whether the total amount of data in the table to be statistically analyzed significantly exceeds or falls below the average amount of data in the table. If so, it indicates an anomaly in the total amount of data in the current table, and this anomaly information is reported to the data management end so that the data management end can find the cause of the anomaly. If the deviation between the total amount of data in the table to be statistically analyzed and the average amount of data in the current table is small or non-existent, it indicates that the total amount of data in the current table is normal, and the statistical results can be output normally.

[0107] The data statistics device provided in the embodiments of this application is described below. The data statistics device described below can be referred to in correspondence with the data statistics method described above.

[0108] First, combined Figure 4 This section introduces data statistics devices used in the Spark computing engine platform, such as... Figure 4 As shown, this data statistics device may include:

[0109] The plan generation unit 100 is used to obtain the data volume query instruction of the data storage tool Hive and generate a data volume query instruction corresponding to the data volume query instruction.

[0110] The data reading unit 200 is used to execute the query execution plan to read the table data information in the Hive data storage tool corresponding to the query execution plan. The table data information includes at least one table to be counted, as well as the table dataset and table name corresponding to the table to be counted.

[0111] Rdd creation unit 300 is used to create at least one distributed dataset Rdd corresponding to each of the tables to be counted, and to determine the name and partition of each Rdd. The name of the Rdd corresponds to the table name of the table to be counted corresponding to the Rdd, and each partition of the Rdd includes at least one subtask.

[0112] The data volume reading unit 400 is used to read the data volume of each task;

[0113] The partition matching unit 500 is used to match the partition corresponding to each task with the partition of each Rdd respectively, and determine the total amount of data of all tasks corresponding to each Rdd;

[0114] The table data total determination unit 600 is used to determine the total table data of each table to be counted based on the correspondence between the name of each Rdd and the table name of the table to be counted, and the total data of all tasks corresponding to each Rdd.

[0115] The statistical result determination unit 700 is used to determine the total amount of table data of each table to be statistically analyzed as the data volume statistical result.

[0116] Optionally, the partition matching unit 500 includes:

[0117] The first relation structure generation subunit is used to generate a first relation structure representing the correspondence between the data volume of a task and the partition based on the partition and the data volume corresponding to each task;

[0118] The second relation structure generation subunit is used to generate a second relation structure representing the correspondence between the name and partition of each Rdd based on the name and partition of each Rdd;

[0119] The relational structure traversal sub-unit is used to traverse the first relational structure and the second relational structure to determine the amount of data belonging to the same partition and the name of the Rdd;

[0120] The total data volume determination subunit is used to determine the total data volume of all tasks corresponding to each Rdd based on the data volume with the same name as the corresponding Rdd.

[0121] Optional, also includes:

[0122] The historical statistics data acquisition unit is used to acquire the average total amount of table data for each table to be statistically analyzed within a preset time period;

[0123] The condition judgment unit is used to determine the difference between the total amount of data in each table to be counted and the average total amount of data in the table, and to determine whether the difference corresponding to each table to be counted satisfies the threshold condition.

[0124] An anomaly alarm unit is used to determine that there is a data statistical anomaly in the total amount of data in the table when the judgment result of the condition judgment unit is negative, and to send an alarm signal to the user terminal.

[0125] Optionally, the Rdd creation unit 300 includes:

[0126] The mode determination subunit is used to determine the execution mode for executing the query execution plan, and the execution mode includes: a preset deployment mode and a local execution mode;

[0127] The first Rdd creation subunit is used to create an Rdd at the first execution entry point and determine the name and partition of the Rdd when the execution mode is the preset deployment mode.

[0128] The second creation subunit of Rdd is used to create Rdd at the second execution entry point and determine the name and partition of Rdd when the execution mode is the local execution mode.

[0129] Optionally, the plan generation unit 100 includes:

[0130] The query instruction acquisition subunit is used to acquire the data volume query instruction of Hive, a data volume query tool for statistical data warehouses. The data volume query instruction includes at least one of a data volume query statement or a data volume query task instruction.

[0131] The instruction parsing subunit is used to parse the data query instruction into an abstract syntax tree;

[0132] The analysis subunit is used to transform the abstract syntax tree into a first logical plan through a preset analyzer. The first logical plan includes table data information to be analyzed.

[0133] The logic optimization subunit is used to optimize the logic plan according to the preset optimizer and generate a second logic plan. The second logic plan includes: an execution strategy for querying the amount of data in the table data information to be counted.

[0134] The plan confirmation subunit is used to convert the second logical plan into a Spark execution plan based on a preset planner, and to confirm the Spark execution plan as a query execution plan.

[0135] Optional, also includes:

[0136] The data feedback unit is used to feed back the statistical results of the data volume to the data management terminal.

[0137] In summary, this application embodiment performs data statistics on Hive data based on the Spark platform without developing a new program. It generates a query execution plan corresponding to the data volume query command obtained from the Hive data storage tool, directly reading table data information from Hive for statistical analysis. During the statistical process, each table to be analyzed is represented by an Rdd, and the data volume of one or more tasks corresponding to the partition of each Rdd is read to determine the total data volume for each Rdd. Since the names of the Rdds created in this application correspond to the names of the tables to be analyzed, the total table data volume for each table can be determined.

[0138] This application embodiment utilizes the Spark platform to respond to the data volume query command and count the total amount of data in the Hive table. There is no need to develop a separate calculation program. It can achieve timely response to query requests with high efficiency and near real-time. Furthermore, by automatically reading task data through Spark, the accuracy of table data statistics can be improved.

[0139] The data statistics device provided in this application embodiment can be applied to data statistics equipment.

[0140] Figure 5 A schematic diagram of the data statistics device is shown. (Refer to...) Figure 5 The structure of the data statistics device may include: at least one processor 10, at least one memory 20, at least one communication bus 30 and at least one communication interface 40.

[0141] In this embodiment, the number of processor 10, memory 20, communication bus 30 and communication interface 40 is at least one, and processor 10, memory 20 and communication interface 40 communicate with each other through communication bus 30.

[0142] Processor 10 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0143] The memory 20 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0144] The memory stores a program, and the processor can call the program stored in the memory. The program is used to implement the various processing flows in the aforementioned data statistics scheme.

[0145] This application embodiment also provides a storage medium that can store a program suitable for execution by a processor, the program being used to implement the various processing flows in the aforementioned data statistics scheme.

[0146] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0147] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0148] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data statistics method, characterized in that, Applied to the Spark computing engine platform, including: Obtain the data volume query command for the Hive data storage tool and generate a query execution plan corresponding to the data volume query command; Execute the query execution plan to read the table data information in the Hive data storage tool corresponding to the query execution plan. The table data information includes: at least one table to be counted, and the table dataset and table name corresponding to the table to be counted. Create at least one distributed dataset Rdd corresponding to each of the tables to be counted, and determine the name and partition of each Rdd. The name of the Rdd corresponds to the table name of the table to be counted corresponding to the Rdd, and each partition of the Rdd includes at least one subtask. Read the amount of data for each task; The partition corresponding to each task is matched with the partition of each Rdd to obtain the total amount of data for all tasks corresponding to each Rdd; Based on the correspondence between the name of each Rdd and the table name of the table to be counted, and the total amount of data of all tasks corresponding to each Rdd, determine the total amount of table data of each table to be counted; The total amount of data in each of the tables to be statistically analyzed is determined as the data volume statistical result.

2. The data statistics method according to claim 1, characterized in that, The step of matching the partition corresponding to each task with the partition of each Rdd to determine the total amount of data for all tasks corresponding to each Rdd includes: Based on the partition and data volume corresponding to each task, a first relational structure representing the correspondence between the data volume and partition of the task is generated; Based on the name and partition of each Rdd, a second relation structure is generated that represents the correspondence between the name and partition of the Rdd; Traverse the first relation structure and the second relation structure to determine the amount of data belonging to the same partition and the name of the Rdd; Based on the data volume with the same name as the corresponding Rdd, determine the total data volume of all tasks corresponding to each Rdd.

3. The data statistics method according to claim 1, characterized in that, Also includes: Obtain the average total amount of data in each of the tables to be statistically analyzed within a preset time period; Determine the difference between the total amount of data in each table to be statistically analyzed and the average total amount of data in the table, and determine whether the difference for each table to be statistically analyzed satisfies the threshold condition; When there is a difference in the difference corresponding to each of the tables to be counted that does not meet the threshold condition, it is determined that there is a data statistical anomaly in the total amount of data in the table, and an alarm signal is sent to the user terminal.

4. The data statistics method according to claim 1, characterized in that, The step of creating at least one distributed dataset Rdd corresponding to each of the tables to be statistically analyzed, and determining the name and partition of each Rdd, includes: The execution mode for executing the query execution plan is determined, and the execution mode includes: a preset deployment mode and a local execution mode; When the execution mode is the preset deployment mode, an Rdd is created at the first execution entry point, and the name and partition of the Rdd are determined. When the execution mode is the local execution mode, an Rdd is created at the second execution entry point, and the name and partition of the Rdd are determined.

5. The data statistics method according to claim 1, characterized in that, The step of obtaining the data volume query command for the Hive data storage tool and generating a query execution plan corresponding to the data volume query command includes: Obtain a data volume query instruction from Hive, a data storage tool for statistical data. The data volume query instruction includes at least one of a data volume query statement or a data volume query task instruction. The data volume query command is parsed into an abstract syntax tree; The abstract syntax tree is transformed into a first logical plan by a preset analyzer. The first logical plan includes table data information to be analyzed. The logical plan is optimized by a preset optimizer to generate a second logical plan, which includes an execution strategy for querying the amount of data in the table to be statistically analyzed. Based on the preset planner, the second logical plan is transformed into a Spark execution plan, and the Spark execution plan is confirmed as a query execution plan.

6. The data statistics method according to claim 1, characterized in that, Also includes: The statistical results of the data volume are fed back to the data management terminal.

7. A data statistics device, characterized in that, Applied to the Spark computing engine platform, including: The plan generation unit is used to obtain the data volume query instruction of the Hive data storage tool and generate a query execution plan corresponding to the data volume query instruction; The data reading unit is used to execute the query execution plan to read the table data information in the Hive data storage tool corresponding to the query execution plan. The table data information includes at least one table to be counted, as well as the table dataset and table name corresponding to the table to be counted. The Rdd creation unit is used to create at least one distributed dataset Rdd corresponding to each table to be counted, and to determine the name and partition of each Rdd. The name of the Rdd corresponds to the table name of the table to be counted corresponding to the Rdd, and each partition of the Rdd includes at least one subtask. A data volume reading unit is used to read the data volume of each task. A partition matching unit is used to match the partition corresponding to each task with the partition of each Rdd respectively, and determine the total amount of data of all tasks corresponding to each Rdd; The table data total quantity determination unit is used to determine the total table data quantity of each table to be counted based on the correspondence between the name of each Rdd and the table name of the table to be counted, and the total data quantity of all tasks corresponding to each Rdd. The statistical result determination unit is used to determine the total amount of data in each of the tables to be statistically analyzed as the data volume statistical result.

8. The data statistics device according to claim 7, characterized in that, The partition matching unit includes: The first relation structure generation subunit is used to generate a first relation structure representing the correspondence between the data volume of a task and the partition based on the partition and the data volume corresponding to each task; The second relation structure generation subunit is used to generate a second relation structure representing the correspondence between the name and partition of each Rdd based on the name and partition of each Rdd; The relational structure traversal sub-unit is used to traverse the first relational structure and the second relational structure to determine the amount of data belonging to the same partition and the name of the Rdd; The total data volume determination subunit is used to determine the total data volume of all tasks corresponding to each Rdd based on the data volume with the same name as the corresponding Rdd.

9. A data statistics device, characterized in that, Including memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement each step of the data statistics method as described in any one of claims 1-6.

10. A storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the various steps of the data statistics method as described in any one of claims 1-6.

Images