Mass data statistical method and system based on multi-dimensional depth strategy optimization

Abstract

The invention provides a mass data statistical method and system based on multi-dimensional depth strategy optimization, and the method comprises the steps: carrying out the data volume statistics of tables under different components, employing different statistical rules according to the types of the components, and enabling the component statistics to comprise full-table statistics, partition statistics and warehousing time statistics, performing process statistics on unstructured and semi-structured components; in the task execution process of process statistics, generating an input quantity and an output quantity according to the operation condition of each node; and customizing and configuring statistical rules for different components, and completing data volume statistics in a mode of combining the component statistics and the process statistics. According to the method, data volume statistics of semi-structured and non-structured data is realized in a big data scene mainly based on the statistical capability of the component and in combination with the data volume change condition generated by continuous operation in the task execution process, and flexible configuration of statistical rules and scheduling is used.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and specifically relates to a massive data statistics method and system based on multi-dimensional deep strategy optimization. Background Art

[0002] With the rapid development of information technology, we have entered the era of big data, and components and data formats are becoming increasingly complex and diverse. Today, it not only includes common components and structured data, but also covers many domestic components, such as Renmin University of China Jincang, Tianyunxing, DM8, etc., as well as different types of storage forms, such as Alibaba Cloud's OSS object storage. This diversity has greatly expanded the scope of data processing, but it has also brought many challenges.

[0003] Traditional data volume statistics methods mainly focus on structured data, but are unable to cope with the statistics of semi-structured and unstructured data. In the current technical environment, the demand for data volume statistics is growing, because data volume is not only an important indicator reflecting the scale of data storage and resource usage, but also an indispensable basic element for data reconciliation. From the perspective of enterprise management, data volume can provide management with a basis for decision-making and help them gain an in-depth understanding of business development trends and characteristic performance. However, faced with the diversity of data components, the complexity of statistical rules, and the massive scale of data, how to efficiently store, manage, and process massive amounts of data at the level of tens of billions or hundreds of billions, and accurately calculate the amount of data, has become a problem that needs to be solved urgently.

[0004] In view of this, it is very meaningful to propose a massive data statistical method and system based on multi-dimensional deep strategy optimization. Summary of the invention

[0005] In order to overcome the shortcomings of traditional data volume statistics methods, and to solve the flexibility and accuracy of data volume statistics when the data volume is huge, the component types are various, and the data formats are diversified, the present invention provides a massive data statistics method and system based on multi-dimensional deep strategy optimization to solve the problems of data component diversity, statistical rule complexity, huge data volume, etc., and to achieve accurate statistics of different types of data volumes.

[0006] In a first aspect, the present invention proposes a massive data statistics method based on multi-dimensional deep strategy optimization, the method comprising the following steps:

[0007] Data volume statistics are collected for tables under different components, including relational databases, distributed computing storage components, document databases and message queues, full-text databases, key-value databases, and unstructured and semi-structured components. Different statistical rules are used according to the component type. Component statistics include full table statistics, partition statistics, and storage time statistics, as well as unstructured and semi-structured component configuration process statistics.

[0008] During the task execution of process statistics, input and output quantities are generated according to the operation status of each node;

[0009] Customize statistical rules for different components, and complete data volume statistics by combining the component statistics and the process statistics; wherein, the relational database is configured with full-table statistics, storage time statistics and process statistics; if the distributed computing storage component is a partitioned table, full-table statistics, partition statistics and process statistics are configured; if it is a non-partitioned table, full-table statistics, storage time statistics and process statistics can be performed; document databases and message queues are configured with full-table statistics, storage time statistics and process statistics; full-text databases are configured with full-table statistics and process statistics; key-value databases are configured with full-table statistics, storage time statistics and process statistics.

[0010] Preferably, the component statistics are used to count the complexity of scheduling and data processing, specifically including:

[0011] Initialize the default statistical rules of the table and judge their rationality and completeness according to the default statistical rules of different component types. If the rules are incomplete, further determine whether to initialize statistics. If in the initialization state, clear all current logs that are older than this statistical time and execute the next step. Otherwise, terminate the statistics.

[0012] Determine whether the table definition information is complete, that is, whether the table structure information is complete. If not, end the statistics, otherwise proceed to the next step;

[0013] Determine whether there is a running task blocking. If so, wait until the task is completed before ending the statistics. Otherwise, proceed to the next step.

[0014] If there is no task blocking, further determine whether the current table is only configured with storage time statistics and whether it is the first statistics. If so, the system automatically completes the statistics of the entire table. Otherwise, proceed to the next step.

[0015] Further determine the statistical type and process the three types separately: full table statistics, partition statistics, and storage time statistics.

[0016] Preferably, the implementation flow of the process statistics specifically includes: when constructing data access or processing tasks, determining whether to enable the statistics configuration, if the statistics configuration is enabled, proceeding to the next step, otherwise only enabling the task and recording the execution log before ending the process;

[0017] The source node, operator node and target node of the operator are marked, including obtaining component environment variables, defining statistical indicator groups and specific indicators, implementing batch or streaming statistical operations, pushing indicators to prometheus, parsing logs to ES and processing and outputting them to MYSQL, providing data support for comparison with component statistical accuracy. The MYSQL table structure contains fields such as task-related information and input and output quantities.

[0018] Further preferably, the three types of full table statistics, partition statistics, and storage time statistics are processed separately, specifically including: if the statistical type is full table statistics, the full table count (*) is directly performed by default, and the next step is entered;

[0019] If the statistics type is zonal statistics, then:

[0020] Get all partitions of the table by calling the abstract layer interface;

[0021] Obtain the range of all partitions of the table that have been successfully counted and have not been deleted from the statistics log;

[0022] Compare the differences between the latest partition and the partition in the statistical log, and identify which are historical partitions and newly added partitions; if they are newly added partitions, add them to the statistical task; if the historical partition still exists in the partition list and there is data in the log, no processing is done; if the historical partition is not in the partition list, the partition is marked as deleted;

[0023] For cross-partition processing, the partition rules are obtained according to T-1 time, and the partition name is converted to determine whether the partition is in the partition list. If it is, it is added to the statistics. If it is not, it is directly recorded as 0 and the partition statistics task is marked as failed.

[0024] Continue to determine whether there are partitions that have not been counted. If so, further analyze whether they are asynchronously counted.

[0025] If the statistics type is inventory entry time statistics, obtain the current statistics log, encapsulate the inventory entry time query condition as 00:00:00-23:59:59, and proceed to the next step.

[0026] Further preferably, it also includes: judging whether asynchronous statistics are required for the table, if asynchronous statistics are required, requesting the asynchronous statistics interface of the unified data operation layer, recording the unique identifier returned by the abstract layer and encapsulating the main log record, and finally completing the abstract layer callback interface; if asynchronous statistics are not required, directly calling the abstract layer statistics interface and encapsulating the statistics log to obtain the log details;

[0027] Add or update statistical logs based on the results of the previous step and output atomic indicators.

[0028] Further preferably, the method further includes: calculating derived indicators according to the atomic indicators of the statistical log, and calculating the full amount, incremental amount and deletion amount results using different calculation rules according to different situations of full table statistics, partition statistics, and full table statistics combined with storage time statistics. The calculation rules are as follows:

[0029] For full table statistics, if full table statistics fail, the entire calculation result remains unchanged; if full table statistics succeed and the storage time increment rule statistics fail, the full amount is the full amount of data that has been successfully counted in the current batch; the increment is the full amount of data that has been successfully counted in the current batch minus the full amount of data that was successfully counted last time. If it is a negative number, it is 0; the deletion amount is the full amount of data that was successfully counted last time minus the full amount of data that was successfully counted in the current batch. If it is a negative number, it is 0;

[0030] For statistics on the entire table, first clarify the scheduling scope: the first scheduling includes all partitions in the partition list, and the Nth scheduling includes all partitions in the range from the last scheduling date to T and the partitions that were not successfully run in the previous batch of statistics logs, where N is greater than or equal to 2;

[0031] Next, confirm the calculation rules: If all partition statistics fail, the entire calculation result remains unchanged; however, if there is a log of successful partition statistics, the total amount is the sum of the latest successful data of each partition corresponding to all partitions not marked as deleted; the increment is the sum of the successful data of the partitions in this batch minus the sum of the latest successful data of the corresponding historical batch statistics of the partitions that have successfully counted this batch, and 0 is not taken; the deleted amount is the partition that was successfully run in the original log but is no longer in the latest partition list compared with the partition list of this batch;

[0032] For statistics on the time of adding a full table to the database, the incremental rule for the database entry time is the increment within the time range from the start time of the last successful statistics of the full table to the time when the statistics are currently initiated. If there is no log of the last successful statistics of the full table, the time less than the current statistical initiation time is used as the basis, which is divided into the following situations:

[0033] If the statistics of the entire table fail, the entire calculation result remains unchanged;

[0034] If statistics of the entire table are successful and statistics of the incremental rule for storage time fail, the full amount is the full amount of data that has been successfully counted in the current batch; the incremental amount is the full amount of data that has been successfully counted in the current batch minus the full amount of data that was successfully counted last time. If it is a negative number, it is 0; the deleted amount is the full amount of data that was successfully counted last time minus the full amount of data that was successfully counted in the current batch. If it is a negative number, it is 0;

[0035] If both the full table statistics and the storage time increment rule are successful, the full amount is the full amount of data that has been successfully counted in the current batch; the increment is the result calculated by the storage time increment rule; the deleted amount is the full amount of data that was successfully counted last time plus the current increment plus the current full amount. If it is a negative number, it is set to 0.

[0036] Preferably, the default statistical rules of the initialization table, and the constraints for judging its rationality and integrity are the default full-table statistics for relational databases, document databases, message queues, full-text databases and key-value databases, and the default partition statistics for distributed partition tables.

[0037] In a second aspect, an embodiment of the present invention provides a massive data statistics system based on multi-dimensional deep strategy optimization, including:

[0038] The component statistics module is configured to perform data volume statistics for tables under different components, including relational databases, distributed computing storage components, document databases and message queues, full-text databases, key-value databases, and unstructured and semi-structured components. Different statistical rules are used according to the component type. Component statistics include full table statistics, partition statistics, and storage time statistics, as well as statistics on the configuration process of unstructured and semi-structured components.

[0039] A process statistics module is configured to generate input and output quantities according to the operation status of each node during the execution of the process statistics task;

[0040] The statistical rule configuration module is configured to customize the statistical rules for different components, and complete the statistics of the data volume by combining the component statistics and the process statistics; wherein, the relational database configures the full table statistics, the storage time statistics and the process statistics; if the distributed computing storage component is a partitioned table, the full table statistics, the partition statistics and the process statistics are configured; if it is a non-partitioned table, the full table statistics, the storage time statistics and the process statistics can be performed; the document database and the message queue configure the full table statistics, the storage time statistics and the process statistics; the full-text database configures the full table statistics and the process statistics; the key-value database configures the full table statistics, the storage time statistics and the process statistics.

[0041] Compared with the prior art, the beneficial results of the present invention are:

[0042] The present invention is mainly based on the statistical capabilities of the components themselves, and combined with the changes in the amount of data generated by the continuous operation during the task execution process, so as to achieve data volume statistics of semi-structured and unstructured data in the big data scenario, and use statistical rules and flexible configuration of scheduling. It can not only achieve fast batch configuration, but also make the completeness and accuracy of the physical table data volume statistics, further improve the value of data indicators, and provide data support for decision makers. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated into and constitute a part of this specification. The accompanying drawings illustrate the embodiments and are used together with the description to explain the principles of the present invention. It will be easy to recognize other embodiments and many expected advantages of the embodiments because they become better understood by reference to the following detailed description. The elements of the drawings are not necessarily to scale with each other. The same reference numerals refer to corresponding similar parts.

[0044] Figure 1 It is a flowchart of a massive data statistics method based on multi-dimensional deep strategy optimization according to an embodiment of the present invention;

[0045] Figure 2 A schematic diagram of the implementation logic of component statistics of a specific embodiment of the present invention;

[0046] Figure 3 A schematic diagram of the implementation logic of process statistics of a specific embodiment of the present invention;

[0047] Figure 4 It is a schematic diagram of the implementation logic of process statistics in a specific embodiment of the present invention, taking Hive->Hive as an example;

[0048] Figure 5 A schematic diagram of a massive data statistics system based on multi-dimensional deep strategy optimization according to an embodiment of the present invention;

[0049] Figure 6 It is a schematic diagram of the structure of a computer device suitable for implementing an electronic device of an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0051] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0052] In today's big data era, components and data formats are becoming increasingly diverse, breaking through the previously limited scope of some common components and structured data, and including some domestically produced components such as Renmin University of China Jincang, Tianyunxing, DM8, VERTICA and Alibaba Cloud's OSS object storage.

[0053] Traditional data volume statistics are based on structured data. For different data structures, such as text, pictures, videos, etc., the diversity of big data brings great complexity to data volume statistics. With the rapid development of information technology, the speed and scale of data generation are constantly increasing. How to efficiently store, manage and process these tens of billions or hundreds of billions of massive data and calculate the data volume has also become a difficult problem. Therefore, it is necessary to develop new statistical methods and tools to achieve different types of data volume statistics.

[0054] In order to further solve the problems of data component diversity, complexity of statistical rules, huge data volume, etc., this paper proposes a method of massive data statistics with multi-dimensional deep strategy optimization. This data volume statistics method adopts different statistical rules for different components, and introduces the statistical method of the task process to supplement it, thereby improving the completeness and accuracy of data volume statistics, and providing data support and basis for subsequent data analysis in various business dimensions.

[0055] The present invention is mainly aimed at solving the flexibility and accuracy of data volume statistics when the data volume is huge, the components are diverse, and the data formats are diversified. The invention can not only customize the statistical rules of different components, but also compare the accuracy of data volume under different rules, providing data support for subsequent data reconciliation and dimensional analysis.

[0056] In the first aspect, the embodiments of the present invention disclose a method for calculating the statistics of massive data based on multi-dimensional deep strategy optimization, such as Figure 1 As shown, the method comprises the following steps:

[0057] S1. Perform data volume statistics for tables under different components, including relational databases, distributed computing storage components, document databases and message queues, full-text databases, key-value databases, and unstructured and semi-structured components. Different statistical rules are used according to component types. Component statistics include full table statistics, partition statistics, and storage time statistics. Statistics are collected on the configuration process of unstructured and semi-structured components.

[0058] S2. During the execution of the task of process statistics, the input and output are generated according to the operation status of each node;

[0059] S3. Customize the statistical rules for different components, and complete the data volume statistics by combining the component statistics and the process statistics; among them, the relational database is configured with full-table statistics, storage time statistics and process statistics; if the distributed computing storage component is a partitioned table, the full-table statistics, partition statistics and process statistics are configured; if it is a non-partitioned table, the full-table statistics, storage time statistics and process statistics can be performed; the document database and message queue are configured with full-table statistics, storage time statistics and process statistics; the full-text database is configured with full-table statistics and process statistics; the key-value database is configured with full-table statistics, storage time statistics and process statistics.

[0060] Specifically, component statistics are used to calculate the complexity of scheduling and data processing. Figure 2 , including:

[0061] S11, initialize the default statistical rules of the table, judge its rationality and completeness, and make judgments based on the default statistical rules of different component types. If the rules are incomplete, further judge whether to initialize statistics. If in the initialization state, clear all current logs that are earlier than this statistical time and then execute the next step, otherwise end the statistics;

[0062] Among them, the default statistical rules for the initialized table and the constraints for judging its rationality and completeness are the default full-table statistics for relational databases, document databases, message queues, full-text databases, and key-value databases, and the default partition statistics for distributed partition tables.

[0063] S12, judging whether the table definition information is complete, that is, whether the table structure information is complete, if not, then ending the statistics, otherwise proceeding to the next step;

[0064] S13, determine whether there is a task blocking in the running, if so, wait until the task is completed before ending the statistics, otherwise proceed to the next step;

[0065] S14. If there is no task blocking, further determine whether the current table is only configured with storage time statistics and whether it is the first statistics. If so, the system automatically completes the statistics of the entire table. Otherwise, proceed to the next step.

[0066] S15. Further determine the statistical type, and process the three types of statistics, namely, full table statistics, partition statistics, and storage time statistics, respectively.

[0067] Among them, the three types of statistics, namely, full table statistics, partition statistics, and storage time statistics, are processed separately, including:

[0068] S151. If the statistics type is full table statistics, the default is to count the entire table directly (*) and proceed to the next step;

[0069] S152. If the statistical type is partition statistics, then:

[0070] S1521. Obtain all partitions of the table by calling the abstract layer interface;

[0071] S1522. Obtain the range of all partitions of the table that have been successfully counted and have not been deleted from the statistics log;

[0072] S1523. Compare the differences between the latest partition and the partition in the statistical log, and identify which are historical partitions and newly added partitions; if they are newly added partitions, add them to the statistical task; if the historical partition still exists in the partition list and there is data in the log, do not process it; if the historical partition is not in the partition list, mark the partition as deleted;

[0073] S1524: For cross-partition processing, the partition rule is obtained according to T-1 time, and the partition name is converted to determine whether the partition is in the partition list. If it is, it is added to the statistics. If it is not, it is directly recorded as 0 and the partition statistics task is marked as failed.

[0074] S1525, continue to determine whether there are partitions that have not been counted, and if so, further analyze whether they are asynchronously counted;

[0075] S153. If the statistical type is storage time statistics, obtain the current statistical log, encapsulate the storage time query condition (00:00:00-23:59:59), and proceed to the next step.

[0076] S16. Determine whether asynchronous statistics are required for the table. If asynchronous statistics are required, request the asynchronous statistics interface of the unified data operation layer, record the unique identifier returned by the abstract layer and encapsulate the main log record, and finally complete the abstract layer callback interface; if asynchronous statistics are not required, directly call the abstract layer statistics interface and encapsulate the statistics log to obtain the log details;

[0077] S17. Add or update the statistical log according to the result of the previous step and output the atomic index;

[0078] S18. Calculate derived indicators based on the atomic indicators of the statistical log. Use different calculation rules to calculate the full amount, incremental amount, and deletion amount results according to different situations of full table statistics, partition statistics, and full table statistics combined with storage time statistics. The calculation rules are as follows:

[0079] S181. For statistics of the entire table, if the statistics of the entire table fails, the entire calculation result remains unchanged; if the statistics of the entire table succeed and the statistics of the storage time increment rule fail, the full amount is the full amount of data that has been successfully counted in the current batch; the increment is the full amount of data that has been successfully counted in the current batch minus the full amount of data that was successfully counted last time. If it is a negative number, it is 0; the deletion amount is the full amount of data that was successfully counted last time minus the full amount of data that was successfully counted in the current batch. If it is a negative number, it is 0;

[0080] S182. For statistics of the entire table, a. First, clarify the scheduling scope: the first scheduling includes all partitions in the partition list, and the Nth scheduling includes all partitions in the range from the last scheduling date to T and the partitions that have not been successfully run in the last batch of statistics logs, where N is greater than or equal to 2;

[0081] b. Next, confirm the calculation rules: If all partition statistics fail, the entire calculation result remains unchanged; however, if there is a successful partition statistics log, the total amount is the sum of the latest successful data count of each partition corresponding to all partitions not marked for deletion; the increment is the sum of the successful partition data count of this batch minus the sum of the latest successful data count of the partitions with successful statistics in this batch corresponding to the historical batch, and 0 is not taken; the deleted amount is the partition that was successfully run in the original log but is no longer in the latest partition list compared with the partition list of this batch;

[0082] S183. For statistics on the entry time of the whole table, the increment rule of the entry time is the increment within the time range from the start time of the last successful statistics of the whole table to the time of the current statistics initiation. If there is no log of the last successful statistics of the whole table, the time is less than the current statistics initiation time, which is divided into the following situations:

[0083] a. If the statistics of the entire table fail, the entire calculation result remains unchanged;

[0084] b. If statistics of the entire table are successful and the statistics of the incremental rule for storage time fail, the full amount is the full amount of data that has been successfully counted in the current batch; the incremental amount is the full amount of data that has been successfully counted in the current batch minus the full amount of data that was successfully counted last time. If it is a negative number, it is 0; the deleted amount is the full amount of data that was successfully counted last time minus the full amount of data that was successfully counted in the current batch. If it is a negative number, it is 0;

[0085] c. If both the full table statistics and the storage time increment rule are successful, the full amount is the full amount of data that was successfully counted in the current batch; the increment is the result calculated by the storage time increment rule; the deleted amount is the full amount of data that was successfully counted last time plus the current increment plus the current full amount. If it is a negative number, it is set to 0.

[0086] Specifically, the implementation process of process statistics includes:

[0087] S21. When building a data access or processing task, determine whether to enable the statistical configuration. If the statistical configuration is enabled, proceed to the next step. Otherwise, only enable the task and record the execution log before ending the process.

[0088] S22. Identify the source nodes, operator nodes, and target nodes of the operator, including obtaining component environment variables, defining statistical indicator groups and specific indicators, implementing batch or streaming statistical operations, pushing indicators to prometheus, parsing logs to ES, and processing and outputting them to MYSQL, to provide data support for comparison with component statistical accuracy. The MYSQL table structure contains fields such as task-related information and input and output quantities.

[0089] In a specific embodiment, data volume statistics are divided into two modes: component statistics and process statistics, which means:

[0090] 1. Component statistics: Summarize and calculate the data volume of tables under different components, including statistics of the entire table, partition statistics, and storage time statistics;

[0091] 2. Process statistics: During the task execution, the input and output are generated according to the operation status of each node.

[0092] The following is the preliminary configuration of statistical rules:

[0093] [1] Relational database. In addition to the traditional Oracle, Mysql, and PostgreSQL, relational databases also have domestic components, including DM8, KingbaseES, Vertica, ClickHouse, etc., which can be configured with full table statistics, storage time statistics, and process statistics.

[0094] [2] Distributed computing, including distributed computing storage components such as Hive and Maxcompute, is classified according to whether it is partitioned. If it is a partitioned table, you can configure full table statistics, partition statistics, and process statistics; if it is a non-partitioned table, you can perform full table statistics, storage time statistics, and process statistics.

[0095] 【3】Document database, message queue, for components such as MongoDB, Kafka, and DataHub, you can configure full table statistics, storage time statistics, and process statistics.

[0096] 【4】Full-text databases: For Elasticsearch, Solr, and other data sources, you can configure full-table statistics and process statistics.

[0097] 【5】Key-value database. For Redis key-value database, you can configure statistics for the entire table, statistics for storage time, and statistics for the process.

[0098] [6] Others: Statistics on the configuration process of unstructured and semi-structured components, including object storage (OSS), graph databases (GraphBase, Neo4j), columnar storage (Hbase), file systems (FTP, HDFS), etc.

[0099] In this embodiment, the core part of the component statistics of the present invention lies in the complexity of statistical scheduling and data processing, and the implementation logic is as follows Figure 2 The specific instructions are as follows:

[0100] Step 1: Initialize the default statistical rules of the table and determine whether the statistical rules are reasonable and complete (constraints: relational databases, document databases, message queues, full-text databases, and key-value databases use full-table statistics by default, and distributed partitioned tables use partition statistics by default). If the rules are incomplete, determine whether to initialize statistics. If so, clear all logs in the current table that are older than the statistical time of this batch and proceed to the next step. Otherwise, terminate the statistics.

[0101] Step 2: After step 1, determine whether the table definition information is complete, that is, whether the table structure information is complete; if not, end the statistics, otherwise proceed to the next step;

[0102] Step 3: Determine whether there is a block in the running task. If so, wait until the task is completed before ending the statistics. Otherwise, proceed to the next step.

[0103] Step 4: Based on the fact that there is no task blocking in step 3, further determine whether the current table is only configured with storage time statistics and whether it is the first time statistics. If it is confirmed to be "yes", the system will automatically complete and add "full table statistics", otherwise proceed to the next step;

[0104] Step 5: Determine the statistical type, which is divided into three types: full table statistics, partition statistics, and storage time statistics;

[0105] 1) If the statistics type = whole table statistics, the whole table count (*) is performed by default and the next step is performed;

[0106] 2) If the statistical type = partition statistics, it is divided into the following processes:

[0107] a. Get all partitions of the table by calling the abstract layer interface;

[0108] b. Obtain the range of all partitions in the table that have been successfully counted and not deleted according to the statistics log;

[0109] c. Compare the differences between the latest partition and the partition in the statistical log, and identify which are historical partitions (including deleted partitions) and newly added partitions. If it is a newly added partition, add it to the statistical task; if the historical partition still exists in the partition list and there is data in the log, no processing is done; if the historical partition is not in the partition list, the partition is marked as deleted;

[0110] d. Cross-partition processing: Get the partition rules according to T-1 time, convert the partition name, and determine whether the partition is in the partition list. If it is, add it to the statistics. If it is not, directly record it as 0 and mark the partition statistics task log as "failed";

[0111] e. Continue to determine whether there are partitions that have not been counted. If so, further analyze whether they are asynchronously counted.

[0112] 3) If the statistics type = storage time statistics, obtain the current statistics log, encapsulate the storage time query condition (00:00:00-23:59:59), and proceed to the next step;

[0113] Step 6: Determine whether asynchronous statistics are required for the table. If asynchronous statistics are required, request the asynchronous statistics interface of the unified data operation layer (hereinafter referred to as the abstract layer, which refers to the data operation platform between the big data component storage layer and the data application layer), record the unique identifier returned by the abstract layer and encapsulate the main log record, and finally complete the abstract layer callback interface; if asynchronous statistics are not required, directly call the abstract layer statistics interface and encapsulate the statistics log to obtain the log details;

[0114] Step 7: Add / update statistical logs based on the results of step 6 and output atomic indicators (i.e., the amount of data in each batch);

[0115] Step 8: Calculate derived indicators based on the atomic indicators of the statistical log, and calculate the full amount, incremental amount, and deleted amount results in real time. The specific rules are as follows:

[0116] 1) Statistics of the entire table

[0117] a. If the statistics of the entire table fail, the entire calculation result remains unchanged, that is, the result will not be recalculated.

[0118] b. If the statistics of the entire table are successful and the statistics of the storage time increment rule fail, then:

[0119] i. Full amount = the total amount of data that has been successfully counted in the current batch;

[0120] ii. Increment = the total amount of data successfully counted in the current batch - the total amount of data successfully counted last time. If it is a negative number, it is 0;

[0121] iii. Deleted amount = the total amount of data successfully counted last time - the total amount of data successfully counted in the current batch. If it is a negative number, it is 0.

[0122] 2) Zonal statistics

[0123] a. First, clarify the dispatch scope:

[0124] i. The scope of the first scheduling includes all partitions in the partition list;

[0125] ii. The scope of the Nth (N>=2) scheduling includes all partitions in the range from the last scheduling date to T and the partitions that have not been successfully run in the previous batch of statistical logs.

[0126] b. Next, confirm the calculation rules: If all partition statistics fail, the entire calculation result remains unchanged, that is, the result is not recalculated; but if there is a partition statistics success log, the calculation is performed according to the following rules:

[0127] i. Total amount = the sum of the latest successful data count for each partition corresponding to all partitions not marked for deletion;

[0128] ii. Increment = the sum of the data volume of the partitions that have been successfully counted in this batch - (the sum of the data volume of the partitions that have been successfully counted in this batch and the latest successful data volume of the corresponding historical batch statistics, not 0);

[0129] iii. Deleted amount = compared with the partition list of this batch, the partitions that were successfully run in the original log but are no longer in the latest partition list are considered as deleted amounts.

[0130] 3) Statistics of the entire table + statistics of storage time

[0131] When the statistics rules are both full table statistics and storage time statistics, the storage time increment rule is the increment within the time range (from the start time of the last successful full table statistics to the current statistics initiation time) (left closed and right open, that is, time>=the start time of the last successful full table statistics and time<the current statistics initiation time). If there is no last successful full table statistics log, the start time cannot be obtained, and time<the current statistics initiation time is used as the basis. It can be divided into the following three situations:

[0132] a. If the statistics of the entire table fail, the entire calculation result remains unchanged, that is, the result will not be recalculated.

[0133] b. If the statistics of the entire table are successful and the statistics of the storage time increment rule fail, then:

[0134] i. Full amount = the total amount of data that has been successfully counted in the current batch;

[0135] ii. Increment = the total amount of data successfully counted in the current batch - the total amount of data successfully counted last time. If it is a negative number, it is 0;

[0136] iii. Deleted amount = the total amount of data successfully counted last time - the total amount of data successfully counted in the current batch. If it is a negative number, it is 0.

[0137] c. If the statistics of the entire table and the incremental rules of storage time are both successful, then:

[0138] i. Full amount = the total amount of data that has been successfully counted in the current batch;

[0139] ii. Increment = the result calculated by "warehouse entry time increment rule";

[0140] iii. Deleted amount = the total amount of data successfully counted last time + the current increment + the current total amount. If it is a negative number, it is set to 0.

[0141] The implementation flow chart of the process statistics of this embodiment is as follows: Figure 3 As shown, the following takes Hive->Hive as an example to introduce the implementation steps:

[0142] Step 1: Build a data access or data processing task and enable the statistics configuration. If the statistics configuration is enabled, proceed to step 2; if the statistics configuration is not enabled, enable the task and record the execution log before ending the process.

[0143] Step 2: Identify the operators, including source nodes, operator nodes, and target nodes. The specific process is as follows: Figure 4 As shown:

[0144] a) First obtain the hive component environment variables and other information.

[0145] b) Define the statistical indicator group metricGroup. MetricManage is used to manage the indicator statistics monitoring. The statistical indicators are obtained through the MetricConstants definition, and the operator node indicators are identified and defined. The metricGroup monitoring group is obtained through MetricManage. The metricGroup is used to organize and group monitoring indicators, support hierarchical creation to form a tree structure, define monitoring indicators of different granularities, and implement statistical monitoring of each granularity.

[0146] c) Define specific grouping indicators, i.e. input and output indicators such as inCount / outCount.

[0147] d) Implement batch and stream processing according to different processing flows, and implement corresponding read and write statistical operations.

[0148] e) After the task is enabled, the statistical indicator content is automatically pushed to the operation monitoring tool prometheus.

[0149] f) Through Prometheus, the logs are parsed and pushed to ES according to source->operator->target.

[0150] g) Finally, the ES data is processed and output to the relational database MYSQL to provide data support for subsequent accuracy comparison with component statistics. The MYSQL table structure is shown in Table 1 below.

[0151]

[0152] Table 1

[0153] Further references Figure 5 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a system, which is similar to Figure 1 Corresponding to the method embodiment shown, the system can be specifically applied to various electronic devices.

[0154] In a second aspect, the embodiment of the present invention also discloses a massive data statistics system based on multi-dimensional deep strategy optimization, such as Figure 5 As shown, the system includes: a component statistics module 51, a process statistics module 52 and a statistics rule configuration module 53.

[0155] In a specific embodiment, a component statistics module 51 is configured to perform data volume statistics for tables under different components, wherein the components include relational databases, distributed computing storage components, document databases and message queues, full-text databases, key-value databases, and unstructured and semi-structured components. Different statistical rules are used according to the component type. Component statistics include full table statistics, partition statistics, and storage time statistics. Process statistics are configured for unstructured and semi-structured components. A process statistics module 52 is configured to generate input and output quantities according to the operation status of each node during the execution of the process statistics task.

[0156] The statistical rule configuration module 53 is configured to customize the statistical rules for different components, and complete the statistics of the data volume by combining the component statistics and the process statistics; wherein, the relational database configures the full table statistics, the storage time statistics and the process statistics; if the distributed computing storage component is a partitioned table, the full table statistics, the partition statistics and the process statistics are configured; if it is a non-partitioned table, the full table statistics, the storage time statistics and the process statistics can be performed; the document database and the message queue configure the full table statistics, the storage time statistics and the process statistics; the full-text database configures the full table statistics and the process statistics; the key-value database configures the full table statistics, the storage time statistics and the process statistics.

[0157] The functions of the above modules correspond to the methods and will not be repeated here.

[0158] The embodiment of the present invention is mainly based on the statistical capabilities of the components themselves, and combined with the data volume changes generated by the continuous operation during the task execution process, so as to realize the data volume statistics of semi-structured and unstructured data in the big data scenario, and use the flexible configuration of statistical rules and scheduling. It can not only realize fast batch configuration, but also make the completeness and accuracy of the physical table data volume statistics, further improve the value of data indicators, and provide data support for decision makers.

[0159] Reference below Figure 6, which shows an electronic device (eg Figure 1 A schematic diagram of the structure of a computer device 600 (a server or terminal device as shown). Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0160] like Figure 6 As shown, the computer device 600 includes a central processing unit (CPU) 601 and a graphics processing unit (GPU) 602, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 603 or a program loaded from a storage part 609 to a random access memory (RAM) 604. In the RAM 604, various programs and data required for the operation of the device 600 are also stored. The CPU 601, GPU 602, ROM 603 and RAM 604 are connected to each other through a bus 605. An input / output (I / O) interface 606 is also connected to the bus 605.

[0161] The following components are connected to the I / O interface 606: an input section 607 including a keyboard, a mouse, etc.; an output section 608 including a display such as a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 609 including a hard disk, etc.; and a communication section 610 including a network interface card such as a LAN card, a modem, etc. The communication section 610 performs communication processing via a network such as the Internet. A drive 611 may also be connected to the I / O interface 606 as needed. A removable medium 612, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 611 as needed so that a computer program read therefrom is installed into the storage section 609 as needed.

[0162] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 610, and / or installed from the removable medium 612. When the computer program is executed by the central processing unit (CPU) 601 and the graphics processing unit (GPU) 602, the above-mentioned functions defined in the method of the present invention are executed.

[0163] It should be noted that the computer-readable medium described in the present invention may be a computer-readable signal medium or a computer-readable medium or any combination of the above two. The computer-readable medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor device, apparatus or device, or any combination of the above. More specific examples of computer-readable media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution device, apparatus or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable medium that can send, propagate or transmit a program for use by or in conjunction with an instruction execution device, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0164] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0165] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the device, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based device that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0166] The modules involved in the embodiments of the present invention may be implemented in software or hardware, and the modules described may also be arranged in a processor.

[0167] As another aspect, the present invention further provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by the electronic device, the electronic device: executes the method and steps described in the first aspect.

[0168] The above description is only a preferred embodiment of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) to form a technical solution.

Claims

1. A massive data statistics method based on multi-dimensional deep strategy optimization, characterized in that: The method comprises the following steps: Data volume statistics are collected for tables under different components, including relational databases, distributed computing storage components, document databases and message queues, full-text databases, key-value databases, and unstructured and semi-structured components. Different statistical rules are used according to the component type. Component statistics include full table statistics, partition statistics, and storage time statistics, as well as unstructured and semi-structured component configuration process statistics. During the task execution of process statistics, input and output quantities are generated according to the operation status of each node; Customize statistical rules for different components, and complete data volume statistics by combining the component statistics and the process statistics; wherein, the relational database is configured with full-table statistics, storage time statistics and process statistics; if the distributed computing storage component is a partitioned table, full-table statistics, partition statistics and process statistics are configured; if it is a non-partitioned table, full-table statistics, storage time statistics and process statistics can be performed; document databases and message queues are configured with full-table statistics, storage time statistics and process statistics; full-text databases are configured with full-table statistics and process statistics; key-value databases are configured with full-table statistics, storage time statistics and process statistics.

2. The method for calculating the mass data statistics based on multi-dimensional deep strategy optimization according to claim 1 is characterized in that: The component statistics are used to count the complexity of scheduling and data processing, including: Initialize the default statistical rules of the table and judge their rationality and completeness according to the default statistical rules of different component types. If the rules are incomplete, further determine whether to initialize statistics. If in the initialization state, clear all current logs that are older than this statistical time and execute the next step. Otherwise, terminate the statistics. Determine whether the table definition information is complete, that is, whether the table structure information is complete. If not, end the statistics, otherwise proceed to the next step; Determine whether there is a running task blocking. If so, wait until the task is completed before ending the statistics. Otherwise, proceed to the next step. If there is no task blocking, further determine whether the current table is only configured with storage time statistics and whether it is the first statistics. If so, the system automatically completes the statistics of the entire table. Otherwise, proceed to the next step. Further determine the statistical type and process the three types separately: full table statistics, partition statistics, and storage time statistics.

3. The method for calculating the mass data statistics based on multi-dimensional deep strategy optimization according to claim 1 is characterized in that: The implementation flow of the process statistics specifically includes: When building data access or processing tasks, determine whether to enable statistical configuration. If so, proceed to the next step. Otherwise, only enable the task and record the execution log before ending the process. The source node, operator node and target node of the operator are marked, including obtaining component environment variables, defining statistical indicator groups and specific indicators, implementing batch or streaming statistical operations, pushing indicators to prometheus, parsing logs to ES and processing and outputting them to MYSQL, providing data support for comparison with component statistical accuracy. The MYSQL table structure contains fields such as task-related information and input and output quantities.

4. The method for calculating the mass data statistics based on multi-dimensional deep strategy optimization according to claim 2 is characterized in that: The three types of statistics, namely, full table statistics, partition statistics, and storage time statistics, are processed separately, including: If the statistics type is full table statistics, the default is to count the entire table directly (*) and proceed to the next step; If the statistics type is zonal statistics, then: Get all partitions of the table by calling the abstract layer interface; Obtain the range of all partitions of the table that have been successfully counted and have not been deleted from the statistics log; Compare the differences between the latest partition and the partition in the statistical log, and identify which are historical partitions and newly added partitions; if they are newly added partitions, add them to the statistical task; if the historical partition still exists in the partition list and there is data in the log, no processing is done; if the historical partition is not in the partition list, the partition is marked as deleted; For cross-partition processing, the partition rules are obtained according to T-1 time, and the partition name is converted to determine whether the partition is in the partition list. If it is, it is added to the statistics. If it is not, it is directly recorded as 0 and the partition statistics task is marked as failed. Continue to determine whether there are partitions that have not been counted. If so, further analyze whether the statistics are asynchronous; If the statistics type is inventory entry time statistics, obtain the current statistics log, encapsulate the inventory entry time query condition as 00:00:00-23:59:59, and proceed to the next step.

5. The method for calculating the mass data statistics based on multi-dimensional deep strategy optimization according to claim 2 is characterized in that: Also includes: Determine whether asynchronous statistics are needed for the table. If asynchronous statistics are needed, request the asynchronous statistics interface of the unified data operation layer, record the unique identifier returned by the abstract layer and encapsulate the main log record, and finally complete the abstract layer callback interface; if asynchronous statistics are not needed, directly call the abstract layer statistics interface and encapsulate the statistics log to obtain the log details; Add or update statistical logs based on the results of the previous step and output atomic indicators.

6. The method for calculating the mass data statistics based on multi-dimensional deep strategy optimization according to claim 5 is characterized in that: Also includes: Derived indicators are calculated based on the atomic indicators of the statistical log. Different calculation rules are used to calculate the full amount, incremental amount, and deletion amount results according to different situations of full table statistics, partition statistics, and the combination of full table statistics and storage time statistics. The calculation rules are as follows: For full table statistics, if the full table statistics fails, the entire calculation result remains unchanged; If statistics of the entire table are successful and statistics of the incremental rule for storage time fail, the full amount is the full amount of data that has been successfully counted in the current batch; the incremental amount is the full amount of data that has been successfully counted in the current batch minus the full amount of data that was successfully counted last time. If it is a negative number, it is 0; the deleted amount is the full amount of data that was successfully counted last time minus the full amount of data that was successfully counted in the current batch. If it is a negative number, it is 0; For statistics on the entire table, first clarify the scheduling scope: the first scheduling includes all partitions in the partition list, and the Nth scheduling includes all partitions in the range from the last scheduling date to T and the partitions that were not successfully run in the previous batch of statistics logs, where N is greater than or equal to 2; Next, confirm the calculation rules: If all partition statistics fail, the entire calculation result remains unchanged; however, if there is a successful partition statistics log, the total amount is the sum of the latest successful data count of each partition corresponding to all partitions not marked for deletion; the increment is the sum of the successful partition data count of this batch minus the sum of the latest successful data count of the partitions successfully counted in this batch corresponding to the historical batch, and 0 is not taken; the deleted amount is the partition that was successfully run in the original log but is no longer in the latest partition list compared with the partition list of this batch; For statistics on the time of adding a full table to the database, the incremental rule for the database entry time is the increment within the time range from the start time of the last successful statistics of the full table to the time when the statistics are currently initiated. If there is no log of the last successful statistics of the full table, the time less than the current statistical initiation time is used as the basis, which is divided into the following situations: If the statistics of the entire table fail, the entire calculation result remains unchanged; If statistics of the entire table are successful and statistics of the incremental rule for storage time fail, the full amount is the full amount of data that has been successfully counted in the current batch; the incremental amount is the full amount of data that has been successfully counted in the current batch minus the full amount of data that was successfully counted last time. If it is a negative number, it is 0; the deleted amount is the full amount of data that was successfully counted last time minus the full amount of data that was successfully counted in the current batch. If it is a negative number, it is 0; If both the full table statistics and the storage time increment rule are successful, the full amount is the full amount of data that has been successfully counted in the current batch; the increment is the result calculated by the storage time increment rule; the deleted amount is the full amount of data that was successfully counted last time plus the current increment plus the current full amount. If it is a negative number, it is set to 0.

7. The method for calculating the mass data statistics based on multi-dimensional deep strategy optimization according to claim 2 is characterized in that: The default statistical rules for the initialization table and the constraints for judging their rationality and completeness are the default full-table statistics for relational databases, document databases, message queues, full-text databases and key-value databases, and the default partition statistics for distributed partition tables.

8. A massive data statistics system based on multi-dimensional deep strategy optimization, characterized in that: include: The component statistics module is configured to perform data volume statistics for tables under different components, including relational databases, distributed computing storage components, document databases and message queues, full-text databases, key-value databases, and unstructured and semi-structured components. Different statistical rules are used according to the component type. Component statistics include full table statistics, partition statistics, and storage time statistics, as well as statistics on the configuration process of unstructured and semi-structured components. A process statistics module is configured to generate input and output quantities according to the operation status of each node during the execution of the process statistics task; The statistical rule configuration module is configured to customize the statistical rules for different components, and complete the statistics of the data volume by combining the component statistics and the process statistics; wherein, the relational database configures the full table statistics, the storage time statistics and the process statistics; if the distributed computing storage component is a partitioned table, the full table statistics, the partition statistics and the process statistics are configured; if it is a non-partitioned table, the full table statistics, the storage time statistics and the process statistics can be performed; the document database and the message queue configure the full table statistics, the storage time statistics and the process statistics; the full-text database configures the full table statistics and the process statistics; the key-value database configures the full table statistics, the storage time statistics and the process statistics.

9. An electronic device, comprising: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images