A monitoring method, device and storage medium for big data fusion process

Abstract

The present application proposes a monitoring method, device and storage medium for a big data fusion process, which can ensure that data can support high-quality business decisions and analysis, and avoid losses and risks caused by data quality problems. The method includes: based on the business scenario, determining a target hive table that needs to be monitored, and creating and initializing a data monitoring reference table for the target hive table; before the big data fusion process applies the target hive table for data processing, renaming the target hive table to modify it into a process hive result table; performing time verification and data volume verification on the process hive result table based on the data monitoring reference table; when it is determined that the process hive result table passes the time verification and data volume verification, marking it as normal, and executing the business logic processing process in the big data fusion process; when it is determined that the process hive result table fails to pass the time verification or fails to pass the data volume verification, marking it as abnormal, and truncating the business logic processing process.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a monitoring method, device and storage medium for a big data fusion process. Background Art

[0002] In order to accelerate the implementation of the cloud-to-digital transformation strategy, assist enterprises in high-quality development, fully promote the digital transformation of cloud network operations, promote the improvement of cloud network value and operational efficiency, and accelerate the construction of a world-class enterprise, we will take cloud network operation self-intelligence as the starting point, carry out the digital transformation of cloud network operations, draw up a self-intelligence blueprint, and compile implementation plans for the three major projects of service digitalization, maintenance digitalization, and cost digitalization, as well as two basic guarantee plans, data governance and Kunlun platform, to guide the construction of self-intelligence and digital transformation capabilities across the entire network. On this basis, we will carry out in-depth development and continuous evolution and improvement of the self-intelligence level and update and improve it.

[0003] Based on this, it is necessary to create digital applications based on big data, integrate cloud and network data, unify the data base, meet analytical businesses such as batch processing, promote the implementation of cross-industry and cross-domain data element circulation cases, and carry out unified benefit evaluation in accordance with the business goals of improving quality, reducing costs and increasing efficiency, to help enterprises achieve high-quality development.

[0004] However, in the current process of big data fusion, billions of massive data are collected, and these data come from multiple sources and in various formats, showing explosive growth. The quality of the data source will affect the accuracy of the fused data results, which puts higher requirements on data quality. The traditional common data fusion process can no longer meet the business data processing requirements. Summary of the invention

[0005] The present application provides a monitoring method, device and storage medium for a big data fusion process to solve the problems existing in the related technologies. The technical solution is as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for monitoring a big data fusion process, including:

[0007] Based on the business scenario, determine the target hive table that needs to be monitored, and create and initialize a data monitoring benchmark table for the target hive table;

[0008] Before the target hive table is applied to the big data fusion process for data processing, a rename operation is performed on the target hive table to modify the target hive table into a process hive result table;

[0009] Performing time verification and data volume verification on the process hive result table based on the data monitoring benchmark table;

[0010] When it is determined that the process hive result table passes the time verification and the data volume verification, the process hive result table is marked as normal, and the business logic processing flow in the big data fusion process is executed;

[0011] When it is determined that the process hive result table fails to pass the time verification or the data volume verification, the process hive result table is marked as abnormal, and the business logic processing flow is interrupted.

[0012] In one implementation, creating and initializing a data monitoring reference table for the target hive table includes:

[0013] Based on the business requirements of the business scenario, the target data volume of the target hive table is evaluated, and the target data volume is used as the benchmark data volume of the target hive table;

[0014] Obtain yesterday's timestamp and current timestamp of the target hive table, and use the yesterday's timestamp and current timestamp as initialization variables of the target hive table;

[0015] Using a target editing tool to create the data monitoring reference table, the target editing tool is an editing tool for a database client linked to Hive;

[0016] After the reference data volume and the initialization variables are stored in the data monitoring reference table, the data monitoring reference table is initialized.

[0017] In one embodiment, the data monitoring benchmark table includes the benchmark data volume and yesterday's timestamp of the target hive table; and performing time verification and data volume verification on the process hive result table based on the data monitoring benchmark table includes:

[0018] Get the last update time of the hive result table of the process;

[0019] Performing time verification on the process hive result table based on the last update time and the yesterday's timestamp in the data monitoring reference table;

[0020] When it is determined that the process hive result table passes the time verification, the current data volume of the process hive result table is obtained;

[0021] The data volume of the process hive result table is verified based on the current data volume and the benchmark data volume in the data monitoring benchmark table.

[0022] In one implementation, obtaining the last update time of the target hive table includes:

[0023] Call Hive's metadata API to query Hive's metadata;

[0024] Call the show table extended command to obtain the lastUpdateTime parameter of the process hive result table from the metadata to obtain the last update time.

[0025] In one embodiment, time verification of the process hive result table based on the last update time and the yesterday's timestamp in the data monitoring reference table includes: comparing the last update time with the yesterday's timestamp to time verify the process hive result table;

[0026] Verifying the data volume of the process hive result table based on the current data volume and the benchmark data volume in the data monitoring benchmark table includes: comparing the difference between the current data volume and the benchmark data volume with a specified error threshold to verify the data volume of the process hive result table.

[0027] In one embodiment, determining that the process hive result table passes the time verification and the data volume verification includes: when it is determined that the last update time is less than the yesterday's timestamp, and the difference is less than or equal to the error threshold, determining that the process hive result table passes the time verification and the data volume verification;

[0028] Determining that the process hive result table fails the time verification or fails the data volume verification includes: when it is determined that the last update time is greater than or equal to the yesterday's timestamp, determining that the process hive result table fails the time verification; or when it is determined that the difference is greater than the error threshold, determining that the process hive result table fails the data volume verification.

[0029] In one embodiment, the method further comprises:

[0030] Record the relevant verification process information of the hive result table of the process;

[0031] The relevant verification process information is recorded in the verification result table.

[0032] In a second aspect, the embodiment of the present application further provides a monitoring device for a big data fusion process, including:

[0033] A creation unit, used to determine a target hive table to be monitored based on a business scenario, and to create and initialize a data monitoring reference table for the target hive table;

[0034] A modification unit, used for renaming the target hive table before applying the target hive table to perform data processing in the big data fusion process, and modifying the target hive table to a process hive result table;

[0035] A verification unit, configured to perform time verification and data volume verification on the process hive result table based on the data monitoring reference table;

[0036] An execution unit, used for marking the process hive result table as normal when the verification unit determines that the process hive result table passes the time verification and the data volume verification, and executing the business logic processing flow in the big data fusion process;

[0037] The truncation unit is used to mark the process hive result table as abnormal and truncate the business logic processing flow when the verification unit determines that the process hive result table fails to pass the time verification or the data volume verification.

[0038] In one implementation, when the creation unit is used to create and initialize a data monitoring reference table for the target hive table, it is specifically used to:

[0039] Based on the business requirements of the business scenario, the target data volume of the target hive table is evaluated, and the target data volume is used as the benchmark data volume of the target hive table;

[0040] Obtain yesterday's timestamp and current timestamp of the target hive table, and use the yesterday's timestamp and current timestamp as initialization variables of the target hive table;

[0041] Using a target editing tool to create the data monitoring reference table, the target editing tool is an editing tool for a database client linked to Hive;

[0042] After the reference data volume and the initialization variables are stored in the data monitoring reference table, the data monitoring reference table is initialized.

[0043] In one embodiment, the data monitoring reference table includes the reference data volume and yesterday's timestamp of the target hive table; when the verification unit is used to perform time verification and data volume verification on the process hive result table based on the data monitoring reference table, it is specifically used to:

[0044] Get the last update time of the hive result table of the process;

[0045] Performing time verification on the process hive result table based on the last update time and the yesterday's timestamp in the data monitoring reference table;

[0046] When it is determined that the process hive result table passes the time verification, the current data volume of the process hive result table is obtained;

[0047] The data volume of the process hive result table is verified based on the current data volume and the benchmark data volume in the data monitoring benchmark table.

[0048] In one implementation, when the verification unit is used to obtain the last update time of the target hive table, it is specifically used to:

[0049] Call Hive's metadata API to query Hive's metadata;

[0050] Call the show table extended command to obtain the lastUpdateTime parameter of the process hive result table from the metadata to obtain the last update time.

[0051] In one embodiment, when the verification unit is used to perform time verification on the process hive result table based on the last update time and the yesterday's timestamp in the data monitoring reference table, it is specifically used to: compare the last update time with the yesterday's timestamp to perform time verification on the process hive result table;

[0052] When the verification unit is used to verify the data volume of the process hive result table based on the current data volume and the benchmark data volume in the data monitoring benchmark table, it is specifically used to: compare the difference between the current data volume and the benchmark data volume with a specified error threshold to verify the data volume of the process hive result table.

[0053] In one embodiment, when the verification unit is used to determine whether the process hive result table passes the time verification and the data volume verification, it is specifically used to: when it is determined that the last update time is less than the yesterday's timestamp, and the difference is less than or equal to the error threshold, determine that the process hive result table passes the time verification and the data volume verification;

[0054] When the verification unit is used to determine that the process hive result table fails the time verification or fails the data volume verification, it is specifically used to: when it is determined that the last update time is greater than or equal to the yesterday's timestamp, determine that the process hive result table fails the time verification; or when it is determined that the difference is greater than the error threshold, determine that the process hive result table fails the data volume verification.

[0055] In one embodiment, the monitoring device for the big data fusion process further includes a recording unit, and the recording unit is used to:

[0056] Record the relevant verification process information of the hive result table of the process;

[0057] The relevant verification process information is recorded in the verification result table.

[0058] In a third aspect, an embodiment of the present application further provides a computer device, comprising: a memory and a processor, wherein the memory stores instructions, and the instructions are loaded and executed by the processor to implement a method in any one of the above-mentioned aspects, wherein the memory and the processor communicate with each other through an internal connection path.

[0059] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program runs on a computer, the method in any one of the above-mentioned embodiments is implemented.

[0060] The advantages or beneficial effects of the above technical solution include at least:

[0061] This application verifies the data volume and data time before applying the data source table (i.e., the target hive table) in the big data fusion process, and can parse relevant indicators (such as baseline data volume and yesterday's timestamp, etc.) to make abnormal judgments. It can facilitate monitoring of the progress of relevant data processing flows, and truncate the business logic processing flow in the big data fusion process for abnormal situations. It can reduce the pressure of data operation and maintenance, reduce the risk of rework data reprocessing or distorted data being applied due to data anomalies, and make data processing reliable and stable, and the data has reliable and available disaster recovery capabilities, thereby ensuring that the data can support high-quality business decisions and analysis, and avoid losses and risks caused by data quality issues.

[0062] In practical application, this application can be applied to a variety of business scenarios of big data fusion, and will not affect the data processing flow of normal business, while ensuring that business data will not be incorrect or unstable due to large fluctuations in source data, especially the guarantee mechanism applied to massive data processing. That is, the monitoring method of the big data fusion process provided in the embodiment of this application can not only be applied to the big data processing of cloud network fusion data, but also can be extended to other scenarios with big data hive and large data volume processing.

[0063] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present application will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in the present application and should not be regarded as limiting the scope of the present application.

[0065] Figure 1 A flowchart of a method for monitoring a big data fusion process provided in an embodiment of the present application;

[0066] Figure 2 A flowchart of another method for monitoring a big data fusion process provided in an embodiment of the present application;

[0067] Figure 3 A structural block diagram of a monitoring device for a big data fusion process provided in an embodiment of the present application;

[0068] Figure 4 A structural block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the present application. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.

[0070] Before introducing the embodiments of the present application, some terms in the present application are first explained to facilitate understanding by those skilled in the art.

[0071] MD (Metadata) refers to data about data, which describes the characteristics, structure, location, permissions and other related information of the data. In this article, MD is the data describing the hive table.

[0072] DL (Data Lake) is an architecture for storing large amounts of raw data. The data stored in DL can be structured, semi-structured, or unstructured. The main feature of a data lake is that it can store massive amounts of data at a low cost, and does not perform predefined schemas or structured processing when ingesting data. Instead, it uses a "schema-on-read" approach to dynamically parse the data structure according to specific needs when reading data. In terms of technical implementation, it usually relies on ecological tools such as Hadoop, Spark, Hive, and Flink.

[0073] Hive is a data warehouse tool based on Hadoop, which is used for data extraction, transformation and loading. It is a mechanism for storing, querying and analyzing large-scale data stored in Hadoop. It can map structured data files into a database table and provide SQL query functions, which can convert SQL statements into MapReduce tasks for execution.

[0074] Various technical solutions provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0075] Figure 1 FIG. 1 is a flow chart showing a method for monitoring a big data fusion process according to an embodiment of the present application. Figure 1 As shown, the method may include the following steps:

[0076] S110: Based on the business scenario, determine the target hive table that needs to be monitored, and create and initialize a data monitoring benchmark table for the target hive table.

[0077] In one implementation, before monitoring the big data fusion process, it is first necessary to determine the business process monitoring target. For example, after determining the business scenario of big data fusion, it is possible to clarify which hive tables need to be monitored in the business scenario, and thus determine the business process monitoring target (i.e., the target hive table).

[0078] In one implementation, the implementation process of creating and initializing a data monitoring reference table for a target hive table may include the following steps:

[0079] S111. Based on the business requirements of the business scenario, the target data volume of the target hive table is evaluated, and the target data volume is used as the benchmark data volume of the target hive table.

[0080] During specific implementation, users can evaluate the target data volume of the target hive table based on the business requirements of the business scenario.

[0081] The target data volume can be understood as the amount of data that the target hive table needs to process (including extraction, conversion, loading, storage, etc.) based on the business needs.

[0082] S112: Get yesterday's timestamp and current timestamp of the target hive table, and use yesterday's timestamp and current timestamp as initialization variables of the target hive table.

[0083] In specific implementation, step S112 can be executed in a Linux shell environment.

[0084] S113. Use the target editing tool to create a data monitoring benchmark table.

[0085] In a specific implementation, the target editing tool may be an editing tool (ie, an editor) of a database client linked to Hive.

[0086] In specific implementation, the data monitoring benchmark table can be created by the user in the target editing tool.

[0087] As an example, the data monitoring base table may be dws_kz_object_table_base_nums.

[0088] In the embodiment of the present application, by executing step S113, a data monitoring benchmark can be created for the corresponding accessed database.

[0089] As an example, creating a data monitoring benchmark table can be achieved through the following code:

[0090]

[0091] S114: After storing the reference data volume and the initialization variable in the data monitoring reference table, the data monitoring reference table is initialized.

[0092] In a specific implementation, the reference data volume may be stored in a core field of a data monitoring reference table, and the initialization variable may be stored in a maintenance auxiliary field of the data monitoring reference table.

[0093] In specific implementation, the core fields of the data monitoring benchmark table may include the table name of the target hive table in addition to the benchmark data volume. Also, the maintenance auxiliary fields of the data monitoring benchmark table may include the creator, creation time, modifier, modification time, etc. of the target hive table in addition to the initialization variable.

[0094] In a specific implementation, by initializing the data monitoring reference table, the data monitoring reference table can be used as a data monitoring reference.

[0095] As an example, initializing the data monitoring benchmark table can be implemented by the following code:

[0096]

[0097] As an example, the relevant benchmark data (such as benchmark data volume, yesterday's timestamp, current timestamp, etc.) of the data monitoring benchmark table can be initialized by the following code:

[0098]

[0099] In the embodiment of the present application, by executing step S110, after determining the business scenario, the benchmark value of the business process monitoring target can be initialized, and the target hive table to be monitored and its data monitoring benchmark can be clearly identified.

[0100] S120. Before the target hive table is applied to the big data fusion process for data processing, a rename operation is performed on the target hive table to modify the target hive table into a process hive result table.

[0101] Since the current data in the target hive table may be historically calculated data and differ from the results of this batch of calculations, the existing target hive table needs to be deleted first. At the same time, in order to reference the calculated results and provide them for subsequent processes, the database table renaming technology needs to be used to rename the process table of this calculation corresponding to the target hive table to the process result table, so that the subsequent processes can continue as the results of this calculation.

[0102] Based on this, in an embodiment of the present application, the target hive table can be renamed by using the database table name update technology, the target data result table of the target hive table can be suffixed with _pro, and then the target hive table can be modified to a process hive result table.

[0103] In the embodiment of the present application, by executing step S120, it is possible to easily distinguish the original data results and the process data results of the target hive table, thereby avoiding affecting the business data of the existing process.

[0104] S130, based on the data monitoring benchmark table, perform time verification and data volume verification on the process hive result table. When it is determined that the process hive result table passes the time verification, perform data volume verification, otherwise, execute step S150; when it is determined that the process hive result table passes the data volume verification, execute step S140, otherwise, execute step S150.

[0105] In one implementation, before the target hive table is applied to the big data fusion process for data processing, a monitoring logic processing flow can be compiled for the target hive table. The monitoring logic processing flow is used to perform time verification and data volume verification on the process hive result table corresponding to the target hive table based on the data monitoring benchmark table.

[0106] In one implementation, a shell node may be added and used as a process control node. The process control node may be used to execute the monitoring logic processing flow, thereby facilitating the monitoring process to change the hive result table.

[0107] In one implementation, the implementation process of performing time verification and data volume verification on the process hive result table based on the data monitoring reference table (i.e., the monitoring logic processing flow) may include the following steps:

[0108] S131. Get the last update time of the hive result table.

[0109] In specific implementation, the implementation process of step S131 may include the following steps:

[0110] S1311. Call the Hive metadata API to query Hive metadata.

[0111] As an example, a shell node can be added and used as a point for obtaining source table information. Afterwards, the Hive metadata API can be called through the point for obtaining source table information to query Hive metadata.

[0112] API refers to the Application Programming Interface. Hive's metadata API allows programs to directly query Hive's metadata.

[0113] S1312. Call the show table extended command to obtain the lastUpdateTime parameter of the process hive result table from the metadata to obtain the last update time.

[0114] As an example, the show table extended command is a hive client command (such as show table extended like 'tablename').

[0115] As an example, the show table extended command can be called through the source table information acquisition point to obtain the lastUpdateTime parameter of the process hive result table from the metadata to obtain the last update time.

[0116] For example, because the queried metadata includes many parameters, based on this, the show table extended command can be called through the source table information point to extract the lastUpdateTime parameter of the process hive result table from the metadata using a string interception method. The string interception method can use the shell command echo${string#*lastUpdateTime:} to perform string interception processing. By extracting the data value corresponding to the lastUpdateTime parameter, the last update time can be obtained.

[0117] As an example, in order to consider the accuracy of the data change time, when extracting the data value corresponding to the lastUpdateTime parameter, the data value can be converted into a timestamp in seconds, and then the converted data value can be formatted using date. In other words, the last update time is in seconds, and its format can be "%Y-%m-%d %H:%M:%S".

[0118] In the embodiment of the present application, by executing step S1311 and step S1312, the last update time of the process hive result table (ie, the latest update time of the data source table data) can be obtained through the hive metadata mechanism.

[0119] In specific implementation, after obtaining the last update time, the last update time may be written into the table (dws_kz_procedure_monitor_record).

[0120] S132: Perform time verification on the process hive result table based on the last update time and yesterday's timestamp in the data monitoring reference table.

[0121] In specific implementation, the last update time can be read from the table (dws_kz_procedure_monitor_record). And the yesterday's timestamp can be read from the data monitoring benchmark table. Afterwards, the process hive result table can be time-verified by comparing the last update time with the yesterday's timestamp.

[0122] In the embodiment of the present application, by executing step S132, the data update status of the hive result table of the monitoring process can be facilitated.

[0123] S133. When it is determined that the process hive result table passes the time verification, the current data volume of the process hive result table is obtained.

[0124] In specific implementation, when determining whether the process hive result table has passed the time verification, the monitored object, namely the process hive result table, can be read, and then the address width table calculation can be performed on the process hive result table to obtain the current data volume of the process hive result table.

[0125] S134: Verify the data volume of the process hive result table based on the current data volume and the benchmark data volume in the data monitoring benchmark table.

[0126] In specific implementation, the benchmark data volume can be read from the data monitoring benchmark table. The benchmark data volume and the above-mentioned yesterday's timestamp can be read from the data monitoring benchmark table at the same time, or can be read out in sequence, which is not limited in this embodiment of the application.

[0127] In a specific implementation, the data volume of the process hive result table can be verified by comparing the difference between the current data volume and the benchmark data volume with a specified error threshold.

[0128] The error threshold can be set according to actual needs, and the present application embodiment does not limit this. The error threshold is set to 5% as an example.

[0129] That is, in the embodiment of the present application, by executing step S134, the data volume fluctuation of the process hive result table can be monitored according to whether the data volume fluctuation of the process hive result table is within a specified error range, such as whether it is within 5%.

[0130] In one implementation, when it is determined that the last update time is less than the yesterday's timestamp and the difference is less than or equal to the error threshold, it can be determined that the process hive result table passes the time verification and the data volume verification.

[0131] In one embodiment, when it is determined that the last update time is greater than or equal to yesterday's timestamp, it is determined that the process hive result table fails the time verification; or, when it is determined that the difference is greater than the error threshold, it is determined that the process hive result table fails the data volume verification.

[0132] In the implementation of this application, by executing step S130, before the target hive table is applied to the big data fusion process for data processing, and before the source data in the target hive table is referenced, it can be verified and judged (such as data time and data volume verification and judgment). In this way, the hive table metadata analysis related indicators can be used to make abnormal judgments, thereby ensuring that the data can support high-quality business decisions and analysis, and avoiding losses and risks caused by data quality issues.

[0133] S140, marking the hive result table as normal, and executing the business logic processing flow in the big data fusion process.

[0134] In one implementation, a normal label may be added to the process hive result table to mark the process hive result table as normal.

[0135] S150, marking the hive result table of the process as abnormal, and terminating the business logic processing flow.

[0136] In one implementation, an abnormal label may be added to the process hive result table to mark the process hive result table as abnormal.

[0137] In the specific implementation, in step S150, an exception can be thrown by calling a shell script. For example, exit1 can be executed through the characteristics of the shell to output exception information to terminate the operation of subsequent nodes. In this way, the business logic processing flow can be interrupted.

[0138] In an embodiment of the present application, by executing step S140 and step S150, when an abnormal situation occurs in the target hive table, the subsequent business logic processing flow in the big data fusion process can be truncated, and the subsequent business logic processing flow in the big data fusion process can be limited to only apply the target hive table whose table data update time is within 1 day and whose data volume offset is within the specified error threshold to support business production, which can reduce the pressure of data operation and maintenance, reduce the risk of rework data reprocessing or data distortion being applied due to data anomalies, and make data processing reliable and stable, and the data has reliable and available disaster recovery capabilities, thereby ensuring that the data can support high-quality business decisions and analysis, and avoid losses and risks caused by data quality issues.

[0139] In an applicable scenario provided in an embodiment of the present application, combined with Figure 1 and Figure 2 As shown, the monitoring method of the big data fusion process provided in the embodiment of the present application may also include the following steps:

[0140] S160, record the relevant verification process information of the process hive result table.

[0141] In one embodiment, the relevant verification process information may include, but is not limited to: batch number, date, process name, node name, monitoring object table name, source table, actual number of monitoring records, standard number of monitoring records, actual number of source table records, standard number of source table records, audit results, source table update time. The batch number is used to uniquely identify the current batch calculation of the process hive result table. In other words, the batch number is the unique identification information of the current batch calculation of the process hive result table.

[0142] As an example, the current 10-digit timestamp may be used as the batch number.

[0143] As an example, the implementation process of step S160 can be implemented by the following code:

[0144]

[0145] S170. Record the relevant verification process information into the verification result table.

[0146] As an example, the verification result table may be: dws_kz_procedure_monitor_record.

[0147] In the embodiment of the present application, by executing step S160 and step S170, relevant judgment data in the processing process of the target hive table can be recorded, the data processing status of the target hive table can be timely understood, and the data processing status of the target hive table can be easily traced, so as to help quickly locate the data processing process problems of the target hive table.

[0148] In summary, the monitoring method of the big data fusion process provided in the embodiment of the present application can parse relevant indicators (such as baseline data volume and yesterday's timestamp, etc.) to make abnormal judgments by verifying the data volume and data time before applying the data source table (i.e., the target hive table) in the big data fusion process. It can facilitate the monitoring of the progress of relevant data processing flows and truncate the business logic processing flow in the big data fusion process for abnormal situations. It can reduce the pressure of data operation and maintenance, reduce the risk of rework data reprocessing or distorted data being applied due to data anomalies, and make data processing reliable and stable, and the data has reliable and available disaster recovery capabilities, thereby ensuring that the data can support high-quality business decisions and analysis, and avoid losses and risks caused by data quality issues.

[0149] In practical applications, the monitoring method of the big data fusion process provided by the embodiment of the present application can be applied to a variety of business scenarios of big data fusion, and does not affect the data processing process of normal business, while ensuring that the business data will not be incorrect and unstable due to large fluctuations in source data, especially the guarantee mechanism applied to massive data processing. That is, the monitoring method of the big data fusion process provided by the embodiment of the present application can not only be applied to the big data processing of cloud network fusion data, but also can be extended to other scenarios with big data hive and large data volume processing.

[0150] Figure 3 A structural block diagram of a monitoring device for a big data fusion process according to an embodiment of the present application is shown.

[0151] like Figure 3 As shown, the device may include:

[0152] A creation unit 210 is used to determine a target hive table to be monitored based on a business scenario, and to create and initialize a data monitoring reference table for the target hive table;

[0153] The modification unit 220 is used to rename the target hive table before the target hive table is applied to the big data fusion process for data processing, so as to modify the target hive table into a process hive result table;

[0154] A verification unit 230, configured to perform time verification and data volume verification on the process hive result table based on the data monitoring reference table;

[0155] The execution unit 240 is used to mark the process hive result table as normal when the verification unit 230 determines that the process hive result table passes the time verification and the data volume verification, and execute the business logic processing process in the big data fusion process;

[0156] The truncation unit 250 is used to mark the process hive result table as abnormal and truncate the business logic processing flow when the verification unit 230 determines that the process hive result table fails to pass the time verification or the data volume verification.

[0157] In one implementation, when the creation unit 210 is used to create and initialize a data monitoring reference table for a target hive table, it is specifically used to:

[0158] Based on the business requirements of the business scenario, evaluate the target data volume of the target hive table and use the target data volume as the benchmark data volume of the target hive table;

[0159] Get the yesterday's timestamp and current timestamp of the target hive table, and use the yesterday's timestamp and current timestamp as the initialization variables of the target hive table;

[0160] Use the target editing tool to create a data monitoring benchmark table. The target editing tool is an editing tool for the database client linked to Hive.

[0161] After storing the benchmark data volume and the initialization variables in the data monitoring benchmark table, the data monitoring benchmark table is initialized.

[0162] In one embodiment, the data monitoring reference table includes the reference data volume and yesterday's timestamp of the target hive table; when the verification unit 230 is used to perform time verification and data volume verification on the process hive result table based on the data monitoring reference table, it is specifically used to:

[0163] Get the last update time of the hive result table of the process;

[0164] Time verification is performed on the process hive result table based on the last update time and yesterday's timestamp in the data monitoring benchmark table;

[0165] When determining whether the process hive result table has passed the time verification, obtain the current data volume of the process hive result table;

[0166] Verify the data volume of the process hive result table based on the current data volume and the benchmark data volume in the data monitoring benchmark table.

[0167] In one implementation, when the verification unit 230 is used to obtain the last update time of the target hive table, it is specifically used to:

[0168] Call Hive's metadata API to query Hive's metadata;

[0169] Call the show table extended command to obtain the lastUpdateTime parameter of the process hive result table from the metadata to get the last update time.

[0170] In one embodiment, when the verification unit 230 is used to perform time verification on the process hive result table based on the last update time and yesterday's timestamp in the data monitoring reference table, it is specifically used to: compare the last update time with yesterday's timestamp to perform time verification on the process hive result table;

[0171] When the verification unit 230 is used to verify the data volume of the process hive result table based on the current data volume and the benchmark data volume in the data monitoring benchmark table, it is specifically used to: compare the difference between the current data volume and the benchmark data volume with a specified error threshold to verify the data volume of the process hive result table.

[0172] In one embodiment, when the verification unit 230 is used to determine whether the process hive result table passes the time verification and the data volume verification, it is specifically used to: when it is determined that the last update time is less than yesterday's timestamp, and the difference is less than or equal to the error threshold, determine that the process hive result table passes the time verification and the data volume verification;

[0173] When the verification unit 230 is used to determine whether the process hive result table has failed the time verification or the data volume verification, it is specifically used to: when it is determined that the last update time is greater than or equal to yesterday's timestamp, determine that the process hive result table has failed the time verification; or when it is determined that the difference is greater than the error threshold, determine that the process hive result table has failed the data volume verification.

[0174] In one embodiment, the monitoring device of the big data fusion process further includes a recording unit 260, and the recording unit 260 is used to:

[0175] Record the relevant verification process information of the hive result table;

[0176] Record relevant verification process information in the verification result table.

[0177] The functions of each unit in the monitoring device of the big data fusion process in the embodiment of the present application can be found in the corresponding description in the above method, and will not be repeated here.

[0178] Figure 4 FIG. 2 shows a block diagram of a computer device according to an embodiment of the present application. Figure 4 As shown, the computer device includes: a memory 310 and a processor 320. The memory 310 stores instructions, and the instructions are loaded and executed by the processor 320 to implement the monitoring method of the big data fusion process in the above embodiment. The number of the memory 310 and the processor 320 can be one or more.

[0179] The computer device also includes:

[0180] The communication interface 330 is used to communicate with external devices and perform data exchange transmission.

[0181] If the memory 310, the processor 320 and the communication interface 330 are implemented independently, the memory 310, the processor 320 and the communication interface 330 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0182] Optionally, in a specific implementation, if the memory 310, the processor 320 and the communication interface 330 are integrated on a chip, the memory 310, the processor 320 and the communication interface 330 can communicate with each other through an internal interface.

[0183] An embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program runs on a computer, the method provided in the embodiment of the present application is implemented.

[0184] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in a memory, so that a communication device equipped with the chip executes the method provided by the embodiment of the present application.

[0185] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the application.

[0186] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture.

[0187] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory, and may also include a non-volatile random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct rambus RAM (DR RAM).

[0188] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0189] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0190] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0191] Any process or method description in the flow chart or otherwise described herein can be understood to represent a module, fragment or portion of a code including one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0192] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with these instruction execution systems, apparatuses or devices.

[0193] It should be understood that the various parts of the present application can be implemented with hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0194] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.

[0195] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A monitoring method for a big data fusion process, characterized in that: include: Based on the business scenario, determine the target hive table that needs to be monitored, and create and initialize a data monitoring benchmark table for the target hive table; Before the target hive table is applied to the big data fusion process for data processing, a rename operation is performed on the target hive table to modify the target hive table into a process hive result table; Performing time verification and data volume verification on the process hive result table based on the data monitoring benchmark table; When it is determined that the process hive result table passes the time verification and the data volume verification, the process hive result table is marked as normal, and the business logic processing flow in the big data fusion process is executed; When it is determined that the process hive result table fails to pass the time verification or fails to pass the data volume verification, the process hive result table is marked as abnormal, and the business logic processing flow is interrupted.

2. The method according to claim 1, characterized in that Creating and initializing a data monitoring benchmark table for the target hive table includes: Based on the business requirements of the business scenario, the target data volume of the target hive table is evaluated, and the target data volume is used as the benchmark data volume of the target hive table; Obtain yesterday's timestamp and current timestamp of the target hive table, and use the yesterday's timestamp and current timestamp as initialization variables of the target hive table; Using a target editing tool to create the data monitoring reference table, the target editing tool is an editing tool for a database client linked to Hive; After the reference data volume and the initialization variables are stored in the data monitoring reference table, the data monitoring reference table is initialized.

3. The method according to claim 1, characterized in that The data monitoring benchmark table includes the benchmark data volume and yesterday's timestamp of the target hive table; The time verification and data volume verification of the process hive result table based on the data monitoring reference table includes: Get the last update time of the hive result table of the process; Performing time verification on the process hive result table based on the last update time and the yesterday's timestamp in the data monitoring reference table; When it is determined that the process hive result table passes the time verification, the current data volume of the process hive result table is obtained; The data volume of the process hive result table is verified based on the current data volume and the benchmark data volume in the data monitoring benchmark table.

4. The method according to claim 3, characterized in that Obtaining the last update time of the target hive table includes: Call Hive's metadata API to query Hive's metadata; Call the show table extended command to obtain the lastUpdateTime parameter of the process hive result table from the metadata to obtain the last update time.

5. The method according to claim 3, characterized in that: Performing time verification on the process hive result table based on the last update time and the yesterday's timestamp in the data monitoring reference table includes: performing time verification on the process hive result table by comparing the last update time with the yesterday's timestamp; Verifying the data volume of the process hive result table based on the current data volume and the benchmark data volume in the data monitoring benchmark table includes: comparing the difference between the current data volume and the benchmark data volume with a specified error threshold to verify the data volume of the process hive result table.

6. The method according to claim 5, characterized in that Determining that the process hive result table passes the time verification and the data volume verification includes: when it is determined that the last update time is less than the yesterday's timestamp, and the difference is less than or equal to the error threshold, determining that the process hive result table passes the time verification and the data volume verification; Determining that the process hive result table fails the time verification or fails the data volume verification includes: when it is determined that the last update time is greater than or equal to the yesterday's timestamp, determining that the process hive result table fails the time verification; or when it is determined that the difference is greater than the error threshold, determining that the process hive result table fails the data volume verification.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Record the relevant verification process information of the hive result table of the process; The relevant verification process information is recorded in the verification result table.

8. A monitoring device for a big data fusion process, characterized in that: include: A creation unit, used to determine a target hive table to be monitored based on a business scenario, and to create and initialize a data monitoring reference table for the target hive table; A modification unit, used for renaming the target hive table before applying the target hive table to perform data processing in the big data fusion process, and modifying the target hive table to a process hive result table; A verification unit, configured to perform time verification and data volume verification on the process hive result table based on the data monitoring reference table; An execution unit, used for marking the process hive result table as normal when the verification unit determines that the process hive result table passes the time verification and the data volume verification, and executing the business logic processing flow in the big data fusion process; The truncation unit is used to mark the process hive result table as abnormal and truncate the business logic processing flow when the verification unit determines that the process hive result table fails to pass the time verification or the data volume verification.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory stores instructions, and the instructions are loaded and executed by the processor to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed on a computer, the method according to any one of claims 1 to 7 is implemented.

Images