Task re-running method, device and computer equipment based on data quality detection

Abstract

The present application relates to a method, device and computer equipment for rerunning tasks based on data quality detection, wherein the method comprises: when the Hadoop metadata detection module detects that the input table is ready, triggering the execution of the corresponding task to generate the output table; after the Hadoop metadata detection module detects that the output table is generated, triggering the data quality detection module to detect the indicators in the output table to obtain the quality detection result; when there are abnormal indicators in the quality detection result, triggering the Hadoop metadata detection module to re-detect whether the input table has changed; if the input table has changed, automatically triggering the execution of the corresponding task again until there are no abnormal indicators in the quality detection result of the data quality detection module. Through this application, the problem that the configuration of monitoring files is cumbersome and the timing of rerunning cannot be automatically identified in the related technology, resulting in waste of resources, is solved; automatic triggering of identification of rerunning is achieved, and the timing of identification of rerunning is guaranteed to reduce excessive manual intervention.

Description

Technical Field

[0001] The present application relates to the field of big data and artificial intelligence technology, and in particular to a method, device and computer equipment for rerunning a task based on data quality detection. Background Art

[0002] At present, the big data field is mainly based on the Hadoop ecosystem, and its application field is mainly offline batch tasks (for example, developing a large number of operational reports in enterprises for analysis and decision-making). In actual applications, because the data of a result table may come from multiple input tables, as long as one of the input tables has an abnormality (for example, missing data, abnormal data fields, etc.), it may cause the indicator data generated by the task to be abnormal (for example, the mark is empty or exceeds the reasonable range, etc.), and this abnormal indicator data will lead to misleading.

[0003] At present, for this situation, it is necessary to first detect the amount of data in the upstream table. If the amount of data is abnormal, the task will no longer be executed and an alarm will be issued, and the reasonable monitoring range of the downstream core indicators will be manually set according to historical situations (for example, if it exceeds the reasonable range, an early warning will be issued). However, the configuration of the entire monitoring process is cumbersome and not smart enough. At the same time, after the upstream table data is restored, the task cannot be automatically restored for rerunning, resulting in excessive manual intervention and high costs. Patent CN2021112803665 provides an automatic data business rerun method based on data business monitoring, which monitors the business module by configuring a monitoring file. If the monitoring result is determined to be abnormal, the business module is rerun. This patent requires manual configuration of the monitoring file to monitor the running health of the program. In the case of many indicators and dimensions, if each indicator of each dimension is to be monitored, the configuration of the monitoring file is cumbersome, and the timing of the rerun cannot be automatically identified, resulting in a waste of resources.

[0004] There is currently no effective solution to the problem that the configuration of monitoring files in related technologies is cumbersome and the timing of re-running cannot be automatically identified, resulting in a waste of resources. Summary of the invention

[0005] In this embodiment, a method, device and computer equipment for re-running a task based on data quality detection are provided to solve the problems in the related art that the configuration of monitoring files is cumbersome and the timing of re-running cannot be automatically identified, resulting in waste of resources.

[0006] In a first aspect, a method for rerunning a task based on data quality detection is provided in this embodiment, which is applicable to a device for rerunning a task based on data quality detection; the device comprises a Hadoop metadata detection module and a data quality detection module; the Hadoop metadata detection module is connected to the data quality detection module; the method comprises:

[0007] When the Hadoop metadata detection module detects that the input table is ready, triggering the execution of the corresponding task to generate an output table; the input table is the data source of the task;

[0008] After the Hadoop metadata detection module detects that the output table is generated, the data quality detection module is triggered to detect indicators in the output table to obtain a quality detection result;

[0009] When there are abnormal indicators in the quality detection result, trigger the Hadoop metadata detection module to re-detect whether the input table has been changed;

[0010] If the input table is changed, the corresponding task is automatically triggered to be executed again until there are no abnormal indicators in the quality detection result of the data quality detection module.

[0011] In some of the embodiments, detecting the indicators in the output table to obtain the quality detection results includes:

[0012] Constructing a dynamically changing interval matrix; the interval values ​​in the interval matrix are the reasonable ranges of the indicators;

[0013] Based on the interval matrix, the indicators in the output table are detected to obtain quality detection results.

[0014] In some of the embodiments, constructing a dynamically changing interval matrix includes:

[0015] Assigning periodicity labels to all said indicators;

[0016] Based on the periodicity identifier, determining whether the indicator in the output table is a periodic indicator;

[0017] If it is not a periodic indicator, constructing a first interval value, and constructing the dynamically changing interval matrix based on the first interval value;

[0018] If it is a periodic indicator, determine whether the period length of the corresponding indicator exceeds a preset length threshold;

[0019] If the length does not exceed the threshold, construct a second interval value, and construct the dynamically changing interval matrix based on the second interval value;

[0020] If the length threshold is exceeded, determining whether the current cycle exceeds a preset number of periods within the current cycle;

[0021] If the number of periods is exceeded, a first interval value is constructed, and the dynamically changing interval matrix is ​​constructed based on the first interval value;

[0022] If the number of periods is not exceeded, a second interval value is constructed, and the dynamically changing interval matrix is ​​constructed based on the second interval value.

[0023] In some of the embodiments, constructing the first interval value includes:

[0024] Determine the interval value based on the first interval value formula and historical account periods.

[0025] In some of the embodiments, constructing the second interval value includes:

[0026] Determine the interval value based on the second interval value formula and historical account periods.

[0027] In some of the embodiments, performing the corresponding tasks includes:

[0028] Constructing a relationship matrix between the input table, the output table and the tasks;

[0029] The corresponding tasks are performed according to the relationship matrix and the input table.

[0030] In some embodiments, constructing a relationship matrix among the input table, the output table and the tasks includes:

[0031] Initializing the relationship matrix;

[0032] Edit the task and configure the input table and output table of the task at the same time; the input table is the data input source of the task; the output table is the data output source of the task;

[0033] A relationship matrix between the input table, the output table and the tasks is configured from the tasks, the input table and the output table.

[0034] In a second aspect, in this embodiment, a task rerunning device based on data quality detection is provided, including: a Hadoop metadata detection module and a data quality detection module;

[0035] The Hadoop metadata detection module is connected to the data quality detection module and is used to trigger the execution of a corresponding task to generate an output table when it is detected that the input table is ready; the input table is the data source of the task; after detecting that the output table is generated, the data quality detection module is triggered to detect the indicators in the output table;

[0036] The data quality detection module is used to detect the indicators in the output table to obtain the quality detection result; when there are abnormal indicators in the quality detection result, the Hadoop metadata detection module is triggered to re-detect whether the input table has been changed;

[0037] The Hadoop metadata detection module is further used to automatically re-trigger the execution of the corresponding task if the input table changes, until there is no abnormal indicator in the quality detection result of the data quality detection module.

[0038] In a third aspect, a computer device is provided in this embodiment, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the task rerunning method based on data quality detection described in the first aspect is implemented.

[0039] In a fourth aspect, a storage medium is provided in this embodiment, on which a computer program is stored, and when the program is executed by a processor, the method for rerunning a task based on data quality detection described in the first aspect above is implemented.

[0040] Compared with the related art, the method, device and computer equipment for re-running tasks based on data quality detection provided in this embodiment trigger the execution of corresponding tasks to generate output tables when the Hadoop metadata detection module detects that the input table is ready; the input table is the data source of the task; after the Hadoop metadata detection module detects that the output table is generated, the data quality detection module is triggered to detect the indicators in the output table to obtain the quality detection result; when there are abnormal indicators in the quality detection result, the Hadoop metadata detection module is triggered to re-detect whether the input table has changed; if the input table has changed, the corresponding task is automatically triggered again until there are no abnormal indicators in the quality detection result of the data quality detection module; the problem that the configuration of monitoring files is cumbersome and the timing of re-running cannot be automatically identified in the related art, resulting in waste of resources, is solved; the data quality detection module is used to automatically detect the indicators in the output table, and the automatic triggering and identification of re-running is realized according to the detection result, so as to ensure the timing of identification of re-running and reduce excessive manual intervention.

[0041] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0043] Figure 1 This is a hardware structure block diagram of a terminal device based on a data quality detection task rerun method provided in an embodiment of the present application;

[0044] Figure 2 This is a flowchart of a method for rerunning a task based on data quality detection provided in an embodiment of the present application;

[0045] Figure 3 is a flowchart of step S2110;

[0046] Figure 4 is a schematic diagram of a relationship matrix provided by an embodiment of the present application;

[0047] Figure 5 It is a schematic diagram of a process of constructing an interval matrix provided in an embodiment of the present application;

[0048] Figure 6 It is a structural block diagram of a device for re-running a task based on data quality detection provided in an embodiment of the present application;

[0049] Figure 7 It is a functional schematic diagram of a Hadoop metadata detection module provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0051] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with general skills in the technical field to which this application belongs. The words "one", "a", "the", "these" and the like in this application do not indicate a quantitative limitation, and they may be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone. Generally, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0052] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 is a hardware structure block diagram of a terminal based on the data quality detection task rerunning method of this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 and a memory 104 for storing data, wherein the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0053] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the data quality detection task rerun method in the present embodiment, and the processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0054] The transmission device 106 is used to receive or send data via a network. The above network includes a wireless network provided by a communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0055] In this embodiment, a method for re-running a task based on data quality detection is provided. Figure 2 is a flowchart of the method for re-running a task based on data quality detection in this embodiment. Figure 2 As shown, the process includes the following steps:

[0056] Step S210, when the Hadoop metadata detection module detects that the input table is ready, trigger the execution of the corresponding task to generate the output table; the input table is the data source of the task;

[0057] Step S220, after the Hadoop metadata detection module detects that the output table is generated, the data quality detection module is triggered to detect the indicators in the output table to obtain the quality detection result;

[0058] Step S230, when there are abnormal indicators in the quality detection results, trigger the Hadoop metadata detection module to re-detect whether the input table has changed;

[0059] Step S240: If the input table is changed, the corresponding task is automatically triggered again until there are no abnormal indicators in the quality detection results of the data quality detection module.

[0060] It should be noted that the method embodiment provided in this embodiment is applicable to a task rerun device based on data quality detection; the device includes a Hadoop metadata detection module and a data quality detection module; the Hadoop metadata detection module is connected to the data quality detection module; the Hadoop metadata detection module and the data quality detection module are used to automatically trigger the identification rerun, and ensure the timing of the identification rerun to reduce excessive manual intervention. Among them, Hadoop in Hadoop metadata is a distributed system infrastructure, and Hadoop consists of HDFS, MapReduce, etc. Among them, an HDFS consists of a NameNode and several DataNodes. Among them, NameNode, as the master server, is used to manage the namespace of the file system and the connection of the client; DataNode forms a cluster, and DataNode is used to manage the data stored by each, and Hadoop metadata refers to the namespace of the file system managed by NameNode.

[0061] Among them, tasks are pre-set, including but not limited to offline job tasks, online tasks, etc. The data for task execution comes from the input table; the execution results of the task are stored in the output table. That is, the input table is the data source of the task, and the output table is the storage table of the task results.

[0062] The Hadoop metadata detection module and the data quality detection module both have a trigger mechanism, which can be implemented by scripts or calling third-party tools, and there is no restriction on this. The input table is ready, which means that the input table corresponding to the task has been input into the Hadoop metadata detection module within a preset time period and will not be changed. The preset time period can be set according to the usage scenario.

[0063] Among them, the Hadoop metadata detection module will detect in real time whether the input table is ready, the generation of the output table, and whether the output table has changed. The data quality detection module will detect the indicators in the output table.

[0064] The execution process of the embodiment of the method is as follows: when the Hadoop metadata detection module detects that the input table is ready, it directly triggers the execution of the corresponding task to generate the output table; at this time, the Hadoop metadata detection module detects the generation of the output table, and after detecting the generation of the output table, triggers the data quality detection module to detect the indicators in the output table to obtain the quality detection result; when there are abnormal indicators in the quality detection result, the Hadoop metadata detection module is triggered to re-detect whether the input table has changed; if the input table has changed, the corresponding task is automatically triggered again until there is no abnormal indicator in the quality detection result of the data quality detection module. In this embodiment, by detecting the data quality of the indicators in the output table, it is determined whether it is necessary to automatically re-run the task according to the quality detection result, and at the same time, the Hadoop metadata detection module is combined to detect the changes in the input table of the upstream dependency. If the input table has changed, the related tasks are triggered to start automatically, ensuring the timing of identifying the re-run and reducing the process of manual intervention; it can solve the problem that the configuration monitoring file is cumbersome in the related technology, and the timing of re-run cannot be automatically identified, resulting in waste of resources.

[0065] The above steps are described in detail below:

[0066] In some embodiments, the executing of the corresponding task in step S210 includes the following steps:

[0067] Step S211, constructing a relationship matrix between the input table, the output table and the tasks;

[0068] Step S212, executing corresponding tasks according to the relationship matrix and the input table.

[0069] Specifically, the relationship matrix refers to the relationship between the input table and the task, the relationship between the task and the output table, and the matrix formed by these relationships. For example, the relationship matrix is ​​a relationship table, in which 0, 1 and 2 are used to represent the relationship between the input table and the task, and the relationship between the task and the output table. 0 means no relationship; 1 means there is a relationship in the input table; 2 means there is a relationship in the output table, etc.

[0070] Through this embodiment, a relationship matrix between the input table, the output table and the task is first constructed, and then based on the relationship matrix and the input table, the task execution knows the data source of the task and the storage object of the task execution result, so as to smoothly execute the corresponding task. In other embodiments, the relationship matrix can also be replaced by mapping relationships, setting rules, etc., which is not limited to this.

[0071] In some of the embodiments, Figure 3 As shown in the figure, constructing the relationship matrix between the input table, output table and tasks includes the following steps:

[0072] Step S2111, initializing the relationship matrix;

[0073] Step S2112, edit the task and configure the input table and output table of the task at the same time; the input table is the data input source of the task; the output table is the data output source of the task;

[0074] Step S2113, configuring a relationship matrix between the input table, the output table and the tasks from the tasks, the input table and the output table.

[0075] Initializing the relationship matrix means initializing the relationship matrix to a zero matrix; then editing offline jobs and other tasks, and configuring the input and output tables of the tasks; so that the relationship between different tasks and the input and output tables can be confirmed. Then, the relationship matrix between the input and output tables and the tasks can be configured from the tasks, input and output tables.

[0076] For example, the relationship matrix is ​​as follows Figure 4 As shown, it can be considered that Table 1 and Table 2 are the input tables of Task 1, Table 4 is the output table of Task 1, Table 3 and Table 4 are the input tables of Task 2, and Table 5 is the output table of Task 2.

[0077] In some of the embodiments, Figure 5 As shown, the indicators in the detection output table in step S220 are detected to obtain the quality detection results, including the following steps:

[0078] Construct a dynamically changing interval matrix; the interval values ​​in the interval matrix are the reasonable range of the indicator;

[0079] Based on the interval matrix, the indicators in the output table are detected to obtain the quality detection results.

[0080] Specifically, the interval matrix is ​​dynamically changing, which indicates the possible range of data for each indicator in each dimension in the output table generated in the future account period. The expression of the interval matrix is:

[0081] ;

[0082] In the formula, the interval values ​​in the interval matrix are Represents the reasonable range of the i-th dimension and the j-th indicator.

[0083] The quality inspection result includes whether there are abnormalities in each indicator in the output table. If the indicator in the output table generated in a future account period exceeds the corresponding range in this dimension, it means that the indicator is abnormal; after all the indicators in the output table are detected, the quality inspection result is obtained. That is, the quality inspection result includes whether each indicator in the output table is abnormal or normal. If it is abnormal, the corresponding task needs to be re-executed. The timing of re-execution is controlled by the Hadoop metadata detection module. When the Hadoop metadata detection module detects that there is a change in the input table, it automatically triggers the execution of the corresponding task again until there are no abnormal indicators in the quality inspection results of the data quality detection module; otherwise, the task will not be re-run to save cluster resource overhead.

[0084] Through this embodiment, the constructed interval matrix is ​​used to detect the indicators in the output table, thereby improving the detection accuracy and the detection efficiency.

[0085] In some of the embodiments, constructing a dynamically changing interval matrix includes the following steps:

[0086] Assign periodicity labels to all indicators;

[0087] Based on the periodicity flag, determine whether the indicator in the output table is a periodic indicator;

[0088] If it is not a periodic indicator, a first interval value is constructed, and a dynamically changing interval matrix is ​​constructed based on the first interval value;

[0089] If it is a periodic indicator, determine whether the period length of the corresponding indicator exceeds the preset length threshold;

[0090] If the length threshold is not exceeded, a second interval value is constructed, and a dynamically changing interval matrix is ​​constructed based on the second interval value;

[0091] If the length threshold is exceeded, it is determined whether the current period exceeds the preset number of periods within the current period;

[0092] If the number of periods is exceeded, a first interval value is constructed, and a dynamically changing interval matrix is ​​constructed based on the first interval value;

[0093] If the number of periods is not exceeded, a second interval value is constructed, and a dynamically changing interval matrix is ​​constructed based on the second interval value.

[0094] Specifically, all indicators will be assigned a periodic identifier; for example: the periodic identifiers are 0, 1 (w, m), etc. 0 represents a non-periodic indicator; 1 represents a periodic indicator; for the periodic indicator 1, it is also divided into a weekly indicator w and a monthly indicator m. In other words, for the indicators of this month and this week, it is also assigned to which specific periodic identifier it belongs, whether it is a weekly indicator or a monthly indicator. Based on the periodic identifier, traverse the indicators in the output table to determine whether these indicators are periodic indicators; if the periodic identifier is 0, it is considered that the indicator is not a periodic indicator, then construct a first interval value, and construct a dynamically changing interval matrix based on the first interval value; then the periodic identifier is 1, and it is considered that the indicator is a periodic indicator, then determine whether the period length of the corresponding indicator exceeds the preset length threshold (if it is a periodic indicator, it must have w; or, w and m, so it can be determined whether the period length exceeds the preset length threshold; for example: the length threshold is 15; if the period length is 14, then it is not Exceeds the length threshold); if it does not exceed the length threshold, construct the second interval value, and construct a dynamically changing interval matrix based on the second interval value; if it exceeds the length threshold, determine whether the current cycle exceeds the preset number of periods in this cycle (for example: the preset number of periods is 7, if the current cycle is 6, it does not exceed the preset number of periods); if it does not exceed the number of periods, construct the second interval value, and construct a dynamically changing interval matrix based on the second interval value (for example: the starting period on the 1st, 2nd, etc. at the beginning of the month, it is necessary to use the historical month-end indicators to obtain; at the end of the month, it can be obtained using the situation of the previous 7 days). If it exceeds the number of periods, construct the first interval value, and construct a dynamically changing interval matrix based on the first interval value.

[0095] Through this embodiment, different methods are used to construct a dynamically changing interval matrix, reducing the huge resource consumption caused by using machine learning, deep learning and other models for processing.

[0096] In some embodiments, constructing the first interval value includes the following steps:

[0097] Determine the interval value based on the first interval value formula and historical account periods.

[0098] Specifically, the first interval value formula is: interval value = previous account period value + average value of the difference between the two account periods before and after the previous n account periods ± b × standard deviation of the difference between the two account periods before and after the previous n account periods. Where b is a coefficient, which can be 2 or 3, etc.

[0099] In the big data indicator system, we often encounter cumulative indicators, monthly indicators, weekly indicators, etc. Cumulative indicators often show a continuous upward trend, and the above calculation method can be used to obtain interval values. However, some indicators have strong cyclical characteristics. For example, the number of new users this month has a large difference between the end of last month and the beginning of this month. The interval values ​​of the first few days of the month cannot be obtained by the data of the previous few days. The above calculation method can still be used to obtain the interval values ​​near the end of the month.

[0100] Through this embodiment, the accuracy of the interval value is improved, thereby improving the accuracy of quality detection.

[0101] In some embodiments, constructing the second interval value includes the following steps:

[0102] Determine the interval value based on the second interval value formula and historical account periods.

[0103] The calculation method of the starting account period interval value of the periodic indicator is as follows:

[0104] Specifically, the second interval value formula is: interval value = value of the same period of the previous cycle + average value b of the difference between the two account periods before and after the same period of the previous n cycles × standard deviation of the difference between the two account periods before and after the same period of the previous n cycles. Wherein, b is a coefficient, which can be 2 or 3, etc.

[0105] Through this embodiment, the accuracy of the interval value is improved, thereby improving the accuracy of quality detection.

[0106] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0107] In the present embodiment, a task rerunning device based on data quality detection is also provided, and the device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. The terms "module", "unit", "subunit", etc. used below can implement a combination of software and / or hardware of predetermined functions. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0108] Figure 6 is a structural block diagram of the data quality detection task re-running device of this embodiment, such as Figure 6 As shown, the device includes: a Hadoop metadata detection module 210 and a data quality detection module 220;

[0109] The Hadoop metadata detection module 210 is connected to the data quality detection module 220 and is used to trigger the execution of the corresponding task to generate the output table when the input table is detected to be ready; the input table is the data source of the task; after the output table is detected to be generated, the data quality detection module 220 is triggered to detect the indicators in the output table;

[0110] The data quality detection module 220 is used to detect the indicators in the output table and obtain the quality detection results; when there are abnormal indicators in the quality detection results, the Hadoop metadata detection module 210 is triggered to re-detect whether the input table has changed;

[0111] The Hadoop metadata detection module 210 is also used to automatically re-trigger the execution of the corresponding task if the input table changes, until there is no abnormal indicator in the quality detection result of the data quality detection module 220.

[0112] Through the above-mentioned device, the problem of cumbersome configuration of monitoring files and inability to automatically identify the timing of re-running, resulting in waste of resources, existing in the related technology, is solved; the data quality detection module 220 is used to automatically detect the indicators in the output table, and the automatic triggering of re-running is realized according to the detection results, thereby ensuring the timing of re-running identification and reducing excessive manual intervention.

[0113] The Hadoop metadata detection module 210 can collect Hadoop metadata in real time, collect Hadoop NameNode edit logs in real time for analysis, obtain basic information of the input table from the Hadoop metadata, such as file size and update time, and load the basic information of the input table into the cache, so that the Hadoop metadata detection module 210 can detect whether the input table is ready, whether it has been changed, etc.

[0114] The format of the edit log is as follows:

[0115] <record>

[0116] <opcode> OP_DELETE< / opcode>

[0117] <data><h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr">

[0118] <h2 style=";text-align:left;direction:ltr"> <txid> 2421< / txid> <h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr">

[0119] <h2 style=";text-align:left;direction:ltr"> <length> 0< / length> <h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr">

[0120] <h2 style=";text-align:left;direction:ltr"> <path> / user / stat_user / lizc / flink-checkpoint / 641af83aeed141fcdca5b3e144b70e60 / chk-5 / _metadata< / path> <h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr">

[0121] <h2 style=";text-align:left;direction:ltr"> <timestamp> 1695038139517< / timestamp> <h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr">

[0122] <h2 style=";text-align:left;direction:ltr"><RPC_CLIENTID> 14d5dbdb-03af-4f31-8186-faa75ca875bc< / RPC_CLIENTID><h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr">

[0123] <h2 style=";text-align:left;direction:ltr"><RPC_CALLID> 181< / RPC_CALLID><h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr">

[0124] < / data>

[0125] < / record>

[0126] <record>

[0127] <opcode> OP_DELETE< / opcode>

[0128] <data><h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr">

[0129] <h2 style=";text-align:left;direction:ltr"> <txid> 2422< / txid> <h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr">

[0130] <h2 style=";text-align:left;direction:ltr"> <length> 0< / length> <h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr">

[0131] <h2 style=";text-align:left;direction:ltr"> <path> / user / stat_user / lizc / flink-checkpoint / 641af83aeed141fcdca5b3e144b70e60 / chk-5< / path> <h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr">

[0132] <h2 style=";text-align:left;direction:ltr"> <timestamp> 1695038139522< / timestamp> <h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr">

[0133] <h2 style=";text-align:left;direction:ltr"><RPC_CLIENTID> 14d5dbdb-03af-4f31-8186-faa75ca875bc< / RPC_CLIENTID><h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr">

[0134] <h2 style=";text-align:left;direction:ltr"><RPC_CALLID> 182< / RPC_CALLID><h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr">

[0135] < / data>

[0136] < / record> .

[0137] The above example contains all operation records of the HDFS file system, including operations of adding, deleting, and modifying files. For example, OP_DELETE represents deleting a file, and OP_MKDIR represents creating a folder.

[0138] Among them, Figure 7As shown, the Hadoop metadata detection module 210 can also read the relationship matrix between tasks and tables; read the dynamic update information of metadata in the input table and output table; when the metadata of a task input table is read and all updates are completed, the execution of the corresponding task will be triggered; when the output table is generated after the task is executed, it can detect that the output table is completed and trigger the data quality detection module 220 to perform detection.

[0139] In some of the embodiments, the data quality detection module 220 is further used to construct a dynamically changing interval matrix; the interval values ​​in the interval matrix are reasonable ranges of the indicators;

[0140] Based on the interval matrix, the indicators in the output table are detected to obtain the quality detection results.

[0141] In some of the embodiments, the data quality detection module 220 is further used to assign periodic identification to all indicators;

[0142] Based on the periodicity flag, determine whether the indicator in the output table is a periodic indicator;

[0143] If it is not a periodic indicator, a first interval value is constructed, and a dynamically changing interval matrix is ​​constructed based on the first interval value;

[0144] If it is a periodic indicator, determine whether the period length of the corresponding indicator exceeds the preset length threshold;

[0145] If the length threshold is not exceeded, a second interval value is constructed, and a dynamically changing interval matrix is ​​constructed based on the second interval value;

[0146] If the length threshold is exceeded, it is determined whether the current period exceeds the preset number of periods within the current period;

[0147] If the number of periods is exceeded, a first interval value is constructed, and a dynamically changing interval matrix is ​​constructed based on the first interval value;

[0148] If the number of periods is not exceeded, a second interval value is constructed, and a dynamically changing interval matrix is ​​constructed based on the second interval value.

[0149] In some of the embodiments, the data quality detection module 220 is further used to determine the interval value according to the first interval value formula and the historical account period.

[0150] In some of the embodiments, the data quality detection module 220 is further used to determine the interval value according to the second interval value formula and the historical account period.

[0151] In some of the embodiments, the Hadoop metadata detection module 210 is further used to construct a relationship matrix between the input table, the output table and the task;

[0152] Execute corresponding tasks according to the relationship matrix and input table.

[0153] In some of the embodiments, the Hadoop metadata detection module 210 is also used to initialize the relationship matrix;

[0154] Edit the task and configure the input table and output table of the task at the same time; the input table is the data input source of the task; the output table is the data output source of the task;

[0155] From the tasks, input tables, and output tables, configure the relationship matrix between the input tables, output tables, and tasks.

[0156] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0157] In this embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0158] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0159] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0160] S1, when the Hadoop metadata detection module detects that the input table is ready, it triggers the execution of the corresponding task to generate the output table; the input table is the data source of the task;

[0161] S2, after the Hadoop metadata detection module detects the generation of the output table, the data quality detection module is triggered to detect the indicators in the output table and obtain the quality detection result;

[0162] S3, when there are abnormal indicators in the quality detection results, trigger the Hadoop metadata detection module to re-detect whether the input table has changed;

[0163] S4: If the input table changes, the corresponding task is automatically triggered again until there are no abnormal indicators in the quality inspection results of the data quality inspection module.

[0164] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0165] In addition, in combination with the data quality detection task rerun method provided in the above embodiment, a storage medium can also be provided in this embodiment to implement. The storage medium stores a computer program; when the computer program is executed by the processor, any one of the data quality detection task rerun methods in the above embodiment is implemented.

[0166] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0167] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0168] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various locations in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0169] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A method for re-running a task based on data quality detection, characterized in that: Applicable to a task re-running device based on data quality detection; the device includes a Hadoop metadata detection module and a data quality detection module; The Hadoop metadata detection module is connected to the data quality detection module; the method includes: When the Hadoop metadata detection module detects that the input table is ready, triggering the execution of the corresponding task to generate an output table; the input table is the data source of the task; After the Hadoop metadata detection module detects that the output table is generated, the data quality detection module is triggered to detect indicators in the output table to obtain a quality detection result; When there are abnormal indicators in the quality detection result, triggering the Hadoop metadata detection module to re-detect whether the input table has been changed; If the input table is changed, the corresponding task is automatically triggered again to execute until there are no abnormal indicators in the quality detection result of the data quality detection module; Wherein, detecting the indicators in the output table to obtain the quality detection results includes: Constructing a dynamically changing interval matrix; the interval values ​​in the interval matrix are the reasonable range of the indicator; the interval matrix is ​​a matrix composed of the interval values; Based on the interval matrix, the indicators in the output table are detected to obtain quality detection results.

2. The method for rerunning a task based on data quality detection according to claim 1, characterized in that: Construct a dynamically changing interval matrix, including: Assigning periodicity labels to all said indicators; Based on the periodicity identifier, determining whether the indicator in the output table is a periodic indicator; If it is not a periodic indicator, constructing a first interval value, and constructing the dynamically changing interval matrix based on the first interval value; If it is a periodic indicator, determine whether the period length of the corresponding indicator exceeds a preset length threshold; If the length does not exceed the threshold, construct a second interval value, and construct the dynamically changing interval matrix based on the second interval value; If the length threshold is exceeded, determining whether the current cycle exceeds a preset number of periods within the current cycle; If the number of periods is exceeded, a first interval value is constructed, and the dynamically changing interval matrix is ​​constructed based on the first interval value; If the number of periods does not exceed the number, a second interval value is constructed, and the dynamically changing interval matrix is ​​constructed based on the second interval value; The first interval value is constructed according to the first interval value formula and the historical account period; wherein the first interval value formula is: first interval value = previous account period value + average value of the difference between the two account periods before and after the previous n account periods ± b × standard deviation of the difference between the two account periods before and after the previous n account periods; wherein b is a coefficient; Wherein, the second interval value is constructed according to the second interval value formula and the historical account period; Among them, the numerical formula for the second interval is: the numerical value of the second interval = the numerical value of the same period of the previous cycle + the average value b of the difference between the two account periods before and after the same period of the previous n cycles × the standard deviation of the difference between the two account periods before and after the same period of the previous n cycles; among them, b is the coefficient.

3. The method for rerunning a task based on data quality detection according to claim 1, characterized in that: Executing the corresponding tasks includes: Constructing a relationship matrix between the input table, the output table and the tasks; According to the relationship matrix and the input table, the corresponding task is executed.

4. The method for re-running a task based on data quality detection according to claim 3, characterized in that: Constructing a relationship matrix between the input table, the output table and the task, including: Initializing the relationship matrix; Edit the task and configure the input table and output table of the task at the same time; the input table is the data input source of the task; the output table is the data output source of the task; A relationship matrix between the input table, the output table and the tasks is configured from the tasks, the input table and the output table.

5. A task re-running device based on data quality detection, characterized in that: include: Hadoop metadata detection module and data quality detection module; The Hadoop metadata detection module is connected to the data quality detection module and is used to trigger the execution of a corresponding task to generate an output table when it is detected that the input table is ready; The input table is the data source of the task; After detecting that the output table is generated, triggering the data quality detection module to detect indicators in the output table; The data quality detection module is used to detect the indicators in the output table to obtain the quality detection result; when there are abnormal indicators in the quality detection result, the Hadoop metadata detection module is triggered to re-detect whether the input table has been changed; The data quality detection module is further used to construct a dynamically changing interval matrix; the interval values ​​in the interval matrix are the reasonable range of the indicator; the interval matrix is ​​a matrix composed of the interval values; Based on the interval matrix, detect the indicators in the output table to obtain quality detection results; The Hadoop metadata detection module is further used to automatically re-trigger the execution of the corresponding task if the input table changes, until there is no abnormal indicator in the quality detection result of the data quality detection module.

6. A computer device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps of the data quality detection task rerun method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data quality detection task rerun method described in any one of claims 1 to 4 are implemented.

Images