Batch task quality supervision method
By introducing batch task quality supervision methods into the data quality management system, and using in-process inspection and post-process inspection mechanisms, the problem that existing systems cannot achieve high cost of interception and quality management is solved, real-time quality supervision of batch tasks and efficient data quality management are achieved.
Patent Information
- Application Number
- CN202411548682.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-01
- Publication Date
- 2025-03-04
AI Technical Summary
The existing data quality management system is not complete enough in batch task quality supervision, cannot achieve in-process interception, and the quality management cost is high.
Provide a batch task quality supervision method, and conduct quality supervision of batch tasks through two levels: in-process inspection and post-process inspection. In-process inspection responds to in-process inspection instructions, invokes in-process inspection tables, determines the batch tasks to be inspected based on the configured task name and task group, and executes in-process inspection rules. If the inspection fails, the execution of the batch task to be inspected will be aborted. After the batch tasks to be checked are all executed, the post-check table is called, the post-check rules are executed, and the inspection results are stored.
Real-time quality supervision of batch tasks is realized, prevents the spread of wrong data, reduces quality management costs, and simplifies the data quality supervision process of batch tasks in data warehouses.
Smart Images
Figure CN119557292A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of this specification relate to the field of data management technology, and in particular to a method for monitoring the quality of batch tasks. Background Art
[0002] The existing independently constructed data quality management system needs to be integrated with many peripheral systems, such as the scheduling system. After the batch task, the scheduling system notifies the data quality management system to perform data verification, and stores the verification completion in the database of the data quality management system. The trigger conditions, threshold judgment, alarm recipient configuration and other functions of the quality verification during data verification are all implemented in the data quality verification system. This makes the coverage scenarios of the existing data quality management system incomplete, and the management of batch tasks only has post-event quality verification, and it is impossible to intercept in the process; and when there are new additions or changes to the rules, the new verification rules need to be put into production and changed online, which makes the use of the new verification rules delayed. In addition, due to the need to involve integration with peripheral systems, the construction cycle of the quality management system is long and the maintenance cost is high.
[0003] In view of this, the embodiments of this specification aim to provide a batch task quality supervision method. Summary of the invention
[0004] In view of the above problems in the prior art, the purpose of the embodiments of this specification is to provide a batch task quality supervision method to solve the problems in the prior art that the quality supervision of batch tasks is incomplete and the quality management cost is high.
[0005] In order to solve the above technical problems, the specific technical solutions of the embodiments of this specification are as follows:
[0006] In a first aspect, an embodiment of the present specification provides a batch task quality supervision method, including:
[0007] In response to the in-process inspection instruction, calling the in-process inspection table;
[0008] Determine a number of batch tasks to be checked according to the task names and the task groups to which they belong configured in the in-process checklist, wherein there is a dependency relationship between the batch tasks to be checked;
[0009] Perform an in-process inspection on the first item to be inspected in the data table obtained by executing the batch task to be inspected according to the in-process inspection rule configured in the in-process inspection table;
[0010] Determine whether the in-process inspection has passed;
[0011] If not, the execution of the batch task to be checked is terminated and a reminder is fed back to the operation and maintenance personnel;
[0012] If yes, then determine whether all the batch tasks to be checked have been completed;
[0013] If so, in response to the post-check instruction, calling the post-check table;
[0014] A post-inspection is performed on the second item to be inspected in the data table obtained by executing the batch task to be inspected according to the post-inspection rule configured in the post-inspection table.
[0015] Specifically, the method further includes:
[0016] storing the first item to be inspected, the item number of the first item to be inspected, and the inspection date in an inspection log table; and
[0017] The second item to be inspected, the item number of the second item to be inspected, and the inspection date are stored in the inspection log table.
[0018] Further, performing an in-process inspection on the first item to be inspected in the data table obtained by executing the batch task to be inspected according to the in-process inspection rule configured in the in-process inspection table includes:
[0019] If the in-process inspection rule is an inspection rule based on a predetermined threshold, comparing the first item to be inspected with the predetermined threshold according to a comparison rule in the in-process inspection rule;
[0020] If the in-process inspection rule is an inspection rule based on historical data, the inspection log table is called and the historical inspection item consistent with the project number of the first item to be inspected is obtained according to the comparison date in the in-process inspection rule, and the first item to be inspected is compared with the historical inspection item.
[0021] Furthermore, comparing the first item to be checked with the historical checked item further includes:
[0022] Obtaining a comparison coefficient and a comparison rule corresponding to the first item to be checked in the in-process inspection rule;
[0023] Calculating a comparison value according to the comparison coefficient and the historical inspection item;
[0024] Based on the comparison rule, the first item to be checked is compared with the comparison value.
[0025] Specifically, in response to the in-process inspection instruction, calling the in-process inspection table includes:
[0026] Execute batch tasks to be checked;
[0027] According to the execution result of the batch task to be checked, the execution identifier of the batch task to be checked is updated;
[0028] Determine whether the execution flag of the batch task to be checked is completed;
[0029] If so, respond to the in-progress inspection instruction to conduct an in-progress investigation on the data table obtained by executing the batch task to be inspected.
[0030] Specifically, when the plurality of batch tasks to be checked are all executed, the method further includes:
[0031] In response to the abnormal data sampling instruction, calling the abnormal data sampling table;
[0032] According to the abnormal data, the sampling rules configured in the table are used to splice and obtain sampling statements;
[0033] The data table obtained by executing the batch task to be checked is sampled according to the sampling statement, and the sampling result is stored in the sampling data table.
[0034] Further, after the post-inspection and / or the abnormal data sampling is completed, the method includes:
[0035] In response to the alarm instruction, calling the alarm configuration table;
[0036] According to the alarm items configured in the alarm configuration table, searching for the post-inspection results and / or the abnormal data sampling results;
[0037] According to the alarm rules configured in the alarm configuration table, an alarm is issued for the post-inspection result and / or the abnormal sampling result.
[0038] Specifically, the method further includes:
[0039] The in-process inspection result of the first item to be inspected and the post-inspection result of the second item to be inspected are stored in the inspection log table.
[0040] In a second aspect, an embodiment of the present specification provides a batch task quality supervision device, including:
[0041] A first response module is used to respond to the in-process inspection instruction and call the in-process inspection table;
[0042] A determination module, used to determine a number of batch tasks to be checked according to the task names and the task groups to which they belong configured in the in-process inspection table, wherein there is a dependency relationship between the batch tasks to be checked;
[0043] An in-process inspection module, used for performing an in-process inspection on the first item to be inspected of each batch task to be inspected according to the in-process inspection rules configured in the in-process inspection table;
[0044] A first judgment module is used to judge whether the in-process inspection has passed;
[0045] A suspension and reminder module, used to suspend the execution of the batch tasks to be inspected and provide feedback to the operation and maintenance personnel when the in-process inspection fails;
[0046] A second judgment module is used to judge whether all the batch tasks to be checked have been executed when the in-process inspection passes;
[0047] A second response module is used to call a post-check table in response to a post-check instruction when all the batch tasks to be checked are completed;
[0048] The post-check module is used to perform a post-check on the second item to be checked of the batch task to be checked according to the post-check rule configured in the post-check table.
[0049] In a third aspect, an embodiment of the present specification provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method provided by the above technical solution when executing the computer program.
[0050] In a fourth aspect, an embodiment of the present specification provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method provided by the above technical solution is implemented.
[0051] In a fifth aspect, an embodiment of the present specification provides a computer program product, comprising at least one instruction or at least one program, wherein the at least one instruction or the at least one program is loaded and executed by a processor to implement the method provided by the above technical solution.
[0052] By adopting the above technical solution, the batch task quality supervision method provided in the embodiment of this specification performs quality supervision on the execution of batch tasks from two levels: in-process inspection and post-process inspection. When the in-process inspection result fails, the continued execution of the batch task to be inspected will be suspended to prevent the erroneous data generated by the batch task to be inspected from polluting the downstream and avoiding the continuous spread of erroneous data. In addition, this method only needs to modify the public template script to realize the in-process inspection of batch tasks; post-process quality inspection can be realized by creating a new batch and scheduling tasks, realizing agile and lightweight supervision of the data quality of batch tasks in the data warehouse, with the characteristics of agility and lightweight, efficient operation, short construction cycle, and easy maintenance.
[0053] In order to make the above and other purposes, features and advantages of the embodiments of this specification more obvious and easy to understand, the following specifically cites preferred embodiments and describes them in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0055] Figure 1 A schematic diagram of the steps of a batch task quality supervision method provided in an embodiment of this specification is shown;
[0056] Figure 2 Another schematic diagram of a step of the batch task quality supervision method in the embodiment of this specification is shown;
[0057] Figure 3 Another schematic diagram of the steps of the batch task quality supervision method in the embodiment of this specification is shown;
[0058] Figure 4 A schematic diagram showing another step of the batch task quality supervision method in the embodiment of this specification;
[0059] Figure 5 A schematic diagram of the structure of a batch task quality monitoring device provided in an embodiment of this specification is shown;
[0060] Figure 6 A schematic diagram of the structure of a computer device provided in an embodiment of this specification is shown.
[0061] Description of the accompanying symbols:
[0062] 51. First response module;
[0063] 52. Determine the module;
[0064] 53. In-process inspection module;
[0065] 54. A first judgment module;
[0066] 55. Suspension and reminder module;
[0067] 56. Second judgment module;
[0068] 57. Second response module;
[0069] 58. Post-inspection module;
[0070] 602. Computer equipment;
[0071] 604, processor;
[0072] 606. Memory;
[0073] 608, driving mechanism;
[0074] 610, input / output module;
[0075] 612. Input devices;
[0076] 614. Output devices;
[0077] 616. Presentation equipment;
[0078] 618. Graphical user interface;
[0079] 620, network interface;
[0080] 622, communication link;
[0081] 624. Communication bus. DETAILED DESCRIPTION
[0082] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.
[0083] It should be noted that the terms "first", "second", etc. in this specification and claims and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this specification described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.
[0084] In order to solve the above problems, the embodiments of this specification provide a batch task quality supervision method, which can solve the problems of incomplete quality supervision of batch tasks and high quality management cost in the prior art. Figure 1This is a step diagram of a batch task quality supervision method provided in the embodiment of this specification. This specification provides method operation steps as described in the embodiment or flowchart, but may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the steps among many orders, and does not represent the only order of execution. When the actual system or device product is executed, it can be executed in the order or in parallel according to the method shown in the embodiment or the accompanying drawings. Specifically, Figure 1 As shown, the method may include:
[0085] S110: In response to the in-process check instruction, call the in-process check table.
[0086] S120: Determine a number of batch tasks to be checked according to the task names and the task groups to which they belong configured in the in-progress checklist, wherein there is a dependency relationship between the batch tasks to be checked.
[0087] S130: Perform an in-process inspection on the first item to be inspected in the data table obtained by executing the batch task to be inspected according to the in-process inspection rules configured in the in-process inspection table.
[0088] S140: Determine whether the in-process inspection has passed.
[0089] S150: If not, the execution of the batch task to be checked is terminated and a reminder is fed back to the operation and maintenance personnel.
[0090] It should be noted that there is a dependency relationship between the batch tasks to be checked in the embodiments of this specification. Therefore, when one of the batch tasks to be checked is terminated, since the batch task to be checked cannot provide output data as input data for other batch tasks to be checked, other batch tasks that depend on the batch task to be checked will also be terminated.
[0091] S160: If yes, determine whether all the batch tasks to be checked have been completed.
[0092] S170: If yes, in response to the post-check instruction, call the post-check table.
[0093] S180: Perform a post-inspection on the second item to be inspected in the data table obtained by executing the batch task to be inspected according to the post-inspection rule configured in the post-inspection table.
[0094] That is, in the embodiments of the present specification, the in-process check is relative to the post-check. The in-process check is a check on the data table obtained by executing any batch task (the execution of a batch task is usually a process of processing an original data table and generating a new data table); the post-check is a check on the data tables obtained by executing each batch task according to all the post-check items configured in the post-check table (i.e., the second items to be checked) after all the batch tasks with dependencies have been executed.
[0095] In the embodiments of this specification, both the in-process checklist and the post-process checklist are pre-configured. It should be noted that not every batch task must undergo both in-process and post-process checks. For some scenarios with higher quality requirements, such as batch tasks that must be repaired on the same day or have a greater impact on downstream systems, it is necessary to configure an in-process check; while some batch tasks with relatively loose handling requirements can be configured with a post-process check to achieve tracking, recording and reminders of data quality.
[0096] For any batch task to be checked that is configured with an in-process check, there can be one or more first items to be checked, which are pre-configured by business personnel and development and operation personnel according to actual application needs. Similar to the first item to be checked, for any batch task to be checked that is configured with a post-check, there can be one or more second items to be checked.
[0097] A batch task quality supervision method provided in the embodiments of this specification can supervise the execution of batch tasks from two levels: in-process inspection and post-inspection. When the in-process inspection result fails, the continued execution of the batch task to be inspected will be suspended to prevent the erroneous data generated by the batch task to be inspected from polluting the downstream and avoiding the continuous spread of erroneous data. After the operation and maintenance personnel check, correct and confirm the data of the batch task to be inspected, the continued operation of the batch task to be inspected can be restored. In addition, the method provided in the embodiments of this specification only needs to modify the public template script to realize the in-process inspection of batch tasks; post-event quality inspection can be realized by creating a new batch and scheduling tasks, realizing agile and lightweight supervision of the data quality of batch tasks in the data warehouse.
[0098] Furthermore, in the embodiments of this specification, the method further includes:
[0099] storing the first item to be inspected, the item number of the first item to be inspected, and the inspection date in an inspection log table; and
[0100] The second item to be inspected, the item number of the second item to be inspected, and the inspection date are stored in the inspection log table.
[0101] Furthermore, in the embodiment of the present specification, step S130: performing an in-process inspection on the first item to be inspected in the data table obtained by executing the batch task to be inspected according to the in-process inspection rule configured in the in-process inspection table, includes:
[0102] If the in-process inspection rule is an inspection rule based on a predetermined threshold, comparing the first item to be inspected with the predetermined threshold according to a comparison rule in the in-process inspection rule;
[0103] For example, the first item to be checked (numbered 001) in a batch task data table to be checked is the sales of product A. If the in-process check rule for the first item to be checked is a check rule based on a predetermined threshold, the sales of product A are compared with the predetermined threshold (for example, 1000) according to the comparison rule in the in-process check rule (i.e., one of greater than, less than, greater than or equal to, and less than or equal to).
[0104] If the in-process inspection rule is an inspection rule based on historical data, the inspection log table is called and the historical inspection items in the inspection log table that are consistent with the project number of the first item to be inspected are obtained according to the comparison date in the in-process inspection rule, and the first item to be inspected is compared with the historical inspection items.
[0105] Furthermore, comparing the first item to be checked with the historical check record further includes:
[0106] Obtaining a comparison coefficient and a comparison rule corresponding to the first item to be checked in the in-process inspection rule;
[0107] Calculating a comparison value according to the comparison coefficient and the historical inspection item;
[0108] Based on the comparison rule, the first item to be checked is compared with the comparison value.
[0109] Exemplarily, if the comparison date is "YES (meaning previous day)", then obtain the historical inspection item in the inspection log table that corresponds to the project number of the first item to be inspected (i.e., number 001) and whose date is the comparison date (i.e., previous day), and multiply the historical inspection item by the comparison coefficient to obtain a comparison value, and compare the comparison value with the first item to be inspected based on the comparison rule (one of greater than, less than, greater than or equal to, and less than or equal to).
[0110] That is, in the embodiment of this specification, the fields configured in the in-process inspection table are as shown in Table 1:
[0111] Table 1 In-process inspection list
[0112]
[0113]
[0114] In the embodiment of the present specification, the post-check table is shown in Table 2, and step S180: performing a post-check on the second item to be checked in the data table obtained by executing the batch task to be checked according to the post-check rule configured in the post-check table, specifically:
[0115] Each second item to be checked is checked according to the post-check rules (ie, check statements) configured in the post-check table.
[0116] Table 2 Post-event checklist
[0117] Serial number Field Name illustrate 1 Inspection item number Each inspection item has a unique number. 2 Explanation of inspection items Briefly describe the inspection items 3 Check Statement Quality check statements can only return a single row and a single column 4 Is it enabled? 1 means enabled, 0 means disabled
[0118] like Figure 2 As shown, in the embodiment of this specification, before S110: responding to the in-process inspection instruction and calling the in-process inspection table, the method further includes:
[0119] S210: Execute the batch task to be checked;
[0120] S220: updating the execution identifier of the batch task to be checked according to the execution result of the batch task to be checked.
[0121] S230: Determine whether the execution mark of the batch task to be checked is completed.
[0122] S240: If yes, respond to the in-progress inspection instruction to conduct an in-progress inspection on the data table obtained by executing the batch task to be inspected.
[0123] The embodiment of the present specification provides a batch task quality supervision method, which supervises the execution process of each batch task by setting an execution identifier for the execution of each batch task, and automatically calls a post-check table to perform a post-check on each batch task after all batch tasks with dependencies are completed.
[0124] Furthermore, if Figure 3 As shown, in the embodiment of this specification, when the plurality of batch tasks to be checked are all executed, the method further includes:
[0125] S310: In response to the abnormal data sampling instruction, calling the abnormal data sampling table;
[0126] S320: Using the sampling rules configured in the table according to the abnormal data, splicing to obtain a sampling statement;
[0127] S330: Sampling the data table obtained by executing the batch task to be checked according to the sampling statement, and storing the sampling result in a sampling data table.
[0128] In the embodiments of this specification, the abnormal data sampling table is shown in Table 3, and the sampling data table is shown in Table 4.
[0129] Table 3 Abnormal data sampling table
[0130] Serial number Field Name illustrate 1 Rule Number Unique 2 Rule Description Briefly describe the sampling rules 3 Rule Statements Quality check statements can only return a single row and a single column 4 Label Used to identify record characteristics 5 Is it enabled? 1 means enabled, 0 means disabled
[0131] Table 4 Sampling data table
[0132] Serial number Field Name illustrate 1 Rule Number Unique 2 Data date Data date 3 Label Used to identify record characteristics 4 Sampling Data Save after processing through json serialization
[0133] It should be noted that, in the embodiments of this specification, the abnormal data sampling can be performed on at least part of the batch tasks according to actual application needs, that is, not all batch tasks must be sampled.
[0134] Further, in the embodiment of this specification, after the post-inspection and / or the abnormal data sampling is completed, the method further includes:
[0135] S410: In response to the alarm instruction, calling the alarm configuration table;
[0136] S420: searching the post-inspection result and / or the abnormal data sampling result according to the alarm items configured in the alarm configuration table;
[0137] S430: According to the alarm rules configured in the alarm configuration table, an alarm is issued for the post-inspection result and / or the abnormal data sampling result.
[0138] The alarm configuration table in the embodiment of this specification is shown in Table 5:
[0139] Table 5 Alarm configuration table
[0140]
[0141] As shown in Table 5, in the embodiment of this specification, the alarm rules include alarm rules based on predetermined thresholds and alarm rules based on historical data.
[0142] If the alarm rule for a certain post-inspection result or a certain abnormal data sampling result is an alarm rule based on a predetermined threshold, then the post-inspection result or the abnormal data sampling result is compared with the predetermined threshold;
[0143] Exemplarily, if a certain abnormal data sampling result has 5 abnormal data, this abnormal data sampling result is compared with a predetermined threshold. If it is greater than the predetermined threshold, an alarm is issued for the abnormal data sampling result and the alarm information content is fed back to the alarm recipient configured in the alarm configuration table.
[0144] If the alarm rule for a certain post-inspection result or a certain abnormal data sampling result is an alarm rule based on historical data, the post-inspection result or the abnormal data sampling result is compared with the historical data.
[0145] Exemplarily, if the historical data configured in the alarm configuration table is "EOM (meaning the end of the previous month)", then the historical data of the abnormal data sampling result corresponding to the end of the previous month in the sampling data table is obtained, and the current abnormal data sampling result is compared with the abnormal data sampling result of the corresponding historical data, and based on the comparison result, it is determined whether to issue an alarm for the current abnormal data sampling result.
[0146] The batch task quality supervision method provided in the embodiment of this specification also includes:
[0147] The in-process inspection result of the first item to be inspected and the post-inspection result of the second item to be inspected are stored in the inspection log table.
[0148] Among them, the inspection log table is shown in Table 6:
[0149] Table 6 Inspection log table
[0150]
[0151] In summary, the batch task quality supervision method provided in the embodiment of this specification provides a lightweight solution for supervising the data quality of batch tasks, which divides the data warehouse quality check into two parts: in-process and post-process, and realizes in-process data quality inspection by modifying the public template script; post-process quality inspection can be realized by creating a new batch of post-scheduling tasks, which greatly improves the convenience of data quality inspection. And if the in-process inspection fails, it can be stopped and blocked to prevent the wrong data from polluting the downstream.
[0152] At the same time, this method also summarizes various quality inspection scenarios into a unified configuration template. Quality inspection can be added by adding new configuration items. The quality inspection requirements in most scenarios are met by customizing the inspection statements. The operation is simple and convenient. In addition, this method also has the function of sampling abnormal data. It can retain some sample data while performing quality inspection to facilitate subsequent problem location.
[0153] Based on the above-mentioned method for quality supervision of batch tasks, an embodiment of this specification also provides a corresponding device for quality supervision of batch tasks. The device may include a system (including a distributed system), software (application), module, component, server, client, etc. using the method described in the embodiment of this specification and a device combined with necessary implementation hardware. Based on the same innovative concept, the device in one or more embodiments provided in the embodiment of this specification is as described in the following embodiments. Since the implementation scheme and method of the device to solve the problem are similar, the implementation of the specific device in the embodiment of this specification can refer to the implementation of the aforementioned method, and the repetitions will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.
[0154] like Figure 5 As shown, the embodiment of this specification provides a batch task quality supervision device, including:
[0155] The first response module 51 is used to respond to the in-process inspection instruction and call the in-process inspection table;
[0156] A determination module 52, configured to determine a number of batch tasks to be checked according to the task names and the task groups to which they belong configured in the in-progress checklist, wherein there is a dependency relationship between the batch tasks to be checked;
[0157] An in-process inspection module 53, configured to perform an in-process inspection on the first item to be inspected in the data table obtained by executing the batch task to be inspected according to the in-process inspection rules configured in the in-process inspection table;
[0158] A first judging module 54 is used to judge whether the in-process inspection has passed;
[0159] The suspending and reminding module 55 is used to suspend the execution of the batch task to be inspected and provide feedback to the operation and maintenance personnel when the in-process inspection fails;
[0160] The second judgment module 56 is used to judge whether all the batch tasks to be checked have been executed when the in-process inspection passes;
[0161] The second response module 57 is used to call the post-check table in response to the post-check instruction when all the batch tasks to be checked are completed;
[0162] The post-check module 58 is used to perform a post-check on the second item to be checked in the data table obtained by executing the batch task to be checked according to the post-check rule configured in the post-check table.
[0163] The beneficial effects obtained by the device provided in the embodiments of this specification are consistent with the beneficial effects obtained by the above method and will not be repeated here.
[0164] like Figure 6 As shown, a computer device provided by an embodiment of this specification, the batch task quality supervision device in this specification can be a computer device in this embodiment, and the above method of this specification is executed. The computer device 602 may include one or more processors 604, such as one or more central processing units (CPUs), and each processing unit may implement one or more hardware threads. The computer device 602 may also include any memory 606, which is used to store any kind of information such as code, settings, data, etc. Non-limiting, for example, the memory 606 may include any one or more combinations of the following: any type of RAM, any type of ROM, flash memory device, hard disk, optical disk, etc. More generally, any memory can use any technology to store information. Further, any memory can provide volatile or non-volatile retention of information. Further, any memory can represent a fixed or removable component of the computer device 602. In one case, when the processor 604 executes an associated instruction stored in any memory or a combination of memories, the computer device 602 can perform any operation of the associated instruction. The computer device 602 also includes one or more drive mechanisms 608 for interacting with any storage, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.
[0165] The computer device 602 may also include an input / output module 610 (I / O) for receiving various inputs (via input devices 612) and for providing various outputs (via output devices 614). A specific output mechanism may include a presentation device 616 and an associated graphical user interface (GUI) 618. In other embodiments, the input / output module 610 (I / O), input device 612, and output device 614 may not be included, and the computer device 602 may be used as a computer device in a network. The computer device 602 may also include one or more network interfaces 620 for exchanging data with other devices via one or more communication links 622. One or more communication buses 624 couple the components described above together.
[0166] The communication link 622 may be implemented in any manner, for example, through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 622 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc. governed by any protocol or combination of protocols.
[0167] Corresponding to Figures 1 to 4In addition to the method shown, an embodiment of the specification also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are executed.
[0168] The embodiment of the present specification also provides a computer-readable instruction, wherein when the processor executes the instruction, the program therein causes the processor to execute the following Figures 1 to 4 The method shown.
[0169] The embodiments of the present specification also provide a computer program product, including at least one instruction or at least one program, wherein the at least one instruction or the at least one program is loaded and executed by a processor to implement the following Figures 1 to 4 The method shown.
[0170] It should be understood that in the various embodiments of this specification, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this specification.
[0171] It should also be understood that in the embodiments of this specification, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this specification generally indicates that the associated objects before and after are in an "or" relationship.
[0172] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this specification.
[0173] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0174] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.
[0175] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this specification.
[0176] In addition, each functional unit in each embodiment of this specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0177] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0178] Specific embodiments are used in this specification to illustrate the principles and implementation methods of this specification. The description of the above embodiments is only used to help understand the methods and core ideas of this specification. At the same time, for those skilled in the art, according to the ideas of this specification, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on this specification.
Claims
1. A batch task quality supervision method, characterized in that: include: In response to the in-process inspection instruction, calling the in-process inspection table; Determine a number of batch tasks to be checked according to the task names and the task groups to which they belong configured in the in-process checklist, wherein there is a dependency relationship between the batch tasks to be checked; Perform an in-process inspection on the first item to be inspected in the data table obtained by executing the batch task to be inspected according to the in-process inspection rule configured in the in-process inspection table; Determine whether the in-process inspection has passed; If not, the execution of the batch task to be checked is terminated and a reminder is fed back to the operation and maintenance personnel; If yes, then determine whether all the batch tasks to be checked have been completed; If so, in response to the post-check instruction, calling the post-check table; A post-inspection is performed on the second item to be inspected in the data table obtained by executing the batch task to be inspected according to the post-inspection rule configured in the post-inspection table.
2. The method according to claim 1, characterized in that The method further comprises: storing the first item to be inspected, the item number of the first item to be inspected, and the inspection date in an inspection log table; and The second item to be inspected, the item number of the second item to be inspected, and the inspection date are stored in the inspection log table.
3. The method according to claim 2, characterized in that Performing an in-process inspection on the first item to be inspected in the data table obtained by executing the batch task to be inspected according to the in-process inspection rule configured in the in-process inspection table further includes: If the in-process inspection rule is an inspection rule based on a predetermined threshold, comparing the first item to be inspected with the predetermined threshold according to a comparison rule in the in-process inspection rule; If the in-process inspection rule is an inspection rule based on historical data, the inspection log table is called and the historical inspection items in the inspection log table that are consistent with the project number of the first item to be inspected are obtained according to the comparison date in the in-process inspection rule, and the first item to be inspected is compared with the historical inspection items.
4. The method according to claim 3, characterized in that Comparing the first to-be-checked item with the historical checked item further includes: Obtaining a comparison coefficient and a comparison rule corresponding to the first item to be checked in the in-process inspection rule; Calculating a comparison value according to the comparison coefficient and the historical inspection item; Based on the comparison rule, the first item to be checked is compared with the comparison value.
5. The method according to claim 1, characterized in that In response to the in-process check instruction, calling the in-process check list further includes: Execute batch tasks to be checked; According to the execution result of the batch task to be checked, the execution identifier of the batch task to be checked is updated; Determine whether the execution flag of the batch task to be checked is completed; If so, respond to the in-progress inspection instruction to conduct an in-progress investigation on the data table obtained by executing the batch task to be inspected.
6. The method according to claim 1, characterized in that When the plurality of batch tasks to be checked are all executed, the method further includes: In response to the abnormal data sampling instruction, calling the abnormal data sampling table; According to the abnormal data, the sampling rules configured in the table are used to splice and obtain sampling statements; The data table obtained by executing the batch task to be checked is sampled according to the sampling statement, and the sampling result is stored in the sampling data table.
7. The method according to claim 6, characterized in that After the post-inspection and / or the abnormal data sampling is completed, the method further comprises: In response to the alarm instruction, calling the alarm configuration table; According to the alarm items configured in the alarm configuration table, searching for the post-inspection results and / or the abnormal data sampling results; According to the alarm rules configured in the alarm configuration table, an alarm is issued for the post-inspection result and / or the abnormal sampling result.
8. The method according to claim 2, characterized in that: The method further comprises: The in-process inspection result of the first item to be inspected and the post-inspection result of the second item to be inspected are stored in the inspection log table.
9. A batch task quality monitoring device, characterized in that: include: A first response module is used to respond to the in-process inspection instruction and call the in-process inspection table; A determination module, used to determine a number of batch tasks to be checked according to the task names and the task groups to which they belong configured in the in-process inspection table, wherein there is a dependency relationship between the batch tasks to be checked; An in-process inspection module, used for performing an in-process inspection on the first item to be inspected in the data table obtained by executing the batch task to be inspected according to the in-process inspection rules configured in the in-process inspection table; A first judgment module is used to judge whether the in-process inspection has passed; A suspension and reminder module, used to suspend the execution of the batch tasks to be inspected and provide feedback to the operation and maintenance personnel when the in-process inspection fails; A second judgment module is used to judge whether all the batch tasks to be checked have been executed when the in-process inspection passes; A second response module is used to call a post-check table in response to a post-check instruction when all the batch tasks to be checked are completed; The post-check module is used to perform a post-check on the second item to be checked in the data table obtained by executing the batch task to be checked according to the post-check rule configured in the post-check table.
10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.
11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.
12. A computer program product, characterized in that The method comprises at least one instruction or at least one program, wherein the at least one instruction or the at least one program is loaded and executed by a processor to implement the method according to any one of claims 1 to 8.