Scheduling task monitoring method and device, electronic equipment and readable storage medium

Abstract

Embodiments of the present application disclose a scheduling task monitoring method and device, electronic equipment and a readable storage medium, and relate to the technical field of data monitoring processing. After obtaining the time consumption of the current instance of a target scheduling task and the time consumption of historical instances within a preset time period, the time consumption threshold is obtained according to the time consumption of the historical instances within the preset time period, the time consumption of the current instance is compared with the time consumption threshold, and the target scheduling task is monitored according to the comparison result. In this way, the time consumption threshold is calculated according to the time consumption of the historical instances of the target scheduling task, the monitoring of the target scheduling task is realized, the abnormality of the scheduling task to be monitored can be automatically discovered, and the timeliness of the abnormality processing is improved.

Description

Technical Field

[0001] This invention relates to the field of data monitoring and processing, and more specifically, to a method, apparatus, electronic device, and readable storage medium for monitoring scheduling tasks. Background Technology

[0002] With the rapid development of international logistics business, the daily data volume is increasing rapidly, and customer requirements are also rising. However, the existing big data platform (BDP data platform) cannot automatically monitor scheduling tasks. When scheduling tasks are abnormal, managers often cannot be notified in time. The discovery of abnormal data often requires feedback from business parties, resulting in untimely handling of abnormal situations and an increase in complaints from business parties. Summary of the Invention

[0003] Based on the above research, embodiments of the present invention provide a scheduling task monitoring method, apparatus, electronic device, and readable storage medium to improve the above-mentioned problems.

[0004] In a first aspect, embodiments of the present invention provide a method for monitoring scheduled tasks, the method comprising:

[0005] Get the time taken for the current instance of the target scheduled task and the time taken for historical instances within a preset time period.

[0006] The time consumption threshold is obtained based on the time consumption of historical instances within the preset time period.

[0007] The time consumed by the current instance is compared with the time consumption threshold, and the target scheduling task is monitored based on the comparison result.

[0008] In one optional implementation, the step of obtaining the time consumption threshold based on the time consumption of historical instances within the preset time period includes:

[0009] Obtain the execution node information of historical instances of the target scheduling task within the preset time period.

[0010] Based on the execution node information, the historical instances are classified to obtain completed instances.

[0011] The time threshold is obtained based on the time taken to complete the instance.

[0012] In one optional implementation, the step of obtaining the time threshold based on the time taken to complete the instance includes:

[0013] Based on the time taken to complete the instance, calculate the time consumption index of the completed instance, and set the time consumption index as the time consumption threshold.

[0014] Alternatively, based on the time consumption index, the completed instances can be filtered to obtain target instances, and the time consumption of the target instances can be input into a preset time consumption prediction model to obtain a time consumption threshold.

[0015] In one optional implementation, before inputting the time consumption of the target instance into a preset time consumption prediction model to obtain a time consumption threshold, the method further includes:

[0016] Obtain sample time data of the scheduled tasks and preprocess the sample time data.

[0017] The preprocessed sample time consumption data is input into at least one initial time consumption prediction model to obtain the predicted value corresponding to each initial time consumption prediction model. Each initial time consumption prediction model corresponds to a parameter combination.

[0018] Based on the predicted values ​​corresponding to each of the initial time-consuming prediction models, the evaluation index of each initial time-consuming prediction model is calculated.

[0019] Based on the evaluation metrics of each initial time-consuming prediction model, a time-consuming prediction model is obtained.

[0020] In one optional implementation, the step of classifying the historical instances based on the execution node information includes:

[0021] Detect whether the execution node information of the historical instance includes start node information and end node information.

[0022] If start node information and end node information are included, then the historical instance is set as a completed instance.

[0023] If the start node information is included but the end node information is not included, then the historical instance is set as the current instance.

[0024] If the start node information is not included, the historical instance is set as an unexecuted instance.

[0025] In one optional implementation, after obtaining the time consumption threshold based on the historical instance consumption within the preset time period, the method further includes:

[0026] The time consumed by the historical instance is compared with the time consumption threshold, and the target scheduling task is monitored based on the comparison result.

[0027] In an optional implementation, after monitoring the target scheduling task based on the comparison results, the method further includes:

[0028] In response to the condition selection operation in the data quality monitoring configuration interface, at least one preset condition is determined.

[0029] Based on the preset conditions, generate script code for detecting data quality anomalies.

[0030] Run the script code to monitor the quality of the data generated by the target scheduling task.

[0031] In one optional implementation, the step of monitoring the quality of the data generated by the target scheduling task includes:

[0032] Detect whether the data generated by the target scheduling task meets preset conditions.

[0033] If the preset conditions are met, the test results data will be summarized into the normal table.

[0034] If the preset conditions are not met, the detection results data will be summarized in the alarm table.

[0035] Secondly, embodiments of the present invention provide a scheduling task monitoring device, comprising:

[0036] The data acquisition module is used to acquire the time consumption of the current instance of the target scheduling task and the time consumption of historical instances within a preset time period.

[0037] A threshold acquisition module is used to obtain a time consumption threshold based on the time consumption of historical instances within the preset time period.

[0038] The monitoring and processing module is used to compare the time consumed by the current instance with the time consumption threshold, and monitor the target scheduling task based on the comparison result.

[0039] Thirdly, embodiments of the present invention also provide an electronic device, the electronic device comprising:

[0040] The system includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the scheduling task monitoring method described in any of the preceding embodiments.

[0041] Fourthly, embodiments of the present invention provide a readable storage medium storing a computer program, which, when executed by a processor, implements the scheduling task monitoring method described in any of the foregoing embodiments.

[0042] The present invention provides a scheduling task monitoring method, apparatus, electronic device, and readable storage medium. After obtaining the time consumption of the current instance of the target scheduling task and the time consumption of historical instances within a preset time period, a time consumption threshold is obtained based on the time consumption of historical instances within the preset time period. The time consumption of the current instance is compared with the time consumption threshold. Based on the comparison result, the target scheduling task is monitored. In this way, the time consumption threshold is calculated based on the time consumption of historical instances of the target scheduling task, thereby realizing the monitoring of the target scheduling task. It can automatically detect anomalies in the monitored scheduling task and improve the timeliness of anomaly handling. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0045] Figure 2 This is a schematic diagram of data transmission for a target scheduling task provided in an embodiment of the present invention.

[0046] Figure 3 This is a flowchart illustrating a scheduling task monitoring method provided in an embodiment of the present invention.

[0047] Figure 4 This is another flowchart illustrating the scheduling task monitoring method provided in an embodiment of the present invention.

[0048] Figure 5 This is a flowchart illustrating the classification of historical instances provided in an embodiment of the present invention.

[0049] Figure 6 This is a schematic diagram of the process for obtaining the time prediction model provided in an embodiment of the present invention.

[0050] Figure 7 This is a schematic diagram of the data quality monitoring configuration interface provided in an embodiment of the present invention.

[0051] Figure 8 This is a block diagram of a scheduling task monitoring device provided in an embodiment of the present invention.

[0052] Icons: 100 - Electronic device; 10 - Scheduling task monitoring device; 11 - Data acquisition module; 12 - Threshold acquisition module; 13 - Monitoring and processing module; 20 - Memory; 30 - Processor; 40 - Communication unit. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0054] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0055] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0056] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0057] With the rapid development of logistics operations, the daily data volume is increasing rapidly, posing a significant challenge to the supporting management platform. The existing BDP big data platform is not yet capable of automatically monitoring scheduling tasks. Some instances exhibit anomalies, such as instances taking excessively long times, significantly exceeding the time taken by most historical instances; or instances containing only start node information but lacking end node information, yet still exhibiting unusually long execution times. When these scheduling tasks malfunction, managers often fail to notice in a timely manner, frequently only discovering the abnormal instances after feedback from business stakeholders. This leads to delayed handling of anomalies and an increase in customer complaints. Therefore, the current data platform's monitoring and management of scheduling tasks needs improvement.

[0058] Based on the above research, this embodiment provides a scheduling task monitoring method, device, electronic device, and readable storage medium. It acquires the time consumption of the current instance of the target scheduling task and the time consumption of historical instances within a preset time period. Based on the time consumption of historical instances within the preset time period, a time consumption threshold is obtained. The time consumption of the current instance is compared with the time consumption threshold. Based on the comparison result, the target scheduling task is monitored, thereby automatically detecting whether the monitored scheduling task has anomalies and improving the timeliness of anomaly handling.

[0059] Please see Figure 1 , Figure 1 This is a structural block diagram of an electronic device 100 provided in this embodiment. Figure 1 As shown, the electronic device may include a scheduling task monitoring device 10, a memory 20, a processor 30, and a communication unit 40. The memory 20 stores machine-readable instructions that can be executed by the processor 30. When the electronic device 100 is running, the processor 30 and the memory 20 communicate with each other via a bus. The processor 30 executes the machine-readable instructions and performs the scheduling task monitoring method.

[0060] The memory 20, processor 30, and communication unit 40 are electrically connected directly or indirectly to each other to achieve signal transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines. The scheduling task monitoring device 10 includes at least one software function module that can be stored in the memory 20 in the form of software or firmware. The processor 30 is used to execute the executable module (e.g., the software function module or computer program included in the scheduling task monitoring device 10) stored in the memory 20.

[0061] The memory 20 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0062] In some embodiments, processor 30 is used to perform one or more functions described in this embodiment. In some embodiments, processor 30 may include one or more processing cores (e.g., a single-core processor (S) or a multi-core processor (S)). By way of example only, processor 30 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), an application-specific instruction-set processor (ASIP), a graphics processing unit (GPU), a physical processing unit (PPU), a digital signal processor (DSP), a field-programmable gate array (FPGA), a programmable logic device (PLD), a controller, a microcontroller unit, a reduced instruction set computing (RISC) computer, or a microprocessor, or any combination thereof.

[0063] For ease of explanation, only one processor is described in electronic device 100. However, it should be noted that electronic device 100 in this embodiment may also include multiple processors, and therefore the steps performed by one processor as described in this embodiment may also be performed jointly or individually by multiple processors. For example, if the server's processor performs steps A and B, it should be understood that steps A and B may also be performed jointly by two different processors or individually by one processor. For example, one processor performs step A, and a second processor performs step B, or the first and second processors jointly perform steps A and B.

[0064] In this embodiment, the memory 20 is used to store the program, and the processor 30 is used to execute the program after receiving the execution instruction. The process definition method disclosed in any implementation of this embodiment can be applied to the processor 30, or implemented by the processor 30.

[0065] The communication unit 40 is used to establish a communication connection between the electronic device 100 and other devices via a network, and to send and receive data via the network.

[0066] In some implementations, the network can be any type of wired or wireless network, or a combination thereof. By way of example only, the network may include wired networks, wireless networks, fiber optic networks, telecommunications networks, intranets, the Internet, local area networks (LANs), wide area networks (WANs), wireless local area networks (WLANs), metropolitan area networks (MANs), public switched telephone networks (PSTNs), Bluetooth networks, ZigBee networks, or near field communication (NFC) networks, or any combination thereof.

[0067] In this embodiment, the electronic device 100 may be, but is not limited to, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), or other electronic devices. This embodiment does not impose any restrictions on the specific type of electronic device.

[0068] Understandably, Figure 1 The structure shown is for illustrative purposes only. The electronic device 100 may also have... Figure 1 Showing more or fewer components, or having with Figure 1 The different configurations shown. Figure 1 The components shown can be implemented using hardware, software, or a combination thereof.

[0069] It should also be noted that, see reference Figure 2 As shown, Figure 2 This diagram illustrates the data transmission of the scheduling task provided in this embodiment. From left to right, the diagram shows the acquisition of scheduling data for the scheduling task, the storage of the acquired scheduling data in a database, and the loading of the scheduling data into the BDP big data platform via the database. The aforementioned scheduling data includes log information, execution node information, and data generated during execution.

[0070] In one alternative implementation, custom UDF functions can be written in a script and injected into HIVE (the library for executing scheduled tasks). The custom UDF functions in the script retrieve scheduling data from the scheduled tasks. In offline scenarios, data computation can be performed using Hive scripts. The retrieved scheduling data is stored in a database, such as a MySQL database. The scheduling data in the database is then extracted, transformed, and loaded (ETL) before being fed into the BDP big data platform for processing. It should be noted that in HIVE, each SQL (Structured Query Language) statement requires one or more MapReduce tasks (a programming model suitable for distributed parallel computing in big data environments).

[0071] In this embodiment, Figure 1 The electronic device 100 shown can be deployed on the BDP big data platform or can communicate with the BDP big data platform to obtain information sent by the BDP big data platform, and realize the monitoring of target scheduling tasks and data processing, etc.

[0072] based on Figure 1 , Figure 2 The implementation architecture of this embodiment provides a method for monitoring scheduled tasks, which is based on the following: Figure 1 The electronic device shown performs the following based on Figure 1 The structural diagram of the electronic device 100 shown illustrates in detail the steps of the scheduling task monitoring method provided in this embodiment. Please refer to the attached diagram. Figure 3 The scheduling task monitoring method provided in this embodiment includes steps S101 to S103.

[0073] Step S101: Obtain the time consumption of the current instance of the target scheduled task and the time consumption of historical instances within a preset time period.

[0074] The target scheduling task is the scheduling task to be monitored. The target scheduling task can be determined based on actual conditions, and one or more can be selected. Once the target scheduling task is determined, it is used as the target scheduling task, and the target scheduling tasks are collected. The collected target scheduling tasks are then aggregated into the BDP big data platform. Next, the time consumption of the current instance of the target scheduling task and the time consumption of historical instances within a preset time period are obtained.

[0075] It should be noted that after the scheduled task has a set runtime, each run will generate an instance. In this embodiment, the instance includes the current instance and historical instances. The preset time period mentioned above generally uses a certain time in the present or the past as a node, and the period before this node is used as the preset time period.

[0076] In this embodiment, the current instance is used as the node, and a period of time preceding the current instance is used as the preset time period. For example, the week, month, or year preceding the current instance is used as the preset time period. Preferably, this embodiment uses one month preceding the current instance as the preset time period, and instances within the preset time period are historical instances within the preset time period.

[0077] Step S102: Obtain the time consumption threshold based on the historical instance consumption time within the preset time period.

[0078] Among them, after obtaining the time consumption of historical instances within a preset time period, the time consumption threshold can be obtained based on the time consumption of historical instances within the preset time period.

[0079] In one alternative implementation, the preset time period can be set to one month prior to the current instance, the time consumption of historical instances within this month can be obtained, and the time consumption of historical instances can be calculated or processed according to the set calculation method or processing method.

[0080] In one implementation, the calculation method is assumed to be to obtain the average time of historical instances within a preset time period, and to obtain the time of historical instances a, b, c, d...n within one month prior to the current instance. The time of historical instances a, b, c, d...n is then subjected to mathematical operations to obtain a value m. The value m can be the average of a, b, c, d...n or the median of a, b, c, d...n. The obtained value m is then used as the time consumption threshold.

[0081] In another implementation, the processing method is to classify the time consumption of historical instances within a preset time period, obtain historical instances within one month prior to the current instance, select a portion of the historical instances' time consumption as sample data, input it into a machine learning model, and use the sample data to train the machine learning model to obtain the time consumption threshold.

[0082] Step S103: Compare the time consumed by the current instance with the time consumption threshold, and monitor the target scheduling task based on the comparison result.

[0083] After obtaining the time consumption threshold, the time consumption of the current instance is compared with the time consumption threshold. Based on the comparison result, the target scheduling task can be monitored.

[0084] In this embodiment, the time consumption threshold can be used as a metric. The time consumption of the current instance is compared with the time consumption threshold. Based on the comparison result, the target scheduling task can be monitored to determine whether the current instance is abnormal. When comparing the time consumption of the current instance with the time consumption threshold, the time consumption of the current instance may be greater than the time consumption threshold, or it may be less than or equal to the time consumption threshold. If the time consumption of the current instance is greater than the time consumption threshold, it indicates that the big data platform has detected an anomaly in the current instance, and the big data platform issues an alarm; if the time consumption of the current instance is less than or equal to the time consumption threshold, it indicates that the current instance is normal.

[0085] This embodiment provides a scheduling task monitoring method. After obtaining the time consumption of the current instance of the target scheduling task and the time consumption of historical instances within a preset time period, a time consumption threshold is obtained based on the time consumption of historical instances within the preset time period. The time consumption of the current instance is compared with the time consumption threshold. Based on the comparison result, the target scheduling task is monitored. In this way, the time consumption threshold is calculated based on the time consumption of historical instances of the target scheduling task, thereby realizing the monitoring of the target scheduling task. It can automatically detect anomalies in the monitored scheduling task and improve the timeliness of anomaly handling.

[0086] To monitor scheduled tasks and improve alarm accuracy, this embodiment classifies historical instances to obtain completed instances. Based on the time taken for each completed instance, a time consumption standard is calculated to obtain a time consumption threshold. Therefore, this embodiment combines... Figure 4 As shown, the time consumption threshold is obtained based on the historical instance consumption within a preset time period, including steps S201 to S203:

[0087] Step S201: Obtain the execution node information of historical instances of the target scheduling task within a preset time period;

[0088] Step S202: Based on the execution node information, classify the historical instances to obtain the completed instances.

[0089] Step S203: Obtain the time threshold based on the time taken to complete the instance.

[0090] The scheduling data includes execution node information, task IDs, and responsible personnel. Therefore, execution node information can be obtained through a custom UDF function in the script and stored in a database. Then, the database's execution node information undergoes data extraction, transformation, and loading into the BDP big data platform. It's important to note that the database storing execution node information filters and loads the collected information into the BDP big data platform at fixed intervals, such as every hour or every three hours.

[0091] In this embodiment, the execution node information mentioned above generally refers to the time node of the execution of the historical instance stored in the log information. Common execution node information includes start node information and end node information. For example, a historical instance A started execution at 8:00 AM on February 1, 2020, and ended execution at 8:00 AM on February 3, 2020. Then the execution node information of historical instance A includes start node information and end node information, with the start node information being 8:00 AM on February 1, 2020, and the end node information being 8:00 AM on February 3, 2020. If the log information of a historical instance B only shows that it started execution at 8:00 AM on March 1, 2020, then the execution node information of historical instance B only includes start node information and does not include end node information; the start node information of historical instance B is 8:00 AM on March 1, 2020.

[0092] Optionally, in this embodiment, after obtaining the corresponding execution node information, historical instances can be classified based on the obtained execution node information. There are several methods for classifying historical instances based on execution node information. For example, the first classification method involves classifying historical instances whose execution node information includes both start and end node information into a first category, and classifying historical instances whose execution node information only includes start node information into a second category. The second classification method involves classifying historical instances whose execution node information shows a time interval greater than a set value between the end and start node information into a first category; classifying historical instances whose execution node information shows a time interval less than a set value into a second category; and classifying the remaining instances into a third category.

[0093] The above methods classify historical instances based on the characteristics of execution node information. In this embodiment, to facilitate understanding and implementation by those skilled in the art, combined with... Figure 5 As shown in the figure, this embodiment provides the specific steps of the first classification method:

[0094] Step S301: Detect whether the execution node information of the historical instance includes start node information and end node information.

[0095] Step S302: If start node information and end node information are included, then set the historical instance as a completed instance.

[0096] Step S303: If the start node information is included but the end node information is not included, then the historical instance is set as the current instance.

[0097] Step S304: If the start node information is not included, set the historical instance as an unexecuted instance.

[0098] The execution node information mentioned above is used to store the time of each node during the execution of a historical instance. It can be understood that execution node information includes start node information, pause node information, resume node information, stage node information, and end node information. It's easy to see that some historical instances include both start and end node information, some only include start node information, and some have neither.

[0099] For example, historical instance A started execution at 8:00 AM on July 1, 2020, completed the first phase at 9:00 AM on July 1, 2020, paused execution at 2:00 PM on July 1, 2020, resumed execution at 3:00 PM on July 1, 2020, and ended execution at 4:00 PM on July 1, 2020. Therefore, the execution node information for historical instance A includes 8:00 AM on July 1, 2020, 9:00 AM on July 1, 2020, 2:00 PM on July 1, 2020, 3:00 PM on July 1, 2020, and 4:00 PM on July 1, 2020. Historical instance B started execution at 8:00 AM on August 1, 2020, and ended execution at 9:00 AM on August 1, 2020. Therefore, the execution node information for historical instance B includes 8:00 AM and 9:00 AM on August 1, 2020. Historical instance C started execution at 8:00 AM on September 1, 2020. Therefore, the execution node information for historical instance C includes 8:00 AM on September 1, 2020. Among the examples listed above, historical instances A and B include both start and end node information, while historical instance C only includes start node information.

[0100] In this embodiment, based on the characteristics of execution node information, historical instances can be easily classified. Specifically, historical instances that include both start and end node information are set as completed instances. It is easy to understand that a completed instance means that the execution process of the historical instance is complete. Even if special circumstances occur during the execution process, such as pausing or resuming execution, as long as the historical instance eventually completes execution and has end node information, then the historical instance can be set as a completed instance.

[0101] Historical instances that include only start node information but not end node information are set as ongoing instances. This is because some historical instances may have started executing in the past, but their execution time and status may still be ongoing. In other words, the historical instance is still in a state of ongoing execution. Therefore, historical instances that include only start node information but not end node information are set as ongoing instances.

[0102] Historical instances that have neither start nor end node information are set as unexecuted instances. Following on from the above, it's easy to see that some historical instances, although planned for execution, have not yet reached their scheduled start time. Therefore, these historical instances have neither start nor end node information. This embodiment sets such historical instances as unexecuted instances.

[0103] After classifying historical instances, completed instances can be filtered out. Once completed instances are obtained, the time threshold can be calculated based on their completion time.

[0104] Specifically, in this embodiment, the step of obtaining the time threshold based on the time taken to complete the instance includes:

[0105] Calculate the time metric for completing an instance based on the time taken, and set the time metric as the time threshold.

[0106] After classifying historical instances, completed instances can be obtained. The execution time of a completed instance can be calculated from the start and end node information. For example, a historical instance A started execution at 8:00 AM on February 1, 2020, and ended execution at 8:00 AM on February 3, 2020. The start node information for historical instance A is 8:00 AM on February 1, 2020, and the end node information is 8:00 AM on February 3, 2020. The calculated execution time for historical instance A is 48 hours.

[0107] In this implementation, a time consumption index can be calculated by measuring the time taken to complete an instance. This index can be at least one of the mean, median, or mode of instance completion times within a preset time period. For example, if the mean time taken to complete instances within the preset time period is used as the time consumption index, and the time taken to complete instance A is 48 hours, instance B is 42 hours, instance C is 44 hours, and instance D is 50 hours, then the calculated time consumption index for completing instances A, B, and C is 46 hours. This time consumption index is then used as a time consumption threshold. If the current instance's time consumption exceeds the threshold, it indicates an abnormal timeliness of the current instance, and the big data platform can issue an alert.

[0108] It is easy to imagine that if we take a past instance Q as the monitoring object, obtain the time taken to complete the instance within a preset time period before instance Q, calculate the time consumption index by the time taken to complete the instance, use the time consumption index as the time consumption threshold, and then compare the time taken of instance Q with the time consumption threshold, we can also determine the timeliness of the time taken by instance Q.

[0109] It's worth considering that while using time-consuming metrics as time thresholds to monitor the current instance can detect some abnormal instances, the time consumption of some completed instances (either too long or too short) may not accurately reflect the typical time consumption of the scheduled task. Using the time consumption of these completed instances as the basis for calculating the time-consuming metric and then deriving the time-consuming threshold inevitably leads to inaccuracies in the threshold's accuracy. Therefore, the accuracy of monitoring the current instance using time-consuming metrics can be further improved.

[0110] Based on the above, in this embodiment, completed instances can be further filtered, and a time consumption threshold can be obtained using a time consumption prediction model.

[0111] Optionally, in this embodiment, the step of obtaining the time threshold based on the time taken to complete the instance can also be:

[0112] Based on the time consumption index, completed instances are filtered to obtain target instances. The time consumption of the target instances is then input into the preset time consumption prediction model to obtain the time consumption threshold.

[0113] The time consumption metric can be at least one of the mean, median, and mode of historical instance consumption within a preset time period, as mentioned above. This has already been explained and will not be repeated here. The difference lies in the fact that, in this implementation, the completed instances need further processing based on the time consumption metric. Specifically, the completed instances are filtered according to the time consumption metric, and only those that meet the requirements are retained as target instances.

[0114] Specifically, the execution node information of a completed instance in this embodiment generally includes start node information and end node information. However, considering that in some cases, special or abnormal situations may occur during the execution of historical instances, resulting in historical instances executing completely but with unique time consumption characteristics that cannot represent the true time consumption, it is necessary to filter completed instances based on time consumption metrics.

[0115] There are multiple ways to filter. In one implementation, the average time 'a' of completed instances within a preset time period is used as the time consumption indicator. If the time consumption of a completed instance is less than or equal to the average 'a', the completed instance is selected as the target instance.

[0116] In another implementation, the median b of the time taken to complete an instance within a preset time period is used as the time consumption indicator. If the time taken to complete an instance is less than or equal to the median b, the completed instance is taken as the target instance.

[0117] In another implementation, the average time d of completed instances within a preset time period is used as the time consumption index. A first threshold d1 and a second threshold d2 are set, and the second threshold d2 is greater than the first threshold d1. It is determined whether the time consumption of a completed instance is between the first threshold d1 and the second threshold d2. If the time consumption of a completed instance is between the first threshold d1 and the second threshold d2, the completed instance is taken as the target instance.

[0118] The above methods can all be used to filter completed instances. It should be noted that this embodiment does not impose specific limitations on the specific filtering method.

[0119] After obtaining the target instance, inputting the target instance into the time consumption prediction model yields the time consumption threshold. Optionally, in this embodiment, the time consumption prediction model can be a time series prediction model or an integrated moving average autoregressive model (ARIMAM model).

[0120] Optionally, the time consumption prediction model can be trained using both supervised and unsupervised training methods. In this embodiment, the time consumption prediction model can be trained using supervised training, that is, by training an optimal model using sample data with known true values, and then using this model to map all inputs to corresponding outputs, judging the outputs to achieve the purpose of prediction. The trained model then has the ability to predict unknown data.

[0121] Optional, please refer to Figure 6 As shown, the training process of the time-consuming prediction model may include steps S401 to S404:

[0122] Step S401: Obtain sample time data of the scheduled task and preprocess the sample time data.

[0123] Step S402: Input the preprocessed sample time data into at least one initial time prediction model to obtain the prediction value corresponding to each initial time prediction model; wherein, each initial time prediction model corresponds to a parameter combination.

[0124] Step S403: Calculate the evaluation index of each initial time-consuming prediction model based on the predicted values ​​corresponding to each initial time-consuming prediction model.

[0125] Step S404: Obtain the time prediction model based on the evaluation index of each initial time prediction model.

[0126] Before obtaining sample time data, it is necessary to first obtain historical instances within a preset time period of the target scheduling task, obtain completed instances from the historical instances, filter the completed instances, select the completed instances that are generally representative, and use the time of the filtered completed instances as sample time data.

[0127] It should be noted that the acquisition of completion instances can be based on the execution node information of historical instances. For example, the execution node information of historical instances can be obtained. Execution node information generally includes start node information and end node information. Historical instances whose execution node information includes both start and end node information are selected as completion instances. Considering that in some cases, the execution time of completion instances may be too long or too short due to special reasons, making the execution time of completion instances not universally representative, it is necessary to screen completion instances to reduce the impact on the initial execution time prediction model.

[0128] In this embodiment, completed instances can be filtered based on the time consumption index. The specific filtering method is described in the implementation method for obtaining target instances, and will not be repeated here. After obtaining the sample time consumption data of the scheduling task, the sample time consumption data is cleaned, and then preprocessed by stationarity analysis and white noise analysis.

[0129] After preprocessing the sample time-consuming data, the preprocessed sample time-consuming data is input into at least one initial time-consuming prediction model. It should be noted that multiple initial time-consuming prediction models are typically pre-established, each corresponding to a parameter combination (p, d, q), where p represents the hierarchy, q represents the order, and d represents the difference order. The input data for each initial time-consuming prediction model is the sample time-consuming data, and the output data is the predicted value. The parameter combinations of each initial time-consuming prediction model are not entirely the same; the same sample time-consuming data input into initial time-consuming prediction models with different parameter combinations will yield different predicted values. After establishing the initial time-consuming prediction models, the preprocessed sample time-consuming data is input into the initial time-consuming prediction models to obtain the predicted values ​​corresponding to each initial time-consuming prediction model.

[0130] After obtaining the predicted values, accuracy (ACC) can be used as an evaluation metric to determine which initial time-consuming prediction model is more suitable. When using accuracy as the evaluation metric for the initial time-consuming prediction model, the predicted values ​​of the initial time-consuming prediction model should be compared with the corresponding true values. The true values ​​are included in the sample time-consuming data. If the predicted value is within the preset tolerance range of the true value, the predicted value is considered correct. Finally, the proportion of predicted values ​​within the tolerance range to the total number of predicted values ​​is calculated to obtain the accuracy. The initial time-consuming prediction model with the highest accuracy is selected as the time-consuming prediction model in this embodiment. Commonly, this embodiment can also use the Akaike Information Content (AIC) as the evaluation index of the model, with the formula: AIC = (2k - 2L) / n, where n represents the amount of sample time-consuming data. The larger the amount of sample time-consuming data, the smaller this value is, meaning that the larger the amount of sample time-consuming data, the more convincing the corresponding initial time-consuming prediction model is; k represents the number of parameters in the initial time-consuming prediction model. It can be seen that the fewer the parameters in the initial time-consuming prediction model, the smaller the value of AIC; L represents the log-likelihood value. The larger L is, the more accurate the initial time-consuming prediction model is. Finally, the initial time-consuming prediction model with the smallest AIC is selected as the time-consuming prediction model in this embodiment.

[0131] It should be understood that other metrics that can evaluate the initial time-consuming prediction model can also be used as evaluation metrics in this embodiment, such as accuracy or Bayesian information content (BIC).

[0132] In an optional implementation, sample time consumption data can be input into the time consumption prediction model to obtain predicted values. These predicted values ​​are then compared with the corresponding known true values. Based on the difference between the predicted and true values, the parameters of the time consumption prediction model are adjusted. After parameter adjustment, the model iterates through the sample time consumption data, repeating the parameter adjustment process until a satisfactory time consumption prediction model is obtained. In this embodiment, a satisfactory time consumption prediction model is one where the difference between the predicted and true values ​​is minimized or convergence is achieved.

[0133] In practical applications, some abnormal historical instances of scheduling tasks may be discovered, but others may remain undetected. To improve the user experience for business users and reduce their complaints, it is necessary to monitor historical instances of scheduling tasks as well.

[0134] Optionally, in this embodiment, after obtaining the time consumption threshold based on the historical instance consumption time within a preset time period, the method further includes:

[0135] The time taken by historical instances is compared with the time threshold, and the target scheduling task is monitored based on the comparison results.

[0136] As mentioned earlier, the time consumed by a historical instance can be calculated using the end node information and start node information of the historical instance. After obtaining the time consumed by the historical instance, it is compared with a time consumption threshold. The acquisition of the time consumption threshold has been explained in the foregoing content of this embodiment and will not be repeated here. After comparison, if the time consumed by the historical instance is greater than the time consumption threshold, it indicates that the historical instance is abnormal. If the time consumed by the historical instance is less than or equal to the time consumption threshold, it indicates that the historical instance is normal.

[0137] For example, if a historical instance W is used as the monitoring object, the first step is to obtain the execution node information of historical instance W. The execution time t of historical instance W is calculated using the start and end node information. Taking the month preceding historical instance W as a preset time period, the execution time of historical instances within that month is obtained, and a time threshold T is calculated based on the execution time of historical instances within that month. The execution time t of the historical instance is compared with the time threshold T. If the execution time t of historical instance W is greater than the time threshold T, it indicates that historical instance W is abnormal. If the execution time t of historical instance W is less than or equal to the time threshold T, it indicates that historical instance W is normal.

[0138] Optionally, in this embodiment, after monitoring the target scheduling task based on the comparison results, the method further includes:

[0139] In response to the condition selection operation in the data quality monitoring configuration interface, at least one preset condition is determined.

[0140] Based on various preset conditions, generate script code for detecting data quality anomalies.

[0141] Run the script code to monitor the quality of the data generated by the target scheduling task.

[0142] In order to monitor the quality of data generated by scheduling tasks, a data quality monitoring configuration tool was developed, combined with... Figure 7 As shown, Figure 7The configuration interface for the data quality monitoring of this tool includes various controls such as rule template controls, scan range controls, alarm threshold controls, database name controls, table name controls, operator controls, custom rule controls, add rule controls, delete rule controls, and generate controls. The rule template controls allow you to select data for monitoring across technical quality dimensions, including completeness, accuracy, timeliness, uniqueness, and validity. The scan range controls include partition scan and full table scan. It's important to understand that data generated by the target scheduling task is stored in different tables within corresponding databases. Selecting partition scan monitors only a portion of the selected table, while full table scan monitors the entire table. The alarm threshold controls allow you to select specific data quality monitoring items, such as row count or time consumption. After specifying the monitoring item, you can choose the threshold value. The database name control allows you to select different database names, displaying the database names in the dropdown menu. After selecting the database name, the table name control displays the names of the corresponding data tables in the dropdown menu. The operator control displays the information of the operator performing this data quality monitoring. The custom rule control allows users to define their own quality monitoring rules. The add rule control allows users to add additional rules that need to monitor data quality. The delete rule control allows users to delete existing rules. The generate control generates a set of script code after the above control selection operations are completed. This script code is executed on the BDP big data platform.

[0143] Specifically, when monitoring data quality, users select from the corresponding controls based on preset conditions. For example, in the rule template control, they select "Consistency"; in the scan range control, they select "Partition Scan"; in the alarm threshold control, they select "Number of Rows 10"; in the database name control, they select "dwd_ibu"; and in the table name control, they select "dwd_ibu.alarm.process.intermediate_info". After completing the selection of the preset conditions on the controls, the data quality monitoring configuration interface responds to the selection of preset conditions, adds the required preset conditions to the rule summary, and then generates a set of script code. After the script code is executed on the BDP big data platform, it can complete the quality monitoring of the data generated by the target scheduling task stored in the database named "dwd_ibu".

[0144] Optionally, in this embodiment, the step of monitoring the quality of the data generated by the target scheduling task includes:

[0145] Check whether the data generated by the target scheduling task meets the preset conditions.

[0146] If the preset conditions are met, the test results data will be summarized in the normal table.

[0147] If the preset conditions are not met, the test results data will be summarized in the alarm table.

[0148] The preset conditions can be selected and set by the operator according to the data quality monitoring requirements. After the preset conditions are determined, the execution time can also be configured. For example, the execution time can be selected as daily, weekly, or monthly; or the execution time can be selected as a fixed time, which can be selected as a specific date and time; or the execution time can be selected as immediate. It is understood that there are various options for the execution time, and this embodiment does not limit it.

[0149] In this embodiment, after detecting the data generated by the target scheduling task, the detection results need to be summarized into a result table. The result table includes a normal table and an alarm table. The detection results include the detection results and related data, such as the database name, table name, preset detection conditions, alarm thresholds, etc.

[0150] The target scheduling task may or may not meet preset conditions. If it meets the preset conditions, the detection results are aggregated in the normal results table; if it does not, they are aggregated in the alarm table. Detection results aggregated in the normal table indicate that the monitored scheduling task meets the preset conditions, while those aggregated in the alarm table indicate that it does not. When a monitored scheduling task fails to meet the preset conditions, the BDP big data platform generates an alarm and periodically calls the communication interface to notify the responsible party in real time. There are several ways to notify the responsible party, such as sending an SMS to inform them that the monitored scheduling task does not meet the preset conditions and requires immediate attention. Alternatively, messages can be sent via social media apps, such as WeChat or QQ, to inform them that the monitored scheduling task does not meet the preset conditions and requires immediate attention. After receiving the alarm message, the responsible party needs to process the data of the alarmed scheduling task, analyze the cause of the alarm, and check the impact of the abnormal data quality. Once the responsible person has completed the task, the processed result will be inserted into the alarm processing result table. If the responsible person still hasn't handled it by the next day, the BDP big data platform will still notify the responsible person the following day, reminding them to handle it promptly.

[0151] After completing the above data quality monitoring, the processed information can be pushed to the BDP big data platform, where all results can be summarized and visualized in the data portal.

[0152] The scheduling task monitoring method provided in this embodiment obtains the time consumption of the current instance of the target scheduling task and the time consumption of historical instances within a preset time period. Based on the time consumption of historical instances within the preset time period, a time consumption threshold is obtained. The time consumption of the current instance is compared with the time consumption threshold. Based on the comparison result, the target scheduling task is monitored. In this way, the time consumption threshold is calculated based on the time consumption of historical instances of the target scheduling task, realizing the monitoring of the target scheduling task and automatically detecting anomalies in the monitored scheduling task, thus improving the timeliness of anomaly handling.

[0153] Based on the same inventive concept, please refer to the following: Figure 8 This embodiment also provides a scheduling task monitoring device 10, which applies... Figure 1 The electronic devices shown, such as Figure 8 As shown, the scheduling task monitoring device 10 provided in this embodiment includes a data acquisition module 11, a threshold acquisition module 12, and a monitoring processing module 13.

[0154] The data acquisition module 11 is used to acquire the time consumption of the current instance of the target scheduling task and the time consumption of historical instances within a preset time period.

[0155] The threshold acquisition module 12 is used to obtain the time consumption threshold based on the time consumption of historical instances within a preset time period.

[0156] The monitoring and processing module 13 is used to compare the time consumed by the current instance with the time consumption threshold, and monitor the target scheduling task based on the comparison result.

[0157] In an optional implementation, the threshold acquisition module 12 is used for:

[0158] Obtain the execution node information of historical instances of the target scheduling task within a preset time period; classify the historical instances based on the execution node information to obtain the completed instances.

[0159] The time threshold is obtained based on the time taken to complete the instance.

[0160] In an optional implementation, the threshold acquisition module 12 is used for:

[0161] Calculate the time metric for completing an instance based on the time taken, and set the time metric as the time threshold.

[0162] In an optional implementation, the threshold acquisition module 12 is used for:

[0163] Based on the time consumption index, completed instances are filtered to obtain target instances. The time consumption of the target instances is then input into the preset time consumption prediction model to obtain the time consumption threshold.

[0164] In an optional implementation, the scheduling task monitoring device provided in this embodiment further includes a model training module, which is used for:

[0165] Before inputting the time consumption of the target instance into the preset time consumption prediction model to obtain the time consumption threshold, sample time consumption data of the scheduled task is obtained and the sample time consumption data is preprocessed.

[0166] The preprocessed sample time data is input into at least one initial time prediction model to obtain the prediction value corresponding to each initial time prediction model; wherein, each initial time prediction model corresponds to a parameter combination.

[0167] Based on the predicted values ​​corresponding to each initial time-consuming prediction model, the evaluation index of each initial time-consuming prediction model is calculated.

[0168] Based on the evaluation metrics of each initial time-consuming prediction model, the time-consuming prediction model is obtained.

[0169] In an optional implementation, the threshold acquisition module 12 is used for:

[0170] Check whether the execution node information of historical instances includes start node information and end node information.

[0171] If start node information and end node information are included, then the historical instance is set as the completed instance.

[0172] If start node information is included but end node information is not included, then the historical instance is set as the current instance.

[0173] If the start node information is not included, the historical instance will be set as an unexecuted instance.

[0174] In an optional implementation, the monitoring and processing module 13 is used to: compare the time consumed by historical instances with the time consumption threshold, and monitor the target scheduling task based on the comparison result.

[0175] In an optional implementation, the scheduling task monitoring device provided in this embodiment further includes a quality detection module, which is used for:

[0176] Based on the comparison results, after monitoring the target scheduling task, in response to the condition selection operation of the data quality monitoring configuration interface, at least one preset condition is determined.

[0177] Based on various preset conditions, generate script code for detecting data quality anomalies.

[0178] Run the script code to monitor the quality of the data generated by the target scheduling task.

[0179] In an optional implementation, the quality inspection module is used for:

[0180] Check whether the data generated by the scheduled task to be monitored meets the preset conditions.

[0181] If the preset conditions are met, the test results data will be summarized in the normal table.

[0182] If the preset conditions are not met, the test results data will be summarized in the alarm table.

[0183] The scheduling task monitoring device provided in this embodiment, after obtaining the time consumption of the current instance of the target scheduling task and the time consumption of historical instances within a preset time period, obtains a time consumption threshold based on the time consumption of historical instances within the preset time period, compares the time consumption of the current instance with the time consumption threshold, and monitors the target scheduling task based on the comparison result. In this way, the time consumption threshold is calculated based on the time consumption of historical instances of the target scheduling task, realizing the monitoring of the target scheduling task, and can automatically detect anomalies in the monitored scheduling task, improving the timeliness of anomaly handling.

[0184] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the scheduling task monitoring device described above can be referred to the corresponding process in the aforementioned method, and will not be elaborated further here.

[0185] Based on the above, this embodiment provides a readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements the scheduling task monitoring method of any of the aforementioned embodiments.

[0186] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the readable storage medium described above can be referred to the corresponding process in the aforementioned method, and will not be elaborated further here.

[0187] In summary, the scheduling task monitoring method, apparatus, electronic device, and readable storage medium provided in this embodiment of the invention, after obtaining the time consumption of the current instance of the target scheduling task and the time consumption of historical instances within a preset time period, obtains a time consumption threshold based on the time consumption of historical instances within the preset time period, compares the time consumption of the current instance with the time consumption threshold, and monitors the target scheduling task based on the comparison result. In this way, by calculating the time consumption threshold based on the time consumption of historical instances of the target scheduling task, the monitoring of the target scheduling task is realized, and anomalies in the monitored scheduling task can be automatically detected, improving the timeliness of anomaly handling.

[0188] The above descriptions are merely various embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for monitoring scheduled tasks, characterized in that, The method includes: Get the time taken for the current instance of the target scheduled task and the time taken for historical instances within a preset time period; Obtain execution node information of historical instances of the target scheduling task within the preset time period; classify the historical instances according to the execution node information to obtain completed instances, including: detecting whether the execution node information of the historical instance includes start node information and end node information; if it includes the start node information and the end node information, then set the historical instance as a completed instance; if it includes the start node information but does not include the end node information, then set the historical instance as a running instance; if it does not include the start node information, then set the historical instance as an unexecuted instance. Calculate the time consumption index of the completed instance based on the time consumption of the completed instance, and obtain the time consumption threshold based on the time consumption index; The time consumed by the current instance is compared with the time consumption threshold, and the target scheduling task is monitored based on the comparison result. In response to the condition selection operation in the data quality monitoring configuration interface, at least one preset condition is determined; Based on the preset conditions, generate script code for detecting data quality anomalies; Run the script code to monitor the quality of the data generated by the target scheduling task.

2. The scheduling task monitoring method according to claim 1, characterized in that, The steps for obtaining the time consumption threshold based on the time consumption index include: Set the time consumption indicator as a time consumption threshold; or... Based on the time consumption index, the completed instances are filtered to obtain target instances. The time consumption of the target instances is then input into a preset time consumption prediction model to obtain a time consumption threshold.

3. The scheduling task monitoring method according to claim 2, characterized in that, Before inputting the time consumption of the target instance into a preset time consumption prediction model to obtain the time consumption threshold, the method further includes: Obtain sample time data of the scheduled task, and preprocess the sample time data; The preprocessed sample time consumption data is input into at least one initial time consumption prediction model to obtain the prediction value corresponding to each initial time consumption prediction model; wherein, each initial time consumption prediction model corresponds to a parameter combination; Based on the predicted values ​​corresponding to each of the initial time consumption prediction models, calculate the evaluation index of each of the initial time consumption prediction models; Based on the evaluation metrics of each initial time-consuming prediction model, a time-consuming prediction model is obtained.

4. The scheduling task monitoring method according to any one of claims 1-3, characterized in that, After obtaining the time consumption threshold based on the historical instance consumption within the preset time period, the method further includes: The time consumed by the historical instance is compared with the time consumption threshold, and the target scheduling task is monitored based on the comparison result.

5. The scheduling task monitoring method according to claim 1, characterized in that, The step of monitoring the quality of the data generated by the target scheduling task includes: Detect whether the data generated by the target scheduling task meets preset conditions; If the preset conditions are met, the test results data will be summarized into the normal table; If the preset conditions are not met, the detection results data will be summarized in the alarm table.

6. A scheduling task monitoring device, characterized in that, include: The data acquisition module is used to acquire the time consumption of the current instance of the target scheduling task and the time consumption of historical instances within a preset time period. A threshold acquisition module is used to acquire the execution node information of historical instances of the target scheduling task within the preset time period; Based on the execution node information, the historical instances are classified to obtain completed instances, including: detecting whether the execution node information of the historical instance includes start node information and end node information; if it includes the start node information and end node information, the historical instance is set as a completed instance; if it includes the start node information but does not include the end node information, the historical instance is set as a progress instance; if it does not include the start node information, the historical instance is set as an unexecuted instance; calculating the time consumption index of the completed instance based on the time consumption of the completed instance, and obtaining a time consumption threshold based on the time consumption index; A monitoring and processing module is used to compare the time consumed by the current instance with the time consumption threshold, and monitor the target scheduling task based on the comparison result. A quality detection module is used to respond to the condition selection operation of the data quality monitoring configuration interface, determine at least one preset condition, generate script code for detecting data quality anomalies based on each preset condition, and run the script code to monitor the quality of the data generated by the target scheduling task.

7. An electronic device, characterized in that, The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the scheduling task monitoring method according to any one of claims 1-5.

8. A readable storage medium, characterized in that, The readable storage medium stores a computer program, which, when executed by a processor, implements the scheduling task monitoring method according to any one of claims 1-5.

Images