Task monitoring method, device, electronic device and readable storage medium

Abstract

Embodiments of the present invention provide a task monitoring method, apparatus, electronic device, and readable storage medium. The method includes: obtaining a Spark task to be monitored; and obtaining a target first execution time for the Spark task when it is executed at the scheduled time to be monitored, where the first execution time is the total execution time of each job included in the Spark task. Therefore, in embodiments of the present invention, the accuracy of Spark task execution time calculation is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a task monitoring method, device, electronic device and readable storage medium. Background Art

[0002] Currently, Spark task scheduling in the Spark Service model uses a fair mode. When concurrent Spark Service requests are made, the execution time of each task can be calculated. Furthermore, in the prior art, the time interval between a task request and its completion time is used as the task execution time.

[0003] Among them, when a task is executed, there are multiple parallel tasks in the scheduling queue of the Spark task, and the execution order is based on the priority of the task with the highest priority. However, since the scheduling queue may be blocked for a long time, the actual execution time of a task in the scheduling queue may exceed its normal execution time (i.e., historical execution time). Then, after the request time of the task is obtained, the normal execution time of the task has passed, but the request completion time of the task has not yet been obtained, resulting in a large difference between the execution time of the task calculated this time and the normal execution time of the task, which leads to inaccurate calculation of the task execution time. As can be seen from the above, the existing method for calculating the execution time of Spark tasks will lead to inaccurate calculation of the task execution time when the scheduling queue is blocked for a long time. Summary of the Invention

[0004] Embodiments of the present invention provide a task monitoring method, device, electronic device, and readable storage medium for improving the accuracy of monitoring Spark tasks.

[0005] In order to solve the above problems, an embodiment of the present invention discloses a task monitoring method, which includes:

[0006] Get the Spark task to be monitored;

[0007] Obtain a target value of a first execution time when the Spark task is executed at the run scheduling time to be monitored, wherein the first execution time is the total execution time of each job included in the Spark task.

[0008] The embodiment of the present invention further discloses a task monitoring device, comprising:

[0009] The first acquisition module is used to obtain the Spark task to be monitored;

[0010] The second acquisition module is used to obtain a target value of a first execution time when the Spark task is executed at the running scheduling time to be monitored, wherein the first execution time is the total execution time of each Job included in the Spark task.

[0011] The embodiments of the present invention include the following advantages:

[0012] In an embodiment of the present invention, a Spark task to be monitored can be obtained; thereby obtaining a target value of a first execution time when the Spark task is executed at the running scheduling time to be monitored, wherein the first execution time is the total execution time of each Job included in the Spark task.

[0013] Among them, a Spark task includes multiple jobs, a job corresponds to multiple stages, a stage corresponds to multiple tasks, and the multiple stages included in a job are executed in parallel. The execution time of the last completed stage in a job is the completion time of the job; similarly, if the multiple tasks included in a stage are executed in parallel, the execution time of the last completed task in a stage is the completion time of the stage. Therefore, the aforementioned first execution time does not include the waiting time of the scheduling queue.

[0014] Thus, in embodiments of the present invention, the aforementioned first execution time can be used to describe the actual execution time required for each job included in a Spark task. Specifically, embodiments of the present invention monitor tasks at the job level within a Spark task, accurately obtaining the actual execution time required for each job included in the Spark task, thereby improving the accuracy of calculating the Spark task execution time. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 is a schematic diagram of a task monitoring method provided by an embodiment of the present invention;

[0017] Figure 2 This is a schematic diagram of the interaction process of each terminal device in the Spark Service mode provided by an embodiment of the present invention;

[0018] Figure 3Schematic diagram of the relationship between Spark indicators provided by an embodiment of the present invention;

[0019] Figure 4 This is a structural block diagram of a task monitoring device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] To facilitate understanding of the task monitoring method provided by the embodiment of the present invention, the following contents are first introduced:

[0022] like Figure 2 As shown in the figure, the Spark Service mode is to start the Spark Service by submitting a task through the Spark client (Client). At the same time, a Hypertext Transfer Protocol server (HttpServer) will be started on the Spark driver (Driver), which can interact with the World Wide Web server (Web Server). The Http Server receives the request from the Web Server, parses the request parameters, generates Spark tasks based on the domain name (GroupId) as a group, and sends them to the executor (Executor) for execution. After the execution is completed, the execution result is returned to the Spark Driver, and finally the execution result is returned to the requesting end, that is, the Web Server.

[0023] like Figure 1 FIG. 1 is a flow chart of a task monitoring method according to an embodiment of the present invention. The method may include the following steps:

[0024] Step 101: Get the Spark task to be monitored.

[0025] Among them, such as Figure 3 As shown, a Spark application (App) includes multiple GroupIds; a Spark task generated by a GroupId includes multiple Jobs, a Job includes multiple Stages, and a Stage includes multiple Tasks. In the embodiments of the present invention, Spark tasks are monitored at the Job level, monitoring each Job within a Spark task separately. Therefore, when monitoring a Spark task, a GroupId corresponds to multiple Jobs, meaning that these multiple Jobs can be considered a single Spark task.

[0026] In addition, Spark is a fast and general computing engine designed for large-scale data processing.

[0027] In an embodiment of the present invention, a preset time interval may be set so that Spark tasks to be monitored are acquired at preset time intervals, and the acquired Spark tasks are then monitored (i.e., the task monitoring method of an embodiment of the present invention may be used to periodically monitor the Spark tasks of the Spark App); alternatively, upon receiving a preset operation of the user (e.g., pressing a physical button or a combination of physical buttons set on the execution end), a Spark task acquisition instruction may be generated, thereby triggering the execution of step 101.

[0028] Step 102: Obtain a target value of a first execution time when the Spark task is executed at the scheduled execution time to be monitored.

[0029] The first execution time is the total execution time of each job included in the Spark task.

[0030] It can be seen from the above steps 101 to 102 that in this embodiment of the present invention, the Spark task to be monitored can be obtained; thereby, the target value of the first execution time when the Spark task is executed at the running scheduling time to be monitored is obtained, wherein the first execution time is the total execution time of each Job included in the Spark task.

[0031] Among them, a Spark task includes multiple jobs, a job corresponds to multiple stages, a stage corresponds to multiple tasks, and the multiple stages included in a job are executed in parallel. The execution time of the last completed stage in a job is the completion time of the job; similarly, if the multiple tasks included in a stage are executed in parallel, the execution time of the last completed task in a stage is the completion time of the stage. Therefore, the aforementioned first execution time does not include the waiting time of the scheduling queue.

[0032] Thus, in embodiments of the present invention, the aforementioned first execution time can be used to describe the actual execution time required for each job included in a Spark task. Specifically, embodiments of the present invention monitor tasks at the job level within a Spark task, accurately obtaining the actual execution time required for each job included in the Spark task, thereby improving the accuracy of Spark task execution time.

[0033] Optionally, the process of obtaining the first execution time includes:

[0034] When x is an integer from 1 to y, obtain the first submission time and first completion time of the xth job included in the Spark task;

[0035] Determine the time interval between the first submission time and the first completion time of the x-th job as the first execution time of the x-th job, where y is the number of jobs included in the Spark task;

[0036] The sum of the execution times of the 1st to yth jobs is calculated as the first execution time.

[0037] Among them, the Spark task includes multiple jobs, so the first execution time (First completion time of the xth job - First submission time of the xth job), where y represents the number of jobs included in the Spark task.

[0038] For example, a Spark task includes two jobs. The first submission time of the first job is T1, and the first completion time of the first job is T2. Then the execution time of the first job is "T2-T1"; the first submission time of the second job is T3, and the first completion time of the second job is T4. Then the execution time of the second job is "T4-T3"; the first execution time is "(T2-T1)+(T4-T3)".

[0039] In addition, if a job includes multiple stages, the execution time of a job is the execution time of the last completed stage during the execution of multiple stages.

[0040] Optionally, the method further includes:

[0041] Input the running scheduling time to be monitored into a predetermined execution time prediction model corresponding to the Spark task, and output a value of the predicted execution time;

[0042] Whether there is a problem in the execution process of the Spark task is determined according to the value of the predicted execution time and the target value of the first execution time.

[0043] The execution time prediction model corresponding to the Spark task is used to predict the execution time of the Spark task within the monitored running schedule. The input of the execution time prediction model is the monitored running schedule, and the output is the value of the predicted execution time of the Spark task. Therefore, in an embodiment of the invention, the monitored running schedule can be input into the execution time prediction model to obtain the value of the predicted execution time of the Spark task within the monitored running schedule. Therefore, based on the predicted execution time value and the target value of the first execution time, it can be determined whether there is a problem in the execution process of the Spark task.

[0044] From the above, it can be seen that the embodiment of the present invention performs task monitoring from the Job level under the Spark task, and can accurately obtain the actual time required for the execution of each Job included in the Spark task (i.e., the first execution time), thereby improving the accuracy of the Spark task execution time, and then based on the first execution time, the Spark task can be monitored more accurately.

[0045] Furthermore, it should be noted that different Spark tasks may have different architectures, i.e., they may include different numbers of jobs, stages, and tasks. Therefore, in the embodiments of the present invention, each Spark task with a specific architecture corresponds to a specific execution time prediction model. Therefore, when determining the predicted execution time of a Spark task with a specific architecture, the Spark task's running schedule time must be input into the execution time prediction model corresponding to the Spark task, thereby meeting the monitoring requirements for different Spark tasks.

[0046] Optionally, the process of establishing the execution time prediction model corresponding to the Spark task includes:

[0047] Obtaining historical values ​​of the first execution time of the Spark task;

[0048] Obtain a historical value of the second execution time of the Spark task, wherein the second execution time is the total execution time of each scheduling stage included in each job;

[0049] Obtain a historical value of the third execution time of the Spark task, wherein the third execution time is the total execution time of each Task included in each scheduling stage;

[0050] Using a clustering algorithm, removing abnormal data from the historical values ​​of the first execution time, the historical values ​​of the second execution time, and the historical values ​​of the third execution time;

[0051] The historical values ​​of the first execution time, the second execution time, and the third execution time corresponding to the same running schedule time are weighted and summed to obtain the historical value of the predicted execution time corresponding to the running schedule time;

[0052] The execution time prediction model is obtained according to historical values ​​of the predicted execution time and the running scheduling time corresponding to the historical values ​​of the predicted execution time.

[0053] Clustering, also known as cluster analysis or group analysis, is a statistical analysis method for studying sample or indicator classification problems and is also a key algorithm in data mining. Therefore, using a clustering algorithm can remove abnormal data from the data used to generate the execution time prediction model, making the resulting execution time prediction model more accurate.

[0054] Furthermore, the execution time prediction model is a functional relationship between the scheduled execution time and the predicted execution time. This functional relationship can be derived using a linear regression algorithm based on the historical values ​​of the predicted execution time and the scheduled execution time corresponding to these historical values. Therefore, by substituting the scheduled execution time into this functional relationship, the predicted execution time value can be obtained.

[0055] From the above, it can be seen that in an embodiment of the present invention, after removing abnormal data through a clustering algorithm, each running scheduling time corresponds to a set of first execution time, second execution time and third execution time. The value obtained by weighted summation of a set of first execution time, second execution time and third execution time represents the predicted execution time of the Spark task, thereby determining the execution time prediction model based on each running scheduling time and the corresponding predicted execution time of the Spark task.

[0056] Optionally, the adopting a clustering algorithm to remove abnormal data from the historical values ​​of the first execution time, the historical values ​​of the second execution time, and the historical values ​​of the third execution time includes:

[0057] Obtain the number of jobs included in the Spark task, the number of stages included in each job, the number of tasks included in each stage, and the termination reasons of the tasks;

[0058] Based on the number of the jobs, the number of the scheduling stages, the number of the tasks, and the reasons for the termination of the tasks, a clustering algorithm is used to remove abnormal data from the historical values ​​of the first execution time, the historical values ​​of the second execution time, and the historical values ​​of the third execution time.

[0059] Among them, the number thresholds of each job, each stage, and each task included in the Spark task can be determined in advance. If the number of each job, each stage, and each task actually executed each time during the historical execution of the Spark task reaches each number threshold, it is represented as normal data; if the number of each job, or the number of each stage, or the number of each task actually executed each time during the historical execution of the Spark task does not reach each number threshold, it is represented as abnormal data.

[0060] In addition, after a Spark task completes, you can also obtain the task's termination reason, which is the reason why the Spark task ended. Based on this termination reason, you can determine whether the Spark task's historical value data is abnormal. For example, if the task's termination reason during a Spark task's history is due to a system error, the data from that execution is abnormal. If the task's termination reason is normal, the data from that execution is normal.

[0061] Optionally, the process of obtaining the second execution time includes:

[0062] When i is an integer from 1 to n, obtain the second submission time and the second completion time of the i-th stage included in each job;

[0063] Determine the time interval between the second submission time and the second completion time of the i-th stage as the execution time of the i-th stage, where n is the number of stages included in each job;

[0064] The sum of the execution times of the 1st to nth stages is calculated as the second execution time.

[0065] Among them, each job includes multiple stages, so the second execution time (The second completion time of the i-th stage - the second submission time of the i-th stage), where n represents the number of stages included in each job.

[0066] For example, a job includes two stages. The second submission time of the first stage is T5, and the second completion time of the first stage is T6. Then the execution time of the first stage is "T6-T5"; the second submission time of the second stage is T7, and the second completion time of the second stage is T8, then the execution time of the second stage is "T8-T7"; the second execution time is "(T6-T5)+(T8-T7)".

[0067] In addition, if a Stage includes multiple Tasks, the execution time of a Stage is the execution time of the last completed Task during the execution of multiple Tasks.

[0068] Optionally, the process of obtaining the third execution time includes:

[0069] When j is an integer from 1 to m, obtain the third submission time and the third completion time of the j-th task included in each scheduling stage;

[0070] Determine the time interval between the third submission time and the third completion time of the j-th Task as the execution time of the j-th Task, where m is the number of Tasks included in each scheduling stage;

[0071] The sum of the execution times of the 1st to mth tasks is calculated as the third execution time.

[0072] Among them, a Job includes multiple Stages, and a Stage includes multiple Tasks. The above third execution time (the third completion time of the jth task - the third submission time of the jth task), m represents the number of tasks included in each scheduling stage.

[0073] For example, a Stage includes two Tasks. The third submission time of the first Task is T9, and the third completion time of the first Task is T10. Then the execution time of the first Task is "T10-T9"; the third submission time of the second Task is T11, and the third completion time of the second Task is T12, then the execution time of the second Task is "T12-T11"; the third execution time is "(T10-T9)+(T12-T11)".

[0074] Optionally, before the Spark task is executed, the method further includes:

[0075] Input the running scheduling time to be monitored into a predetermined execution time prediction model corresponding to the Spark task, and output a value of the predicted execution time;

[0076] During the execution of the Spark task, the method further includes:

[0077] Starting from the scheduled operation time to be monitored, at every preset time interval, determine whether the Spark task is completed;

[0078] When the Spark task is not completed, determining whether there is a problem in the execution process of the Spark task according to the target time interval and the value of the predicted execution time, wherein the target time interval is the target time interval between the current time and the running scheduling time to be monitored;

[0079] When the Spark task is executed, the step of determining whether there is a problem in the execution process of the Spark task according to the target time interval and the value of the predicted execution time is skipped.

[0080] Among them, different preset time intervals can be set for different Spark tasks, which are used to compare the time interval between the current time and the running scheduling time of the monitored Spark task during execution with the value of the predicted execution time output by the execution time prediction model corresponding to the Spark task under the same running scheduling time.

[0081] As can be seen from the above, in embodiments of the present invention, during the execution of a Spark task, its run scheduling time can be input into the execution time prediction model corresponding to the Spark task, and the predicted execution time value can be output. Thus, during the execution of the Spark task, at preset time intervals, it is determined whether the Spark task is completed. If not, the interval between the current time and the run scheduling time (i.e., the start time) of the Spark task is used to determine whether there is a problem in the execution process of the Spark task. In other words, in embodiments of the present invention, it is also possible to determine whether there is a problem in the execution process of the Spark task in real time during the execution of the Spark task, thereby promptly discovering the problem.

[0082] Optionally, determining whether there is a problem in the execution process of the Spark task based on the target time interval and the value of the predicted execution time includes:

[0083] Calculating a difference between the target time interval and the predicted execution time;

[0084] When the difference is greater than a second preset threshold, determining that there is a problem in the execution process of the Spark task;

[0085] When the difference is less than or equal to the second preset threshold, it is determined that there is no problem in the execution process of the Spark task.

[0086] Among them, the difference between the above-mentioned target time interval and the predicted execution time is greater than the second preset threshold, which means that the time elapsed before the Spark task is completed has exceeded the predicted execution time (that is, the normal execution time of the Spark task), that is, the normal execution time of the Spark task has been reached before the Spark task is completed. Therefore, in this case, it can be determined that there is a problem in the execution process of the Spark task.

[0087] Optionally, after determining whether there is a problem in the execution process of the Spark task, the method further includes:

[0088] When it is determined that there is a problem in the execution process of the Spark task, a preset alarm operation is performed.

[0089] Among them, the preset alarm operation can be at least one of displaying a prompt message indicating that there is a problem in the Spark task execution process, sending an alarm message to a predetermined electronic device (i.e., a text message alarm), making a call to a predetermined communication number (i.e., a telephone alarm), and sending an alarm email to a predetermined email address (i.e., an email alarm).

[0090] In summary, the specific implementation of the task monitoring method according to the embodiment of the present invention may include steps H1 to H8 as follows:

[0091] Step H1: Get the Spark task to be monitored;

[0092] Step H2: Get the running schedule time of the Spark task to be monitored;

[0093] Step H3: Input the running schedule time to be monitored into the predetermined execution time prediction model corresponding to the Spark task, and output the predicted execution time value; the process of establishing the execution time prediction model is described above and will not be repeated here;

[0094] Step H4: starting from the scheduled operation time to be monitored, at every preset time interval, determine whether the Spark task is completed;

[0095] Step H5: When the Spark task is not completed, the difference between the target time interval and the predicted execution time is calculated. If the difference is greater than a second preset threshold, it is determined that there is a problem in the execution process of the Spark task; if the difference is less than or equal to the second preset threshold, it is determined that there is no problem in the execution process of the Spark task, wherein the target time interval is the target time interval between the current time and the running schedule time to be monitored;

[0096] Step H6: When or after the Spark task is completed, obtain a target value of the first execution time, where the first execution time is the total execution time of each job included in the Spark task;

[0097] Step H7: Calculate the absolute value of the difference between the target value of the first execution time and the value of the predicted execution time, and if the absolute value of the difference is greater than a first preset threshold, determine that there is a problem in the execution process of the Spark task; if the absolute value of the difference is less than or equal to the first preset threshold, determine that there is no problem in the execution process of the Spark task;

[0098] Step H8: When it is determined that there is a problem in the execution process of the Spark task, a preset alarm operation is executed.

[0099] It can be seen from the above steps that in this embodiment of the present invention, the Spark task to be monitored can be obtained; thereby, the target value of the first execution time when the Spark task is executed at the running scheduling time to be monitored is obtained, wherein the first execution time is the total execution time of each Job included in the Spark task.

[0100] Among them, a Spark task includes multiple jobs, a job corresponds to multiple stages, a stage corresponds to multiple tasks, and the multiple stages included in a job are executed in parallel. The execution time of the last completed stage in a job is the completion time of the job; similarly, if the multiple tasks included in a stage are executed in parallel, the execution time of the last completed task in a stage is the completion time of the stage. Therefore, the aforementioned first execution time does not include the waiting time of the scheduling queue.

[0101] Thus, in embodiments of the present invention, the aforementioned first execution time can be used to describe the actual execution time required for each job included in a Spark task. Specifically, embodiments of the present invention monitor tasks at the Job level under a Spark task, accurately obtaining the actual execution time required for each job included in a Spark task. This improves the accuracy of Spark task execution time and enables more accurate monitoring of Spark tasks.

[0102] It should be noted that for the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0103] Reference Figure 4 , shows a structural block diagram of a task monitoring device according to an embodiment of the present invention. The task monitoring device 400 may include the following modules:

[0104] The first acquisition module 401 is used to acquire the Spark task to be monitored;

[0105] The second acquisition module 402 is used to obtain a target value of a first execution time when the Spark task is executed at the running scheduling time to be monitored, wherein the first execution time is the total execution time of each job included in the Spark task.

[0106] Optionally, the second obtaining module 402 includes:

[0107] A first time acquisition submodule is used to obtain the first submission time and first completion time of the xth job included in the Spark task when x is an integer from 1 to y;

[0108] a first determining submodule, configured to determine a time interval between a first submission time and a first completion time of the x-th job as a first execution time of the x-th job, where y is the number of jobs included in the Spark task;

[0109] The first calculation submodule is configured to calculate the sum of the execution times of the 1st to yth jobs as the first execution time.

[0110] Optionally, the task monitoring device 400 further includes:

[0111] A first output module is configured to input the monitored running scheduling time into a predetermined execution time prediction model corresponding to the Spark task, and output a value of the predicted execution time;

[0112] The first determining module is configured to determine whether there is a problem in the execution process of the Spark task according to the value of the predicted execution time and the target value of the first execution time.

[0113] Optionally, the task monitoring device 400 further includes:

[0114] A first historical value acquisition module, configured to acquire a historical value of the first execution time of the Spark task;

[0115] A second historical value acquisition module is used to obtain historical values ​​of the second execution time of the Spark task, wherein the second execution time is the total execution time of each scheduling stage included in each job;

[0116] a third historical value acquisition module, configured to acquire a historical value of a third execution time of the Spark task, wherein the third execution time is the total execution time of each Task included in each scheduling stage;

[0117] an abnormal data removal module, configured to remove abnormal data from the historical values ​​of the first execution time, the historical values ​​of the second execution time, and the historical values ​​of the third execution time by using a clustering algorithm;

[0118] a fourth historical value acquisition module, configured to add a weighted value to the historical values ​​of the first execution time, the second execution time, and the third execution time corresponding to the same run scheduling time, and sum the sums to obtain a historical value of the predicted execution time corresponding to the run scheduling time;

[0119] The second determining module is configured to obtain the execution time prediction model according to historical values ​​of the predicted execution time and the running scheduling time corresponding to the historical values ​​of the predicted execution time.

[0120] Optionally, the second historical value acquisition module includes:

[0121] A second time acquisition submodule, configured to acquire the second submission time and the second completion time of the i-th stage included in each job when i is an integer from 1 to n;

[0122] a second determining submodule, configured to determine the time interval between the second submission time and the second completion time of the i-th stage as the execution time of the i-th stage, where n is the number of stages included in each job;

[0123] The second calculation submodule is used to calculate the sum of the execution time of the 1st to nth stages as the second execution time.

[0124] Optionally, the third historical value acquisition module includes:

[0125] A third time acquisition submodule is configured to obtain the third submission time and the third completion time of the j-th Task included in each scheduling stage when j is an integer from 1 to m;

[0126] a third determining submodule, configured to determine the time interval between the third submission time and the third completion time of the j-th task as the execution time of the j-th task, where m is the number of tasks included in each scheduling stage;

[0127] The third calculation submodule is configured to calculate the sum of the execution times of the 1st to mth tasks as the third execution time.

[0128] Optionally, the first determining module includes:

[0129] a fourth determining submodule, configured to determine that there is a problem in the execution process of the Spark task when an absolute value of a difference between the target value of the first execution time and the value of the predicted execution time is greater than a first preset threshold;

[0130] The fifth determination submodule is used to determine that there is no problem in the execution process of the Spark task when the absolute value of the difference between the target value of the first execution time and the value of the predicted execution time is less than or equal to the first preset threshold.

[0131] Optionally, the task monitoring device 400 further includes:

[0132] A third output module is configured to input the monitored running scheduling time into a predetermined execution time prediction model corresponding to the Spark task, and output a value of the predicted execution time;

[0133] The task monitoring device 400 further includes:

[0134] A judgment module is used to judge whether the Spark task is completed at a preset time interval starting from the scheduled operation time to be monitored;

[0135] a third determining module, configured to determine, when the Spark task is not completed, whether there is a problem in the execution process of the Spark task according to a target time interval and a value of the predicted execution time, wherein the target time interval is a target time interval between the current time and the running scheduling time to be monitored;

[0136] The fourth determination module is used to skip the step of determining whether there is a problem in the execution process of the Spark task based on the target time interval and the value of the predicted execution time when the Spark task is completed.

[0137] Optionally, the third determining module includes:

[0138] a fourth calculation submodule, configured to calculate a difference between the target time interval and the predicted execution time;

[0139] a sixth determining submodule, configured to determine that there is a problem in the execution process of the Spark task when the difference is greater than a second preset threshold;

[0140] The seventh determining submodule is configured to determine that there is no problem in the execution process of the Spark task when the difference is less than or equal to the second preset threshold.

[0141] Optionally, the task monitoring device 400 further includes:

[0142] The execution module is used to execute a preset alarm operation when it is determined that there is a problem in the execution process of the Spark task.

[0143] It can be seen from this that in an embodiment of the present invention, the Spark task to be monitored can be obtained; thereby, the target value of the first execution time when the Spark task is executed at the running scheduling time to be monitored is obtained, wherein the first execution time is the total execution time of each Job included in the Spark task.

[0144] Among them, a Spark task includes multiple jobs, a job corresponds to multiple stages, a stage corresponds to multiple tasks, and the multiple stages included in a job are executed in parallel. The execution time of the last completed stage in a job is the completion time of the job; similarly, if the multiple tasks included in a stage are executed in parallel, the execution time of the last completed task in a stage is the completion time of the stage. Therefore, the aforementioned first execution time does not include the waiting time of the scheduling queue.

[0145] Thus, in embodiments of the present invention, the aforementioned first execution time can be used to describe the actual execution time required for each job included in a Spark task. Specifically, embodiments of the present invention monitor tasks at the job level within a Spark task, accurately obtaining the actual execution time required for each job included in the Spark task, thereby improving the accuracy of Spark task execution time.

[0146] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0147] An embodiment of the present invention further provides an electronic device, including:

[0148] One or more processors; and one or more machine-readable media having instructions stored thereon, which, when executed by the one or more processors, cause the electronic device to perform the method described in the embodiment of the present invention.

[0149] The embodiments of the present invention further provide one or more machine-readable media having instructions stored thereon, which, when executed by one or more processors, enable the processors to perform the methods described in the embodiments of the present invention.

[0150] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0151] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, apparatus, or computer program products. Thus, embodiments of the present invention may take the form of a fully hardware embodiment, a fully software embodiment, or an embodiment combining software and hardware. Furthermore, embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0152] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the process in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0153] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0154] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0155] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0156] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0157] The above is a detailed introduction to a task monitoring method, device electronic device and readable storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A task monitoring method, characterized in that: The method comprises: Get the Spark task to be monitored; Obtain a target value of a first execution time when the Spark task is executed at the scheduled running time to be monitored, wherein the first execution time is the total execution time of each job included in the Spark task; Input the running scheduling time to be monitored into a predetermined execution time prediction model corresponding to the Spark task, and output a value of the predicted execution time; Determining whether there is a problem in the execution process of the Spark task according to the value of the predicted execution time and the target value of the first execution time; The process of establishing the execution time prediction model corresponding to the Spark task includes: Obtaining historical values ​​of the first execution time of the Spark task; Obtain a historical value of the second execution time of the Spark task, wherein the second execution time is the total execution time of each scheduling stage included in each job; Obtain a historical value of the third execution time of the Spark task, wherein the third execution time is the total execution time of each Task included in each scheduling stage; Using a clustering algorithm, removing abnormal data from the historical values ​​of the first execution time, the historical values ​​of the second execution time, and the historical values ​​of the third execution time; The historical values ​​of the first execution time, the second execution time, and the third execution time corresponding to the same running schedule time are weighted and summed to obtain the historical value of the predicted execution time corresponding to the running schedule time; The execution time prediction model is obtained according to historical values ​​of the predicted execution time and the running scheduling time corresponding to the historical values ​​of the predicted execution time.

2. The method according to claim 1, characterized in that The process of obtaining the first execution time includes: When x is an integer from 1 to y, obtain the first submission time and first completion time of the xth job included in the Spark task; Determine the time interval between the first submission time and the first completion time of the x-th job as the first execution time of the x-th job, where y is the number of jobs included in the Spark task; The sum of the execution times of the 1st to yth jobs is calculated as the first execution time.

3. The method according to claim 1, characterized in that The process of obtaining the second execution time includes: When i is an integer from 1 to n, obtain the second submission time and the second completion time of the i-th stage included in each job; Determine the time interval between the second submission time and the second completion time of the i-th stage as the execution time of the i-th stage, where n is the number of stages included in each job; The sum of the execution times of the 1st to nth stages is calculated as the second execution time.

4. The method according to claim 1, wherein The process of obtaining the third execution time includes: When j is an integer from 1 to m, obtain the third submission time and the third completion time of the j-th task included in each scheduling stage; Determine the time interval between the third submission time and the third completion time of the j-th Task as the execution time of the j-th Task, where m is the number of Tasks included in each scheduling stage; The sum of the execution times of the 1st to mth tasks is calculated as the third execution time.

5. The method according to claim 1, wherein The determining, based on the value of the predicted execution time and the target value of the first execution time, whether there is a problem in the execution process of the Spark task includes: When an absolute value of a difference between the target value of the first execution time and the value of the predicted execution time is greater than a first preset threshold, determining that there is a problem in the execution process of the Spark task; When the absolute value of the difference between the target value of the first execution time and the value of the predicted execution time is less than or equal to the first preset threshold, it is determined that there is no problem in the execution process of the Spark task.

6. The method according to claim 1, characterized in that Before the Spark task is executed, the method further includes: Input the running scheduling time to be monitored into a predetermined execution time prediction model corresponding to the Spark task, and output a value of the predicted execution time; During the execution of the Spark task, the method further includes: Starting from the scheduled operation time to be monitored, at every preset time interval, determine whether the Spark task is completed; When the Spark task is not completed, determining whether there is a problem in the execution process of the Spark task according to the target time interval and the value of the predicted execution time, wherein the target time interval is the target time interval between the current time and the running scheduling time to be monitored; When the Spark task is executed, the step of determining whether there is a problem in the execution process of the Spark task according to the target time interval and the value of the predicted execution time is skipped.

7. The method according to claim 6, characterized in that The determining, based on the target time interval and the value of the predicted execution time, whether there is a problem in the execution process of the Spark task includes: Calculating a difference between the target time interval and the predicted execution time; When the difference is greater than a second preset threshold, determining that there is a problem in the execution process of the Spark task; When the difference is less than or equal to the second preset threshold, it is determined that there is no problem in the execution process of the Spark task.

8. A task monitoring device, characterized in that: The device comprises: The first acquisition module is used to obtain the Spark task to be monitored; A second acquisition module is configured to acquire a target value of a first execution time when the Spark task is executed at the scheduled running time to be monitored, wherein the first execution time is the total execution time of each job included in the Spark task; A first output module is configured to input the monitored running scheduling time into a predetermined execution time prediction model corresponding to the Spark task, and output a value of the predicted execution time; a first determining module, configured to determine whether there is a problem in the execution process of the Spark task according to the value of the predicted execution time and the target value of the first execution time; A first historical value acquisition module, configured to acquire a historical value of the first execution time of the Spark task; A second historical value acquisition module is used to obtain historical values ​​of the second execution time of the Spark task, wherein the second execution time is the total execution time of each scheduling stage included in each job; a third historical value acquisition module, configured to acquire a historical value of a third execution time of the Spark task, wherein the third execution time is the total execution time of each Task included in each scheduling stage; an abnormal data removal module, configured to remove abnormal data from the historical values ​​of the first execution time, the historical values ​​of the second execution time, and the historical values ​​of the third execution time by using a clustering algorithm; a fourth historical value acquisition module, configured to add a weighted value to the historical values ​​of the first execution time, the second execution time, and the third execution time corresponding to the same run scheduling time, and sum the sums to obtain a historical value of the predicted execution time corresponding to the run scheduling time; The second determining module is configured to obtain the execution time prediction model according to historical values ​​of the predicted execution time and the running scheduling time corresponding to the historical values ​​of the predicted execution time.

9. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the task monitoring method according to any one of claims 1 to 7 when executed by the processor.

10. A computer-readable storage medium, characterized in that The computer program stored therein enables the processor to execute the task monitoring method according to any one of claims 1 to 7.

Images