Spark-based slow task analysis method and device for scheduling tasks

Abstract

This invention relates to a method, apparatus, and computer-readable medium for slow task analysis based on Spark scheduling tasks. The method includes: dividing a task phase into multiple segments, further dividing each segment into multiple sub-processes, and predefining slow query reasons corresponding to each sub-process; running multiple tasks in a Spark program, each task including multiple task phases; obtaining the execution time of each sub-process, determining whether the sub-process is an abnormal sub-process based on its execution time; and statistically analyzing the slow query reasons of the multiple abnormal sub-processes, defining the most frequent slow query reason as a common slow query reason. This invention, by predefining the slow query reasons corresponding to sub-processes, enables the program to automatically analyze the slow query reasons of batch tasks; by sorting the frequency of slow query reasons, the slow query reasons of batch tasks can be quickly located.

Description

Technical Field

[0001] This invention mainly relates to the field of big data application performance analysis, specifically to a slow task analysis method, apparatus, and computer-readable medium based on Spark scheduling tasks. Background Technology

[0002] With the rise of big data, the Spark computing framework emerged to address the challenges of big data computation. Based on the concept of elastic distributed computing, this framework iterates through computation data in memory, significantly reducing the need to write intermediate results to disk, thereby decreasing runtime and improving computational performance. Currently, Spark has become a cornerstone of enterprise data computing.

[0003] During operation, the Spark computing framework logs the computation process of individual jobs to monitor performance issues. When a program runs slowly, a common practice is to select a particularly slow job and analyze several stages within that job to estimate the cause of the slowness. However, analyzing only one or a few jobs involves a significant amount of human experience. For engineers unfamiliar with the internal workings of Spark or with limited experience, this often leads to misjudgments due to insufficient expertise. While it's possible to analyze the cause of a slow job from Spark's job logs, when technicians run hundreds or thousands of slow SQL jobs, it's impossible to quickly and comprehensively pinpoint the performance issue within a short timeframe.

[0004] Therefore, there is an urgent need for a method that can quickly and automatically locate the cause of slow queries from multiple abnormal tasks. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a slow task analysis method, device and computer-readable medium based on Spark scheduling tasks, so as to solve the problem of not being able to quickly locate the cause of slow query when there are multiple tasks.

[0006] To address the aforementioned technical problems, this invention provides a slow task analysis method based on Spark scheduling tasks, comprising: dividing a task phase into multiple segments, further dividing each segment into multiple sub-processes, and predefining a slow query reason corresponding to each sub-process; running multiple tasks in a Spark program, each task including multiple task phases; obtaining the execution time of each sub-process, determining whether the sub-process is an abnormal sub-process based on its execution time; and statistically analyzing the slow query reasons of the multiple abnormal sub-processes, defining the slow query reason that occurs most frequently as a common slow query reason.

[0007] In one embodiment of the present invention, the method further includes: the Spark program inserting a timer into each of the sub-processes, the timer being used to record the start time and end time of the sub-process, and calculating the execution duration of the sub-process based on the start time and the end time.

[0008] In one embodiment of the present invention, the method further includes: the Spark program obtaining log data of the plurality of tasks through an interface program, wherein the log data includes the execution duration of each task, the execution duration of each task stage, and the execution duration of each subprocess.

[0009] In one embodiment of the present invention, the step of determining whether a subprocess is an abnormal subprocess based on the execution time of the subprocess includes: calculating an outlier value of the execution time of the subprocess, wherein the outlier value is used to indicate the degree to which the execution time deviates from the average among the execution times of multiple subprocesses, and when the outlier value is greater than a preset threshold, the subprocess is determined to be an abnormal subprocess.

[0010] In one embodiment of the present invention, the step of determining whether a subprocess is an abnormal subprocess based on the execution time of the subprocess includes determining it according to the following formula:

[0011] │x-μ│>3σ

[0012] Where x represents the sample of execution time, μ represents the mean of execution time, and σ represents the standard deviation of execution time. When the absolute value of the difference between the sample x and the mean μ is greater than 3 times the standard deviation 3σ, the subprocess corresponding to the sample is determined to be an abnormal subprocess.

[0013] In one embodiment of the present invention, the method further includes: obtaining the execution time of each task, and determining whether the task is an abnormal task based on the execution time of the task, wherein the step of obtaining the execution time of each sub-process includes: the sub-process belongs to the abnormal task.

[0014] In one embodiment of the present invention, the method further includes: obtaining the execution duration of each task stage, and determining whether the task stage is an abnormal task stage based on the execution duration of the task stage, wherein the step of obtaining the execution duration of each sub-process includes: the sub-process belongs to the abnormal task stage.

[0015] In one embodiment of the present invention, the step of determining whether the subprocess is an abnormal subprocess based on the execution time further includes: extracting the slow query reasons of the abnormal subprocess; and sorting the slow query reasons according to the number of times they occur.

[0016] In one embodiment of the present invention, the segment includes one or any combination of a read data source segment, an execution segment, and a write task segment.

[0017] In one embodiment of the present invention, the sub-processes of the read data source segment include one or any combination of preparing partitions and pulling data; the sub-processes of the execution segment include one or any combination of left join query, right join query, and inner join query; and the sub-processes of the write task segment include one or any combination of writing to HDFS and writing to MySQL.

[0018] To address the aforementioned technical problems, this invention provides a slow task analysis device based on Spark scheduling tasks, comprising: a memory for storing instructions executable by a processor; and a processor for executing the instructions to implement the slow task analysis method based on Spark scheduling tasks as described above.

[0019] To address the aforementioned technical problems, the present invention provides a computer-readable medium storing computer program code, which, when executed by a processor, implements the slow task analysis method based on Spark scheduling tasks as described above.

[0020] The technical solution of this invention divides the task stage into segments and sub-processes, and then predefines the slow query reasons corresponding to the sub-processes, so that the program can automatically analyze the slow query reasons of batch tasks; this invention sorts the slow query reasons by frequency and defines the slow query reasons that occur most frequently as common slow query reasons, which can quickly locate the slow query reasons of batch tasks. Attached Figure Description

[0021] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0022] Figure 1 This is an exemplary flowchart of a slow task analysis method based on Spark scheduling tasks according to an embodiment of the present invention;

[0023] Figure 2 This is a predefined diagram of sub-processes of a slow task analysis method based on Spark scheduling tasks according to an embodiment of the present invention;

[0024] Figure 3 This is a schematic diagram illustrating the execution principle of a task for locating anomalies according to an embodiment of the present invention.

[0025] Figure 4 This is a schematic diagram of the execution principle of the abnormal location task stage according to an embodiment of the present invention.

[0026] Figure 5This is a schematic diagram illustrating the principle of finding the cause of a slow query based on a subprocess according to an embodiment of the present invention;

[0027] Figure 6 This is a system block diagram of a Spark-based batch task processing device according to an embodiment of the present invention. Detailed Implementation

[0028] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0029] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the invention is not limited to the specific embodiments disclosed below.

[0030] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not specifically singular and may include plural forms. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0031] Unless otherwise specifically stated, the relative arrangement of the parts and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present application. At the same time, it should be understood that, for ease of description, the sizes of the various parts shown in the drawings are not drawn according to actual proportional relationships. The techniques, methods and equipment known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, the techniques, methods and equipment should be considered as part of the authorization specification. In all examples shown and discussed here, any specific values ​​should be interpreted as being merely exemplary and not as limitations. Therefore, other examples of the exemplary embodiments may have different values. It should be noted that similar numbers and letters represent similar items in the following figures, and therefore, once an item is defined in one figure, it does not need to be further discussed in subsequent figures.

[0032] Furthermore, it should be noted that the use of terms such as "first" and "second" to define components is merely for the purpose of distinguishing the corresponding components. Unless otherwise stated, these terms have no special meaning and therefore should not be construed as limiting the scope of protection of this application. In addition, although the terminology used in this application is selected from commonly known and used terms, some terms mentioned in this application's specification may have been chosen by the applicant according to his or her judgment, and their detailed meanings are explained in the relevant sections of this description. Moreover, this application should be understood not only through the actual terms used, but also through the meaning implied by each term.

[0033] Flowcharts are used in this application to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, various steps can be processed in reverse order or simultaneously. Furthermore, other operations may be added to these processes, or one or more steps may be removed from these processes.

[0034] Figure 1 This is an exemplary flowchart of a slow task analysis method based on Spark scheduling tasks according to an embodiment of the present invention. Figure 1 As shown, the slow task analysis method 10 based on Spark scheduling tasks in this embodiment includes the following steps:

[0035] Step S11: Divide the task phase into multiple segments, divide each segment into multiple sub-processes, and predefine the slow query reasons corresponding to each sub-process;

[0036] Step S12: The Spark program runs multiple tasks, each of which includes multiple task stages;

[0037] Step S13: Obtain the execution time of each sub-process, and determine whether the sub-process is an abnormal sub-process based on its execution time; and

[0038] Step S14: Analyze the slow query reasons of multiple abnormal sub-processes and define the slow query reasons that occur most frequently as common slow query reasons.

[0039] The following combination Figure 1-Figure 5 The above steps S11-S14 will be explained in detail.

[0040] In step S11, in order to automatically analyze and locate the time consumption and common problems of sub-processes in each task stage and quickly find the cause of slow queries, the time of each sub-process can be timed and the cause of slowness of the sub-process can be defined before the program runs. However, this application is not limited to this; the present invention can time each sub-process and define the cause of slowness of the sub-process at any time. Typically, the task stage can be divided into multiple segments, and each segment can be divided into multiple sub-processes.

[0041] In some embodiments, a segment includes one or any combination of a read data source segment, an execution segment, and a write task segment.

[0042] In some embodiments, the sub-processes of the read data source segment include one or any combination of preparing partitions and pulling data; the sub-processes of the execution segment include one or any combination of left join query, right join query, and inner join query; and the sub-processes of the write task segment include one or any combination of writing to HDFS and writing to MySQL.

[0043] Figure 2 This is a predefined diagram of sub-processes in a slow task analysis method based on Spark scheduling tasks according to an embodiment of the present invention. For example... Figure 2 As shown, task phase 20 is divided into three sections: Read Data Source Section 21, Execution Section 22, and Write Task Section 23. Read Data Source Section 21 includes the sub-processes: Prepare Partition 211 and Fetch Data 212; Execution Section 22 includes the sub-processes: Inner Join Query 221, Left Join Query 222, and Inner Join Query 223; Write Task Section 23 includes the sub-processes: Create Table 231 and Write to HDFS 232.

[0044] In some embodiments, the method further includes: the Spark program sets a timer in each sub-process to record the start and end times of the sub-process, and calculates the execution duration of the sub-process based on the start and end times. The program first sets a timer for the sub-processes in each task phase segment, and when the sub-process completes, the execution duration of the sub-process can be obtained by subtracting the start time of the sub-process from its end time.

[0045] like Figure 2 As shown, the parentheses next to the sub-process indicate the execution time of the sub-process. For example, the execution time for preparing partition 211 is 12 minutes, and the execution time for fetching data 212 is 16 minutes. The execution times of other sub-processes can be obtained similarly, and will not be repeated here. Then, the execution time of each sub-process can be output as a log for use in subsequent slow query analysis.

[0046] For the predefinition of slow query reasons for subprocesses, specifically, the possible causes of slow queries and the attribute information of the subprocesses can be predefined. For example... Figure 2 As shown, the subprocess "Prepare Partition 211" can be predefined as belonging to "Read Data Stage 21", with the job name "test table" and the stage name "test table" for reasons such as "large amount of data to be fetched" and "fetched primary key not automatically sorted". Similarly, the subprocess "Write HDFS 232" can be predefined as belonging to "Write Task Segment 23", with the job name "test1 table" and the stage name "test1 table" for reasons such as "large amount of data to write", "write network latency", and "low write concurrency". Using the execution duration obtained from the timer and the subprocess definitions, the execution duration, segment, stage, and job information of these subprocesses are logged and saved as text on the Spark-supported service disk for easy subsequent location and analysis.

[0047] In step S12, since Spark is a distributed computing framework, multiple tasks can be run simultaneously, and each task includes multiple task stages.

[0048] In step S13, the execution time of each sub-process is obtained, and it is determined whether the sub-process is an abnormal sub-process based on its execution time. In some embodiments, this further includes: the Spark program obtaining log data from multiple tasks through an interface program. The log data includes the execution time of each task, the execution time of each task stage, and the execution time of each sub-process. Specifically, log data from multiple tasks can be obtained through Spark's API. The log data includes the execution time of each sub-process. Each task stage consists of multiple sub-processes, and the execution time of each task stage can be obtained based on the execution time of the sub-processes in the logs and the attribute information of the sub-processes. Similarly, each job consists of multiple task stages, and each task stage consists of multiple sub-processes. The execution time of each task stage can be obtained based on the execution time of the sub-processes in the logs and the attribute information of the sub-processes.

[0049] In some embodiments, the step of determining whether a subprocess is an anomalous subprocess based on its execution duration includes: calculating the outlier of the subprocess's execution duration, where the outlier represents the degree to which its execution duration deviates from the mean among multiple subprocesses; and determining the subprocess as an anomalous subprocess when the outlier exceeds a preset threshold. Defining a subprocess as anomalous using temporal outliers is crucial. While there is no universally accepted definition of outliers, the commonly accepted definition, proposed by Hawkins, is that an observation deviates too much from other observations, suggesting a different mechanism attribution. More simply, in a set of parallel measurements, if individual data points differ significantly from the mean, these are considered suspicious values, also known as outliers. A threshold can be set for the subprocess based on historical data; when the outlier exceeds the preset threshold, the subprocess is determined to be an anomalous subprocess.

[0050] In some embodiments, the step of determining whether a subprocess is an abnormal subprocess based on its execution time includes determining it using the 3 sigma standard deviation method formula:

[0051] │x-μ│>3σ

[0052] Where x represents the sample execution time, μ represents the mean execution time, and σ represents the standard deviation of execution time. When the absolute value of the difference between the sample x and the mean μ is greater than three times the standard deviation (3σ), the subprocess corresponding to the sample is judged to be an abnormal subprocess. According to the normal distribution law, the probability of deviation exceeding 3σ is less than 0.3%. When the number of measurements is small, such data can be considered abnormal. This invention can determine whether a subprocess is abnormal based on the 3 sigma standard deviation method, but this application is not limited to this. This invention can use any outlier detection method. For example, this invention can also use the 4d test to determine whether a subprocess is abnormal. The 4d test is defined as follows:

[0053]

[0054] Where x represents the sample execution time, This represents the average execution time. This represents the average deviation of the execution time.

[0055] This invention can also use the Chauvenet method to determine whether a subprocess is abnormal. The Chauvenet method is defined as follows:

[0056]

[0057] Where x represents the sample execution time, Let represent the mean execution time, and s represent the standard deviation of execution time. The statistic w is calculated using the above formula. nFind the Schauweiler coefficient value ω(n) in the table based on the number of measurements n. If w n If the expression is greater than ω(n), it is considered an outlier; otherwise, no outlier was found.

[0058] In some embodiments, the method further includes: obtaining the execution time of each task, and determining whether a task is an anomalous task based on its execution time. The step of obtaining the execution time of each sub-process includes determining whether the sub-process is an anomalous task. Specifically, firstly, log data for all tasks can be obtained through Spark's API, including the execution time of each task. Then, the execution times of all tasks are sorted, and the sorted tasks are judged from highest to lowest to determine whether they are anomalous tasks. For example, the 3-sigma standard deviation method mentioned above can be used to determine whether a task is an anomalous task, or other outlier detection methods can be used.

[0059] Figure 3 This is a schematic diagram illustrating the execution principle of a task for locating anomalies according to an embodiment of the present invention. Figure 3 As shown, each task has a unique JobID and execution duration. First, all tasks are globally sorted by execution duration. Then, tasks are ranked from longest to shortest to determine if they are abnormal. If abnormal tasks exist, they are selected. If no abnormal tasks exist, the tasks are considered to have executed normally and require no special attention. For example, the selected abnormal tasks have JobIDs of 11, 29, and 41. Next, the task stages of the selected abnormal tasks are sorted. Then, task stages are ranked from longest to shortest to determine if they are abnormal. If abnormal task stages exist, they are selected. Figure 4 As shown, the abnormal task with JobID 11 includes abnormal task stages with StageIDs 23, 21, and 53; the abnormal task with JobID 29 includes abnormal task stages with StageIDs 149 and 2; and the abnormal task with JobID 41 includes abnormal task stages with StageIDs 52 and 34. Since the section and task stage to which the subprocess belongs are printed in the log, all subprocesses can be quickly filtered out from the text based on the obtained abnormal task stages, and the reasons and sections to which these subprocesses belong can be obtained. All filtered subprocesses are then judged to be abnormal. The specific subprocess judgment method is as described above and will not be repeated here.

[0060] In some embodiments, the method further includes: obtaining the execution duration of each task stage, and determining whether a task stage is an abnormal task stage based on the execution duration of the task stage. The step of obtaining the execution duration of each sub-process includes determining whether the sub-process belongs to an abnormal task stage. Specifically, firstly, log data for all tasks can be obtained through Spark's API, including the execution duration of each task stage. Then, the execution durations of all tasks are sorted, and the task stages are determined from highest to lowest to determine whether they are abnormal task stages. For example, the 3-sigma standard deviation method can be used to determine whether a task stage is an abnormal task stage, or other outlier detection methods can be used.

[0061] Figure 4 This is a schematic diagram illustrating the execution principle of the anomaly location task phase according to an embodiment of the present invention. Figure 4 As shown, each task stage has a StageID and execution duration. First, the execution durations of all task stages are globally sorted. Then, tasks are ranked from longest to shortest to determine if they are abnormal task stages. If abnormal task stages exist, they are selected. For example, the StageIDs of the selected abnormal task stages are 42, 91, and 45. Since the section and task stage to which the subprocess belongs are printed in the log, all subprocesses can be quickly filtered out from the text based on the obtained abnormal task stages, and the reasons and sections to which these subprocesses belong can be obtained. All filtered subprocesses are then judged to be abnormal. The specific subprocess judgment method is as described above and will not be repeated.

[0062] In some embodiments, sub-processes belonging to abnormal tasks and sub-processes belonging to abnormal task stages are extracted from all sub-processes, and the execution time of all obtained sub-processes is used to determine whether the sub-process is an abnormal sub-process.

[0063] Figure 5 This is a schematic diagram illustrating the principle of finding the cause of a slow query based on a subprocess according to an embodiment of the present invention. Figure 5 As shown, for ease of understanding, Figure 5The above process has been simplified by extracting exception subprocesses belonging to the exception task and exception subprocesses belonging to the exception task stage from all subprocesses. This includes four exception subprocesses, each containing a name, execution duration, section, task stage, task, and reason. For example, the first abnormal sub-process: fetching data, execution time: 17 minutes, segment: reading data source stage, stage: full_test table, job: full_test table, reason: large amount of data fetched; the second abnormal sub-process: fetching data, execution time: 11 minutes, segment: reading data source stage, stage: full_test1 table, job: full_test1 table, reason: large amount of data fetched; the third abnormal sub-process: inner join query, execution time: 9.1 minutes, main stage: execution stage, stage: full_test1 table, job: full_test1 table, reason: large amount of data executed, executing a non-primary key join; the fourth abnormal sub-process: left join query, execution time: 8 minutes, main stage: execution stage, stage: dfg_test1 table, job: dfg_test1 table, reason: executing a non-primary key join.

[0064] In step S14, before statistically analyzing the slow query reasons of multiple abnormal sub-processes, some embodiments further include: extracting the slow query reasons of the abnormal sub-processes and sorting them according to the frequency of occurrence of the slow query reasons. Specifically, rule matching, such as regular expressions, can be used to extract the slow query reasons from the abnormal sub-processes. The extracted abnormal sub-processes only include: name and slow query reason. Then, the frequency of occurrence of the slow query reasons is counted, and the results are sorted according to the frequency of occurrence of the slow query reasons. Figure 5 As shown, the reasons for slow queries are categorized into three types: large data fetch volume, execution of non-primary key joins, and large data execution volume. Sorted by the frequency of occurrence of each slow query reason from highest to lowest, "large data fetch volume" appeared twice, "execution of non-primary key joins" appeared once, and "large data execution volume" appeared once. The most frequent slow query reasons can be defined as common slow query reasons. In this embodiment, it can be concluded that the main reason for slow queries across the entire batch of tasks is "large data fetch volume."

[0065] This invention divides the task phase into segments and sub-processes, and then predefines the slow query reasons corresponding to the sub-processes, enabling the program to automatically analyze the slow query reasons of batch tasks. This invention sorts the slow query reasons by frequency and defines the slow query reasons that occur most frequently as common slow query reasons, which can quickly locate the slow query reasons of batch tasks.

[0066] The present invention also includes a slow task analysis apparatus based on Spark scheduled tasks, comprising a memory and a processor. The memory stores instructions executable by the processor; the processor executes these instructions to implement the slow task analysis method based on Spark scheduled tasks described above.

[0067] Figure 6 This is a system block diagram of a slow task analysis device based on Spark scheduling tasks according to an embodiment of the present invention. (Reference) Figure 6 As shown, the slow task analysis device 600 may include an internal communication bus 601, a processor 602, a read-only memory (ROM) 603, a random access memory (RAM) 604, and a communication port 605. When applied to a personal computer, the slow task analysis device 600 may also include a hard disk 606. The internal communication bus 601 enables data communication between the components of the slow task analysis device 600. The processor 602 can make judgments and issue prompts. In some embodiments, the processor 602 may consist of one or more processors. The communication port 605 enables data communication between the slow task analysis device 600 and external devices. In some embodiments, the slow task analysis device 600 can send and receive information and data from a network through the communication port 605. The slow task analysis device 600 may also include different forms of program storage units and data storage units, such as the hard disk 606, the read-only memory (ROM) 603, and the random access memory (RAM) 604, capable of storing various data files used for computer processing and / or communication, as well as possible program instructions executed by the processor 602. The processor executes these instructions to implement the main part of the method. The results of the processor's processing are transmitted to the user device via the communication port and displayed on the user interface.

[0068] The slow task analysis method based on Spark scheduling described above can be implemented as a computer program, stored on the hard disk 606, and loaded into the processor 602 for execution, so as to implement the slow task analysis method based on Spark scheduling of this application.

[0069] The present invention also includes a computer-readable medium storing computer program code that, when executed by a processor, implements the slow task analysis method based on Spark scheduling described above.

[0070] When the slow task analysis method based on Spark scheduling is implemented as a computer program, it can also be stored as an artifact in a computer-readable storage medium. For example, computer-readable storage media can include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic stripes), optical discs (e.g., compact discs (CDs), digital multifunction discs (DVDs)), smart cards, and flash memory devices (e.g., electrically erasable programmable read-only memory (EPROM), cards, sticks, key drives). Furthermore, the various storage media described herein can represent one or more devices and / or other machine-readable media used for storing information. The term "machine-readable medium" can include, but is not limited to, wireless channels and various other media (and / or storage media) capable of storing, containing, and / or carrying code and / or instructions and / or data.

[0071] It should be understood that the embodiments described above are merely illustrative. The embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or any combination thereof. For hardware implementation, the processor may be implemented within one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, and / or other electronic units designed to perform the functions described herein, or combinations thereof.

[0072] Some aspects of this application can be executed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The aforementioned hardware or software may be referred to as a "data block," "module," "engine," "unit," "component," or "system." The processor may be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DAPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, or combinations thereof. Furthermore, aspects of this application may manifest as computer products residing in one or more computer-readable media, including computer-readable program code. For example, computer-readable media may include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic tapes, etc.), optical discs (e.g., compressed CDs, digital multifunction DVDs, etc.), smart cards, and flash memory devices (e.g., cards, sticks, key drives, etc.).

[0073] A computer-readable medium may contain a propagated data signal containing computer program code, for example, on baseband or as part of a carrier wave. This propagated signal may take various forms, including electromagnetic, optical, and so on, or suitable combinations thereof. A computer-readable medium can be any computer-readable medium other than a computer-readable storage medium, which can be connected to an instruction execution system, apparatus, or device to enable communication, propagation, or transmission of a program for use. The program code located on the computer-readable medium can be propagated through any suitable medium, including radio, cable, fiber optic cable, radio frequency signals, or similar media, or any combination of the above media.

[0074] The basic concepts have been described above. Obviously, for those skilled in the art, the above disclosure is merely illustrative and does not constitute a limitation of this application. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to this application. Such modifications, improvements, and corrections are suggested in this application, and therefore remain within the spirit and scope of the exemplary embodiments of this application.

[0075] Furthermore, this application uses specific terms to describe embodiments of the application. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic related to at least one embodiment of the application. Therefore, it should be emphasized and noted that "an embodiment," "one embodiment," or "an alternative embodiment" mentioned twice or more in different locations in this specification do not necessarily refer to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of the application can be appropriately combined.

[0076] Similarly, it should be noted that, in order to simplify the description of the present application and thus aid in the understanding of one or more embodiments, the foregoing description of the embodiments of the present application sometimes combines multiple features into a single embodiment, drawing, or description thereof. However, this disclosure method does not imply that the subject matter of the present application requires more features than those mentioned in the claims. In fact, the embodiments contain fewer features than all the features of the single embodiments disclosed above.

[0077] Although this application has been described with reference to specific embodiments, those skilled in the art should recognize that the above embodiments are only used to illustrate this application, and various equivalent changes or substitutions can be made without departing from the spirit of this application. Therefore, any changes or modifications to the above embodiments within the essential spirit of this application will fall within the scope of the claims of this application.

Claims

1. A slow task analysis method based on Spark scheduling tasks, comprising: The task phase is divided into multiple segments, and each segment is further divided into multiple sub-processes. Slow query reasons are predefined for each sub-process. The segments include one or any combination of read data source segment, execution segment, and write task segment. The sub-processes of the read data source segment include one or any combination of prepare partition and pull data. The sub-processes of the execution segment include one or any combination of left join query, right join query, and inner join query. The sub-processes of the write task segment include one or any combination of write to HDFS and write to MySQL. A Spark program runs multiple tasks, each of which includes multiple task stages; Obtain the execution time of each sub-process, and determine whether the sub-process is an abnormal sub-process based on its execution time; and The slow query reasons of multiple abnormal sub-processes are statistically analyzed, and the slow query reason that occurs most frequently is defined as the common slow query reason.

2. The slow task analysis method as described in claim 1, characterized in that, Also includes: The Spark program places a timer in each of the sub-processes. The timer is used to record the start time and end time of the sub-process and to calculate the execution time of the sub-process based on the start time and end time.

3. The slow task analysis method as described in claim 1, characterized in that, Also includes: The Spark program obtains log data of the multiple tasks through an interface program. The log data includes the execution time of each task, the execution time of each task stage, and the execution time of each subprocess.

4. The slow task analysis method as described in claim 1, characterized in that, The step of determining whether a subprocess is an abnormal subprocess based on its execution time includes: calculating an outlier value for the execution time of the subprocess, wherein the outlier value is used to indicate the degree to which the execution time deviates from the average among the execution times of multiple subprocesses, and when the outlier value is greater than a preset threshold, the subprocess is determined to be an abnormal subprocess.

5. The slow task analysis method as described in claim 4, characterized in that, The steps for determining whether a subprocess is an abnormal subprocess based on its execution time include determining it according to the following formula: │x-μ│>3σ Where x represents the sample of execution time, μ represents the mean of execution time, and σ represents the standard deviation of execution time. When the absolute value of the difference between the sample x and the mean μ is greater than 3 times the standard deviation 3σ, the subprocess corresponding to the sample is determined to be an abnormal subprocess.

6. The slow task analysis method as described in claim 1, characterized in that, Also includes: The execution time of each task is obtained, and the task is determined as an abnormal task based on the execution time of the task. The step of obtaining the execution time of each sub-process includes: the sub-process belongs to the abnormal task.

7. The slow task analysis method as described in claim 1, characterized in that, Also includes: The execution duration of each task stage is obtained, and the task stage is determined as an abnormal task stage based on the execution duration of the task stage. The step of obtaining the execution duration of each sub-process includes: the sub-process belongs to the abnormal task stage.

8. The slow task analysis method as described in claim 1, characterized in that, The step of determining whether a subprocess is an abnormal subprocess based on the execution time further includes: extracting the slow query reasons of the abnormal subprocess; and sorting the slow query reasons according to the frequency of occurrence.

9. A slow task analysis device based on Spark scheduling tasks, comprising: Memory is used to store instructions that can be executed by the processor; A processor for executing the instructions to implement the method as described in any one of claims 1-8.

10. A computer-readable medium storing computer program code that, when executed by a processor, implements the method as claimed in any one of claims 1-8.

Images