Script execution performance determination method and apparatus, storage medium, and electronic device

Abstract

The application discloses a script execution performance determination method and device, a storage medium and electronic equipment. Relates to the field of big data. The method comprises the following steps: obtaining and analyzing a first log after target script execution to obtain M jobs; obtaining time information of each job, and obtaining N abnormal jobs from the M jobs according to the time information; obtaining a second log generated after the target script execution, obtaining P tasks in each abnormal job and a time information set of each task, and determining first abnormal performance information of the target script according to the P time information sets; obtaining a main task of each abnormal job, and determining an association value of a table associated with a source table of the abnormal job according to the main task, obtaining multiple groups of association values, and determining second abnormal performance information of the target script. Through the application, the problem that the efficiency and accuracy of determining the slow execution reason by manually analyzing the log when the SQL statement is executed slowly in the related art is low is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of big data, in particular, to a script execution performance determination method and device, a storage medium and an electronic device. BACKGROUND

[0002] Hive is a data warehouse infrastructure built on Hadoop (a distributed system infrastructure). It can provide a series of tools that can be used for data extraction, transformation and loading, and is a mechanism that can store, query and analyze large-scale data stored in Hadoop. Hive defines a simple SQL-like (Structured Query Language) query language called HQL (Hibernate Query Language), which allows users familiar with SQL statements to query data. Hive is a crucial framework for big data platforms, and its stability and ease of use also make it one of the most commonly used frameworks for building enterprise-level data warehouses.

[0003] However, when executing SQL statements in Hive, the user of Hive only knows the execution logic of the entire set of SQL statements, and does not know whether the user's submitted SQL statement has a performance problem, which can cause the SQL statement to execute slowly. The existing analysis method for SQL statement performance problems is generally to log in to the system after discovering that the SQL statement is running slowly, analyze the performance problem according to professional experience, and then notify the business personnel to optimize, and then observe whether the optimization is effective, so as to solve the problem of slow SQL statement running.

[0004] In view of the low efficiency and accuracy of the method of manually analyzing logs to determine the slow execution reason when the SQL statement executes slowly in the related art, no effective solution has been proposed so far. SUMMARY

[0005] The present application provides a script execution performance determination method and device, a storage medium and an electronic device to solve the problem of low efficiency and accuracy of the method of manually analyzing logs to determine the slow execution reason when the SQL statement executes slowly in the related art.

[0006] According to an aspect of the present application, a script execution performance determination method is provided. The method comprises: obtaining a first log generated after execution of a target script, and parsing the first log to obtain M jobs in the target script, wherein the first log is a log of a data warehouse tool; obtaining time information of each job, and obtaining abnormal jobs from the M jobs according to the time information of the jobs to obtain N abnormal jobs; obtaining a second log generated after execution of the target script, and obtaining P tasks in each abnormal job and time information of each task in the second log to obtain P sets of time information, and determining first abnormal performance information of the target script according to the P sets of time information, wherein the second log is a log of a resource manager; obtaining a main task of each abnormal job, and determining a source table of the abnormal job according to the main task to obtain N source tables, determining an association value of a table associated with each source table to obtain multiple sets of association values, and determining second abnormal performance information of the target script according to the multiple sets of association values; and determining execution performance of the target script according to the first abnormal performance information and the second abnormal performance information.

[0007] Optionally, obtaining abnormal jobs from the M jobs according to the time information of the jobs to obtain N abnormal jobs comprises: for each job, obtaining a first submission time, a first start time and a first end time of the job; subtracting the first start time from the first end time to obtain a first execution duration, and determining whether the first execution duration is greater than a first preset duration; in a case where the first execution duration is greater than the first preset duration, determining the job as an abnormal job; subtracting the first submission time from the first start time to obtain a first waiting duration, and determining whether the first waiting duration is greater than a second preset duration; in a case where the first waiting duration is greater than the second preset duration, determining the job as an abnormal job.

[0008] Optionally, before determining whether the first execution duration is greater than the first preset duration, the method further comprises: obtaining the first execution duration of each job, and calculating a first average execution duration of the M first execution durations; determining whether the first average execution duration is greater than the first preset duration; in a case where the first average execution duration is greater than the first preset duration, replacing the first preset duration with the first average execution duration to obtain an updated first preset duration; and before determining whether the first waiting duration is greater than the second preset duration, the method further comprises: calculating the waiting duration of each job to obtain M job waiting durations, and calculating a standard deviation of the M job waiting durations to obtain a first standard deviation; obtaining a preset waiting time ratio, and multiplying the first standard deviation by the preset waiting time ratio to obtain the second preset duration.

[0009] Optionally, determining the first abnormal performance information of the target script according to the P sets of time information comprises: for each task, obtaining a second submission time, a second start time and a second end time of the task from the time information of the task; subtracting the second start time from the second end time to obtain a second execution duration, and determining whether the second execution duration is greater than a third preset duration; in the case that the second execution duration is greater than the third preset duration, determining the task as an abnormal task; subtracting the second submission time from the second start time to obtain a second waiting duration, and determining whether the second waiting duration is greater than a fourth preset duration; in the case that the second waiting duration is greater than the fourth preset duration, determining the task as an abnormal task; and determining the log content of the abnormal task as the first abnormal performance information.

[0010] Optionally, before determining whether the second execution duration is greater than the third preset duration, the method further comprises: calculating the second execution duration of each task, and calculating a second average execution duration of the M second execution durations; determining whether the second average execution duration is greater than the third preset duration; in the case that the second average execution duration is greater than the third preset duration, replacing the third preset duration with the second average execution duration to obtain an updated third preset duration; and before determining whether the second waiting duration is greater than the fourth preset duration, the method further comprises: calculating the waiting duration of each task to obtain M task waiting durations, and calculating a standard deviation of the M task waiting durations to obtain a second standard deviation; obtaining a preset waiting time ratio, and multiplying the second standard deviation by the preset waiting time ratio to obtain the fourth preset duration.

[0011] Optionally, determining the source table of the abnormal job according to the main task comprises: obtaining an initial source table of the abnormal job from the log information of the main task, and determining whether the initial source table is a temporary table; in the case that the initial source table is a temporary table, determining a previous execution job of the abnormal job from the M jobs according to a start time of the abnormal job to obtain a previous job; determining an initial source table of the previous job according to the main task of the previous job, and determining the initial source table of the previous job as the source table of the abnormal job; and in the case that the initial source table is not a temporary table, determining the initial source table as the source table of the abnormal job.

[0012] Optionally, determining the second abnormal performance information of the target script according to the multiple sets of correlation values comprises: for a set of correlation values, identifying preset correlation values in the set of correlation values to obtain Q preset correlation values; obtaining a total number of each preset correlation value in the source table and a table associated with the source table to obtain Q quantity values; and determining the source table and the Q quantity values as the second abnormal performance information.

[0013] Optionally, before determining the execution performance of the target script according to the first abnormal performance information and the second abnormal performance information, the method further comprises: obtaining a data volume set of each task of each job from the second log, wherein the data volume of each task comprises a data input volume and a data output volume; obtaining a maximum data input volume and a maximum data output volume in each data volume set, and dividing the maximum data output volume by the maximum data input volume to obtain a target ratio; determining whether the maximum data input volume is greater than a preset input volume, and storing the job to which the maximum data input volume belongs to the second monitoring table in a case where the maximum data input volume is greater than the preset input volume; and determining whether the target ratio is greater than a preset ratio, and determining the log information of the job to which the target ratio belongs as the second abnormal performance information in a case where the target ratio is greater than the preset ratio.

[0014] According to another aspect of the present application, a script execution performance determination apparatus is provided. The apparatus comprises: a first obtaining unit configured to obtain a first log generated after execution of a target script, and parse the first log to obtain M jobs in the target script, wherein the first log is a log of a data warehouse tool; a second obtaining unit configured to obtain time information of each job, and obtain abnormal jobs from the M jobs according to the time information of the jobs to obtain N abnormal jobs; a first determining unit configured to obtain a second log generated after execution of the target script, and obtain P tasks in each abnormal job and time information of each task in the second log to obtain P sets of time information, and determine first abnormal performance information of the target script according to the P sets of time information, wherein the second log is a log of a resource manager; a second determining unit configured to obtain a main task of each abnormal job, and determine a source table of the abnormal job according to the main task to obtain N source tables, determine an association value of a table associated with each source table to obtain multiple groups of association values, and determine second abnormal performance information of the target script according to the multiple groups of association values; and a third determining unit configured to determine execution performance of the target script according to the first abnormal performance information and the second abnormal performance information.

[0015] According to another aspect of the present application, a computer storage medium is provided for storing a program, wherein the program controls a device in which the computer storage medium is located to execute a script execution performance determination method when the program is run.

[0016] According to another aspect of the present application, an electronic device is provided, comprising one or more processors and a memory; the memory stores computer readable instructions, and the processor is configured to run the computer readable instructions, wherein the computer readable instructions execute a script execution performance determination method when the computer readable instructions are run.

[0017] By the present application, the following steps are adopted: obtaining a first log generated after execution of a target script, and parsing the first log to obtain M jobs in the target script, wherein the first log is a log of a data warehouse tool; obtaining time information of each job, and obtaining abnormal jobs from the M jobs according to the time information of the jobs, to obtain N abnormal jobs; obtaining a second log generated after execution of the target script, and obtaining P tasks in each abnormal job and time information of each task in the second log, to obtain P sets of time information, and determining first abnormal performance information of the target script according to the P sets of time information, wherein the second log is a log of a resource manager; obtaining a main task of each abnormal job, and determining a source table of the abnormal job according to the main task, to obtain N source tables, determining an association value of a table associated with each source table, to obtain multiple sets of association values, and determining second abnormal performance information of the target script according to the multiple sets of association values; and determining execution performance of the target script according to the first abnormal performance information and the second abnormal performance information. The problem that the efficiency and accuracy of a method of determining a slow execution reason by manually analyzing a log when a SQL statement is executed slowly in the related art are low is solved. By reading log content in the first log and the second log, time information of each job and task when a SQL statement is executed is determined, and a reason for an abnormality is determined according to the time information, and meanwhile, a source table is determined according to log information related to a task in the second log, and a reason for slow execution is determined according to association content of the source table, and thus the effect that log content is automatically obtained and analyzed according to execution information, and a reason for slow execution of a SQL statement is accurately determined is achieved. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings, which form a part of the present application, are intended to provide further understanding of the present application, and are used to interpret the application and its specification, and do not constitute improper limitations to the present application. In the drawings:

[0019] Figure 1 is a flowchart of a script execution performance determination method provided according to an embodiment of the present application;

[0020] Figure 2 is a flowchart of an optional abnormal job determination method provided according to an embodiment of the present application;

[0021] Figure 3 is a schematic diagram of a script execution performance determination apparatus provided according to an embodiment of the present application;

[0022] Figure 4 is a schematic diagram of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] It should be noted that the embodiments and features of the present application can be combined with each other in the case of no conflict. The technical solutions in the embodiments of the present application will be described in detail below with reference to the drawings and in combination with the embodiments.

[0024] In order for those skilled in the technical field to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0026] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, analyzed data, etc.) involved in the present disclosure are all information and data authorized by the user or authorized by all parties. For example, an interface is provided between the system and the relevant user or institution. Before obtaining the relevant information, the interface needs to send a request to the aforementioned user or institution, and after receiving the consent information fed back by the aforementioned user or institution, the relevant information is obtained.

[0027] It should be noted that the script execution performance determination method, device, storage medium and electronic device determined by the present disclosure can be used in the field of big data, and can also be used in any field other than the field of big data. The application field of the script execution performance determination method, device, storage medium and electronic device determined by the present disclosure is not limited.

[0028] According to the embodiments of the present application, a script execution performance determination method is provided.

[0029] Figure 1 is a flowchart of the script execution performance determination method provided according to the embodiments of the present application. As shown in Figure 1 , the method comprises the following steps:

[0030] In step S101, a first log generated after execution of a target script is acquired, and the first log is parsed to obtain M jobs in the target script, wherein the first log is a log of a data warehouse tool.

[0031] Specifically, when a certain business is executed, an execution script of the business is required, the script including a plurality of SQL statements required to be used when the script is executed, each group of SQL statements including M jobs, i.e., job steps, the first log being a Hive log, the Hive log recording execution information generated when each group of SQL statements in the target script is executed, wherein the target script includes a plurality of groups of SQL statements, and each group of SQL statements includes M jobs. Therefore, when job log information is required to be acquired, the information can be acquired from the Hive log.

[0032] In step S102, time information of each job is acquired, and an abnormal job is acquired from the M jobs according to the time information of the job, to obtain N abnormal jobs.

[0033] Specifically, after the Hive log is obtained, each line of the Hive log to be preprocessed is mapped once, so that the log is parsed, and performance key information, including M job information, is extracted, wherein the job information includes resource queue, job start time, job end time, job abnormal termination time, and job waiting resource time.

[0034] After the time information is obtained, the time information is processed and calculated, so that execution time and waiting time of each job are obtained, and abnormal jobs in the M jobs are determined according to the time information. For example, a job with a waiting time of more than 1 hour is an abnormal job.

[0035] In step S103, a second log generated after execution of the target script is acquired, P tasks in each abnormal job and time information of each task in the second log are acquired, P time information sets are obtained, and first abnormal performance information of the target script is determined according to the P time information sets, wherein the second log is a log of a resource manager.

[0036] Specifically, the second log can be a Yarn log, after the abnormal jobs are obtained, to further determine the abnormal reasons of the abnormal tasks, the corresponding Yarn log is acquired according to the job number, the Yarn log is parsed, execution logs of P tasks under each job recorded in the Yarn log are obtained, and time information of each task under the abnormal job is acquired in the execution logs, so that the reasons for slow execution of the target script during execution can be determined according to the time information of the tasks, and therefore the time information with the abnormality can be taken as the first abnormal performance information, and the reason analysis of the slow execution is completed.

[0037] It should be noted that, since the number of tasks included in each job is different, the value of P corresponding to each task is also different.

[0038] In step S104, the main task of each abnormal job is obtained, and the source table of the abnormal job is determined according to the main task, N source tables are obtained, the association value of the table associated with each source table is determined, a plurality of sets of association values are obtained, and the second abnormal performance information of the target script is determined according to the plurality of sets of association values.

[0039] Specifically, after obtaining the Yarn log, the main task of the abnormal job can be determined first, and the source table corresponding to the main task can be determined in the Yarn log of the main task, and the tables associated with the source table can be obtained, so that the association value between the source table and the associated table can be obtained, that is, the association key value pair of the two tables, and then the abnormal reason of the abnormal job can be determined according to the state of the association key value pair.

[0040] For example, after the source table is determined, the SQL statement can be located through the source table to find the performance problem of a certain associated table, that is, the data skew problem. Each set of SQL statements will search for multiple tables, and data will be taken from multiple tables. The processing of each set of SQL statements will be divided into M jobs, and each job will be divided into P tasks (tasks). The current read source table is obtained through the Yarn log corresponding to each job, and whether there is a data skew phenomenon is determined through the key value pair between the source table and the associated table, and then whether the job abnormality is caused by data skew is determined.

[0041] In step S105, the execution performance of the target script is determined according to the first abnormal performance information and the second abnormal performance information.

[0042] Specifically, after obtaining the first abnormal performance information, that is, the time information, and obtaining the second abnormal performance information, that is, the source table information, the reason for slow execution of the target script can be determined according to the time information and the source table information, so that manual log screening and abnormal phenomenon screening can be avoided, and the data with abnormality and the job and task corresponding to the data can be directly displayed, thereby improving the efficiency and accuracy of determining the reason for slow execution.

[0043] The script execution performance determination method provided in the embodiments of the present application comprises the following steps: obtaining a first log generated after execution of a target script, and parsing the first log to obtain M jobs in the target script, wherein the first log is a log of a data warehouse tool; obtaining time information of each job, and obtaining abnormal jobs from the M jobs according to the time information of the jobs to obtain N abnormal jobs; obtaining a second log generated after execution of the target script, and obtaining P tasks in each abnormal job and time information of each task in the second log to obtain P time information sets, and determining first abnormal performance information of the target script according to the P time information sets, wherein the second log is a log of a resource manager; obtaining a main task of each abnormal job, and determining a source table of the abnormal job according to the main task to obtain N source tables, determining an association value of a table associated with each source table to obtain multiple sets of association values, and determining second abnormal performance information of the target script according to the multiple sets of association values; and determining execution performance of the target script according to the first abnormal performance information and the second abnormal performance information. The method solves the problem of low efficiency and accuracy of determining slow execution reasons by manually analyzing logs when a SQL statement is executed slowly in the related art. By reading log contents in the first log and the second log, time information of each job and task when a SQL statement is executed is determined, and reasons for abnormality are determined according to the time information. In addition, a source table is determined according to log information related to the task in the second log, and reasons for slow execution are determined according to association contents of the source table, so that the log contents are analyzed according to the execution information by automatic acquisition, and the reasons for slow execution of the SQL statement are accurately determined.

[0044] Optionally, Figure 2 is a flowchart of an optional abnormal job determination method provided according to the embodiments of the present application, as shown in Figure 2 In the script execution performance determination method provided in the embodiments of the present application, in step S102, the abnormal jobs are obtained from the M jobs according to the time information of the jobs to obtain the N abnormal jobs, which comprises the following steps:

[0045] In step S201, for each job, a first submission time, a first start time and a first end time of the job are obtained.

[0046] In step S202, the first execution duration is obtained by subtracting the first start time from the first end time, and it is determined whether the first execution duration is greater than a first preset duration.

[0047] In step S203, in the case where the first execution duration is greater than the first preset duration, the job is determined as an abnormal job.

[0048] In step S204, the first waiting duration is obtained by subtracting the first start time from the first submission time, and it is determined whether the first waiting duration is greater than a second preset duration.

[0049] Step S205, in the case that the first waiting duration is greater than the second preset duration, the job is determined as an abnormal job.

[0050] Specifically, after obtaining the time information, the first submission time, the first start time and the first end time of the job can be obtained, and the start time and the end time are subtracted to obtain the execution duration, the submission time and the start time are subtracted to obtain the waiting duration, and it is judged whether the execution duration is greater than the first preset duration, in the case that the first execution duration is greater than the first preset duration, the job is determined as an abnormal job, in the case that the first execution duration is less than or equal to the first preset duration, it is judged whether the first waiting duration is greater than the second preset duration, wherein the second preset duration and the first preset duration have no size relationship, that is, the first preset duration can be greater than the second preset duration, or less than the second preset duration, or equal to the second preset duration. In the case that the first waiting duration is greater than the second preset duration, the waiting time of the job is too long, so there is an abnormality. The embodiment accurately determines the execution abnormal job causing the SQL statement to execute slowly by analyzing the time information of the job.

[0051] Further, the number of tasks contained in each job can also be determined and compared with the preset task number, in the case that the number of tasks is greater than the preset task number, the job can also be determined as an abnormal job.

[0052] For example, the number of Mappers and the number of Reducers of each job are obtained, according to the correspondence between the job number and the Stage, the Stage and the number of Mappers and the number of Reducers, the mapping key-value pair of the job number and the number of Mappers and the number of Reducers is obtained, the threshold value of the number of Mappers is set as M1, the threshold value of the number of Reducers is set as R1, when the Mappers value of the job number is greater than M1 or the Reducers value is greater than R1, the job number is an abnormal job.

[0053] It should be noted that Mapper is the first stage of Hadoop data processing, which is used to process input records and generate intermediate results in the form of key-value pairs, Reducer receives the output of Mapper (that is, the intermediate key-value pair) and processes it, and finally generates the output.

[0054] Optionally, in the script execution performance determination method provided in the embodiments of the present application, before determining whether the first execution duration is greater than the first preset duration, the method further comprises: obtaining the first execution duration of each job, and calculating a first average execution duration of the M first execution durations; determining whether the first average execution duration is greater than the first preset duration; in the case where the first average execution duration is greater than the first preset duration, replacing the first preset duration with the first average execution duration to obtain an updated first preset duration; before determining whether the first waiting duration is greater than the second preset duration, the method further comprises: calculating the waiting duration of each job to obtain M job waiting durations, and calculating a standard deviation of the M job waiting durations to obtain a first standard deviation; obtaining a preset waiting time ratio, and multiplying the first standard deviation by the preset waiting time ratio to obtain the second preset duration.

[0055] It should be noted that, before determining whether the first execution duration is greater than the first preset duration and determining whether the first waiting duration is greater than the second preset duration, the first preset duration and the second preset duration need to be determined first, and then the above determination operations can be performed.

[0056] Specifically, when determining the first preset duration, the time information of each job can be obtained from the log first, so as to determine the execution duration of each job, obtain a plurality of execution durations, and calculate an average execution duration of the plurality of execution durations. At this time, it can be determined whether the average execution duration is greater than the first preset duration. In the case where it is greater, it indicates that the threshold of the first preset duration set at the initial moment is too small. At this time, the average execution duration can be determined as a new first preset duration, so as to complete the update of the first preset duration.

[0057] Further, after the first preset duration is determined, the resource waiting time ratio a needs to be set, and the resource waiting time standard deviation T D of the job is calculated, wherein the standard deviation of the waiting duration can be calculated by the waiting duration of each job, and the standard deviation of the waiting duration T D is multiplied by the resource waiting time ratio a, so as to obtain the second preset duration. Through the calculation of the first preset duration and the second preset duration, the appropriate determination threshold is determined, and the effect of accurately determining the abnormal job is achieved.

[0058] Optionally, in the script execution performance determination method provided in the embodiments of the present application, the first abnormal performance information of the target script is determined according to the P sets of time information, including: for each task, obtaining a second submission time, a second start time and a second end time of the task from the time information of the task; subtracting the second start time from the second end time to obtain a second execution duration, and determining whether the second execution duration is greater than a third preset duration; in the case where the second execution duration is greater than the third preset duration, determining the task as an abnormal task; subtracting the second submission time from the second start time to obtain a second waiting duration, and determining whether the second waiting duration is greater than a fourth preset duration; in the case where the second waiting duration is greater than the fourth preset duration, determining the task as an abnormal task; and determining the log content of the abnormal task as the first abnormal performance information.

[0059] It should be noted that after the abnormal job is determined, the abnormal task needs to be determined in the abnormal job, and the determination method of the abnormal task is the same as that of the abnormal job.

[0060] Specifically, after the time information is obtained, the second submission time, the second start time and the second end time of the task can be obtained therefrom, and the start time and the end time are subtracted to obtain the execution duration, and the submission time and the start time are subtracted to obtain the waiting duration, and it is determined whether the execution duration is greater than a third preset duration, in the case where the second execution duration is greater than the third preset duration, the task is determined as an abnormal task, in the case where the second execution duration is less than or equal to a fourth preset duration, it is determined whether the second waiting duration is greater than the fourth preset duration, wherein the third preset duration and the fourth preset duration have no size relationship, that is, the third preset duration can be greater than, less than or equal to the fourth preset duration. In the case where the second waiting duration is greater than the fourth preset duration, the waiting time of the task is too long, and thus there is an abnormality. The embodiments accurately determine the execution abnormal task causing the slow execution of the SQL statement by analyzing the time information of the task.

[0061] For example, in Hadoop, a job is submitted to an RM (Resource Manager, resource manager), and the RM selects a suitable scheduling node AM (ApplicationMaster, application host) in the NM (Node Manager, node manager) cluster according to the load of the NM, and the AM is responsible for the initialization of the job task and applies for resources from the RM, a job is divided into P tasks, and the RM schedules a suitable NM to start a Container (container), and the Container executes the task. Therefore, the execution log of the scheduling node AM of each job needs to be found in the Yarn log, and the start time, the end time, the waiting resource time and the abnormally terminated task of each task are obtained therefrom.

[0062] Optionally, in the script execution performance determination method provided in the embodiments of the present application, before determining whether the second execution duration is greater than the third preset duration, the method further comprises: calculating the second execution duration of each task, and calculating a second average execution duration of the M second execution durations; determining whether the second average execution duration is greater than the third preset duration; in the case where the second average execution duration is greater than the third preset duration, replacing the third preset duration with the second average execution duration to obtain an updated third preset duration; before determining whether the second waiting duration is greater than the fourth preset duration, the method further comprises: calculating the waiting duration of each task to obtain M task waiting durations, and calculating a standard deviation of the M task waiting durations to obtain a second standard deviation; obtaining a preset waiting time ratio, and multiplying the second standard deviation by the preset waiting time ratio to obtain the fourth preset duration.

[0063] Specifically, when determining the third preset duration, the time information of each task can be obtained from the log first to determine the execution duration of each task, to obtain a plurality of execution durations, and to calculate an average execution duration of the plurality of execution durations. At this time, it can be determined whether the average execution duration is greater than the third preset duration. In the case where it is greater, it indicates that the threshold of the third preset duration set at the initial moment is too small. At this time, the average execution duration can be determined as a new third preset duration, thereby completing the update of the third preset duration.

[0064] Further, after the third preset duration is determined, the resource waiting time ratio β needs to be set, and the resource waiting time standard deviation T D2 of the task is calculated. The standard deviation of the waiting duration can be calculated by the waiting duration of each task. Multiplying the standard deviation of the waiting duration T D2 by the resource waiting time ratio β, the second preset duration can be obtained. Through the calculation of the third preset duration and the second preset duration, the appropriate determination threshold is determined, and the effect of accurately determining the abnormal task is achieved.

[0065] Optionally, in the script execution performance determination method provided in the embodiments of the present application, determining the source table of the abnormal job according to the main task comprises: obtaining the initial source table of the abnormal job from the log information of the main task, and determining whether the initial source table is a temporary table; in the case where the initial source table is a temporary table, determining the previous execution job of the abnormal job from the M jobs according to the start time of the abnormal job to obtain a previous job; determining the initial source table of the previous job according to the main task of the previous job, and determining the initial source table of the previous job as the source table of the abnormal job; in the case where the initial source table is not a temporary table, determining the initial source table as the source table of the abnormal job.

[0066] Specifically, in the determination of the source table, the source table can be directly obtained in the log of the main task of each job, so as to determine whether there is a data skew problem according to the source table, wherein the data skew represents that the data dispersion is not enough, resulting in that a large amount of cache data is concentrated on one service node.

[0067] For example, after the source table is determined, the source table can be used to locate the performance problem of the SQL statement to find a certain associated table, that is, the data skew problem. Each group of SQL statements will find multiple tables, and data will be taken from the multiple tables. The processing of each group of SQL statements will be divided into M jobs, and each job will be divided into P tasks. The source table currently read is obtained through the Yarn log corresponding to each job, and whether there is a data skew phenomenon is determined through the key-value pair between the source table and the associated table, so as to determine whether the job exception is caused by the data skew.

[0068] Further, in the determination of the source table, it is also necessary to determine whether the obtained source table is a temporary table. In the case of a temporary table, the main task of the previous job of the current job needs to be determined, and the source table corresponding to the temporary table is obtained in the log of the main task of the previous job, so as to accurately obtain the source table and lay a foundation for subsequent judgment of whether there is a data skew.

[0069] For example, in the determination of the source table, the hdfs path can be detected, and then the table name after the db field is matched by regular expression, so that the source table of the job can be obtained. However, some jobs store temporary tables of previous jobs. For the hdfs path of the temporary table, the temporary table name after / tmp / hive-scratch is matched by regular expression, and then the previous job is searched again until the final source table is found.

[0070] Optionally, in the script execution performance determination method provided in the embodiments of the present application, the second abnormal performance information of the target script is determined according to a plurality of associated values, comprising: for a group of associated values, identifying a preset associated value in the group of associated values to obtain Q preset associated values; obtaining the total number of each preset associated value in the source table and the table associated with the source table to obtain Q quantity values; and determining the Q quantity values and the source table as the second abnormal performance information.

[0071] Specifically, for a source table and a corresponding association table, that is, for a pair of association tables, since the tables are associated in a key-value pair association manner, it is necessary to determine whether the association value in each key-value pair is an abnormal association value. At this time, the abnormal association value can be determined as a preset association value, and the key-value pair is screened by the preset association value to determine whether there is an abnormal association value. The abnormal association value can be a null value, a negative value, or a meaningless value. In the case of obtaining an abnormal association value, it is necessary to determine the total number of abnormal association values in the table, and when the number is too large, it is determined that the table will cause data skew when the job is executed, which will cause the job to be executed too slowly.

[0072] Optionally, in the script execution performance determination method provided in the embodiments of the present application, before determining the execution performance of the target script according to the first abnormal performance information and the second abnormal performance information, the method further comprises: obtaining a data volume set of each task of each job from the second log, wherein the data volume of each task includes data input volume and data output volume; obtaining the maximum data input volume and the maximum data output volume in each data volume set, and dividing the maximum data output volume by the maximum data input volume to obtain a target ratio; determining whether the maximum data input volume is greater than a preset input volume, and storing the job to which the maximum data input volume belongs to the second monitoring table in the case that the maximum data input volume is greater than the preset input volume; determining whether the target ratio is greater than a preset ratio, and determining the log information of the job to which the target ratio belongs as the second abnormal performance information in the case that the target ratio is greater than the preset ratio.

[0073] Specifically, the data volume set of each task of each job can also be obtained from the second log, and whether there is an unreasonable phenomenon in the number of processing quantities is determined according to the data volume processed by each task. First, since there are P tasks in each job, it is necessary to determine the input volume and the output volume of each task, and to obtain the maximum input volume and the maximum output volume in each job. When the maximum input volume is greater than the preset input volume, it indicates that the input volume of the job is too large, and the job is an abnormal job.

[0074] Further, the maximum data output volume is divided by the maximum data input volume to obtain a target ratio, and it is determined whether the target ratio is greater than a preset value, for example, 100. In the case of being greater, it indicates that the job is an abnormal job, and the above data is stored together, so that the operation and maintenance personnel can determine the abnormal reason according to the above data.

[0075] For example, map tasks typically determine the number of data records read by searching `records read` / `RECORDS_IN`. A map task handling tens of millions of data records is considered high-performance, and struggling with hundreds of millions. Reduce tasks, on the other hand, determine the number of output data records by searching `records written` / `RECORDS_OUT`. The maximum value N is calculated by examining the `records read` / `RECORDS_IN` fields in all container logs under the job's Yarn logs. read This refers to the amount of input data for the task; the maximum value N is calculated from the detection field `records written / RECORDS_OUT`. out This refers to the output data volume of the job; a threshold of N is set for the data volume read into a single job. When N... read If the value is greater than N, then the data volume of the job number will be written to the monitoring table. When N... out / N read If the value is greater than 100, the input and output data volume will be written to the monitoring table.

[0076] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0077] This application also provides a script execution performance determination device. It should be noted that this device can be used to execute the script execution performance determination method provided in this application. The script execution performance determination device provided in this application is described below.

[0078] Figure 3 This is a schematic diagram of a script execution performance determination device provided according to an embodiment of this application. Figure 3 As shown, the device includes: a first acquisition unit 31, a second acquisition unit 32, a first determination unit 33, a second determination unit 34, and a third determination unit 35.

[0079] The first acquisition unit 31 is used to acquire the first log generated after the target script is executed, and parse the first log to obtain M jobs in the target script, wherein the first log is the log of the data warehouse tool.

[0080] The second acquisition unit 32 is used to acquire the time information of each job, and to acquire abnormal jobs from M jobs based on the job time information, thereby obtaining N abnormal jobs.

[0081] The first determination unit 33 is configured to acquire a second log generated after execution of the target script, acquire P tasks in each abnormal job and time information of each task in the second log, obtain P time information sets, and determine first abnormal performance information of the target script according to the P time information sets, wherein the second log is a log of a resource manager.

[0082] The second determination unit 34 is configured to acquire a main task of each abnormal job, determine a source table of the abnormal job according to the main task, obtain N source tables, determine an association value of a table associated with each source table, obtain multiple sets of association values, and determine second abnormal performance information of the target script according to the multiple sets of association values.

[0083] The third determination unit 35 is configured to determine execution performance of the target script according to the first abnormal performance information and the second abnormal performance information.

[0084] The script execution performance determination apparatus provided in the embodiment of the present application comprises a first acquisition unit 31 configured to acquire a first log generated after execution of a target script, and parse the first log to obtain M jobs in the target script, wherein the first log is a log of a data warehouse tool; a second acquisition unit 32 configured to acquire time information of each job, and acquire abnormal jobs from the M jobs according to the time information of the jobs to obtain N abnormal jobs; a first determination unit 33 configured to acquire a second log generated after execution of the target script, acquire P tasks in each abnormal job and time information of each task in the second log, obtain P time information sets, and determine first abnormal performance information of the target script according to the P time information sets, wherein the second log is a log of a resource manager; a second determination unit 34 configured to acquire a main task of each abnormal job, determine a source table of the abnormal job according to the main task, obtain N source tables, determine an association value of a table associated with each source table, obtain multiple sets of association values, and determine second abnormal performance information of the target script according to the multiple sets of association values; and a third determination unit 35 configured to determine execution performance of the target script according to the first abnormal performance information and the second abnormal performance information. The script execution performance determination apparatus provided in the embodiment of the present application solves the problem of low efficiency and accuracy of a method of determining a slow execution reason by manually analyzing a log when a SQL statement is executed slowly in the related art. By reading log content in the first log and the second log, time information of each job and task when a SQL statement is executed is determined, and a reason for an abnormality is determined according to the time information. In addition, a source table is determined according to log information related to a task in the second log, and a reason for slow execution is determined according to association content of the source table, thereby achieving the effect of automatically acquiring and analyzing log content according to execution information, and accurately determining a reason for slow execution of a SQL statement.

[0085] Optionally, in the script execution performance determination apparatus provided in the embodiments of the present application, the second obtaining unit 32 comprises: a first obtaining module, configured to obtain, for each job, a first submission time, a first start time and a first end time of the job; a first calculating module, configured to subtract the first start time from the first end time to obtain a first execution duration, and determine whether the first execution duration is greater than a first preset duration; a first determining module, configured to determine the job as an abnormal job in the case that the first execution duration is greater than the first preset duration; a second calculating module, configured to subtract the first submission time from the first start time to obtain a first waiting duration, and determine whether the first waiting duration is greater than a second preset duration; and a second determining module, configured to determine the job as an abnormal job in the case that the first waiting duration is greater than the second preset duration.

[0086] Optionally, in the script execution performance determination apparatus provided in the embodiments of the present application, the apparatus further comprises: a third obtaining unit, configured to obtain a first execution duration of each job, and calculate a first average execution duration of the M first execution durations; a first judging unit, configured to determine whether the first average execution duration is greater than a first preset duration; a first replacing unit, configured to replace the first preset duration with the first average execution duration to obtain an updated first preset duration in the case that the first average execution duration is greater than the first preset duration; a first calculating unit, configured to calculate a waiting duration of each job to obtain M job waiting durations, and calculate a standard deviation of the M job waiting durations to obtain a first standard deviation; and a fourth obtaining unit, configured to obtain a preset waiting time ratio, and multiply the first standard deviation by the preset waiting time ratio to obtain a second preset duration.

[0087] Optionally, in the script execution performance determination apparatus provided in the embodiments of the present application, the first determining unit 33 comprises: a second obtaining module, configured to obtain, for each task, a second submission time, a second start time and a second end time of the task from time information of the task; a third calculating module, configured to subtract the second start time from the second end time to obtain a second execution duration, and determine whether the second execution duration is greater than a third preset duration; a third determining module, configured to determine the task as an abnormal task in the case that the second execution duration is greater than the third preset duration; a fourth calculating module, configured to subtract the second submission time from the second start time to obtain a second waiting duration, and determine whether the second waiting duration is greater than a fourth preset duration; a fourth determining module, configured to determine the task as an abnormal task in the case that the second waiting duration is greater than the fourth preset duration; and a fifth determining module, configured to determine log content of the abnormal task as the first abnormal performance information.

[0088] Optionally, in the script execution performance determination apparatus provided in the embodiments of the present application, the apparatus further comprises: a second calculation unit, configured to calculate a second execution duration of each task, and calculate a second average execution duration of the M second execution durations; a second judgment unit, configured to judge whether the second average execution duration is greater than a third preset duration; a second replacement unit, configured to replace the third preset duration with the second average execution duration to obtain an updated third preset duration, in the case that the second average execution duration is greater than the third preset duration; a third calculation unit, configured to calculate a waiting duration of each task to obtain M task waiting durations, and calculate a standard deviation of the M task waiting durations to obtain a second standard deviation; and a fifth acquisition unit, configured to acquire a preset waiting time ratio, and multiply the second standard deviation by the preset waiting time ratio to obtain a fourth preset duration.

[0089] Optionally, in the script execution performance determination apparatus provided in the embodiments of the present application, the second determination unit 34 comprises: a third acquisition module, configured to acquire an initial source table of the abnormal job from the log information of the main task, and judge whether the initial source table is a temporary table; a sixth determination module, configured to determine a previous execution job of the abnormal job from the M jobs according to a start time of the abnormal job, to obtain a previous job, in the case that the initial source table is a temporary table; a seventh determination module, configured to determine the initial source table of the previous job according to the main task of the previous job, and determine the initial source table of the previous job as the source table of the abnormal job; and an eighth determination module, configured to determine the initial source table as the source table of the abnormal job, in the case that the initial source table is not a temporary table.

[0090] Optionally, in the script execution performance determination apparatus provided in the embodiments of the present application, the second determination unit 34 comprises: an identification module, configured to identify a preset association value in a set of association values to obtain Q preset association values, for a set of association values; a fourth acquisition module, configured to acquire a total number of each preset association value in the source table and a table associated with the source table to obtain Q quantity values; and a ninth determination module, configured to determine the Q quantity values and the source table as the second abnormal performance information.

[0091] Optionally, in the script execution performance determination apparatus provided by the embodiment of the present application, the apparatus further comprises: a sixth obtaining unit, configured to obtain a data volume set of each task of each job from the second log, wherein the data volume of each task comprises a data input volume and a data output volume; a seventh obtaining unit, configured to obtain a maximum data input volume and a maximum data output volume in each data volume set, and divide the maximum data output volume by the maximum data input volume to obtain a target ratio; a third judging unit, configured to judge whether the maximum data input volume is greater than a preset input volume, and in the case that the maximum data input volume is greater than the preset input volume, store the job to which the maximum data input volume belongs into the second monitoring table; and a fourth judging unit, configured to judge whether the target ratio is greater than a preset ratio, and in the case that the target ratio is greater than the preset ratio, determine the log information of the job to which the target ratio belongs as the second abnormal performance information.

[0092] The script execution performance determination apparatus comprises a processor and a memory, the first obtaining unit 31, the second obtaining unit 32, the first determining unit 33, the second determining unit 34, the third determining unit 35 and the like are stored in the memory as program units, and the corresponding functions are realized by the processor executing the program units stored in the memory.

[0093] The processor comprises a core, and the core retrieves the corresponding program units from the memory. One or more than one core can be set, and the efficiency and accuracy of the method for determining the slow execution reason by manually analyzing the log when the SQL statement is executed slowly in the related art can be improved by adjusting the core parameters.

[0094] The memory can comprise a non-permanent memory in a computer readable medium, a random access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM) or a flash memory (flash RAM), and the memory comprises at least one memory chip.

[0095] The embodiment of the present application provides a computer readable storage medium, which stores a program, and the program is executed by a processor to realize the script execution performance determination method.

[0096] The embodiment of the present application provides a processor, which is used for running a program, and the program is executed to realize the script execution performance determination method.

[0097] As Figure 4As shown, the embodiment of the present application provides an electronic device, the electronic device 40 comprises a processor, a memory and a program stored on the memory and executable on the processor, when the processor executes the program, the following steps are implemented: obtaining a first log generated after execution of a target script, and parsing the first log to obtain M jobs in the target script, wherein the first log is a log of a data warehouse tool; obtaining time information of each job, and obtaining abnormal jobs from the M jobs according to the time information of the jobs to obtain N abnormal jobs; obtaining a second log generated after execution of the target script, and obtaining P tasks in each abnormal job and time information of each task in the second log to obtain P sets of time information, and determining first abnormal performance information of the target script according to the P sets of time information, wherein the second log is a log of a resource manager; obtaining a main task of each abnormal job, and determining a source table of the abnormal job according to the main task to obtain N source tables, determining an association value of a table associated with each source table to obtain multiple sets of association values, and determining second abnormal performance information of the target script according to the multiple sets of association values; and determining execution performance of the target script according to the first abnormal performance information and the second abnormal performance information. The device in the present document can be a server, a PC, a PAD, a mobile phone, etc.

[0098] The present application also provides a computer program product adapted to execute a program that initializes the following method steps when executed on a data processing device: obtaining a first log generated after execution of a target script, and parsing the first log to obtain M jobs in the target script, wherein the first log is a log of a data warehouse tool; obtaining time information of each job, and obtaining abnormal jobs from the M jobs according to the time information of the jobs to obtain N abnormal jobs; obtaining a second log generated after execution of the target script, and obtaining P tasks in each abnormal job and time information of each task in the second log to obtain P sets of time information, and determining first abnormal performance information of the target script according to the P sets of time information, wherein the second log is a log of a resource manager; obtaining a main task of each abnormal job, and determining a source table of the abnormal job according to the main task to obtain N source tables, determining an association value of a table associated with each source table to obtain multiple sets of association values, and determining second abnormal performance information of the target script according to the multiple sets of association values; and determining execution performance of the target script according to the first abnormal performance information and the second abnormal performance information.

[0099] Those skilled in the art will understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0100] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0101] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0102] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0103] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0104] The memory can include non-persistent memory and / or volatile memory, such as a random access memory (RAM) including a cache area for the temporary storage of data. The memory can also include non-volatile memory, such as read only memory (ROM), electrically programmable read only memory (EPROM), or electrically erasable programmable read only memory (EEPROM), for the storage of software that is read during runtime. The memory is an example of computer readable media.

[0105] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0106] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusions, so that a process, method, article or apparatus that includes a list of elements does not only include those elements, but also includes other elements not explicitly listed, or further includes elements inherent in such a process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0107] The above only is an embodiment of the present application, and is not used to limit the present application. For those skilled in the art, the present application can have various changes and variations. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A method for determining script execution performance, characterized in that, include: Obtain the first log generated after the target script is executed, and parse the first log to obtain M jobs in the target script, wherein the first log is the log of the data warehouse tool; Obtain the time information of each job, and based on the job's time information, identify abnormal jobs from the M jobs to obtain N abnormal jobs, where N is less than or equal to M; Obtain the second log generated after the target script is executed, and obtain P tasks in each abnormal job and the time information of each task from the second log to obtain a set of P time information. Then, determine the first abnormal performance information of the target script based on the set of P time information. The second log is the log of the resource manager. The determination of the first abnormal performance information of the target script based on the P sets of time information includes: for each task, obtaining the second submission time, second start time, and second end time of the task from the task's time information; subtracting the second start time from the second end time to obtain a second execution duration, and determining whether the second execution duration is greater than a third preset duration; if the second execution duration is greater than the third preset duration, determining the task as an abnormal task; subtracting the second submission time from the second start time to obtain a second waiting duration, and determining whether the second waiting duration is greater than a fourth preset duration; if the second waiting duration is greater than the fourth preset duration, determining the task as an abnormal task; and determining the log content of the abnormal task as the first abnormal performance information. Obtain the main task of each abnormal job, and determine the source table of the abnormal job based on the main task to obtain N source tables. Determine the association value of the table associated with each source table to obtain multiple sets of association values, and determine the second abnormal performance information of the target script based on the multiple sets of association values. The method of determining the second abnormal performance information of the target script based on the multiple sets of associated values ​​includes: for a set of associated values, identifying preset associated values ​​in the set of associated values ​​to obtain Q preset associated values; obtaining the total number of each preset associated value in the source table and the table associated with the source table to obtain Q quantity values; and determining the Q quantity values ​​and the source table as the second abnormal performance information. The execution performance of the target script is determined based on the first abnormal performance information and the second abnormal performance information.

2. The method according to claim 1, characterized in that, Based on the time information of the tasks, abnormal tasks are obtained from the M tasks, resulting in N abnormal tasks, including: For each job, obtain the first submission time, the first start time, and the first end time of the job; Subtract the first start time from the first end time to obtain the first execution duration, and determine whether the first execution duration is greater than the first preset duration; If the first execution duration exceeds the first preset duration, the job will be identified as an abnormal job. Subtract the first submission time from the first start time to obtain the first waiting time, and determine whether the first waiting time is greater than the second preset time; If the first waiting time exceeds the second preset time, the job will be identified as an abnormal job.

3. The method according to claim 2, characterized in that, Before determining whether the first execution time is greater than the first preset time, the method further includes: Obtain the first execution time of each job, and calculate the first average execution time of the M first execution times; Determine whether the first average execution time is greater than the first preset time; If the first average execution time is greater than the first preset time, the first preset time is replaced by the first average execution time to obtain the updated first preset time; Before determining whether the first waiting time is greater than the second preset time, the method further includes: Calculate the waiting time for each job to obtain the waiting time for M jobs, and calculate the standard deviation of the waiting time for the M jobs to obtain the first standard deviation; Obtain the preset waiting time ratio, and multiply the first standard deviation by the preset waiting time ratio to obtain the second preset duration.

4. The method according to claim 1, characterized in that, Before determining whether the second execution time is greater than the third preset time, the method further includes: Calculate the second execution time for each task, and calculate the second average execution time for the M second execution times; Determine whether the second average execution time is greater than the third preset time; If the second average execution time is greater than the third preset time, the third preset time is replaced by the second average execution time to obtain an updated third preset time. Before determining whether the second waiting time is greater than the fourth preset time, the method further includes: Calculate the waiting time for each task to obtain the waiting time for M tasks, and calculate the standard deviation of the waiting time for the M tasks to obtain the second standard deviation; Obtain the preset waiting time ratio, and multiply the second standard deviation by the preset waiting time ratio to obtain the fourth preset duration.

5. The method according to claim 1, characterized in that, The source table for determining abnormal jobs based on the main task includes: Obtain the initial source table of the abnormal job from the log information of the main task, and determine whether the initial source table is a temporary table; If the initial source table is a temporary table, the previous execution job of the abnormal job is determined from the M jobs according to the start time of the abnormal job, and the previous job is obtained. The initial source table of the preceding task is determined based on the main task of the preceding task, and the initial source table of the preceding task is determined as the source table of the abnormal task. If the initial source table is not a temporary table, the initial source table is determined as the source table of the abnormal operation.

6. The method according to claim 1, characterized in that, Before determining the execution performance of the target script based on the first abnormal performance information and the second abnormal performance information, the method further includes: Obtain the data volume set for each task of each job from the second log, wherein the data volume of each task includes the data input volume and the data output volume; Obtain the maximum data input and maximum data output in each data set, and divide the maximum data output by the maximum data input to obtain the target ratio. Determine whether the maximum data input amount is greater than the preset input amount, and if the maximum data input amount is greater than the preset input amount, store the job to which the maximum data input amount belongs in the second monitoring table; Determine whether the target ratio is greater than the preset ratio, and if the target ratio is greater than the preset ratio, determine the log information of the job to which the target ratio belongs as the second abnormal performance information.

7. A script execution performance determination device, characterized in that, include: The first acquisition unit is used to acquire the first log generated after the target script is executed, and parse the first log to obtain M jobs in the target script, wherein the first log is the log of the data warehouse tool; The second acquisition unit is used to acquire the time information of each job, and to acquire abnormal jobs from the M jobs based on the time information of the jobs, thereby obtaining N abnormal jobs; The first determining unit is used to obtain the second log generated after the target script is executed, and to obtain P tasks in each abnormal job and the time information of each task in the second log to obtain a set of P time information, and to determine the first abnormal performance information of the target script based on the set of P time information, wherein the second log is the log of the resource manager; The first determining unit includes: a second acquisition module, used to acquire, for each task, a second submission time, a second start time, and a second end time from the task's time information; a third calculation module, used to subtract the second start time from the second end time to obtain a second execution duration, and determine whether the second execution duration is greater than a third preset duration; a third determining module, used to determine the task as an abnormal task if the second execution duration is greater than the third preset duration; a fourth calculation module, used to subtract the second submission time from the second start time to obtain a second waiting duration, and determine whether the second waiting duration is greater than a fourth preset duration; a fourth determining module, used to determine the task as an abnormal task if the second waiting duration is greater than the fourth preset duration; and a fifth determining module, used to determine the log content of the abnormal task as the first abnormal performance information. The second determining unit is used to obtain the main task of each abnormal job, determine the source table of the abnormal job according to the main task, obtain N source tables, determine the association value of the table associated with each source table, obtain multiple sets of association values, and determine the second abnormal performance information of the target script according to the multiple sets of association values. The second determining unit includes: an identification module, used to identify a preset associated value in a set of associated values ​​to obtain Q preset associated values; a fourth acquisition module, used to acquire the total number of each preset associated value in the source table and the table associated with the source table to obtain Q quantity values; and a ninth determining module, used to determine the Q quantity values ​​and the source table as the second abnormal performance information. The third determining unit is used to determine the execution performance of the target script based on the first abnormal performance information and the second abnormal performance information.

8. A computer storage medium, characterized in that, The computer storage medium is used to store a program, wherein the program, when running, controls the device where the computer storage medium is located to execute the script execution performance determination method according to any one of claims 1 to 6.

9. An electronic device, characterized in that, The system includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the script execution performance determination method according to any one of claims 1 to 6.

Images