Task processing method and device, equipment and storage medium

Abstract

The present disclosure provides a task processing method and device, equipment and storage medium, which relates to the technical field of computer. The method comprises: obtaining a to-be-processed script; analyzing the to-be-processed script, matching the to-be-processed script with a keyword library to obtain a keyword matching result; obtaining first prompt information according to the keyword matching result to perform first optimization processing on the to-be-processed script according to the first prompt information; obtaining log information of a task of executing the to-be-processed script; scanning and analyzing the log information to obtain task execution information corresponding to the to-be-processed script; determining an index corresponding to the task execution information as a to-be-optimized index according to an optimization rule in an optimization rule library; and obtaining second prompt information according to the to-be-optimized index to perform second optimization processing on a task execution parameter according to the second prompt information. The method improves the utilization rate of computing resources.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to a task processing method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] As data volumes grow exponentially, the demands on big data processing also increase. To ensure timely data output, methods such as increasing computing resources can improve task execution efficiency. However, with increased computing resources, costs also rise. Therefore, optimizing tasks is necessary to maximize resource efficiency while maintaining timeliness.

[0003] Due to the complexity of the business and computational logic, optimizing tasks requires developers with extensive theoretical knowledge and practical experience, a skill that is relatively scarce. Most developers are only at the level of knowing how to use the tools, and their understanding of task optimization is superficial, failing to achieve the goal of making efficient use of resources.

[0004] As mentioned above, how to optimize tasks to improve resource utilization is an urgent problem to be solved.

[0005] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this disclosure is to provide a task processing method, apparatus, electronic device, and readable storage medium that improves resource utilization to at least some extent.

[0007] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part by practice of this disclosure.

[0008] According to one aspect of this disclosure, a task processing method is provided, comprising: acquiring a script to be processed; parsing the script to be processed and matching the script to be processed with a keyword library to obtain a keyword matching result; obtaining a first prompt information based on the keyword matching result, so as to perform a first optimization processing on the script to be processed based on the first prompt information; acquiring log information of a task executing the script to be processed; scanning and parsing the log information to obtain task execution information corresponding to the script to be processed; determining an indicator corresponding to the task execution information as an indicator to be optimized according to optimization rules in an optimization rule library; and obtaining a second prompt information based on the indicator to be optimized, so as to perform a second optimization processing on the task execution parameters based on the second prompt information.

[0009] According to one embodiment of this disclosure, the script to be processed includes a first script uploaded to the task execution platform for the first time; the log information of the task executing the script to be processed includes the log information of the task that runs the first script; obtaining first prompt information based on the keyword matching result includes: if the keyword matching result is that no command to be optimized is matched, then the first script is run on the task execution platform; obtaining the log information of the task executing the script to be processed includes: obtaining the log information of the task that runs the first script.

[0010] According to one embodiment of this disclosure, obtaining the first prompt information based on the keyword matching result further includes: if the keyword matching result matches the command to be optimized, then obtaining the first prompt information, wherein the first prompt information includes information for prompting the optimization scheme of the command to be optimized.

[0011] According to an embodiment of this disclosure, the command to be optimized includes at least one of a global sorting command, a data deduplication command, a first data filtering command, a multi-table merging command without predicate pushdown, a partition creation command without specified partitions, and a selection command without corresponding column pruning operations; wherein, the optimization scheme for the global sorting command is to replace the global sorting command with a local sorting command, the optimization scheme for the data deduplication command is to replace the data deduplication command with a data grouping command, the optimization scheme for the first data filtering command is to replace the first data filtering command with a second data filtering command, the optimization scheme for the multi-table merging command without predicate pushdown is predicate pushdown, the optimization scheme for the partition creation command without specified partitions is specified partitions, and the optimization scheme for the selection command without corresponding column pruning operations is column pruning.

[0012] According to one embodiment of this disclosure, parsing the script to be processed and matching the script to be processed with a keyword library to obtain a keyword matching result includes: parsing the first script and performing a legality check on the first script; if the legality check result of the first script is passed, then matching the first script with the keyword library to obtain the keyword matching result.

[0013] According to an embodiment of this disclosure, the task execution information includes the size information of the input table; the optimization rules include a table size threshold; determining the indicator corresponding to the task execution information as the indicator to be optimized according to the optimization rules in the optimization rule base includes: judging whether the size of the input table is less than the table size threshold based on the size information of the input table; if the size of the input table is less than the table size threshold, then determining that there is a small table in the task execution, and the size of the input table is the indicator to be optimized; the second prompt information includes information prompting the addition of broadcast merging parameters; obtaining the second prompt information according to the indicator to be optimized includes: if the size of the input table is determined to be the indicator to be optimized, then obtaining the information prompting the addition of broadcast merging parameters.

[0014] According to one embodiment of this disclosure, the task execution information further includes stage execution indicator values, which include the amount of data read and the startup time; the optimization rules further include stage execution indicator thresholds, which include a data read threshold and a startup time threshold; determining the indicator corresponding to the task execution information as the indicator to be optimized according to the optimization rules in the optimization rule base further includes: determining the corresponding execution indicator as the indicator to be optimized based on the data read and the corresponding data read threshold; determining the corresponding execution indicator as the indicator to be optimized based on the startup time and the corresponding startup time threshold; the second prompt information includes information prompting the addition of a first optimization parameter and information prompting the adjustment of the task execution time; obtaining the second prompt information based on the indicator to be optimized includes: if the execution indicator corresponding to the data read is determined to be the indicator to be optimized, then obtaining the corresponding information prompting the addition of a first optimization parameter; if the execution indicator corresponding to the startup time is determined to be the indicator to be optimized, then obtaining the information prompting the adjustment of the task execution time.

[0015] According to one embodiment of this disclosure, the task execution information further includes execution exception information; the optimization rules further include preset exception information; determining the indicator corresponding to the task execution information as the indicator to be optimized according to the optimization rules in the optimization rule base further includes: matching the execution exception information with the preset exception information, and determining the corresponding indicator as the indicator to be optimized based on the matching result; the second prompt information includes information prompting adjustment of the second optimization parameter; obtaining the second prompt information according to the indicator to be optimized includes: if it is determined that the execution indicator corresponding to the execution exception information is the indicator to be optimized, then obtaining the corresponding information prompting adjustment of the second optimization parameter.

[0016] According to one embodiment of this disclosure, the script to be processed includes a second script that has already been run on the task execution platform; the log information of the task executing the script to be processed includes the log information of the task executing the second script; obtaining first prompt information based on the keyword matching result includes: if the keyword matching result is that no command to be optimized is matched, then obtaining the first prompt information, the first prompt information including information for prompting the acquisition of platform logs; obtaining the log information of the task executing the script to be processed includes: obtaining the log information of the task trial running the second script based on the information for prompting the acquisition of platform logs.

[0017] According to another aspect of this disclosure, a task processing apparatus is provided, comprising: a first acquisition module for acquiring a script to be processed; a first matching module for parsing the script to be processed and matching the script to be processed with a keyword library to obtain a keyword matching result; a first optimization module for obtaining a first prompt message based on the keyword matching result, and performing a first optimization process on the script to be processed based on the first prompt message; a second acquisition module for acquiring log information of a task executing the script to be processed; a log scanning module for scanning and parsing the log information to acquire task execution information corresponding to the script to be processed; a second matching module for determining an indicator corresponding to the task execution information as an indicator to be optimized according to optimization rules in an optimization rule library; and a second optimization module for obtaining a second prompt message based on the indicator to be optimized, and performing a second optimization process on task execution parameters based on the second prompt message.

[0018] According to another aspect of this disclosure, an electronic device is provided, comprising: a memory, a processor, and executable instructions stored in the memory and executable in the processor, wherein the processor, when executing the executable instructions, implements any of the methods described above.

[0019] According to another aspect of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, which, when executed by a processor, implement any of the methods described above.

[0020] The task processing method provided in the embodiments of this disclosure parses the acquired script to be processed, matches the script to be processed with a keyword library to obtain keyword matching results, and then obtains first prompt information based on the keyword matching results. Based on the first prompt information, a first optimization process is performed on the script to be processed. Then, log information of the task executing the script to be processed is obtained, and the log information is scanned and parsed to obtain the task execution information corresponding to the script to be processed. Then, according to the optimization rules in the optimization rule library, the indicator corresponding to the task execution information is determined as the indicator to be optimized. Based on the indicator to be optimized, a second prompt information is obtained, and a second optimization process is performed on the task execution parameters based on the second prompt information, thereby improving resource utilization.

[0021] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this disclosure. Attached Figure Description

[0022] The above and other objects, features and advantages of this disclosure will become more apparent from a detailed description of exemplary embodiments thereof with reference to the accompanying drawings.

[0023] Figure 1 A flowchart of a task processing method according to an embodiment of this disclosure is shown.

[0024] Figure 2 A flowchart illustrating another task processing method in an embodiment of this disclosure is shown.

[0025] Figure 3 It shows Figure 2 The step S204 shown is a schematic diagram of the processing procedure in one embodiment.

[0026] Figure 4 A flowchart of another task processing method in an embodiment of this disclosure is shown.

[0027] Figure 5 It shows Figure 1 The steps S110 to S114 shown are schematic diagrams of the processing procedure in one embodiment.

[0028] Figure 6 It shows Figure 1 The steps S110 to S114 shown are schematic diagrams of the processing procedure in another embodiment.

[0029] Figure 7 It shows Figure 1 The steps S110 to S114 shown are schematic diagrams of the processing procedure in another embodiment.

[0030] Figure 8 It is based on Figures 1 to 7 The diagram illustrates a task optimization process.

[0031] Figure 9 A block diagram of a task processing apparatus according to an embodiment of the present disclosure is shown.

[0032] Figure 10 A block diagram of another task processing apparatus according to an embodiment of the present disclosure is shown.

[0033] Figure 11 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0034] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0035] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, apparatuses, steps, etc., can be employed. In other instances, well-known structures, methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0036] Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this disclosure, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. The symbol " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0037] In this disclosure, unless otherwise expressly specified and limited, the term "connection" and similar terms should be interpreted broadly, for example, it can refer to an electrical connection or the ability to communicate with each other; it can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this disclosure according to the specific circumstances.

[0038] The terminology used in the embodiments of this disclosure will be explained below.

[0039] Apache Spark: Apache Spark is a fast and general-purpose computing engine designed for large-scale data processing. Spark is an open-source, Hadoop MapReduce-like general-purpose parallel framework developed by the UC Berkeley AMP lab. While possessing the advantages of Hadoop MapReduce, Spark differs in that intermediate job outputs can be stored in memory, eliminating the need to read and write to HDFS. Therefore, Spark is better suited for iterative MapReduce algorithms such as data mining and machine learning.

[0040] Jobhistory: Spark history server, which records the logs of Spark execution. Logs after Spark execution are recorded and saved through the history server.

[0041] Executor: A Spark Executor is a Java Virtual Machine (JVM) process within a worker node of the Spark cluster. It is responsible for running specific tasks within a Spark job. These tasks are independent of each other. Executor nodes are started simultaneously with the Spark application and exist throughout the entire lifecycle of the Spark application. If an Executor node fails or crashes, the Spark application can continue execution by rescheduling tasks from the failed node to other Executor nodes.

[0042] Executors have two core functions: running the tasks that make up a Spark application and returning the results to the Driver process. They provide in-memory storage for RDDs that user programs request to be cached, through their own Block Manager. RDDs are cached directly in the Executor process, allowing tasks to fully utilize cached data at runtime to accelerate computation.

[0043] With the advent of the big data era, a wide variety of technologies have emerged for big data. Commonly used offline computing engines for big data are MapReduce and Spark, both of which divide the computation process into map and reduce phases. The map phase can be understood as reading and cleaning data to prepare for the reduce phase, while the reduce phase performs aggregation calculations and outputs the results. Apache Spark is one of the most popular offline computing engines today, characterized by its speed, ease of development, and versatility. Spark SQL is Spark's module for processing structured data, providing interactive query functionality. Big data development and processing can be performed using SQL statements, greatly simplifying the work of big data developers and data analysts and improving their efficiency.

[0044] As mentioned above, task optimization aims to ensure the timeliness of big data processing while maximizing resource conservation and utilization. However, optimization in related technologies often fails to achieve the desired resource efficiency. Furthermore, differences in developers' skill levels and development habits lead to problems such as long task execution times, high resource consumption, poor readability, and high maintenance costs.

[0045] Therefore, this disclosure provides a task processing method that diagnoses problems in script and task execution by performing rule-based comparative analysis on the script to be processed and the corresponding task execution logs, and provides optimization suggestions to reduce the development threshold and improve resource utilization.

[0046] Figure 1 This is a flowchart illustrating a task processing method according to an exemplary embodiment. For example... Figure 1 The methods shown can be applied to task execution platforms such as MapReduce, Spark, etc.

[0047] refer to Figure 1 The method 10 provided in this embodiment may include the following steps.

[0048] In step S102, the script to be processed is obtained.

[0049] In some embodiments, the script to be processed may include a first script uploaded to the task execution platform for the first time. Specific implementation methods for scanning and test-run monitoring of the first script can be found in [reference needed]. Figure 2 and Figure 3 .

[0050] In some embodiments, the script to be processed may include a second script that has already been run on the task execution platform. Specific implementation methods for inspecting the second script and its logs can be found in [reference needed]. Figure 4 .

[0051] In step S104, the script to be processed is parsed, and the script to be processed is matched with the keyword database to obtain the keyword matching results.

[0052] In some embodiments, if the script to be processed is the first script, the specific implementation of step S104 can be referred to step S204.

[0053] In some embodiments, if the script to be processed is a second script, the specific implementation of step S104 can be referred to step S404.

[0054] In step S106, first prompt information is obtained based on the keyword matching results, and the script to be processed is optimized based on the first prompt information.

[0055] In some embodiments, the specific implementation of step S106 can be referred to step S206.

[0056] In step S108, log information of the task executing the script to be processed is obtained.

[0057] In some embodiments, the log information for the task executing the script to be processed may include the log information for the task that runs the first script in trial mode.

[0058] In some embodiments, the log information of the task executing the script to be processed may include the log information of the task executing the second script. For example, the most recent execution log file of the task executing the script to be processed in Spark's history can be obtained.

[0059] In step S110, the log information is scanned and parsed to obtain the task execution information corresponding to the script to be processed.

[0060] In some embodiments, task execution information may include the size information of the input table.

[0061] In some embodiments, task execution information may include stage execution metric values, which may include the amount of data read and the start time.

[0062] For example, after a script is submitted to the task execution platform, the platform analyzes and generates a corresponding number of jobs. Each job is then divided into one or more stages, which are executed sequentially. Each stage is further divided into multiple tasks. Job execution metrics can be obtained, such as job runtime and the corresponding stage identifier (ID). Stage execution metrics can also be obtained, such as start and end times, runtime, input or shuffle read volume, task ID, number of tasks, start and end times, runtime, and input or shuffle read volume for each task.

[0063] In some embodiments, task execution information may include execution metrics of the execution unit (executor) that executes the task, such as the number of executors, the CPU and memory requested by each executor, etc.

[0064] In some embodiments, task execution information may include execution exception information, such as error and exception information.

[0065] In step S112, the indicators corresponding to the task execution information are determined as indicators to be optimized according to the optimization rules in the optimization rule base.

[0066] In some embodiments, the optimization rules for inputting table size information may include a table size threshold. Specific implementation methods for determining the corresponding optimization metrics can be found in [reference needed]. Figure 5 .

[0067] In some embodiments, the optimization rules for stage execution indicator values ​​may include stage execution indicator thresholds, which include a data read volume threshold and a startup time threshold. Specific implementation methods for determining the indicators to be optimized can be found in [reference needed]. Figure 6 .

[0068] In some embodiments, the optimization rules for executing exception information also include preset exception information. Specific implementation methods for determining the corresponding indicators to be optimized can be found in [reference needed]. Figure 7 .

[0069] In step S114, second prompt information is obtained based on the indicator to be optimized, and the task execution parameters are optimized based on the second prompt information.

[0070] In some embodiments, when the metric to be optimized is a small table, the second prompt message may include a prompt to add broadcast merging parameters. Specific implementation details can be found in [reference needed]. Figure 5 .

[0071] In some embodiments, when the metric to be optimized is a stage execution metric, the second prompt information may include information prompting the addition of the first optimization parameter and information prompting the adjustment of the task execution time. Specific implementation details can be found in [reference needed]. Figure 6 .

[0072] In some embodiments, when the metric to be optimized is the metric corresponding to the execution anomaly information, the second prompt information may include information prompting the adjustment of the second optimization parameter. Specific implementation details can be found in [reference needed]. Figure 7 .

[0073] According to the task processing method provided in this embodiment, the obtained script to be processed is parsed, matched with a keyword library to obtain keyword matching results, and then a first prompt is obtained based on the keyword matching results. Based on the first prompt, a first optimization process is performed on the script to be processed. Then, log information of the task executing the script to be processed is obtained, and the log information is scanned and parsed to obtain the task execution information corresponding to the script to be processed. Then, according to the optimization rules in the optimization rule library, the indicators corresponding to the task execution information are determined as indicators to be optimized. Based on the indicators to be optimized, a second prompt is obtained, and based on the second prompt, a second optimization process is performed on the task execution parameters. By diagnosing the script and task, the development threshold and development cost are greatly reduced, and the utilization rate of cluster resources is improved.

[0074] Figure 2 This is a flowchart illustrating another task processing method according to an exemplary embodiment. Figure 2 and Figure 1 The relationship is that, Figure 2 yes Figure 1 The script to be processed is the first script implemented by the platform for the first upload task.

[0075] refer to Figure 2 The method 20 provided in this embodiment may include the following steps.

[0076] In step S202, the first script of the first upload task execution platform is obtained.

[0077] In step S204, the first script is parsed and matched with the keyword database to obtain keyword matching results.

[0078] In some embodiments, after parsing the first script using an SQL parser, a validity check can be performed first, followed by keyword matching. For specific implementation details, please refer to [reference needed]. Figure 3 .

[0079] In some embodiments, keywords / words in the first script can be scanned and matched against a list of keywords / words that are not recommended for use, which may be keywords / words in command statements.

[0080] In step S206, if the keyword matching result matches the command to be optimized, then the first prompt information is obtained. The first prompt information includes information for prompting the optimization scheme of the command to be optimized, so as to execute the optimization scheme of the command to be optimized.

[0081] In some embodiments, the command to be optimized may include at least one of the following: a global sorting command, a data deduplication command, a first data filtering command, a multi-table merging command without predicate pushdown, a partition creation command without specified partitions, and a selection command without corresponding column pruning operations.

[0082] In some embodiments, the optimization scheme for the global sorting command can be to replace the global sorting command with a local sorting command; the optimization scheme for the data deduplication command can be to replace the data deduplication command with a data grouping command; the optimization scheme for the first data filtering command can be to replace the first data filtering command with a second data filtering command; the optimization scheme for the multi-table merge command without predicate pushdown can be predicate pushdown; the optimization scheme for the partition creation command without specifying a partition can be specifying a partition; and the optimization scheme for the selection command without performing the corresponding column pruning operation can be column pruning.

[0083] For example, the global sorting command can be `order by`. This command sorts the input data globally, putting all data into the same reducer for processing. Regardless of the amount of data or the number of files, only one reducer task is used for processing. All data is processed on the same reducer, which may exceed the disk and memory storage capacity of a single node when dealing with large amounts of data, leading to task failure or excessively long execution times. `order by` can be replaced with the local sorting command `sort by`. This command sorts the data within each reducer, meaning the data output from each reducer is ordered, but the global data may not be ordered (unless there is only one reducer, in which case global ordering is possible). Generally, performing local sorting first, followed by global sorting, can significantly improve processing efficiency.

[0084] For example, the data deduplication command can be `distinct`. This command processes all data in a single reducer, which can easily lead to performance bottlenecks or even Out of Memory (OOM) errors. `distinct` can be replaced with the data grouping command `group by`. `group by` groups the data and distributes it to different reducers for computation, and finally uses the `count` operator to count the results. When dealing with large datasets, the combination of `group by` and `count` is significantly more efficient than the combination of `count` and `distinct`. Examples are as follows:

[0085] select count(distinct column_1)from t_table;

[0086] After optimization:

[0087] select

[0088] count(column_1)

[0089] from(

[0090] select column_1from t_table group by column_1

[0091] )t;

[0092] For example, the first data filtering command could be `not in`, which is used to filter data and retrieve data that does not match the conditional dataset. However, this keyword results in a Cartesian product calculation, causing the task to run slowly. Replacing the `not in` statement with a second data filtering command such as `left join`, `semi join`, or `anti join` can significantly improve execution efficiency. Examples are as follows:

[0093] SELECT t1.column_1FROM t_table_1t1 WHERE t1.column_1NOT IN(SELECTcolumn_1FROM t_table_2);

[0094] After optimization:

[0095] SELECT t1.column_1FROM t_table_1t1 LEFT JOIN t_table_2t2 ONt1.column_1=t2.column_1WHERE COALESCE(t2.column_1,”)=”;

[0096] For example, when merging multiple tables (join), a WHERE clause is typically used for filtering after the join. Predicate pushdown can be used, moving the filtering expression as close to the data source as possible so that irrelevant data is skipped during actual execution. In other words, filtering conditions should be prioritized in appropriate scenarios. Examples are as follows:

[0097] SELECT t1.column_1,t1.column_2,t2.column_3FROM t_table_1t1LEFT JOINt_table_2t2 ON t1.column_1=t2.column_1WHERE t1.column_2='1';

[0098] After optimization:

[0099] SELECT t1.column_1,t1.column_2,t2.column_3FROM(SELECT t.column_1,t.column_2FROM t_table_1t WHERE t.column_2='1')t1

[0100] LEFT JOIN t_table_2t2

[0101] ON t1.column_1=t2.column_1;

[0102] For example, if no partition is specified when creating a partitioned table, then a specific partition must be specified.

[0103] For example, when using select* for a query, column pruning is required.

[0104] In step S208, if the keyword matching result is that no command to be optimized is matched, the first script is run on the task execution platform.

[0105] In step S210, the log information of the task of trial running the first script is obtained.

[0106] In step S212, the log information is scanned and parsed to obtain the task execution information corresponding to the first script.

[0107] In step S214, the indicators corresponding to the task execution information are determined as indicators to be optimized according to the optimization rules in the optimization rule base.

[0108] In step S216, second prompt information is obtained based on the indicator to be optimized, and the task execution parameters are then optimized based on the second prompt information.

[0109] In some embodiments, the specific implementation of steps S210 to S216 can be referred to steps S108 to S114.

[0110] According to the method provided in this disclosure, by verifying and testing newly uploaded scripts and providing optimization solutions, the upload of non-standard and low-quality scripts is greatly reduced.

[0111] Figure 3 It shows Figure 2 The diagram shown illustrates step S204 in one embodiment. Figure 3 As shown in the present embodiment, step S204 may further include the following steps.

[0112] Step S302: Parse the first script and perform a validity check on the first script.

[0113] In some embodiments, the first script can be parsed using an SQL parser.

[0114] Step S304: If the validity check result of the first script is passed, then the first script is matched with the keyword database to obtain the keyword matching result.

[0115] Step S306: If the validity check result of the first script is not passed, the process ends.

[0116] In some embodiments, the first script can be validated for legality, checking for syntax errors. If a syntax error is found, the process terminates immediately and a syntax error message is displayed. If a syntax error is found, keyword matching continues.

[0117] According to the method provided in this disclosure, by performing legality verification on newly uploaded scripts, scripts with grammatical problems can be filtered out before keyword matching, reducing the processing load of script diagnosis and improving the efficiency of the task optimization process.

[0118] Figure 4 This is a flowchart illustrating another task processing method according to an exemplary embodiment. Figure 4 and Figure 2 The difference is that, Figure 4 yes Figure 1 The script to be processed is an implementation of a second script that has already run on the task execution platform.

[0119] refer to Figure 4 The method 40 provided in this embodiment may include the following steps.

[0120] In step S402, a second script that has already been run on the task execution platform is obtained.

[0121] In step S404, the second script is parsed and matched with the keyword database to obtain keyword matching results.

[0122] In some embodiments, the second script can be parsed by an SQL parser, scanning for keywords / terms in the second script and matching them with a list of keywords / terms that are not recommended for use. These keywords / terms can be keywords / terms in command statements.

[0123] In step S406, if the keyword matching result matches the command to be optimized, information on the optimization scheme for prompting the command to be optimized is obtained, and the optimization scheme for the command to be optimized is executed.

[0124] In some embodiments, the specific implementation of step S406 can be referred to step S206.

[0125] In step S408, if the keyword matching result is that no command to be optimized is matched, information is obtained to prompt for obtaining platform logs.

[0126] In step S410, the log information of the task for trial running the second script is obtained based on the information used to prompt for obtaining the platform log.

[0127] In step S412, the log information is scanned and parsed to obtain the task execution information corresponding to the second script.

[0128] In step S414, the indicators corresponding to the task execution information are determined as indicators to be optimized according to the optimization rules in the optimization rule base.

[0129] In step S416, second prompt information is obtained based on the indicator to be optimized, and the task execution parameters are optimized based on the second prompt information.

[0130] In some embodiments, the specific implementation of steps S410 to S416 can be referred to steps S108 to S114.

[0131] According to the method provided in the embodiments of this disclosure, by inspecting the existing scripts and tasks in the cluster, problems existing in the previous scripts can be found, and optimization solutions can be provided to make more reasonable use of resources.

[0132] Figure 5 It shows Figure 1 The steps S110 to S114 shown are schematic diagrams of the processing procedure in one embodiment. Figure 5 The task execution optimization process shown is for cases where small tables exist.

[0133] Step S502: Scan and parse the log information to obtain the size information of the input table in the task execution information corresponding to the script to be processed.

[0134] Step S504: Determine whether the size of the input table is less than the table size threshold based on the size information of the input table.

[0135] Step S506: If the size of the input table is less than the table size threshold, it is determined that there is a small table in the task execution, and the size of the input table is the indicator to be optimized.

[0136] For example, the table size threshold can be 70MB, 80MB, or 90MB, etc.

[0137] Step S508: If the size of the input table is determined to be the metric to be optimized, then information prompting to add broadcast merging parameters is obtained, and broadcast merging parameters are added according to the prompt.

[0138] In some embodiments, if a smaller table exists, a broadcast join can be used. It is recommended to add the following parameters:

[0139] set Spark.sql.autoBroadcastJoinThreshold=83886080;

[0140] Step S510: If the size of the input table is not less than the table size threshold, then the size of the input table will not be determined as an optimization metric.

[0141] Figure 6 It shows Figure 1 The steps S110 to S114 shown are schematic diagrams of the processing procedure in another embodiment. Figure 6 The task execution optimization process shown addresses situations such as data skew, small files, unreasonable partitioning strategies, unreasonable parallelism settings, and / or limited cluster queue resources.

[0142] Step S602: Scan and parse the log information to obtain the amount of data read and the start time from the stage execution indicator values ​​in the task execution information corresponding to the script to be processed.

[0143] Step S6042: Determine the corresponding execution metrics as metrics to be optimized based on the amount of data read and the corresponding data read threshold.

[0144] In some embodiments, the maximum, minimum and average values ​​of the input or shuffle read data of each task in each stage can be obtained, and the difference can be calculated. If the difference is greater than (average value * 75%) (75% is the corresponding threshold for the amount of data read, and can also be set to 70%, 80% or so, depending on the actual situation), then data skew can be determined.

[0145] In some embodiments, if the average amount of data read by each task in the input stage is less than 50MB (50MB is the corresponding data reading threshold, which can also be set to 40MB, 60MB, etc., depending on the actual situation), it is determined that a small file exists.

[0146] In some embodiments, if the average amount of data read by each task in the input stage is greater than 600MB (600MB is the corresponding data reading threshold, which can also be set to 500MB, 700MB, etc., depending on the actual situation), the splitting strategy is deemed unreasonable.

[0147] In some embodiments, if the average amount of data read in the stage is less than 400MB or greater than 700MB (400MB and 700MB are the corresponding data read thresholds, and can also be set to 300MB and 600MB, or 500MB and 800MB, etc., depending on the actual situation), then the parallelism setting is deemed unreasonable.

[0148] Step S6044: Determine the corresponding execution metrics as the metrics to be optimized based on the startup time and the corresponding startup time threshold.

[0149] In some embodiments, if the startup interval between two adjacent stages exceeds 1 minute (1 minute is the corresponding startup time threshold, and can also be set to a startup time threshold of 0.5 minutes, 1.5 minutes, etc., depending on the actual situation), it can be determined that the cluster queue resources are strained.

[0150] In some embodiments, if the difference between the end time and start time of a task exceeds 3 minutes (3 minutes is the corresponding start time threshold, and can also be set to a start time threshold of 2 minutes, 4 minutes, etc., depending on the actual situation), it can be determined that the cluster queue resources are under pressure.

[0151] Step S6062: If the execution metric corresponding to the amount of data read is determined to be a metric to be optimized, then the corresponding prompt to add the first optimization parameter is obtained.

[0152] In some embodiments, if data skew is determined, it may be recommended to add parameters. If this does not work, the hot key issue needs to be resolved manually.

[0153] set spark.sql.adaptive.enabled=true;

[0154] set spark.sql.adaptive.skewJoin.enabled=true;

[0155] set spark.sql.adaptive.skewJoin.enhance.enabled=true;

[0156] set spark.sql.adaptive.forceOptimizeSkewedJoin=true;

[0157] In Spark, `spark.sql.adaptive.enabled` enables adaptive query execution if it's TRUE, re-optimizing the query plan based on accurate runtime statistics during execution. `spark.sql.adaptive.skewJoin.enabled`: When both this parameter and `spark.sql.adaptive.enabled` are TRUE, Spark dynamically handles skew in random joins (sort-merge and random hashing) by splitting (and replicating as needed) skewed partitions. The `spark.sql.adaptive.enabled` parameter enables Spark's adaptive query functionality, essentially enabling automatic optimization. With this parameter enabled, the Spark engine can obtain runtime information and dynamically optimize during task execution. `spark.sql.adaptive.skewJoin.enabled` is a skew handling switch; when enabled, Spark splits skewed partitions for processing by multiple tasks and finally merges the results using a union.

[0158] In some embodiments, if it is determined that small files exist, it may be recommended to add the following parameters:

[0159] set spark.sql.files.openCostInBytes=4m;

[0160] set spark.sql.files.maxPartitionBytes=600m;

[0161] `spark.files.openCostInBytes` sets the number of bytes that can be scanned simultaneously to measure the estimated cost of opening files. This is used when partitioning multiple files. It's acceptable to overestimate the cost, so partitioning smaller files will be faster than partitioning larger files.

[0162] `spark.sql.files.maxPartitionBytes` represents the maximum number of bytes packed into a single partition when reading files. This configuration is effective when using file-based sources such as Parquet, JSON, and ORC.

[0163] In some embodiments, if the file splitting strategy is deemed unreasonable, it is recommended to add the following parameters:

[0164] set hive.exec.orc.split.strategy=ETL;

[0165] set mapred.max.split.size=629145600;

[0166] set mapred.min.split.size.per.node=629145600;

[0167] set mapred.min.split.size.per.rack=629145600;

[0168] The `set hive.exec.orc.split.strategy` directive sets the splitting strategy Spark uses when reading ORC files. The BI strategy splits files at the file level; the ETL strategy splits files into multiple stripes forming a single split; the HYBRID strategy uses ETL only when the average file size exceeds Hadoop's maximum split size (default 256MB), otherwise it uses the BI strategy. `set mapred.max.split.size` sets the maximum file splitting threshold. `set mapred.min.split.size.per.node` sets the minimum splitting threshold per machine node. `set mapred.min.split.size.per.rack` sets the minimum splitting threshold per rack.

[0169] In some embodiments, if the parallelism setting is deemed unreasonable, the following parameters can be adjusted according to the data volume:

[0170] set spark.sql.shuffle.partitions = (total shuffle reads / 600MB); / / The default number of partitions to use when joining or aggregating shuffled data.

[0171] Step S6064: If the execution metric corresponding to the startup time is determined to be a metric to be optimized, then information prompting to adjust the task execution time is obtained.

[0172] In some embodiments, if queue resources are determined to be scarce, a prompt may be made to check queue monitoring and adjust task execution time.

[0173] Figure 7 It shows Figure 1 The steps S110 to S114 shown are schematic diagrams of the processing procedure in another embodiment. Figure 7 The task execution optimization process shown addresses situations where log errors occur.

[0174] Step S702: Scan and parse the log information to obtain execution exception information from the task execution information corresponding to the script to be processed.

[0175] Step S704: Match the execution exception information with the preset exception information, and determine the corresponding indicator as the indicator to be optimized based on the matching result.

[0176] In some embodiments, if the logs show errors such as java.lang.OutOfMemoryError: Java heap space, org.apache.spark.shuffle.FetchFailedException: Java heap space, or Container killed on request.Exit code is 143, then the program requires more memory than spark.executor.memory. This is usually due to a large amount of data being processed or cached, insufficient existing memory, and a memory allocation rate exceeding the garbage collection rate.

[0177] In some embodiments, if the error message "The threshold of physical resource utilization is exceeded.EXCEEDED_RESOURCE_TYPE_MEM" is found in the log, the memory usage of the physical machine where the executor is located exceeds the physical machine resource utilization threshold, causing the container to be evicted by YARN. YARN's current eviction policy is that the last scheduled container is evicted first.

[0178] In some embodiments, if a large number of fetch failure errors appear in the log, it can be determined that the disk is relatively busy.

[0179] Step S706: If it is determined that the execution indicator corresponding to the execution exception information is an indicator to be optimized, then the corresponding prompt to adjust the second optimization parameter is obtained.

[0180] In some embodiments, if the message "Container killed on request.Exit code is 143" appears, the following adjustments may be suggested: increase `spark.executor.memory`, decrease `spark.executor.cores`, reduce unnecessary cache operations, avoid broadcasting large amounts of data, minimize shuffle operators, or optimize program logic / underlying data. Here, `spark.executor.memory` represents the amount of memory used by each executor process, and `spark.executor.cores` represents the number of cores used by each executor.

[0181] In some embodiments, if the EXCEEDED_RESOURCE_TYPE_MEM message appears, the following adjustments can be made: Since the physical machine memory usage exceeds the threshold, it cannot be well avoided at the task level. You can try to reduce the executor's core and memory proportionally (ensuring that the memory allocated to each core remains unchanged), try not to request large amounts of memory, and avoid the physical machine memory usage rate reaching the threshold.

[0182] In some embodiments, if the disk is determined to be busy, adjustments can be made as follows: select a time when the disk is not busy to execute or enable the RSS mechanism based on disk monitoring.

[0183] Figure 8 It is based on Figures 1 to 7 This diagram illustrates a task optimization process. Figure 8 The task optimization process shown can be executed by the diagnostic optimization device 8002, and may include the following steps S802 to S824.

[0184] Step S802: Obtain the newly uploaded script 8002 or the script 8006 previously uploaded to the periodic inspection platform, and parse the script using the SQL parser.

[0185] Step S804: Perform a validity check on the newly uploaded script. If the newly uploaded script has syntax errors and the validity check fails, the process ends directly (proceeding to step S824) and a syntax error message is displayed. If the validity check passes, proceed to step S806.

[0186] Step S806: Scan the keywords in the script and match them with the keyword library 8008 that is not recommended for use.

[0187] Step S808: Determine whether an undesirable keyword is matched. If an undesirable keyword is matched, the corresponding optimization solution 8010 is suggested. For details on the implementation method, please refer to [link / reference]. Figure 2 If no unrecommended keywords are matched, proceed to step S810.

[0188] Step S810: Obtain the most recent execution log file of the task in Spark jobhistory 8012.

[0189] Step S812: Determine if the task execution log has been obtained. If the task execution log has not been obtained, perform a trial run through the scheduling platform (step S814). During the trial run, Spark execution log information can be obtained through the Spark listener. If the log is obtained, proceed to step S816.

[0190] Step S816: After obtaining the execution logs of the Spark task, scan and parse the log files.

[0191] Step S818: Obtain task execution information from the log file. For a detailed implementation, refer to step S110.

[0192] Step S820: After obtaining the task execution information, the system begins matching the rules in the optimization rule base 8014. For a detailed implementation, please refer to step S112.

[0193] Step S822: Determine whether the scan log matches the corresponding rule. If the corresponding rule is matched, provide the corresponding specific optimization solution 8010, the specific implementation of which can be found in step S114. If the corresponding rule is not matched, proceed to step S824.

[0194] Step S824, process ends.

[0195] Figure 9 This is a block diagram illustrating a task processing apparatus according to an exemplary embodiment. Figure 9 The device shown can be applied, for example, to a task execution platform.

[0196] refer to Figure 9 The apparatus 90 provided in this embodiment may include a first acquisition module 902, a first matching module 904, a first optimization module 906, a second acquisition module 908, a log scanning module 910, a second matching module 912, and a second optimization module 914.

[0197] The first acquisition module 902 can be used to acquire the script to be processed.

[0198] The first matching module 904 can be used to parse the script to be processed, match the script to be processed with the keyword library, and obtain the keyword matching results.

[0199] The first optimization module 906 can be used to obtain first prompt information based on the keyword matching results, and then perform first optimization processing on the script to be processed based on the first prompt information.

[0200] The first optimization module 906 can be used to run the first script on the task execution platform if the keyword matching result does not match the command to be optimized.

[0201] The second acquisition module 908 can be used to acquire log information of the task executing the script to be processed.

[0202] The log scanning module 910 can be used to scan and parse log information to obtain the task execution information corresponding to the script to be processed.

[0203] The second matching module 912 can be used to determine the indicators corresponding to the task execution information as indicators to be optimized according to the optimization rules in the optimization rule base.

[0204] The second optimization module 914 can be used to obtain second prompt information based on the indicator to be optimized, so as to perform second optimization processing on the task execution parameters based on the second prompt information.

[0205] Figure 10 This is a block diagram illustrating another task processing apparatus according to an exemplary embodiment. Figure 10 The device shown can be applied, for example, to a task execution platform.

[0206] refer to Figure 10 The apparatus 100 provided in this embodiment may include a first acquisition module 1002, a legality verification module 1003, a first matching module 1004, a trial operation module 1005, a first optimization module 1006, a second acquisition module 1008, a log scanning module 1010, a second matching module 1012, and a second optimization module 1014.

[0207] The first acquisition module 1002 can be used to acquire scripts to be processed.

[0208] The script to be processed may include the first script uploaded to the task execution platform for the first time.

[0209] The script to be processed may also include a second script that has already been run on the task execution platform.

[0210] The legality verification module 1003 can be used to parse the first script and perform legality verification on the first script.

[0211] The first matching module 1004 can be used to parse the script to be processed, match the script to be processed with the keyword library, and obtain the keyword matching results.

[0212] The first matching module 1004 can also be used to match the first script with the keyword database to obtain the keyword matching result if the validity check result of the first script is passed.

[0213] The trial run module 1005 can be used to run the first script on the task execution platform if the keyword matching result does not match the command to be optimized.

[0214] The commands to be optimized may include at least one of the following: global sorting command, data deduplication command, first data filtering command, multi-table merging command without predicate pushdown, partition creation command without specified partition, and selection command without corresponding column pruning operation.

[0215] The first optimization module 1006 can be used to obtain first prompt information based on the keyword matching results, and to perform first optimization processing on the script to be processed based on the first prompt information.

[0216] The first optimization module 1006 can also be used to obtain first prompt information if the keyword matching result matches the command to be optimized. The first prompt information includes information on the optimization scheme for prompting the command to be optimized.

[0217] The optimization solutions for the global sort command can be to replace it with a local sort command; the optimization solutions for the data deduplication command can be to replace it with a data grouping command; the optimization solutions for the first data filtering command can be to replace it with a second data filtering command; the optimization solutions for the multi-table merge command without predicate pushdown can be to use predicate pushdown; the optimization solutions for the partition creation command without specifying a partition can be to specify a partition; and the optimization solutions for the selection command without performing corresponding column clipping operations can be to use column clipping.

[0218] The first optimization module 1006 can also be used to obtain a first prompt message if the keyword matching result is that no command to be optimized is matched. The first prompt message includes information for prompting the acquisition of platform logs.

[0219] The second acquisition module 1008 can be used to acquire log information of the task executing the script to be processed.

[0220] The log information for tasks executing pending scripts may include the log information for the task that runs the first script in trial mode.

[0221] The log information for the task executing the script to be processed may include the log information for the task executing the second script.

[0222] The second acquisition module 1008 can also be used to obtain log information of the task of trial running the second script based on the information used to prompt the acquisition of platform logs.

[0223] The log scanning module 1010 can be used to scan and parse log information to obtain the task execution information corresponding to the script to be processed.

[0224] Task execution information may include the size of the input table.

[0225] Task execution information may also include stage execution metrics, which include the amount of data read and the start time.

[0226] Task execution information also includes execution exception information.

[0227] The second matching module 1012 can be used to determine the indicators corresponding to the task execution information as indicators to be optimized according to the optimization rules in the optimization rule base.

[0228] Optimization rules can include table size thresholds.

[0229] The optimization rules may also include stage execution indicator thresholds, which include data read volume thresholds and startup time thresholds.

[0230] Optimization rules can also include preset exception information.

[0231] The second matching module 1012 can also be used to determine whether the size of the input table is less than the table size threshold based on the size information of the input table; if the size of the input table is less than the table size threshold, it is determined that there is a small table in the task execution, and the size of the input table is an indicator to be optimized.

[0232] The second matching module 1012 can also be used to determine the corresponding execution indicators as indicators to be optimized based on the amount of data read and the corresponding data read threshold; and to determine the corresponding execution indicators as indicators to be optimized based on the startup time and the corresponding startup time threshold.

[0233] The second matching module 1012 can also be used to match execution exception information with preset exception information, and determine the corresponding indicator as the indicator to be optimized based on the matching result.

[0234] The second optimization module 1014 can be used to obtain second prompt information based on the indicator to be optimized, so as to perform second optimization processing on the task execution parameters based on the second prompt information.

[0235] The second prompt message may include information prompting you to add broadcast merging parameters.

[0236] The second prompt message may also include information prompting the addition of the first optimization parameter and information prompting the adjustment of the task execution time.

[0237] The second prompt message may also include information suggesting adjustments to the second optimization parameters.

[0238] The second optimization module 1014 can also be used to obtain information prompting the addition of broadcast merging parameters if the size of the input table is determined to be the metric to be optimized.

[0239] The second optimization module 1014 can also be used to obtain information prompting to add the first optimization parameter if the execution indicator corresponding to the amount of data read is determined to be an indicator to be optimized; and to obtain information prompting to adjust the task execution time if the execution indicator corresponding to the start time is determined to be an execution indicator to be optimized.

[0240] The second optimization module 1014 can also be used to obtain information on adjusting the second optimization parameters if it is determined that the execution indicator corresponding to the execution exception information is an indicator to be optimized.

[0241] The specific implementation of each module in the apparatus provided in this embodiment can be referred to the content of the above method, and will not be repeated here.

[0242] Figure 11 A schematic diagram of the structure of an electronic device according to an embodiment of this disclosure is shown. It should be noted that... Figure 11 The devices shown are merely examples of computer systems and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0243] like Figure 11 As shown, device 1100 includes a central processing unit (CPU) 1101, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1102 or a program loaded from storage section 1108 into random access memory (RAM) 1103. The RAM 1103 also stores various programs and data required for the operation of device 1100. CPU 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. Input / output (I / O) interface 1105 is also connected to bus 1104.

[0244] The following components are connected to I / O interface 1105: an input section 1106 including a keyboard, mouse, etc.; an output section 1107 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1108 including a hard disk, etc.; and a communication section 1109 including a network interface card such as a LAN card, modem, etc. The communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to I / O interface 1105 as needed. Removable media 1110, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1110 as needed so that computer programs read from it can be installed into storage section 1108 as needed.

[0245] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1109, and / or installed from removable medium 1110. When the computer program is executed by central processing unit (CPU) 1101, it performs the functions defined above in the system of this disclosure.

[0246] It should be noted that the computer-readable medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0247] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0248] The modules described in the embodiments of this disclosure can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor may be described as including a first acquisition module, a first matching module, a first optimization module, a second acquisition module, a log scanning module, a second matching module, and a second optimization module. The names of these modules do not necessarily limit the module itself; for example, the first acquisition module may also be described as a "module for acquiring scripts to be processed."

[0249] In another aspect, this disclosure also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs that, when executed by the device, cause the device to perform:

[0250] The process involves: acquiring the script to be processed; parsing the script and matching it against a keyword database to obtain keyword matching results; obtaining a first prompt based on the keyword matching results, and performing a first optimization on the script based on the first prompt; acquiring the log information of the task executing the script; scanning and parsing the log information to obtain the task execution information corresponding to the script; determining the metrics corresponding to the task execution information as the metrics to be optimized according to the optimization rules in the optimization rule database; obtaining a second prompt based on the metrics to be optimized, and performing a second optimization on the task execution parameters based on the second prompt.

[0251] Exemplary embodiments of this disclosure have been specifically shown and described above. It should be understood that this disclosure is not limited to the detailed structures, arrangements, or implementations described herein; rather, this disclosure is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. A task processing method, characterized in that, include: Get the script to be processed; The script to be processed is parsed, and the script to be processed is matched with the keyword database to obtain the keyword matching results; If the keyword matching result matches the command to be optimized, then a first prompt message is obtained, and the script to be processed is optimized according to the first prompt message. The first prompt message includes information for prompting the optimization scheme of the command to be optimized. The command to be optimized includes at least one of the following: global sorting command, data deduplication command, first data filtering command, multi-table merging command without predicate pushdown, partition creation command without specified partition, and selection command without corresponding column pruning operation. Obtain the log information of the task executing the script to be processed; The log information is scanned and parsed to obtain the task execution information corresponding to the script to be processed. The task execution information includes at least one of the following: the size information of the input table, the stage execution indicator value, and the execution exception information. The stage execution indicator value includes the amount of data read and the startup time. The indicators corresponding to the task execution information are determined as indicators to be optimized according to the optimization rules in the optimization rule base. The optimization rules include at least one of the following: table size threshold, stage execution indicator threshold, and preset abnormal information. The stage execution indicator threshold includes a data reading amount threshold and a startup time threshold. The second prompt information is obtained based on the indicator to be optimized, and the task execution parameters are then optimized based on the second prompt information. The second prompt information includes at least one of the following: information prompting the addition of broadcast merging parameters, information prompting the addition of first optimization parameters, information prompting the adjustment of task execution time, and information prompting the adjustment of second optimization parameters.

2. The method according to claim 1, characterized in that, The script to be processed includes the first script uploaded to the task execution platform for the first time; The log information of the task executing the script to be processed includes the log information of the task that runs the first script in trial mode; The method includes: If the keyword matching result is that the command to be optimized is not matched, then the first script is run on the task execution platform. Obtain log information of the task executing the script to be processed, including: Obtain the log information of the task that runs the first script.

3. The method according to claim 1, characterized in that, The optimization scheme for the global sorting command is to replace it with a local sorting command; the optimization scheme for the data deduplication command is to replace it with a data grouping command; the optimization scheme for the first data filtering command is to replace it with a second data filtering command; the optimization scheme for the multi-table merge command without predicate pushdown is to use predicate pushdown; the optimization scheme for the partition creation command without specifying a partition is to specify a partition; and the optimization scheme for the selection command without performing corresponding column pruning operation is to use column pruning.

4. The method according to claim 2, characterized in that, The script to be processed is parsed, and the script is matched with a keyword database to obtain keyword matching results, including: The first script is parsed and its validity is verified. If the validity check of the first script passes, then the first script is matched with the keyword database to obtain the keyword matching result.

5. The method according to claim 1, characterized in that, The task execution information includes the size information of the input table; The optimization rules include the table size threshold; The metrics corresponding to the task execution information are determined as metrics to be optimized according to the optimization rules in the optimization rule base, including: Determine whether the size of the input table is less than the table size threshold based on the size information of the input table; If the size of the input table is less than the table size threshold, it is determined that there is a small table in the task execution, and the size of the input table is the indicator to be optimized. The second prompt message includes information about adding broadcast merging parameters as prompted; The second prompt information is obtained based on the indicator to be optimized, including: If the size of the input table is determined to be the target metric, then the information prompting the addition of broadcast merging parameters is obtained.

6. The method according to claim 1, characterized in that, The task execution information includes the stage execution indicator values; The optimization rules also include the stage execution indicator thresholds; According to the optimization rules in the optimization rule base, the indicators corresponding to the task execution information are determined as indicators to be optimized, and the following are also included: Based on the amount of data read and the corresponding data read threshold, the corresponding execution metric is determined as the metric to be optimized. Based on the startup time and the corresponding startup time threshold, the corresponding execution metric is determined as the metric to be optimized. The second prompt information includes the information prompting the addition of the first optimization parameter and the information prompting the adjustment of the task execution time; The second prompt information is obtained based on the indicator to be optimized, including: If the execution metric corresponding to the amount of data read is determined to be the metric to be optimized, then the corresponding prompt to add the first optimization parameter is obtained; If the execution metric corresponding to the startup time is determined to be the metric to be optimized, then the information prompting to adjust the task execution time is obtained.

7. The method according to claim 1, characterized in that, The task execution information includes the execution exception information; The optimization rules also include the preset anomaly information; According to the optimization rules in the optimization rule base, the indicators corresponding to the task execution information are determined as indicators to be optimized, and the following are also included: The execution exception information is matched with preset exception information, and the corresponding indicator is determined as the indicator to be optimized based on the matching result; The second prompt message includes information prompting the adjustment of the second optimization parameter; The second prompt information is obtained based on the indicator to be optimized, including: If the execution metric corresponding to the execution anomaly information is determined to be the metric to be optimized, then the corresponding prompt to adjust the second optimization parameter is obtained.

8. The method according to claim 1, characterized in that, The script to be processed includes a second script that has already been run on the task execution platform; The log information of the task executing the script to be processed includes the log information of the task executing the second script; The method further includes: If the keyword matching result is that the command to be optimized is not matched, then the first prompt information is obtained, and the first prompt information includes information for prompting the acquisition of platform logs; Obtain log information of the task executing the script to be processed, including: The log information of the task for trial running the second script is obtained based on the information provided for prompting the acquisition of platform logs.

9. A task processing device, characterized in that, include: The first acquisition module is used to acquire the script to be processed. The first matching module is used to parse the script to be processed, match the script to be processed with the keyword library, and obtain the keyword matching result; The first optimization module is used to obtain first prompt information if the keyword matching result matches the command to be optimized, so as to perform first optimization processing on the script to be processed according to the first prompt information. The first prompt information includes information for prompting the optimization scheme of the command to be optimized. The command to be optimized includes at least one of the following: global sorting command, data deduplication command, first data filtering command, multi-table merging command without predicate pushdown, partition creation command without specified partition, and selection command without corresponding column pruning operation. The second acquisition module is used to acquire log information of the task executing the script to be processed; The log scanning module is used to scan and parse the log information to obtain the task execution information corresponding to the script to be processed. The task execution information includes at least one of the following: the size information of the input table, the stage execution indicator value, and the execution exception information. The stage execution indicator value includes the amount of data read and the start time. The second matching module is used to determine the indicator corresponding to the task execution information as the indicator to be optimized according to the optimization rules in the optimization rule base. The optimization rules include at least one of the following: table size threshold, stage execution indicator threshold, and preset abnormal information. The stage execution indicator threshold includes a data reading amount threshold and a startup time threshold. The second optimization module is used to obtain second prompt information based on the indicator to be optimized, so as to perform a second optimization process on the task execution parameters based on the second prompt information. The second prompt information includes at least one of the following: information prompting the addition of broadcast merging parameters, information prompting the addition of first optimization parameters, information prompting the adjustment of task execution time, and information prompting the adjustment of second optimization parameters.

10. An electronic device, comprising: A memory, a processor, and executable instructions stored in the memory and executable in the processor, characterized in that the processor, when executing the executable instructions, implements the method as described in any one of claims 1-8.

11. A computer-readable storage medium having computer-executable instructions stored thereon, characterized in that, When the executable instructions are executed by the processor, they implement the method as described in any one of claims 1-8.

Images