Data processing script execution method and device

Abstract

The present application provides a method and device for executing a data processing script, the method comprising: parsing the script to be executed according to the target grammatical rules (the grammatical rules of the computer language in which the script to be executed is written) to obtain the logical structure of the script to be executed (including multiple execution units and the connection relationship between each execution unit); splitting the script to be executed into multiple sub-scripts according to the logical structure of the script to be executed, and the input data name and output data name of the script to be executed; determining the execution logic between every two sub-scripts according to the logical structure of the script to be executed; the execution logic between every two sub-scripts is parallel execution or sequential execution; executing multiple sub-scripts according to the execution logic to obtain the output data of the script to be executed. This scheme splits the script to be executed into multiple sub-scripts, and when executing these sub-scripts, multiple sub-scripts in which the execution logic is parallel execution can be executed simultaneously, thereby improving the execution efficiency of the script to be executed.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method and device for executing a data processing script. Background Art

[0002] In the present invention, data processing refers to the process of extracting and / or deriving data with certain meaning and value from a large amount of data stored in a database, which may be disorganized and difficult to understand. In order to realize the processing of large quantities of data, scripts are often needed. Scripts, also known as computer scripts, refer to computer executable files written manually in a certain format and syntax. Writing and executing data processing scripts on a computer can allow the computer to automatically complete specific data processing tasks.

[0003] Currently in the field of data processing, the script execution method is generally to directly load the script into the server for execution after obtaining the manually written script. However, for some highly complex data processing tasks, when the manually written script is directly executed, its execution efficiency is low. Summary of the invention

[0004] In view of the above problems in the prior art, the present invention provides a method and device for executing a data processing script to improve the execution efficiency of the data processing script.

[0005] A first aspect of the present invention provides a method for executing a data processing script, comprising:

[0006] Get the script to be executed;

[0007] Parsing the script to be executed according to the target grammar rule to obtain the logical structure of the script to be executed; wherein the target grammar rule refers to the grammar rule of the computer language used to write the script to be executed; the logical structure of the script to be executed includes multiple execution units of the script to be executed and the connection relationship between each of the execution units;

[0008] According to the logical structure of the script to be executed, and the input data name and output data name of the script to be executed, the script to be executed is split into multiple sub-scripts; wherein the input data name and output data name of each sub-script are determined according to the input data name and output data name of the script to be executed;

[0009] Determine the execution logic between each two sub-scripts according to the logic structure of the script to be executed; wherein the execution logic between each two sub-scripts is parallel execution or sequential execution;

[0010] According to the execution logic between every two of the sub-scripts, the multiple sub-scripts are executed to obtain the output data of the script to be executed.

[0011] Optionally, before parsing the script to be executed according to the target grammar rules to obtain the logical structure of the script to be executed, the method further includes:

[0012] Determine whether there is an optimized script matching the script to be executed in the cache;

[0013] If so, read the optimized script matched by the script to be executed, and execute each sub-script in the optimized script matched by the script to be executed according to the execution logic contained in the optimized script matched by the script to be executed;

[0014] If not, executing the step of parsing the script to be executed according to the target grammar rules to obtain the logical structure of the script to be executed;

[0015] Wherein, after determining the execution logic between every two sub-scripts according to the logical architecture of the script to be executed, the method further includes:

[0016] The execution logics of each sub-script and between every two sub-scripts are summarized into an optimized script matched by the script to be executed, and the optimized script matched by the script to be executed is stored in a cache area.

[0017] Optionally, executing the multiple sub-scripts according to the execution logic between every two of the sub-scripts to obtain output data of the script to be executed includes:

[0018] Execute the multiple sub-scripts according to the execution logic between every two of the sub-scripts to obtain output data of each of the sub-scripts;

[0019] Execute a pre-generated summary script to summarize the output data of each of the sub-scripts into the output data of the script to be executed; wherein the summary script is generated according to the execution logic between each of the sub-scripts and every two of the sub-scripts.

[0020] Optionally, executing the multiple sub-scripts according to the execution logic between every two of the sub-scripts to obtain output data of each of the sub-scripts includes:

[0021] Configuring corresponding runtime and runtime resources for each of the sub-scripts;

[0022] According to the execution logic between every two sub-scripts, each sub-script is executed using the runtime and runtime resources corresponding to each sub-script to obtain output data of each sub-script.

[0023] Optionally, parsing the script to be executed according to the target grammar rule to obtain the logical structure of the script to be executed includes:

[0024] Identify and obtain multiple grammatical keywords in the script to be executed; wherein the grammatical keywords are specified by target grammatical rules;

[0025] The logical structure of the script to be executed is determined according to the grammatical keywords and the target grammatical rules.

[0026] A second aspect of the present invention provides a data processing script execution device, comprising:

[0027] An acquisition unit, used to acquire the script to be executed;

[0028] A parsing unit, configured to parse the script to be executed according to a target grammatical rule to obtain a logical structure of the script to be executed; wherein the target grammatical rule refers to a grammatical rule of a computer language used to write the script to be executed; the logical structure of the script to be executed includes a plurality of execution units of the script to be executed, and a connection relationship between each of the execution units;

[0029] A splitting unit, used for splitting the script to be executed into a plurality of sub-scripts according to the logical structure of the script to be executed, and the input data name and output data name of the script to be executed; wherein the input data name and output data name of each sub-script are determined according to the input data name and output data name of the script to be executed;

[0030] A determination unit, configured to determine the execution logic between each two of the sub-scripts according to the logic structure of the script to be executed; wherein the execution logic between each two of the sub-scripts is parallel execution or sequential execution;

[0031] The execution unit is used to execute the multiple sub-scripts according to the execution logic between every two sub-scripts to obtain the output data of the script to be executed.

[0032] Optionally, the execution device further includes:

[0033] A determination unit, used to determine whether there is an optimized script matching the script to be executed in the cache;

[0034] If so, read the optimized script matched by the script to be executed, and execute each sub-script in the optimized script matched by the script to be executed according to the execution logic contained in the optimized script matched by the script to be executed;

[0035] If not, trigger the parsing unit to parse the script to be executed according to the target grammar rules to obtain the logical structure of the script to be executed;

[0036] The cache unit is used to summarize the execution logic of each sub-script and between every two sub-scripts into an optimized script matched by the script to be executed, and store the optimized script matched by the script to be executed in a cache area.

[0037] Optionally, the execution unit executes the multiple sub-scripts according to the execution logic between every two sub-scripts to obtain the output data of the script to be executed, specifically for:

[0038] Execute the multiple sub-scripts according to the execution logic between every two of the sub-scripts to obtain output data of each of the sub-scripts;

[0039] Execute a pre-generated summary script to summarize the output data of each of the sub-scripts into the output data of the script to be executed; wherein the summary script is generated according to the execution logic between each of the sub-scripts and every two of the sub-scripts.

[0040] Optionally, the execution unit executes the multiple sub-scripts according to the execution logic between every two sub-scripts to obtain the output data of each sub-script, specifically for:

[0041] Configuring corresponding runtime and runtime resources for each of the sub-scripts;

[0042] According to the execution logic between every two sub-scripts, each sub-script is executed using the runtime and runtime resources corresponding to each sub-script to obtain output data of each sub-script.

[0043] Optionally, when the parsing unit parses the script to be executed according to the target grammar rule to obtain the logical structure of the script to be executed, it is specifically used to:

[0044] Identify and obtain multiple grammatical keywords in the script to be executed; wherein the grammatical keywords are specified by target grammatical rules;

[0045] The logical structure of the script to be executed is determined according to the grammatical keywords and the target grammatical rules.

[0046] The present invention provides a method and device for executing a data processing script, the method comprising: obtaining a script to be executed; parsing the script to be executed according to a target grammatical rule to obtain a logical structure of the script to be executed; wherein the target grammatical rule refers to a grammatical rule of a computer language used to write the script to be executed; the logical structure of the script to be executed includes multiple execution units of the script to be executed and a connection relationship between each execution unit; according to the logical structure of the script to be executed and the input data name and output data name of the script to be executed, the script to be executed is split into multiple sub-scripts; wherein the input data name and output data name of each sub-script are determined according to the input data name and output data name of the script to be executed; according to the logical structure of the script to be executed, the execution logic between every two sub-scripts is determined; wherein the execution logic between every two sub-scripts is parallel execution or sequential execution; according to the execution logic between every two sub-scripts, multiple sub-scripts are executed to obtain the output data of the script to be executed. This scheme splits the script to be executed into multiple sub-scripts, and when executing these sub-scripts, multiple sub-scripts in which the execution logic is parallel execution can be executed simultaneously, thereby improving the execution efficiency of the script to be executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0048] Figure 1 A flowchart of a method for executing a data processing script provided in an embodiment of the present application;

[0049] Figure 2 A schematic diagram of a logical architecture obtained by parsing a script to be executed provided in an embodiment of the present application;

[0050] Figure 3 A schematic diagram of multiple sub-scripts after splitting provided in an embodiment of the present application;

[0051] Figure 4 A schematic diagram of the structure of a data processing script execution device provided in an embodiment of the present application;

[0052] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] Data processing refers to the process of extracting and deriving valuable and meaningful data for certain specific users from a large amount of data that may be disorganized and difficult to understand.

[0055] Script: A computer executable file written in a certain format using a specific computer language.

[0056] Parallelism: Multiple executable programs or files run simultaneously within the same period of time.

[0057] Dependency: The execution of this node requires the result of the previous node before it can start executing.

[0058] The existing data processing method is to directly load the manually written data processing script (i.e., the script to be executed) into the server for execution. The execution speed of the data processing script is not only related to factors such as the performance of the database server and the amount of data processed, but also closely related to the quality and rationality of the data processing script. First, with the increase of existing graphical, interface, and visualization tools for generating data processing scripts, the development threshold of data processing has been lowered. However, due to the limitation of the user's level, manually written scripts, especially scripts written for data processing tasks with high complexity, often have problems of poor quality and rationality, resulting in low execution efficiency when the script is directly executed, and the inability to quickly complete the processing of large quantities of data.

[0059] Second, the large talent gap has led to data processing practitioners lacking effective business and technical processing knowledge to optimize data processing scripts.

[0060] Third, with the vigorous development of big data applications, the complexity of data processing continues to rise. The practice of manual script tuning is difficult to standardize and has great limitations. An unsustainable solution is to increase the concurrency of data processing to improve the task processing capability, but this also seriously reduces the processing efficiency of data processing scripts.

[0061] At present, in related fields, data analysis experts can only manually analyze data processing scripts, and manually adjust the processing logic of data scripts after fully understanding the business processing logic of the scripts. In other words, they can manually optimize and adjust the scripts to be executed. There is currently no solution to automatically optimize the scripts to be executed.

[0062] In view of the above problems of the prior art, the present invention provides a method and device for executing a data processing script, which improves the execution efficiency of a data processing script with multiple steps and a long execution time by intelligently optimizing the script. The method and device can determine the dependency between the data processing steps, and parse and decompose the data processing script into concurrent sub-scripts, intelligently schedule and configure multiple sub-scripts, execute multiple sub-scripts that can be executed in parallel at the same time, and put each sub-script into a database for execution in turn for sub-scripts that need to be executed sequentially to obtain the final result.

[0063] The present invention is to intelligently optimize a data processing script with multiple steps and a long execution time in order to improve its execution efficiency. The method and device can determine the dependency between the data processing steps, and parse and decompose the data processing script into sub-scripts that can be executed concurrently, intelligently schedule and configure multiple sub-scripts, and execute multiple sub-scripts that can be executed in parallel at the same time. For sub-scripts that need to be executed sequentially, each sub-script is put into a database for execution in turn to obtain the final result. The present invention automatically parses the content of the data processing script, automatically parses out the statements that can be executed concurrently, and encapsulates the statements into various sub-scripts for intelligent scheduling and execution, and encapsulates the final result, thereby automatically and efficiently improving the execution efficiency of the data processing script, and is suitable for the optimization of data processing scripts with multiple steps and a long execution time.

[0064] Please refer to Figure 1 The execution method of the data processing script provided in this application may include the following steps:

[0065] S101, obtaining a script to be executed.

[0066] Developers can write scripts on the terminal device to meet current data processing requirements, and then upload the written scripts to the script server. The script to be executed in step S101 is a script that has been uploaded to the script server and has not yet been executed.

[0067] S102: Parse the script to be executed according to the target grammar rules to obtain the logical structure of the script to be executed.

[0068] Among them, the target grammatical rules refer to the grammatical rules of the computer language used to write the script to be executed (i.e., the implementation language of the script to be executed); the logical architecture of the script to be executed includes multiple execution units of the script to be executed, and the connection relationship between each execution unit.

[0069] Optionally, step S102, i.e. parsing the script to be executed according to the target grammar rule to obtain the logical structure of the script to be executed, may include:

[0070] A plurality of grammatical keywords are identified in the script to be executed; wherein the grammatical keywords are specified by the target grammatical rules;

[0071] Determine the logical structure of the script to be executed based on the grammatical keywords and target grammar rules.

[0072] The script can be written in any computer language, and for a specific computer language, its grammatical rules are usually fixed. Therefore, the grammatical rules of some common languages ​​(referring to computer languages ​​frequently used for writing scripts) can be set in advance in the device used to execute the present method.

[0073] In this way, when executing step S102, the corresponding target grammar rules can be read according to the implementation language of the script to be executed, and then parsed according to the target grammar rules.

[0074] The execution units of the script to be executed include data processing, data merging and data joining. Among them, data processing refers to conditional filtering, field selection, deduplication, cleaning, clustering, code conversion, formula calculation and other operations on a single data table; data merging refers to associating the results of two or more subqueries through the same primary key, mainly inner association, left association, right association, and full association; data joining refers to vertically joining two or more subqueries together when the field structures are the same. Each execution unit is composed of one or more statements in the script. For a specific computer language, the statements in each execution unit will contain specific syntax keywords defined by the computer language. Therefore, by identifying the syntax keywords of each statement in the script to be executed, it can be determined which execution unit of the script to be executed each statement belongs to.

[0075] Take the PostgreSQL language, which is most commonly used in data processing, as an example. The statements that make up the data processing unit will contain any one or more of the syntax keywords such as where, group, case, having, like, in, null, etc.; the statements that make up the data merging unit will contain any one or more of the syntax keywords such as innerjoin, leftjoin, rightjoin, outerjoin, etc.; the statements that make up the data union unit will contain the union keyword. Therefore, the script to be executed can be divided into multiple execution units by identifying the syntax keywords corresponding to the script to be executed.

[0076] After identifying the execution units, it is necessary to determine the connection relationship between the execution units. The connection relationship between the execution units can also be called the dependency relationship between the execution units. For a specific computer language, the dependency relationship between the execution units is generally expressed using a specific grammatical structure. Still using the PostgreSQL language, assume that there are two data processing units, denoted as data processing 1 and data processing 2, and a data merging unit, denoted as data merging 1. If data merging 1 needs to merge the output of data processing 1 with the output of data processing 2, that is, data merging 1 depends on data processing 1 and data processing 2, then the statements contained in data merging 1 in the script can be:

[0077] Result 1 join on Result 2.

[0078] Result 1 represents the output of data processing 1, and result 2 represents the output of data processing 2. Therefore, by identifying the above statements, the connection relationship (dependency relationship) between data merging 1, data processing 1 and data processing 2 can be determined.

[0079] Assume that there is a script to be executed that needs to perform functions, that is, to process tables A, B, C, D and E separately, then merge the processing results of tables A and B, merge the processing results of tables C, D and E, and then perform data union on the two merged results. Through step S102, the logical structure of the script to be executed can be parsed, such as Figure 2 As shown. Data processing 1 to data processing 5 represent execution units for processing tables A, B, C, D and E respectively, data merging 1 is used to merge the processing results of tables A and B, data merging 2 is used to merge the processing results of tables C, D and E, data union 1 is used to combine the results of data union 1 and data union 2, and data processing 6 is used for the result of data union 1.

[0080] S103: split the script to be executed into multiple sub-scripts according to the logical structure of the script to be executed and the input data name and output data name of the script to be executed.

[0081] The input data name and output data name of each sub-script are determined according to the input data name and output data name of the script to be executed.

[0082] A complete script needs to include input data names (used to refer to the input of the script), output data names (used to refer to the output of the script) and statements for execution. Figure 2 For example, Figure 2The script to be executed corresponding to the logical architecture shown has input data names of A, B, C, D and E tables respectively, and the output data name is final result 1. It should be noted that before the script to be executed is actually executed, the above-mentioned A, B, C, D and E tables, as well as the final result 1, do not contain substantive content, but are merely names used to refer to the input and output of the script to be executed.

[0083] In order to execute step S103, the input data name and input parameter name of the script to be executed may be marked and stored in the input configuration table, and the output data name of the script to be executed may be marked and stored in the output configuration table. Figure 2 Taking the logical architecture shown as an example, the A, B, C, D and E table annotations can be stored in the input configuration table. If specific parameters are required when the script to be executed is executed, these parameters are also stored in the input configuration table, and the final result 1 annotation is stored in the output configuration table.

[0084] After completing the above annotation storage process, you can Figure 2 The logical architecture shown is split into sub-scripts as follows:

[0085] For each execution unit, the first aspect is to determine whether the data used when the execution unit is executed is included in the input data name of the script to be executed recorded in the input configuration table, and the second aspect is to determine whether the output of the execution unit is included in the output data name of the script to be executed recorded in the output configuration table.

[0086] If the judgment result of the first aspect is yes, the input data name recorded in the input configuration table is determined as the input of the execution unit. Otherwise, if the judgment result of the first aspect is no, a temporary input is configured for this execution unit (that is, a temporary table is generated as input).

[0087] If the judgment result of the second aspect is yes, the output data name recorded in the output configuration table is determined as the output of the execution unit. Otherwise, if the judgment result of the second aspect is no, a temporary output is configured for this execution unit (that is, a temporary table is generated as the output).

[0088] Then, for Figure 2 In the logical architecture shown, for every two execution units with a dependency relationship, the temporary inputs and temporary outputs of the two execution units are unified, that is, the temporary output of the previous execution unit and the temporary input of the next execution unit are represented by the same data name.

[0089] This process is equivalent to generating a temporary table for the previous sub-script with a dependency relationship, the next script performs data processing based on the temporary table of the previous sub-script, and finally integrates the result sets of each sub-script.

[0090] Finally, for each execution unit, the input, output and execution unit of the execution unit are divided into a sub-script.

[0091] by Figure 2 As an example of the logical architecture shown in FIG. 1 , through step S103, Figure 2 The scripts to be executed corresponding to the logical architecture shown are divided into Figure 3 The nine sub-scripts shown are recorded as Script 1 to Script 9, respectively.

[0092] In the specific splitting process, combined with Figure 2 It can be seen that the inputs of data processing 1 to data processing 5 are included in the input data name of the entire script to be executed, so the A, B, C, D and E tables are directly determined as data processing 1 to data processing 5. Similarly, the final result 1 can be determined as the output of data processing 6.

[0093] Afterwards, for data processing 1 to data processing 5, the outputs of these execution units do not belong to the final result 1, so 5 temporary outputs are configured for data processing 1 to data processing 5 respectively, and the inputs of data merger 1 and data merger 2 do not belong to the input data names of the script to be executed (i.e., A, B, C, D and E tables), therefore, two temporary inputs are configured for data merger 1 and three temporary inputs are configured for data merger 2 respectively.

[0094] Depend on Figure 2 From the logical architecture shown, we can see that there is a dependency relationship between data merging 1 and data processing 1 and 2, so we can determine that the output of data processing 1 and 2 should be equal to the input of data merging 1. Therefore, we configure the same name for the temporary output of data processing 1 and 2 and the temporary input of data merging 1, that is, we configure temporary result A1 as the temporary output of data processing 1, and temporary result B1 as the temporary output of data processing 2. At the same time, temporary result A1 and temporary result B1 are used as temporary inputs of data merging 1. Therefore, table A, data processing 1 and temporary result A1 constitute script 1; table B, data processing 2 and temporary result B1 constitute script 2.

[0095] The splitting methods of other sub-scripts are similar. For the specific process, please refer to Figure 2 and Figure 3 Understood, no further details here.

[0096] S104: Determine the execution logic between every two sub-scripts according to the logic structure of the script to be executed.

[0097] The execution logic between every two sub-scripts is parallel execution or sequential execution.

[0098] Generally, in step S104, the execution logic between multiple sub-scripts before data union or data merging can be determined as parallel execution, and the execution logic between the sub-script corresponding to the data merging (and data union) and the sub-scripts before and after it can be determined as sequential execution.

[0099] Combination Figure 2 and Figure 3 , it can be seen that scripts 1 to 5 are located before data merge 1 and data merge 2, so the execution logic of scripts 1 to 5 is parallel execution, scripts 6 and 7 are located before data union 1, the execution logic between scripts 6 and 7 is parallel execution, and the execution logic between the remaining sub-scripts is sequential execution. Finally, Figure 3 The execution logic of each sub-script shown is as follows:

[0100] Scripts 1 to 5 are executed in parallel, scripts 6 and 7 are executed in parallel, scripts 1, 2 and 6 are executed sequentially, that is, scripts 1 and 2 are executed first, and then script 6 is executed, scripts 3 to 5 and 7 are executed sequentially, that is, scripts 3 to 5 are executed first, and then script 7 is executed, script 7 and script 8 are executed sequentially, and script 8 and script 9 are executed sequentially.

[0101] After executing step S104, the execution logics between the sub-scripts may be marked and stored in the concurrency and dependency configuration table respectively.

[0102] S105 , executing multiple sub-scripts according to the execution logic between every two sub-scripts, and obtaining output data of the script to be executed.

[0103] Optionally, in order to ensure that each sub-script can be executed smoothly, a scheduling configuration for executing these sub-scripts may be generated first, that is, the running sequence and running resources required for running are allocated to these sub-scripts. That is, step S105 may specifically include:

[0104] Configure the corresponding runtime and runtime resources for each sub-script;

[0105] According to the execution logic between every two sub-scripts, each sub-script is executed using the runtime and runtime resources corresponding to each sub-script to obtain output data of each sub-script.

[0106] Finally, the output data of the entire script to be executed is determined by the output data of one or more sub-scripts.

[0107] for Figure 3 The specific process of executing each sub-script shown can be:

[0108] Create threads 1 to 5, and use threads 1 to 5 to execute scripts 1 to 5 in parallel. Each thread is used to execute the corresponding numbered script. After obtaining the output of scripts 1 to 5, use threads 1 and 2 to execute scripts 6 and 7 in parallel. After obtaining the output of scripts 6 and 7, use thread 1 to execute scripts 8 and 9 in sequence, and finally obtain the final result 1.

[0109] by Figure 3 For example, Figure 3 After each sub-script shown is executed according to the established execution logic, the output data obtained is Figure 3 The final result shown is 1.

[0110] It should be noted that after the execution of the multiple sub-scripts obtained by splitting the script to be executed according to the determined execution logic, it is equivalent to that the script to be executed is executed once, and the output data obtained at this time is the substantial content with the aforementioned output data name. For example, the aforementioned output data name may be just a blank table named output table X, which does not contain data, and after executing step S105, a table named output table X and containing specific data can be obtained.

[0111] In some cases, the output of a sub-script obtained by splitting the script to be executed is equivalent to the output of the entire script to be executed before the split, for example, Figure 3 The final result 1 output by script 9 is the final result 1 that the original script to be executed wants to obtain. In this case, the output data of the script can be directly used as the output data of the entire script to be executed.

[0112] In other cases, the output data of the entire script to be executed is composed of the output data of several sub-scripts obtained by splitting. In this case, it is necessary to generate a summary script (ending script) before executing step S105. Then, when executing step S105, the following process is performed:

[0113] According to the execution logic between every two sub-scripts, multiple sub-scripts are executed to obtain the output data of each sub-script;

[0114] Execute the pre-generated summary script to summarize the output data of each sub-script into the output data of the script to be executed.

[0115] The summary script is generated according to the execution logic between each sub-script and every two sub-scripts.

[0116] After executing step S104, that is, splitting the script to be executed into multiple sub-scripts and determining the execution logic between the sub-scripts, the following steps may also be executed:

[0117] The execution logic of each sub-script and between every two sub-scripts is summarized into an optimized script matched by the script to be executed, and the optimized script matched by the script to be executed is stored in a cache area.

[0118] By storing the optimized script in the cache area, when the same script to be executed is obtained again later, the optimized script can be directly read from the cache area, and then step S105 is directly executed, that is, each sub-script is executed according to the established execution logic.

[0119] That is, if a buffer area for storing optimized scripts is set, optionally, before executing step S102, i.e. parsing the script to be executed according to the target grammar rules to obtain the logical structure of the script to be executed, the method may further include:

[0120] Determine whether there is an optimized script matching the script to be executed in the cache;

[0121] If yes, read the optimized script matched by the script to be executed, and execute each sub-script in the optimized script matched by the script to be executed according to the execution logic contained in the optimized script matched by the script to be executed;

[0122] If not, the script to be executed is parsed according to the target grammatical rules to obtain the logical architecture step of the script to be executed, that is, step S102 is executed.

[0123] In this way, if the cache area has already stored the optimized script, the steps of parsing and splitting the script to be executed can be omitted, and each sub-script in the optimized script can be directly executed according to the established execution logic.

[0124] The present invention provides an execution method of a data processing script, the method comprising: obtaining a script to be executed; parsing the script to be executed according to a target grammatical rule to obtain a logical structure of the script to be executed; wherein the target grammatical rule refers to a grammatical rule of a computer language used to write the script to be executed; the logical structure of the script to be executed includes a plurality of execution units of the script to be executed, and a connection relationship between the execution units; according to the logical structure of the script to be executed, and the input data name and output data name of the script to be executed, the script to be executed is split into a plurality of sub-scripts; wherein the input data name and output data name of each sub-script are determined according to the input data name and output data name of the script to be executed; according to the logical structure of the script to be executed, the execution logic between every two sub-scripts is determined; wherein the execution logic between every two sub-scripts is parallel execution or sequential execution; according to the execution logic between every two sub-scripts, multiple sub-scripts are executed to obtain the output data of the script to be executed. This scheme splits the script to be executed into a plurality of sub-scripts, and when executing these sub-scripts, multiple sub-scripts in which the execution logic is parallel execution can be executed simultaneously, thereby improving the execution efficiency of the script to be executed.

[0125] The key to the present invention is to realize intelligent parsing of scripts to be executed, intelligently parse data processing scripts, automatically split the scripts to be executed into multiple sub-scripts, reasonably arrange the execution order of multiple sub-scripts, and accurately assemble multiple sub-scripts to return results, thereby automatically realizing script performance tuning through script splitting and combination, improving data processing speed and efficiency, replacing the manual performance tuning method, thereby achieving the improvement of data operation and processing efficiency.

[0126] In conjunction with the data processing script execution method provided in the embodiment of the present application, the embodiment of the present application also provides a data processing script execution device, please refer to Figure 4 , the device may include the following units:

[0127] The acquisition unit 401 is used to acquire the script to be executed.

[0128] The parsing unit 402 is used to parse the script to be executed according to the target grammar rules to obtain the logical structure of the script to be executed.

[0129] The target grammatical rules refer to the grammatical rules of the computer language used to write the script to be executed; the logical architecture of the script to be executed includes multiple execution units of the script to be executed and the connection relationship between the execution units.

[0130] The splitting unit 403 is used to split the script to be executed into multiple sub-scripts according to the logical structure of the script to be executed and the input data name and output data name of the script to be executed.

[0131] The input data name and output data name of each sub-script are determined according to the input data name and output data name of the script to be executed.

[0132] The determination unit 404 is used to determine the execution logic between every two sub-scripts according to the logic structure of the script to be executed.

[0133] The execution logic between every two sub-scripts is parallel execution or sequential execution.

[0134] The execution unit 405 is used to execute multiple sub-scripts according to the execution logic between every two sub-scripts to obtain output data of the script to be executed.

[0135] Optionally, the execution device further includes:

[0136] A determination unit 406 is used to determine whether there is an optimized script matching the script to be executed in the cache;

[0137] If yes, read the optimized script matched by the script to be executed, and execute each sub-script in the optimized script matched by the script to be executed according to the execution logic contained in the optimized script matched by the script to be executed;

[0138] If not, trigger the parsing unit 402 to parse the script to be executed according to the target grammar rules to obtain the logical structure of the script to be executed;

[0139] The cache unit 407 is used to summarize the execution logic of each sub-script and between every two sub-scripts into an optimized script matched by the script to be executed, and store the optimized script matched by the script to be executed in the cache area.

[0140] Optionally, the execution unit 405 executes multiple sub-scripts according to the execution logic between every two sub-scripts to obtain the output data of the script to be executed, specifically for:

[0141] According to the execution logic between every two sub-scripts, multiple sub-scripts are executed to obtain the output data of each sub-script;

[0142] Execute a pre-generated summary script to summarize the output data of each sub-script into the output data of the script to be executed; wherein the summary script is generated according to the execution logic of each sub-script and between every two sub-scripts.

[0143] Optionally, the execution unit 405 executes multiple sub-scripts according to the execution logic between every two sub-scripts to obtain output data of each sub-script, specifically for:

[0144] Configure the corresponding runtime and runtime resources for each sub-script;

[0145] According to the execution logic between every two sub-scripts, each sub-script is executed using the runtime and runtime resources corresponding to each sub-script to obtain output data of each sub-script.

[0146] Optionally, when the parsing unit 402 parses the script to be executed according to the target grammar rule to obtain the logical structure of the script to be executed, it is specifically used to:

[0147] A plurality of grammatical keywords are identified in the script to be executed; wherein the grammatical keywords are specified by the target grammatical rules;

[0148] Determine the logical structure of the script to be executed based on the grammatical keywords and target grammar rules.

[0149] The specific working principle of the data processing script execution device provided in the present application can refer to the relevant steps in the data processing script execution method provided in the embodiment of the present application, and will not be repeated here.

[0150] The present invention provides an execution device for a data processing script, wherein an acquisition unit 401 acquires a script to be executed; a parsing unit 402 parses the script to be executed according to a target grammatical rule to obtain a logical structure of the script to be executed; wherein the target grammatical rule refers to a grammatical rule of a computer language used to write the script to be executed; the logical structure of the script to be executed includes a plurality of execution units of the script to be executed and a connection relationship between the execution units; a splitting unit 403 splits the script to be executed into a plurality of sub-scripts according to the logical structure of the script to be executed and an input data name and an output data name of the script to be executed; wherein the input data name and the output data name of each sub-script are determined according to the input data name and the output data name of the script to be executed; a determination unit 404 determines an execution logic between every two sub-scripts according to the logical structure of the script to be executed; wherein the execution logic between every two sub-scripts is parallel execution or sequential execution; an execution unit 405 executes a plurality of sub-scripts according to the execution logic between every two sub-scripts to obtain output data of the script to be executed. This solution divides the script to be executed into multiple sub-scripts. When executing these sub-scripts, multiple sub-scripts whose execution logic is parallel execution can be executed simultaneously, thereby improving the execution efficiency of the script to be executed.

[0151] An embodiment of the present application also provides a computer storage medium for storing a computer program. When the stored computer program is executed, it is specifically used to implement the execution method of the data processing script provided in any embodiment of the present application.

[0152] The present application also provides an electronic device, such as Figure 5 As shown, the electronic device includes a memory 501 and a processor 502 .

[0153] The memory 501 is used to store computer programs.

[0154] The processor 502 is used to execute the above-mentioned computer program, specifically to implement the execution method of the data processing script provided in any embodiment of the present application.

[0155] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0156] It should be noted that the concepts such as "first" and "second" mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0157] The present application can be implemented or used by professionals and technicians. Various modifications to these embodiments will be apparent to professionals and technicians in the field, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for executing a data processing script, characterized in that: include: Get the script to be executed; Parsing the script to be executed according to the target grammar rules to obtain the logical structure of the script to be executed, which includes: identifying a plurality of grammar keywords in the script to be executed; wherein the grammar keywords are specified by the target grammar rules; determining the logical structure of the script to be executed according to the grammar keywords and the target grammar rules; wherein the target grammar rules refer to the grammar rules of the computer language used to write the script to be executed; the logical structure of the script to be executed includes a plurality of execution units of the script to be executed, and the connection relationship between each of the execution units; According to the logical structure of the script to be executed, and the input data name and output data name of the script to be executed, the script to be executed is split into multiple sub-scripts; wherein the input data name and output data name of each sub-script are determined according to the input data name and output data name of the script to be executed; Determine the execution logic between each two sub-scripts according to the logic structure of the script to be executed; wherein the execution logic between each two sub-scripts is parallel execution or sequential execution; According to the execution logic between every two of the sub-scripts, the multiple sub-scripts are executed to obtain the output data of the script to be executed.

2. The execution method according to claim 1, characterized in that: Before parsing the script to be executed according to the target grammar rules to obtain the logical structure of the script to be executed, the method further includes: Determine whether there is an optimized script matching the script to be executed in the cache; If so, read the optimized script matched by the script to be executed, and execute each sub-script in the optimized script matched by the script to be executed according to the execution logic contained in the optimized script matched by the script to be executed; If not, executing the step of parsing the script to be executed according to the target grammar rules to obtain the logical structure of the script to be executed; Wherein, after determining the execution logic between every two sub-scripts according to the logical architecture of the script to be executed, the method further includes: The execution logics of each sub-script and between every two sub-scripts are summarized into an optimized script matched by the script to be executed, and the optimized script matched by the script to be executed is stored in a cache area.

3. The execution method according to claim 1, characterized in that: The executing the plurality of sub-scripts according to the execution logic between each two of the sub-scripts to obtain output data of the script to be executed includes: Execute the multiple sub-scripts according to the execution logic between every two of the sub-scripts to obtain output data of each of the sub-scripts; Execute a pre-generated summary script to summarize the output data of each of the sub-scripts into the output data of the script to be executed; wherein the summary script is generated according to the execution logic between each of the sub-scripts and every two of the sub-scripts.

4. The execution method according to claim 3, characterized in that: The step of executing the plurality of sub-scripts according to the execution logic between each two of the sub-scripts to obtain output data of each of the sub-scripts includes: Configuring corresponding runtime and runtime resources for each of the sub-scripts; According to the execution logic between every two sub-scripts, each sub-script is executed using the runtime and runtime resources corresponding to each sub-script to obtain output data of each sub-script.

5. A data processing script execution device, characterized in that: include: An acquisition unit, used to acquire the script to be executed; A parsing unit, used for parsing the script to be executed according to the target grammar rule to obtain the logical structure of the script to be executed, which includes: identifying a plurality of grammar keywords in the script to be executed; wherein the grammar keywords are specified by the target grammar rule; determining the logical structure of the script to be executed according to the grammar keywords and the target grammar rule; wherein the target grammar rule refers to the grammar rule of the computer language used to write the script to be executed; the logical structure of the script to be executed includes a plurality of execution units of the script to be executed, and the connection relationship between each of the execution units; A splitting unit, used for splitting the script to be executed into a plurality of sub-scripts according to the logical structure of the script to be executed, and the input data name and output data name of the script to be executed; wherein the input data name and output data name of each sub-script are determined according to the input data name and output data name of the script to be executed; A determination unit, configured to determine the execution logic between each two of the sub-scripts according to the logic structure of the script to be executed; wherein the execution logic between each two of the sub-scripts is parallel execution or sequential execution; The execution unit is used to execute the multiple sub-scripts according to the execution logic between every two sub-scripts to obtain the output data of the script to be executed.

6. The execution device according to claim 5, characterized in that: The execution device also includes: A determination unit, used to determine whether there is an optimized script matching the script to be executed in the cache; If so, read the optimized script matched by the script to be executed, and execute each sub-script in the optimized script matched by the script to be executed according to the execution logic contained in the optimized script matched by the script to be executed; If not, trigger the parsing unit to parse the script to be executed according to the target grammar rules to obtain the logical structure of the script to be executed; The cache unit is used to summarize the execution logic of each sub-script and between every two sub-scripts into an optimized script matched by the script to be executed, and store the optimized script matched by the script to be executed in a cache area.

7. The execution device according to claim 5, characterized in that: The execution unit executes the multiple sub-scripts according to the execution logic between each two sub-scripts to obtain the output data of the script to be executed, specifically for: Execute the multiple sub-scripts according to the execution logic between every two of the sub-scripts to obtain output data of each of the sub-scripts; Execute a pre-generated summary script to summarize the output data of each of the sub-scripts into the output data of the script to be executed; wherein the summary script is generated according to the execution logic between each of the sub-scripts and every two of the sub-scripts.

8. The execution device according to claim 5, characterized in that: The execution unit executes the plurality of sub-scripts according to the execution logic between each two sub-scripts to obtain output data of each sub-script, specifically for: Configuring corresponding runtime and runtime resources for each of the sub-scripts; According to the execution logic between every two sub-scripts, each sub-script is executed using the runtime and runtime resources corresponding to each sub-script to obtain output data of each sub-script.

Images