Compilation method, device, equipment and storage medium

Abstract

The present invention discloses a compilation method, device, equipment and storage medium. The compilation method includes: obtaining a file to be compiled; extracting source code of each parallel computing function of the file to be compiled according to a preset function decorator in the file to be compiled; parsing the source code of each parallel computing function to determine an abstract syntax tree of each parallel computing function; for each abstract syntax tree of the parallel computing function, if a first loop function of the abstract syntax tree and a second loop function of the abstract syntax tree meet a preset condition, merging the first loop function and the second loop function to generate an execution code of the file to be compiled, and providing a compilation method for a parallel computing algorithm, which is simple to program and has high performance, and improves the efficiency of parallel data processing.

Description

Technical Field

[0001] The present invention relates to the field of code compilation technology, and in particular to a compilation method, device, equipment and storage medium. Background Art

[0002] With the continuous development of big data and artificial intelligence, the demand for large-scale data processing and computing is also increasing. Parallel computing technology has developed rapidly with its excellent processing efficiency. The architecture of parallel computing has evolved from single-core, single-processor, and single-node to multi-core, multi-processor, and multi-node, that is, multi-level parallel computing. Multi-level parallel computing has greatly improved the efficiency of data processing.

[0003] At present, when using interpreted language codes for parallel computing, since the interpreted language cannot be directly understood by the computer, the parallel computing algorithm will have a large overhead and low computing efficiency, which cannot meet user needs. Summary of the invention

[0004] The main purpose of the present invention is to provide a compilation method, device, equipment and storage medium. For the files to be compiled for parallel computing, the function decorators of the parallel computing part in the files to be compiled are decorated, the abstract syntax tree is extracted, and then the calculation logic of the loop part in the function is merged, thereby improving the operation efficiency.

[0005] In a first aspect, an embodiment of the present invention provides a compilation method, the compilation method comprising:

[0006] Obtain a file to be compiled; extract source code of each parallel computing function of the file to be compiled according to a preset function decorator in the file to be compiled; parse the source code of each parallel computing function to determine an abstract syntax tree of each parallel computing function; for each abstract syntax tree of the parallel computing function, if a first loop function of the abstract syntax tree and a second loop function of the abstract syntax tree meet a preset condition, merge the first loop function and the second loop function to obtain each merged parallel computing function, so as to generate an execution code of the file to be compiled according to each merged parallel computing function.

[0007] Optionally, the first loop function of the abstract syntax tree and the second loop function of the abstract syntax tree satisfy a preset condition, including:

[0008] The first loop function and the second loop function traverse the same data set.

[0009] Optionally, the compiling method further includes:

[0010] The data reading range of each of the parallel computing functions is obtained; and the reading data of the file to be compiled is determined according to each of the data reading ranges, so as to process the reading data through the parallel computing function.

[0011] Optionally, before generating the execution code of the file to be compiled, the compiling method further includes:

[0012] For each abstract syntax tree of the parallel computing function, identifying a repeated read portion in the abstract syntax tree, wherein the repeated read portion exists at least in a first node and a second node of the abstract syntax tree; deleting the repeated read portion in the second node; connecting a third node that obtains data corresponding to the repeated read portion from the second node to the first node, so that the third node obtains data corresponding to the repeated read portion from the first node, and obtaining a parallel computing function with duplicates deleted;

[0013] Correspondingly, the generating of the execution code of the file to be compiled includes:

[0014] The execution code of the file to be compiled is generated according to each duplicated parallel computing function.

[0015] Optionally, before generating the execution code of the file to be compiled, the compiling method further includes:

[0016] If the data splitting type is not specified for each of the parallel computing functions, a hash function is used to split the data of each of the parallel computing functions; or, if the data splitting type is not specified for any of the parallel computing functions, a second data splitting type of a second parallel computing function is obtained, and the second data splitting type is used to split the data of the parallel computing function, wherein the second parallel computing function is the parallel computing function with a specified data splitting type, and the second data splitting type is the data splitting type of the second parallel computing function;

[0017] Correspondingly, the generating of the execution code of the file to be compiled includes:

[0018] Generate execution code of the file to be compiled according to each of the parallel computing functions after the data is split.

[0019] Optionally, before generating the execution code of the file to be compiled, the compiling method further includes:

[0020] For each of the parallel computing functions, determine whether the structure of the abstract syntax tree of the parallel computing function is a preset structure; if so, insert a merge function into the parallel computing function;

[0021] Correspondingly, the generating of the execution code of the file to be compiled includes:

[0022] Generate the execution code of the file to be compiled according to the parallel computing function after the merge function is inserted.

[0023] Optionally, after generating the execution code of the file to be compiled, the compiling method further includes:

[0024] According to the attributes of each server, the execution code is divided to obtain each execution sub-code;

[0025] Each of the execution sub-codes is sent to the corresponding server.

[0026] In a second aspect, an embodiment of the present invention further provides a compiling device, including:

[0027] A module for obtaining a file to be compiled is used to obtain a file to be compiled; a source code extraction module is used to extract the source code of each parallel computing function of the file to be compiled according to a preset function decorator in the file to be compiled; a source code parsing module is used to parse the source code of each parallel computing function and determine the abstract syntax tree of each parallel computing function; a compilation optimization module is used to merge the first loop function and the second loop function of the abstract syntax tree of each parallel computing function if the first loop function of the abstract syntax tree and the second loop function of the abstract syntax tree meet preset conditions to obtain each merged parallel computing function, so as to generate the execution code of the file to be compiled according to each merged parallel computing function.

[0028] In a third aspect, an embodiment of the present invention further provides a compilation device, comprising: a memory, a processor, and a compilation program stored in the memory and executable on the processor, wherein the compilation program, when executed by the processor, implements the steps of the compilation method provided in any embodiment corresponding to the first aspect of the present invention.

[0029] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a compiler is stored, and when the compiler is executed by a processor, the steps of the compilation method provided in any embodiment corresponding to the first aspect of the present invention are implemented.

[0030] The compilation method, apparatus, device and storage medium provided by the embodiments of the present invention, for a file to be compiled for parallel computing, extracts the source code of each parallel computing function by decorating the parallel computing function in the file to be compiled with a function decorator, and obtains an abstract syntax tree of each parallel computing function based on the source code, and then merges the calculation logic in the loop function in the parallel computing function based on the abstract syntax tree, so as to simplify the code, reduce the running overhead, improve the code running efficiency and performance, and thus improve the efficiency of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is an application scenario diagram of the compilation method provided by an embodiment of the present invention;

[0032] Figure 2 is a flowchart of a compiling method provided by an embodiment of the present invention;

[0033] Figure 3 The present invention Figure 2 A schematic diagram of the structure of a computer cluster provided by the illustrated embodiment;

[0034] Figure 4 is a flowchart of a compiling method provided by another embodiment of the present invention;

[0035] Figure 5 The present invention Figure 4 A schematic diagram of a DAG of a parallel computing function provided by the illustrated embodiment;

[0036] Figure 6 The present invention Figure 5 A schematic diagram of an optimized DAG corresponding to the illustrated embodiment;

[0037] Figure 7 is a schematic diagram of the structure of a compiling device provided by an embodiment of the present invention;

[0038] Figure 8 A schematic diagram of the structure of a compiling device provided by an embodiment of the present invention.

[0039] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0040] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0041] The application scenarios of the embodiments of the present invention are explained below:

[0042] Figure 1 is an application scenario diagram of the compilation method provided by an embodiment of the present invention, such as Figure 1 As shown, in the field of big data processing, users write parallel computing algorithms 120 on the user end 110 according to their own business needs, and send the parallel computing algorithms to each computing node 130 of the computer cluster 120, so that each computing node 130 can perform parallel computing on the distributed stored data by running the parallel computing algorithm, thereby obtaining processed data, and send the processed data to the user end 110, so that the user end 110 can perform subsequent business processing according to the processed data. The business demand can be to determine the training data of the decision tree and the neural network, and the corresponding distributed storage data can be the original data for training, and the processed data is the data directly used for model training, which can be the data after feature extraction.

[0043] Since computers cannot directly understand the statements of interpreted languages, if interpreted languages, such as Python, are used as programming languages ​​for big data parallel computing algorithm codes, the codes of parallel computing algorithms need to be translated into machine languages. Therefore, interpreted languages ​​have a large computational overhead and are not suitable for writing parallel computing algorithms in the field of big data. If the above parallel computing algorithms are written based on interpreted languages, the computational overhead of the parallel computing algorithm codes will be large, resulting in low computational efficiency of parallel computing and failure to meet user needs.

[0044] In order to improve the performance of algorithms written in interpreted languages ​​for use in parallel computing of big data, an embodiment of the present invention provides a compilation method. A parallel computing algorithm is compiled by the compilation method, thereby greatly reducing the computational overhead of the algorithm and improving the efficiency of parallel computing. The main concept of the compilation method is as follows: by adding a function decorator to the parallel computing function of a file to be compiled, and exploring the function code based on the function decorator, an abstract syntax tree is obtained, and then based on the abstract syntax tree, a first loop function and a second loop function in the parallel computing function that meet preset conditions are merged, that is, loop functions with similar computational logic in the parallel function are merged, and by merging function logic, the complexity of the function is reduced, and the computational overhead of the loop function is reduced, thereby reducing the computational overhead of the parallel function, optimizing the performance of the parallel algorithm, and improving the operating efficiency of the parallel algorithm, thereby improving the efficiency of parallel computing.

[0045] Figure 2 is a flowchart of a compiling method provided by an embodiment of the present invention, such as Figure 2 As shown, the compilation method includes the following steps:

[0046] Step S201, obtaining a file to be compiled.

[0047] The file to be compiled may be a program written in an interpretive language, such as a program edited in Python. The file to be compiled may be a file of multi-level parallel computing.

[0048] As an explanatory language, Python has been widely used because of its easy-to-learn characteristics. However, due to its large internal overhead, it is usually not suitable for high-performance data computing. The present invention improves the program based on Python language and converts it into a high-performance underlying implementation.

[0049] Specifically, the file to be compiled may be a program from a user end that needs to be compiled. The program is a program edited in Python language and adopts a multi-level parallel computing mode.

[0050] Among them, the multi-level parallel computing mode means that an algorithm needs to be executed on multiple servers, and each server includes multiple CPUs (Central Processing Units), and each CPU has multiple cores.

[0051] Exemplarily, the file to be compiled may be a file, program or data used to count the average income and income distribution of the enterprise's employees, or a file, program or data used to count the distribution of product parameters of a certain product of the enterprise.

[0052] Step S202: extracting source code of each parallel computing function of the file to be compiled according to the preset function decorator in the file to be compiled.

[0053] The preset function decorator is a decorator set before each function that needs parallel computing in the file to be compiled. The advantage of this setting is that the part of the file to be compiled that needs parallel computing is decorated with the preset function decorator, so that it will not interfere with the operation of other parts of the code, and can be compiled using its corresponding python interpreter. The parallel computing function is the function that needs parallel computing.

[0054] Specifically, a function decorator is essentially a function that can add additional functionality to the decorated function without making any code changes. The preset function decorator @par can be manually added before each parallel computing function in the file to be compiled to decorate each parallel computing function with @par.

[0055] Exemplarily, the specific content of part of the code of the file to be compiled containing the above preset function decorator @par is as follows:

[0056]

[0057] It can be seen that the functions avg(), histo() and program() that need parallel computing are all decorated with the preset function decorator @par, so that the corresponding parallel computing functions, that is, the source codes of the functions avg(), histo() and program(), can be automatically obtained according to the preset function decorator @par.

[0058] Step S203: parse the source code of each of the parallel computing functions to determine the abstract syntax tree of each of the parallel computing functions.

[0059] Among them, the Abstract Syntax Tree (AST) is a tree-like representation of the abstract syntax structure of the source code.

[0060] Specifically, an abstract syntax tree of the parallel computing function may be generated based on the AST editor according to the source code of the parallel computing function.

[0061] Step S204: for the abstract syntax tree of each of the parallel computing functions, if the first loop function of the abstract syntax tree and the second loop function of the abstract syntax tree meet a preset condition, merge the first loop function and the second loop function to obtain each merged parallel computing function, so as to generate the execution code of the file to be compiled according to each merged parallel computing function.

[0062] The first loop function and the second loop function are two different loop functions in the parallel computing function, and can be loop functions based on statements or functions such as for, in, while, and range.

[0063] Specifically, the number of the second round-robin functions may be one or more, that is, if there are three or more round-robin functions that all meet the preset condition, the round-robin functions that meet the preset condition are merged.

[0064] Specifically, the preset condition may be that the data sets corresponding to the first loop function and the second loop function are the same. It may also include that the loop types of the first loop function and the second loop function are the same, such as both are loop functions based on for statements, or both are loop functions based on while statements.

[0065] Optionally, the first loop function of the abstract syntax tree and the second loop function of the abstract syntax tree satisfy a preset condition, including:

[0066] The first loop function and the second loop function traverse the same data set.

[0067] The data set corresponding to the loop function refers to the set of data required to be used when the loop function is traversed.

[0068] Specifically, if the data set traversed by the first loop function and the second loop function is the same, the calculation logic of the first loop function and the second loop function are merged to form a new loop function, and the new loop function has the functions of the first loop function and the second loop function.

[0069] By merging loop functions, the running overhead of the functions to be compiled is reduced, the performance of code compilation is optimized, and the running efficiency of the code is improved.

[0070] Specifically, the generated execution code may be a C language code, a Fortran language code, a CUDA language code, an OpenMP language code or an OpenCL language code.

[0071] Optionally, after generating the execution code of the file to be compiled, the method further includes:

[0072] The execution code is sent to at least two servers, so that the at least two servers execute the execution code.

[0073] The at least two servers form a computer cluster, and the servers in the computer cluster are multi-core servers, which are also called multi-core CPU servers, and are servers including multiple cores.

[0074] Specifically, after the execution code is generated, the execution code is divided into a plurality of parallel sub-codes, and each parallel sub-code is sent to a corresponding server for execution to obtain an execution result.

[0075] Specifically, the generated execution code is sent to a computer cluster via the compilation system, and then the computer cluster performs data exchange through a network to complete the execution of the execution code and output the operation result.

[0076] For example, Figure 3 The present invention Figure 2 The schematic diagram of the structure of the computer cluster provided in the embodiment shown is as follows: Figure 3 As shown, after the computer cluster receives the execution code, it will divide the execution code and then execute it separately by each multi-core CPU server. When different servers need to interact with each other, the data interaction is carried out through the network. For each multi-core CPU server, the task of running the execution code will be executed by a different CPU, and each CPU runs by accessing local files and shared memory.

[0077] In this embodiment, for the parallel computing files to be compiled, the source code of each parallel computing function is extracted by decorating the parallel computing function in the file to be compiled with a function decorator, and an abstract syntax tree of each parallel computing function is obtained based on the source code. Then, based on the abstract syntax tree, the calculation logic in the loop function in the parallel computing function is merged to simplify the code, reduce the running overhead, improve the code running efficiency and performance, and thus improve the efficiency of data processing.

[0078] Figure 4 is a flowchart of a compiling method provided by another embodiment of the present invention. Figure 2 Based on the embodiment shown, steps of data reading and repeated reading optimization are added, such as Figure 4 As shown, the compilation method provided in this embodiment includes the following steps:

[0079] Step S401, obtaining a file to be compiled, wherein the file to be compiled is a program edited based on Python language.

[0080] Step S402: extract source code of each parallel computing function of the file to be compiled according to the preset function decorator in the file to be compiled.

[0081] Step S403: Obtain the data reading range of each of the parallel computing functions.

[0082] Specifically, the data reading range of the parallel function may be determined according to the data reading range corresponding to each statement of the parallel computing function.

[0083] For example, if a statement of the parallel function is: Counter(field(0), field(1)), then the corresponding data reading range is the data range corresponding to field(0) and field(1).

[0084] Step S404: determining the read data of the to-be-compiled file according to each of the data read ranges, so as to process the read data through the parallel computing function.

[0085] Specifically, in order to improve the running efficiency of the code, it is necessary to tailor the read data of each parallel computing function. Since the data of the file to be compiled is often stored in a column-by-column manner, the start and end ranges of the read data of the file to be compiled can be determined according to the columns that each parallel computing function needs to read.

[0086] Specifically, the read data may be processed by any calculation processing method, such as summing, averaging, sorting, etc.

[0087] Exemplarily, the file to be compiled includes parallel computing function f1 and parallel computing function f2, and the initial reading range of the data of the file to be compiled is range A, range B and range C. According to the statements of each function, the initial data reading range of f1 is determined to be range A and range B, and the initial data reading range of f2 is range B. Then, the data reading range of the parallel function is set to range A and range B, that is, the data corresponding to range C in the data of the file to be edited is deleted, thereby avoiding the execution of unnecessary traversal steps to reduce the program overhead.

[0088] Step S405: parse the source code of each of the parallel computing functions to determine the abstract syntax tree of each of the parallel computing functions.

[0089] Step S406: for each abstract syntax tree of the parallel computing function, if the data set traversed by the first loop function of the abstract syntax tree is the same as the data set traversed by the second loop function of the abstract syntax tree, merge the first loop function and the second loop function.

[0090] Step S407: for each abstract syntax tree of the parallel computing function, identify a repeatedly read portion in the abstract syntax tree.

[0091] The repeatedly read portion is a portion that needs to be read at least twice in the parallel computing function. The repeatedly read portion exists at least in the first node and the second node of the abstract syntax tree. The first node and the second node are any two different nodes in the abstract syntax tree.

[0092] Step S408, delete the repeatedly read part in the second node; and connect the third node that obtains the data corresponding to the repeatedly read part from the second node to the first node, so that the third node obtains the data corresponding to the repeatedly read part from the first node, and obtains a parallel computing function that deletes duplications.

[0093] The third node is a node that needs to obtain data corresponding to the repeatedly read part from the second node.

[0094] For example, if the first node of function A needs to read the first column of data D, the second node of function A needs to read the first column and the second column of data D, and the third node of function A needs to process the data involved in the second node, such as averaging, summing, etc., then it is determined that the first column of data D is a repeated read part, and the code corresponding to the first column of read data D in the second node can be deleted, and the third node can implement its data processing based on the first column of read data D in the first node and the second column of read data D in the second node.

[0095] Step S409: Generate execution code of the file to be compiled according to each duplicated parallel computing function.

[0096] Specifically, the execution code of the file to be compiled may be generated according to each parallel computing function after deleting duplication and merging the first loop function and the second loop function.

[0097] Specifically, steps S407 to S408 may be performed before step S406.

[0098] Optionally, before generating the execution code of the file to be compiled, it also includes: if each of the parallel computing functions does not specify a data splitting type, then using a hash function to split the data of each of the parallel computing functions; accordingly, when generating the execution code of the file to be compiled, it includes: generating the execution code of the file to be compiled according to each of the parallel computing functions after the data splitting.

[0099] Specifically, the data split type can be determined based on the call statement of split() in the parallel function. If all parallel computing functions do not contain the call statement of split(), the data of each parallel computing function can be split based on the hash() function.

[0100] Optionally, before generating the execution code of the file to be compiled, it also includes: if any of the parallel computing functions does not specify the data splitting type, obtaining the second data splitting type of the second parallel computing function, and using the second data splitting type to split the data of the parallel computing function; accordingly, when generating the execution code of the file to be compiled, it includes: generating the execution code of the file to be compiled according to each of the parallel computing functions after the data splitting.

[0101] The second parallel computing function is the parallel computing function of the specified data splitting type, and the second data splitting type is the data splitting type of the second parallel computing function.

[0102] Specifically, for a parallel computing function that does not specify a data splitting type, data splitting is performed according to the data splitting type of any parallel computing function that specifies a data splitting type.

[0103] Optionally, before generating the execution code of the file to be compiled, it also includes: for each of the parallel computing functions, determining whether the structure of the abstract syntax tree of the parallel computing function is a preset structure; if so, inserting a merge function into the parallel computing function; accordingly, when generating the execution code of the file to be compiled, it includes: generating the execution code of the file to be compiled according to the parallel computing function after the merge function is inserted.

[0104] The merge function is a data merging function used to merge data. By inserting the merge function to merge data, the efficiency of data transmission and processing can be improved. The preset structure can be that the abstract syntax tree includes at least two branches, a first branch and a second branch, the first branch includes a first parent node and a first child node, and the second branch includes a second parent node. If the first child node of the first branch needs to be connected or merged with the second parent node of the second branch, it means that the abstract syntax tree meets the preset structure.

[0105] Specifically, it may be determined through pattern matching whether the structure of the abstract syntax tree of the parallel computing function is a preset structure.

[0106] Furthermore, if the data stream of the parallel computing function of the file to be compiled is a single instruction multiple data (SIMD) stream, the parallel computing function can also be optimized based on a preset SIMD optimization algorithm.

[0107] The preset SIMD optimization algorithm may be a SIMD optimization method for addition instructions.

[0108] Specifically, when it is identified that the current instruction is an array addition instruction, the instruction is optimized based on the preset SIMD optimization algorithm.

[0109] Exemplarily, when an instruction to add an array to another array is recognized (such as the array is an integer array int32), the conventional calculation process is: create a new result array with a length equal to the length of the two input arrays, traverse the two pairs of input arrays in turn, add the elements of the corresponding indexes of the two arrays in turn, and store the results in the result array. The calculation process optimized based on the preset SIMD optimization algorithm is (assuming that the SIMD bit width is 256 bits, that is, 8 int32): First, according to the total length of the array, each array is logically divided into multiple groups: the number of groups is specifically the quotient of the total length of the array divided by 8 plus 1, such as 17 elements, then there are 3 groups. For the first two groups, each group calls the SIMD 256-bit addition instruction, which can complete the addition of two input groups (each input group has 8 elements) in turn at the chip level and store the results; for the third group, the SIMD 256-bit addition instruction is called, but the bit length is specified as 32 bits to specially handle the last element; the final result is the same as the non-SIMD optimization result, which is to add the two input arrays in turn and store them in the result array. It can be seen that the addition calculation process optimized by the preset SIMD optimization algorithm has higher calculation efficiency and reduces calculation overhead.

[0110] Of course, the preset SIMD optimization algorithm may also include optimization methods for other operation instructions, which is not limited in the present invention.

[0111] For example, Figure 5 The present invention Figure 4 The schematic diagram of the DAG of the parallel computing function provided by the illustrated embodiment, Figure 6 The present invention Figure 5 A schematic diagram of the optimized DAG corresponding to the illustrated embodiment. Figure 5 and Figure 6 This is a schematic diagram of the DAG corresponding to different compilation methods for the same file to be compiled. The file to be compiled is mainly used to calculate the average value of the first and second columns of data and generate the statistical histogram of the first and second columns of data. Figure 5 and Figure 6 It can be seen that Figure 5 The DAG (Directed Acyclic Graph) provided in the traversal of data repeatedly obtains the first column of data. At the same time, when generating the Counter, no merge calculation is performed, resulting in the need to repeatedly read the data to obtain the histogram. Figure 6 In the , the repeated reading part is optimized so that the data corresponding to the first and second columns only need to be read once during traversal. At the same time, when generating Counter, merge logic is inserted to merge data, simplify data transmission, and improve operation and processing efficiency.

[0112] Specifically, the parallel computing framework for executing the code may be based on an MPI (Message passing interface) and OpenMP (Open Multiprocessing) framework, a simulator, or a heterogeneous computing implementation.

[0113] Specifically, the simulator-based implementation can be a multi-threaded or multi-process pool implementation based on Python. The MPI and OpenMP framework-based implementation can be to adapt the parallel task provision and communication to the MPI message communication and shared memory framework, translate the instruction-level optimization into OpenMP preprocessing instructions, and hand it over to the OpenMP compiler to further implement multi-core, SIMD and CPU cache optimization. The implementation based on heterogeneous computing is specifically: adapt the communication part to NVLINK, RDMA (Remote Direct Memory Access) implementation, and map the calculation distribution to the OpenCL (Open Computing Language) compiler implementation. This implementation method can support GPU (Graphics Processing Unit, graphics processor), RDMA and other devices.

[0114] Specifically, on the basis of the merge process, each parallel computing function after at least one of the processes of data splitting, deduplication and inserting merge function, as well as each parallel computing function without any processing, can be further processed to generate the execution code of the file to be compiled.

[0115] Furthermore, the method further includes: sending the execution code to a computer cluster to obtain an execution result.

[0116] Furthermore, after obtaining the execution result, the method further includes: displaying the execution result, or sending the execution result to the client.

[0117] Optionally, after generating the execution code of the file to be compiled, the method further includes:

[0118] According to the attributes of each server, the execution code is divided to obtain various execution sub-codes; and each execution sub-code is sent to the corresponding server.

[0119] The attributes of the server may include the number of CPUs included in the server and the number of cores of each CPU. Each server may be a computing node of a computer cluster.

[0120] In this embodiment, for a file to be compiled based on multi-level parallel computing edited in Python, the source code of each parallel computing function is extracted by decorating the parallel computing function in the file to be compiled with a function decorator, and its data reading is trimmed to reduce unnecessary convenience steps; and an abstract syntax tree of each parallel computing function is obtained based on the source code, and then based on the abstract syntax tree, the calculation logic in the loop function in the parallel computing function is merged, and the repeated reading part in the parallel computing function is optimized to avoid repeated reading, so as to simplify the code. At the same time, the splitting and merging of data are optimized based on hash splitting and merge merging, so as to optimize the compilation of code from multiple dimensions, reduce running overhead, improve code running efficiency and performance, and thus improve data processing efficiency.

[0121] Figure 7 is a schematic diagram of the structure of a compiling device provided by an embodiment of the present invention, such as Figure 7 As shown, the compiling device includes: a to-be-compiled file acquisition module 710 , a source code extraction module 720 , a source code analysis module 730 and a compiling optimization module 740 .

[0122] Among them, the to-be-compiled file acquisition module 710 is used to obtain the to-be-compiled file; the source code extraction module 720 is used to extract the source code of each parallel computing function of the to-be-compiled file according to the preset function decorator in the to-be-compiled file; the source code parsing module 730 is used to parse the source code of each of the parallel computing functions and determine the abstract syntax tree of each of the parallel computing functions; the compilation optimization module 740 is used to merge the first loop function and the second loop function of the abstract syntax tree of each of the parallel computing functions if the first loop function of the abstract syntax tree and the second loop function of the abstract syntax tree meet the preset conditions to obtain the merged parallel computing functions, so as to generate the execution code of the to-be-compiled file according to the merged parallel computing functions.

[0123] Optionally, the compile optimization module 740 is specifically used for:

[0124] For each abstract syntax tree of the parallel computing function, if the data set traversed by the first loop function of the abstract syntax tree is the same as the data set traversed by the second loop function of the abstract syntax tree, the first loop function and the second loop function are merged to generate the execution code of the file to be compiled.

[0125] Optionally, the compiling device further includes:

[0126] The read range clipping module is used to obtain the data read range of each of the parallel computing functions; and determine the read data of the file to be compiled according to each of the data read ranges, so as to process the read data through the parallel computing function.

[0127] Optionally, the compiling device further includes:

[0128] A repeated read optimization module is used to identify the repeated read part in the abstract syntax tree of each parallel computing function before generating the execution code of the file to be compiled, wherein the repeated read part exists in at least the first node and the second node of the abstract syntax tree; delete the repeated read part in the second node; connect a third node that obtains data corresponding to the repeated read part from the second node to the first node so that the third node obtains the data corresponding to the repeated read part from the first node to obtain the parallel computing function with duplicate deletion; a first execution code generation module is used to generate the execution code of the file to be compiled according to each parallel computing function with duplicate deletion.

[0129] Optionally, the compiling device further includes:

[0130] A data splitting optimization module is used to, before generating the execution code of the file to be compiled, if each of the parallel computing functions does not specify a data splitting type, use a hash function to split the data of each of the parallel computing functions; or, if any of the parallel computing functions does not specify a data splitting type, obtain a second data splitting type of a second parallel computing function, and use the second data splitting type to split the data of the parallel computing function, wherein the second parallel computing function is the parallel computing function with a specified data splitting type, and the second data splitting type is the data splitting type of the second parallel computing function; a second execution code generation module is used to generate the execution code of the file to be compiled according to each of the parallel computing functions after data splitting.

[0131] Optionally, the compiling device further includes:

[0132] A data merging and optimization module is used to determine, for each of the parallel computing functions, whether the structure of the abstract syntax tree of the parallel computing function is a preset structure before generating the execution code of the file to be compiled; if so, insert a merge function into the parallel computing function; a third execution code generation module is used to generate the execution code of the file to be compiled according to the parallel computing function after the merge function is inserted.

[0133] Optionally, the compiling device further includes:

[0134] The execution code sending module is used to send the execution code of the file to be compiled to at least two servers after generating the execution code, so that the at least two servers execute the execution code, wherein the servers are multi-core servers.

[0135] Optionally, the compiling device further includes:

[0136] The execution code division module is used to divide the execution code of the file to be compiled according to the attributes of each server after generating the execution code to obtain various execution sub-codes; the sub-code sending module is used to send each of the execution sub-codes to the corresponding server.

[0137] The compiling device provided in the embodiment of the present invention can execute the compiling method provided in any embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method.

[0138] Figure 8 A schematic diagram of the structure of a compiling device provided by an embodiment of the present invention is shown in FIG. Figure 8 As shown, the compiling device includes: a memory 810, a processor 820 and a computer program.

[0139] The computer program is stored in the memory 810 and is configured to be executed by the processor 820 to implement the present invention. Figure 2-Figure 4 The compilation method provided by any embodiment in the corresponding embodiment.

[0140] The memory 810 and the processor 820 are connected via a bus 830 .

[0141] For related instructions, please refer to Figure 2-Figure 4 The relevant descriptions and effects corresponding to the steps are understood, and no further elaboration is given here.

[0142] One embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the present invention. Figure 2-Figure 4 The compilation method provided by any embodiment in the corresponding embodiment.

[0143] Among them, the computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0144] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0145] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0146] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, or each module may exist physically separately, or two or more modules may be integrated into one unit. The above-mentioned module-composed unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0147] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present invention.

[0148] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be implemented by a combination of hardware and software modules in the processor.

[0149] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0150] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present invention is not limited to only one bus or one type of bus.

[0151] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0152] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application-specific integrated circuit (Application Specific Integrated Circuits, referred to as ASIC). Of course, the processor and the storage medium can also be present in an electronic device or a main control device as discrete components. It should be noted that, in this article, the term "includes", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, article or device. In the absence of more restrictions, the elements defined by the statement "including one..." do not exclude the presence of other identical elements in the process, method, article or device including the element.

[0153] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0154] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0155] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A compiling method, characterized in that: include: Obtaining a file to be compiled; the file to be compiled is a program written in an interpreted language, and the program is a program edited in a language and adopts a multi-level parallel computing mode; Extracting source code of each parallel computing function of the file to be compiled according to the preset function decorator in the file to be compiled; Parsing the source code of each of the parallel computing functions to determine an abstract syntax tree of each of the parallel computing functions; For the abstract syntax tree of each parallel computing function, if the first loop function of the abstract syntax tree and the second loop function of the abstract syntax tree meet the preset conditions, the first loop function and the second loop function are merged to obtain the merged parallel computing functions, so as to generate the execution code of the file to be compiled according to the merged parallel computing functions.

2. The method according to claim 1, characterized in that The first loop function of the abstract syntax tree and the second loop function of the abstract syntax tree satisfy a preset condition, including: The first loop function and the second loop function traverse the same data set.

3. The method according to claim 1, characterized in that: Before generating the execution code of the file to be compiled, the method further includes: For each abstract syntax tree of the parallel computing function, identifying a repeated reading portion in the abstract syntax tree, wherein the repeated reading portion exists at least in a first node and a second node of the abstract syntax tree; deleting the repeatedly read portion in the second node; Connecting a third node that obtains the data corresponding to the repeatedly read part from the second node to the first node, so that the third node obtains the data corresponding to the repeatedly read part from the first node, and obtains a parallel computing function for deleting duplication; Correspondingly, the generating of the execution code of the file to be compiled includes: The execution code of the file to be compiled is generated according to each duplicated parallel computing function.

4. The method according to claim 1, characterized in that Before generating the execution code of the file to be compiled, the method further includes: If the data splitting type is not specified for each of the parallel computing functions, a hash function is used to split the data of each of the parallel computing functions; or, If any of the parallel computing functions does not specify a data splitting type, obtaining a second data splitting type of a second parallel computing function, and using the second data splitting type to split the data of the parallel computing function, wherein the second parallel computing function is the parallel computing function with a specified data splitting type, and the second data splitting type is the data splitting type of the second parallel computing function; Correspondingly, the generating of the execution code of the file to be compiled includes: Generate execution code of the file to be compiled according to each of the parallel computing functions after the data is split.

5. The method according to claim 1, characterized in that Before generating the execution code of the file to be compiled, the method further includes: For each of the parallel computing functions, determining whether the structure of the abstract syntax tree of the parallel computing function is a preset structure; If yes, insert a merge function into the parallel computing function; Correspondingly, the generating of the execution code of the file to be compiled includes: Generate the execution code of the file to be compiled according to the parallel computing function after the merge function is inserted.

6. The method according to claim 1, characterized in that Also includes: Obtaining the data reading range of each of the parallel computing functions; The read data of the to-be-compiled file is determined according to each of the data read ranges, so as to process the read data through the parallel computing function.

7. The method according to any one of claims 1 to 6, characterized in that: After generating the execution code of the file to be compiled, the method further includes: According to the attributes of each server, the execution code is divided to obtain each execution sub-code; Each of the execution sub-codes is sent to the corresponding server.

8. A compiling device, characterized in that: include: A module for acquiring files to be compiled, used for acquiring files to be compiled; the files to be compiled are programs written based on an interpreted language, and the programs are programs edited using a language and using a multi-level parallel computing mode; A source code extraction module, used to extract source code of each parallel computing function of the file to be compiled according to a preset function decorator in the file to be compiled; A source code parsing module, used to parse the source code of each of the parallel computing functions and determine an abstract syntax tree of each of the parallel computing functions; A compilation optimization module is used to combine the first loop function and the second loop function of each of the abstract syntax trees of the parallel computing functions to obtain the merged parallel computing functions if the first loop function of the abstract syntax tree and the second loop function of the abstract syntax tree meet the preset conditions, so as to generate the execution code of the file to be compiled according to the merged parallel computing functions.

9. A compiling device, characterized in that: The compiling device comprises: a memory, a processor, and a compiling program stored in the memory and executable on the processor, wherein the compiling program implements the steps of the compiling method according to any one of claims 1 to 7 when executed by the processor.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a compiler program, and when the compiler program is executed by a processor, the steps of the compilation method according to any one of claims 1 to 7 are implemented.

Images