Method and apparatus for generating code for multi-node execution to run a target model

Abstract

This specification provides a method and apparatus for generating code that can be executed on multiple nodes to run a target model. The method includes: obtaining first code for running the target model on a single computing node; converting the first code into a first intermediate representation including a plurality of operators calculated sequentially; inputting target data of the target model that includes at least split first data; determining n target operators from the plurality of operators that can be decomposed and calculated sequentially on k computing nodes according to the splitting method of the first data, the types of the plurality of operators, and the calculation order of the plurality of operators; determining k first calculation parts according to the n target operators; and generating k second codes based on the k first calculation parts, wherein the plurality of second codes are executed on the k computing nodes respectively to run the target model.

Description

Technical Field

[0001] The embodiments in this specification pertain to the field of machine learning, and particularly relate to a method and apparatus for generating code that executes on multiple nodes to run a target model. Background Technology

[0002] Machine learning models are computational models that automatically learn and improve their performance through data training. They can extract patterns, regularities, and relationships from large amounts of data to perform tasks such as prediction, classification, and clustering on new data. Running machine learning models on a single computing node is generally straightforward. However, due to limited computing resources (such as memory) on a single node and the need to protect the privacy of data from different stakeholders, it is often desirable to run machine learning models with the support of computing power or data from multiple computing nodes. However, existing solutions for running machine learning models on multiple computing nodes still suffer from problems such as either requiring significant human effort or insufficient protection of data privacy. Summary of the Invention

[0003] The purpose of this invention is to provide a method and apparatus for generating code that can be executed on multiple nodes to run a target model, which can automatically obtain code for execution on multiple computing nodes to run a target model and protect the privacy data of multiple parties.

[0004] To achieve the above objectives, the first aspect of this specification provides a method for generating code that can be executed on multiple nodes to run a target model, comprising: obtaining first code for running the target model on a single computing node; converting the first code into a first intermediate representation including a plurality of operators computed sequentially; inputting target data of the target model that includes at least split first data; determining n target operators from the plurality of operators that can be decomposed and computed sequentially on k computing nodes according to the splitting method of the first data, the type of the plurality of operators, and the computation order of the plurality of operators; determining k first computation parts according to the n target operators; and generating k second codes based on the k first computation parts, the plurality of second codes being executed on the k computing nodes respectively to run the target model.

[0005] A second aspect of this specification provides an apparatus for generating code that can be executed on multiple nodes to run a target model, comprising: an acquisition unit configured to acquire first code for running the target model on a single computing node, and convert the first code into a first intermediate representation including a plurality of operators computed sequentially; inputting target data of the target model that includes at least split first data, and determining n target operators from the plurality of operators that can be decomposed and computed sequentially on k computing nodes according to the splitting method of the first data, the type of the plurality of operators, and the computation order of the plurality of operators; and a generation unit configured to determine k first computation parts based on the n target operators; and generate k second codes based on the k first computation parts, the plurality of second codes being executed on the k computing nodes respectively to run the target model.

[0006] A third aspect of this specification provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method described in the first aspect.

[0007] A fourth aspect of this specification provides a computing device including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method described in the first aspect.

[0008] This specification provides a computer program product in a fifth aspect, including a computer program / instructions that, when executed by a processor, implement the steps of the method described in the first aspect.

[0009] In the scheme for generating code to run a target model on multiple nodes provided in the embodiments of this specification, first code for running the target model on a single computing node can be obtained. This first code is then converted into a first intermediate representation including multiple operators computed sequentially. Target data for the input target model is obtained, including at least split first data. Based on the first data, the types of the multiple operators, and the computation order of the multiple operators, n target operators are determined from the multiple operators that can be sequentially decomposed and computed on k computing nodes. Based on the n target operators, k first computation parts are determined. Based on the k first computation parts, n second codes are generated. These n second codes are executed on multiple computing nodes to run the target model. This method can automatically obtain code for running the target model on multiple computing nodes while protecting the privacy data of multiple parties during execution. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments in this specification, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of a scheme to run a machine learning model on multiple nodes;

[0012] Figure 2 This is a schematic diagram of another scheme for running machine learning models on multiple nodes;

[0013] Figure 3 This is a schematic diagram of a method for generating code that is executed on multiple nodes to run a target model, as described in one embodiment of this specification.

[0014] Figure 4 This is a flowchart of a method for generating code that is executed on multiple nodes to run a target model, as described in one embodiment of this specification.

[0015] Figure 5 This is a schematic diagram of the splitting of the dot product operator in one embodiment of this specification;

[0016] Figure 6 This is a schematic diagram of another dot product operator splitting in one embodiment of this specification;

[0017] Figure 7 This is a schematic diagram illustrating the acquisition of multi-node execution code through a second intermediate representation in one embodiment of this specification;

[0018] Figure 8 This is a schematic diagram illustrating the determination of the second code based on a structure tree in one embodiment of this specification;

[0019] Figure 9 This is an architectural diagram of an apparatus for generating code that is executed on multiple nodes to run a target model, according to one embodiment of this specification. Detailed Implementation

[0020] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0021] As mentioned earlier, a machine learning model is a computational model that automatically learns and improves its performance through data training. It can extract patterns, rules, and relationships from large amounts of data to perform tasks such as prediction, classification, and clustering on new data. Running machine learning models on a single computing node is usually relatively easy. However, due to the limited computing resources (such as memory) on a single computing node and the need to protect the privacy of data from different stakeholders, it is often desirable to run machine learning models with the support of computing power or data from multiple computing nodes. However, existing solutions for running machine learning models on multiple computing nodes still have problems such as either requiring a large amount of human effort or not providing sufficient protection for data privacy.

[0022] Figure 1 This is a schematic diagram of a scheme for running machine learning models on multiple nodes. For example... Figure 1 As shown, for example, a single worker node may only have limited local memory, while the training dataset is too large to be stored in the local memory of a single worker node. Therefore, the dataset needs to be partitioned and distributed across multiple worker nodes. Each worker node then trains the model using the complete model and its allocated local data. However, this approach has a problem: each compute node stores all model parameters, wasting significant storage resources. Furthermore, the computation process on each compute node consumes even more computing resources, such as CPU or GPU time, due to the need to process all model parameters. Especially when compute node hardware storage or computing resources are limited, this can lead to low efficiency or even failure to run the machine learning model.

[0023] Figure 2 This is a diagram illustrating another approach to running machine learning models on multiple nodes. For example... Figure 2 As shown, the network parameters of each layer of the machine learning model's neural network are stored on different computing nodes. For example, the network parameters of each layer of the neural network are stored on a separate computing node. Each computing node processes the calculation of the parameters of its own neural network layer and sends the processing result to the computing node corresponding to the next layer of the neural network, until the calculation of each layer of the machine learning model is completed on the computing nodes corresponding to each layer. However, the problem with this approach is that when the data involved in the model calculation includes private data from multiple parties, the security of the private data of multiple parties cannot usually be guaranteed. For example, the working node with the first layer of the neural network usually receives data from all parameter parties.

[0024] Another approach involves distributing the parameters of each neural network layer across multiple computation nodes. For example, the parameters of each layer can be equally divided into several parts, and each computation node stores a portion of the parameters from all layers. The computation of each neural network layer is then completed through interaction between these multiple computation nodes. This partitioning scheme is also known as the tensor partitioning scheme. However, the problem with this scheme is that it only splits the parameters of the neural network layers. When the data involved in the model computation includes private data from multiple stakeholders, the security of that private data cannot usually be guaranteed. Furthermore, the tensor partitioning scheme cannot automatically generate the code for splitting the computation based on the code running a machine learning model on a single computation node.

[0025] Another approach involves manually splitting the machine learning model and then executing the different parts on different computing nodes. This approach requires a significant amount of manual work and time. Furthermore, when the deep learning model is updated, it often needs to be manually split again, which is inefficient.

[0026] To address the aforementioned technical problems, embodiments of this specification propose a method for generating code that executes on multiple nodes to run a target model. Figure 3 This is a schematic diagram illustrating a method for generating code that executes on multiple nodes to run a target model, as described in one embodiment of this specification. Figure 3 As shown, code used to run a machine learning model (e.g., a target model) on a single compute node can be converted into an intermediate representation (IR). Since running a machine learning model on a single compute node typically involves centralizing all model parameters and input data of the target model on that single node, this code is also referred to as centralized computation code in this specification. The intermediate representation of a machine learning model is primarily used to convert models trained on different machine learning frameworks into a unified format for deployment and optimization in different environments. Because machine learning models can often have a mathematical model representation, the converted intermediate representation usually also contains multiple operators from the mathematical model. Then, according to preset rules, it is determined whether there are decomposable operators (e.g., a target operator) among the multiple operators. If so, the target operator is decomposed into multiple parts, and based on these parts, multiple codes are obtained that are executed on multiple compute nodes to run the target model.

[0027] The advantages of this method are as follows: First, it can automatically split the machine learning model based on the code that is centrally computed on a single computing node, and generate code that runs on multiple computing nodes. Compared with existing model splitting schemes, this significantly improves the efficiency of obtaining code that runs on multiple computing nodes. Second, the model splitting method based on atomic operators allows this method to support a wide range of machine learning models. Third, since the computational parts allocated to each computing node typically include sub-computational parts decomposed into multiple decomposable operators that can be computed sequentially, and these decomposable operators are computed independently on each computing node in the order of computation, it greatly increases the difficulty for attackers to reverse-engineer the privacy data of each computing node's parameters when the computational results of each computing node are merged. Therefore, this method can protect the security of privacy data belonging to multiple computing nodes from multiple participants when the data involved in model computation includes privacy data from multiple participants.

[0028] The following describes in more detail the method for generating code that can be executed on multiple nodes to run the target model, as provided in the embodiments of this specification. Figure 4 This is a flowchart illustrating a method for generating code that executes on multiple nodes to run a target model, according to one embodiment of this specification. Figure 4 As shown, the method includes at least the following steps:

[0029] Step S401: Obtain first code for running the target model on a single computing node, and convert the first code into a first intermediate representation including multiple operators calculated sequentially; the target data of the input target model includes at least the first data that has been split, and based on the splitting method of the first data, the type of the multiple operators and the calculation order of the multiple operators, determine n target operators that can be decomposed and calculated sequentially on k computing nodes respectively.

[0030] Step S403: Determine k first computation parts based on the n target operators; generate k second codes based on the k first computation parts, wherein the multiple second codes are executed on the k computation nodes respectively to run the target model.

[0031] First, in step S401, first code for running the target model on a single computing node is obtained. Depending on the implementation, the target model can be a different machine learning model. The first code is the code used to run the target model on a single computing node. Depending on the implementation, the first code can be code written in different programming languages, and this specification does not limit this. For example, in one embodiment, the first code can be code written in Python. In another embodiment, the first code can also be code written in other languages. Depending on the implementation, the target model can also be a different machine learning model. For example, in one embodiment, the target model can be one of a logistic regression model, a Transformer model, or a deep factorization machine.

[0032] After obtaining the initial code, it can be transformed into a first intermediate representation comprising multiple operators computed sequentially. In existing technologies, code used to run machine learning models typically runs on specific machine learning frameworks, such as TensorFlow, PyTorch, Caffe (Convolutional Architecture for Fast Feature Embedding), and JAX (Just-in-Time Automatic Differentiation for High-Performance Machine Learning). Because the code is based on a specific machine learning framework, it is often difficult to port it to environments configured with other machine learning frameworks. The intermediate representation of a machine learning model, however, is usually determined based on the mathematical computational model of the machine learning model and is independent of the specific machine learning framework. Therefore, the intermediate representation can convert machine learning models based on different machine learning frameworks into a unified format, enabling them to be transformed into runtime code based on the local machine learning framework in runtime environments configured with different frameworks, thus allowing deployment of the machine learning model in environments with different machine learning frameworks. By transforming the initial code into a first intermediate representation comprising multiple operators computed sequentially, subsequent steps can determine the decomposable operator (target operator) from among the multiple operators.

[0033] Depending on the implementation, the first intermediate representation can be based on different pre-defined intermediate representation instruction sets, and the different intermediate representation instructions in the intermediate representation instruction sets can correspond to different pre-defined operators. In one embodiment, it can be an intermediate representation based on a Stable High-Level Operations (StableHLO) set. Depending on the implementation, the operators included in the multiple operators in the first intermediate representation can have different operator types. In one embodiment, the multiple operators can include multiple operators such as dot product, vector element-wise multiplication, vector element-wise division, vector element-wise addition, vector element type conversion, natural exponentiation, vector element negation, vector element aggregation, and vector maximum element value operator.

[0034] Depending on the implementation method, the specific way of generating the first intermediate representation from the first code can vary. As mentioned above, the machine learning framework on which the first code is based can differ depending on the implementation method. Therefore, depending on the implementation method, the first intermediate representation can be obtained using intermediate representation generation tools for different machine learning frameworks. For example, in one embodiment, the machine learning framework on which the first code is based can be TensorFlow or Keras, and the StableHLO-based intermediate representation of the first code can be obtained using the MLIR-EmitC tool. In another embodiment, the machine learning framework on which the first code is based can be PyTorch, and the StableHLO-based intermediate representation of the first code can be obtained using the torch_xla.stable_hlo module.

[0035] In one example, the first code could be JAX-based Python code used to run a logical "Logistic Regression" model, specifically...

[0036]

[0037]

[0038] Among them, dot_general is the dot product operator, convert is the vector element type conversion operator, broadcast is the vector expansion operator, negate is the vector element negation operator, add is the vector element-wise addition operator, divide is the vector element-wise division operator, broadcast is the vector expansion operator, and const is the constant definition instruction.

[0039] After obtaining the first intermediate representation, the target data of the input target model can be obtained, which includes at least the split first data. Based on the splitting method of the first data, the types of multiple operators, and the calculation order of the multiple operators, multiple target operators (e.g., n, where the value of k may vary in different specific examples) can be determined from the multiple operators, each capable of being decomposed and calculated sequentially on multiple (e.g., k, where the value of k may vary in different specific examples) computation nodes. Depending on the implementation method, the split first data can be data from different participants or specific data used for different specific businesses; this specification does not limit this. In one implementation, the different parts of the split first data can, for example, be privacy data from different participants. Depending on the implementation method, the splitting method of the first data can also be different, for example, it can be split according to dimension or split according to element value. In one example, the complete first data is, for example, a vector with a dimension of 2×4, where a part (e.g., part A, with a dimension of 2×1) comes from participant A, and another part (e.g., part B, with a dimension of 2×3) comes from participant B. In another example, the complete first data is a 2×4 dimension vector, where one part (e.g., part A, 2×4 dimension) comes from participant A, and the other part (e.g., part B, 2×4 dimension) comes from participant B. The value of each element in this first data is equal to the sum or product of its corresponding element in part A and its corresponding element in part B.

[0040] Depending on the implementation method, the specific way to determine n target operators to be calculated sequentially from multiple operators can vary. In one embodiment, each operator among the multiple operators is determined to be a target operator according to the calculation order of the multiple operators, and the input data of at least one operator among the multiple operators includes the first data. Specifically, determining whether any first operator among the multiple operators is a target operator includes: if the first operator does not have a preceding operator, the output result of the preceding operator is used to determine the input data of the first operator; determining whether the first operator is a target operator based on whether there is a first equivalent method for calculating the divided input data on k computing nodes; if the first equivalent method exists, then the first operator is determined to be a target operator, and multiple sub-computation parts corresponding to the first operator on the k computing nodes are determined according to the first equivalent method; if the first equivalent method does not exist, then the first operator is determined not to be a target operator; if the first operator has the preceding operator and the first operator... The preceding operator does not include non-target operators. Based on the combination of the calculation results of the multiple sub-computation parts corresponding to the preceding operator of the first operator on k computation nodes, and the type of the first operator, it is determined whether the first operator has a second equivalent method for calculation on k computation nodes according to the combination method. If the second equivalent method exists, the first operator is determined to be a target operator, and the k sub-computation parts corresponding to the first operator on k computation nodes are determined according to the second equivalent method. If the second equivalent method does not exist, the first operator is not a target operator. If the first operator has the preceding operator and the preceding operator of the first operator includes non-target operators, the first operator is not a target operator.

[0041] For example, in the example above of obtaining the first intermediate representation based on StableHLO using JAX-based Python code, we can first identify the leading dot-multiplication operator (dot_general) among multiple operators. Then, based on the splitting method of the first data (e.g., arg0) and the second data (e.g., arg1), we determine if this dot-multiplication operator has an equivalent computation method (first equivalence method), and determine the multiple sub-computation parts corresponding to this dot-multiplication operator, as well as the combination method of the output results of these sub-computation parts. Next, we sequentially determine whether each operator following the dot-multiplication operator is a target operator. For each operator, if its preceding operator is a target operator, based on the combination method of the output results of the multiple sub-computation parts of the preceding operator and the type of the operator, we determine whether the operator has an equivalent computation method (second equivalence method) that operates on multiple nodes. If it exists, we determine whether the operator is a target operator; if it exists, then the operator is not a target operator. If the preceding operator is not a target operator, then the operator is not a target operator. For example, for a convert operator following a dot product operator, since its preceding dot product operator is the target operator, and the outputs of its multiple sub-computation parts are combined by element-wise vector addition, the convert operator can continue to be calculated on different computation nodes without merging the outputs of the preceding operator's corresponding sub-computation parts. Specifically, different computation nodes can perform convert calculations (i.e., the convert operator's corresponding sub-computation parts) based on their own outputs of the preceding dot product operator's corresponding sub-computation parts. The results of the convert calculations performed by different computation nodes can be combined (e.g., by element-wise vector addition) into the complete convert operator result. Therefore, the convert operator can also be considered the target operator. Similarly, for the subsequent broadcast, add, negate, convert, and exponential operators, since they can all execute their own sub-computational parts on the same computing node based on the output of the preceding operator's sub-computational part on that computing node, and the results of executing the sub-computational parts of the operator on different computing nodes can be combined into the complete computational result of the operator, these operators can also be identified as target operators. It is important to note that this statement about combining the results of executing the sub-computational parts of the operator on different computing nodes into the complete computational result of the operator is only a method for identifying target operators, and does not mean that the results of each identified target operator's sub-computational parts on different computing nodes are all merged.

[0042] In real-world production scenarios, besides the initial data, the target model often includes other input data used for the computation of each operator. The initial state of these other input data may be unsplit. To allow for the split computation of each operator across multiple nodes, the other input data can also be split during the computation process. Therefore, in one embodiment, the target data further includes second data input to the first operator. Based on the splitting method of the first data, the type of the second data, and the type of the first operator, it is determined whether a first equivalent method exists. If a first equivalent method exists, the splitting method of the second data, the k sub-computation parts corresponding to the first operator on the k computation nodes, and the combination method of the output results of the k sub-computation parts corresponding to the first operator on the k computation nodes are determined based on the first equivalent method.

[0043] Depending on the implementation method, the specific method for determining whether a first equivalent method exists can vary. In one specific implementation, for example, if the first operator is a dot product operator, the first data is a first matrix, the second data is a second matrix, and the first data is split by dividing the first matrix into k first sub-matrices according to its dimension, then it is determined that a first equivalent method exists corresponding to the first operator. Furthermore, the second matrix can be split into k second sub-matrices according to the first equivalent method, such that the k first sub-matrices correspond one-to-one with the plurality of second sub-matrices, and the sum of the dot product results between each first sub-matrix and its corresponding second sub-matrix is ​​equal to the dot product result between the first matrix and the second matrix.

[0044] In the example above where the first intermediate representation based on StableHLO is obtained from JAX-based Python code, for instance, the parameter arg0 of the dot_general operator is a vector X. Figure 5 In the diagram, vectors X1 and X2, for example, come from participant A and participant B, respectively. X1 and X2 can be combined to form the complete vector X. Vector W is the second data. Based on the splitting method of the first data (splitting by dimension), W can be split into W1 and W2 by dimension, such that X1·W1 + X2·W2, where (·) represents the dot product. Thus, for example, X1·W1 is calculated at computation node 1 (for example, belonging to participant A) (i.e., the sub-computation part of the dot product operator at computation node 1 is X1·W1), and X2·W2 is calculated at computation node 2 (for example, belonging to participant B) (i.e., the sub-computation part of the dot product operator at computation node 2 is X2·W2). The set of sub-computation results of computation node 1 and computation node 2 is obtained by element-wise addition.

[0045] In another specific implementation, if the first operator is a dot product operator, the first data is a first matrix, the second data is a second matrix, and the first data is split by dividing the first matrix into k first sub-matrices with the same dimension as the first matrix, and the sum of the values ​​of the elements at the same position in the k first sub-matrices equals the value of the elements at the same position in the first matrix, then it is determined that a first equivalent method corresponding to the first operator exists. Figure 6 In the diagram, vectors X3 and X4, for example, come from participant A and participant B, respectively. X3 and X4 can be combined to form the complete vector X (first data). Specifically, X3 and X4 have the same dimension as X, and the sum of corresponding elements in X3 and X4 equals the corresponding element in X. For example, the sum of corresponding elements (at the same position) x11 and x21 in X3 and X4 equals the value of the corresponding element (at the same position) x1 in X. Vector W is the initial, unsplit, full data (second data). Based on the splitting method of X (element-wise splitting) and W, the equivalent computation method of the dot product operator on multiple computation nodes can be determined, for example, X·W = X3·W + X4·W. Thus, X3·W can be calculated, for example, on computation node 1 (e.g., belonging to participant A), and X4·W can be calculated, for example, on computation node 2 (e.g., belonging to participant B). The combination of the two calculation results can be element-wise addition.

[0046] In different examples, determining whether the first operator has a second equivalent method for computation on k computation nodes according to the combination method can vary. In one example, if the type of the first operator is a vector element-wise addition operator, and the combination method of the computation results of the k sub-computation parts of the first operator's predecessor is vector element-wise multiplication, then it is determined that the first operator does not have a second equivalent method. In the example above of obtaining the first intermediate representation based on StableHLO according to JAX-based Python code, for example, the add operator after the exponential operator, since the exponential operator is the predecessor of the add operator (the input data of the add operator includes the output result of the exponential operator "%6"), and the computation method of the exponential operator can be represented as e Y =e Y1 e Y2 Where Y is the input data, Y = Y1 + Y2, and Y1 and Y2 are the partitions of the input data at different computing nodes, e Y1 and e Y2The exponential operator consists of sub-computational parts on different computational nodes. However, since the combination of the computational results of the different sub-computational parts of the exponential operator is element-wise multiplication, it can be determined that there is no equivalent computational method for the add operator that calculates on multiple computational nodes based on this combination method and obtains the correct computational result.

[0047] Then, in step S403, based on the n target operators obtained in step S401, k first computation parts are determined; based on the k first computation parts, k second codes are generated. The k second codes are executed on the k computation nodes respectively to run the target model. In one embodiment, the k computation nodes may belong to k participants. In different specific embodiments, the value of k may be different, and the k participants may be different specific participants, for example, they may be k participants with their own privacy data; this specification does not limit this.

[0048] Depending on the implementation method, the specific way of determining the k first computation parts can be different. In the above embodiment where multiple (e.g., k) sub-computation parts are determined corresponding to multiple (e.g.,) target operators, the k first computation parts are determined based on the k sub-computation parts corresponding to the n target operators. The k first computation parts can each correspond to k computation nodes, and the first computation part corresponding to each computation node can include the sub-computation parts of the n target operators corresponding to that computation node.

[0049] In a plurality of operators, those other than the target operator cannot be assigned to different computing nodes for computation. The computations corresponding to these other operators can be performed on a single computing node. This computing node can be one of the multiple computing nodes executing multiple first computation parts, or a computing node other than those multiple computing nodes. Therefore, in one embodiment, k first computation parts can be determined based on the n target operators; and second computation parts can be determined based on the other operators besides the target operators among the plurality of operators. Furthermore, multiple second codes can be generated based on the k first computation parts; and third codes can be generated based on the second computation parts. The k second codes are executed on the k computing nodes respectively, and the third codes are executed on the first nodes among or outside the k computing nodes.

[0050] In one embodiment, a fourth code may also be generated, which merges the calculation results of the k first computational parts and inputs the merged result into the second computational part; the fourth code is executed on a second node among or other than the k computational nodes. In different specific embodiments, the second node executing the fourth code may be the same as or different from the first node executing the third code in the previous embodiment.

[0051] In one embodiment, after determining k first computational parts and second computational parts, a second intermediate representation indicating the k first computational parts and second computational parts can be generated. The second intermediate representation further includes a merging operator that merges the computation results of the k first computational parts and inputs the merged result into the second computational part. Based on the second intermediate representation, the k second codes and the third code are generated. Compared to the first intermediate representation, since the second intermediate representation indicates both the first and second computational parts, it allows for the generation of second codes (which can be executed on the k computational nodes respectively) for executing multiple (e.g., k) first computational parts, and third codes for executing the second computational parts, as shown below. Figure 7 As shown.

[0052] Depending on the implementation method, the specific form of the generated second intermediate representation can vary. In one example, parameters can be added to the operators of the first intermediate representation to indicate operator partitioning. In the example above of obtaining the first intermediate representation based on StableHLO using JAX-based Python code, the generated second intermediate representation might specifically be, for example, "

[0053] %arg0:{tensor<2x1xi32>,tensor<2x3xi32>}

[0054] %arg1:

[0055] %arg2:

[0056] # Case where one of the inputs in the dotgeneral splitting rule is concat splitting

[0057] %0=dot_general{{%arg0,%arg1},(tensor<2x1xi32>,tensor<1x5xi32>)->tensor<2x5xi32>,(tensor<2x3xi32>,tensor<3x5xi32>)->tensor<2x5xi32>,+}

[0058] %1=convert{{%arg2},(tensor <i32>)->tensor <i32>}

[0059] %2=broadcast{{%1},(tensor <i32>)->tensor<2x5xi32>}

[0060] Since %0 is a splitting operation, performing an addition operation on %0 requires dividing the other operand by the number of splits.

[0061] %intermediate0=divide{{%2,%split_cnt},(tensor<2x5xi32>,tensor <2> )->tensor<2x5xi32>}

[0062] %3=add{{%0,%intermediate0},(tensor<2x5xi32>,tensor<2x5xi32>)->tensor<2x5xi32>,(tensor<2x5xi32>,tensor<2x5xi32>)->tensor<2x5xi32>,+}

[0063] #3, 4, and 5 are all in the form of add splitting. Each split part performs the next, convert, and exponential operations respectively. After the exponential operation, it is transformed into a multiplicative splitting form according to the splitting rules.

[0064] %4=negate{{%3},(tensor<2x5xi32>)->tensor<2x5xi32>,(tensor<2x5xi32>)->tensor<2x5xi32>,+}

[0065] %5=convert{{%4}(tensor<2x5xi32>)->tensor<2x5xf32>,(tensor<2x5xi32>)->tensor<2x5xf32>,+}

[0066] %6=exponential{{%5},(tensor<2x5xf32>)->tensor<2x5xf32>,(tensor<2x5xf32>)->te nsor<2x5xf32>,x}

[0067] %cst=constant{dense<1.000000e+00>,tensor <f32>}

[0068] %7=broadcast_in_dim{{%cst},(tensor <f32>)->tensor<2x5xf32>}

[0069] The variable 6, which is split in a multiplication operation, cannot be further added in its split form. Therefore, a multiplication merging operator, `multiply_merge`, is added to merge the split variables, allowing subsequent calculations to proceed normally.

[0070] %intermediate1=multiply_merge{{%6},(tensor<2x5xf32>,tensor<2x5xf32>)->tensor<2x5xf32>}

[0071] %8=add{{%7,%intermediate1},(tensor<2x5xf32>,tensor<2x5xf32>)->tensor2x5xf32}

[0072] %cst_0=constant{dense<1.000000e+00>,tensor <f32>}

[0073] %9=broadcast_in_dim{{%cst_0},(tensor <f32>)->tensor<2x5xf32>}

[0074] %10=divide{{%9,%8},(tensor<2x5xf32>,tensor<2x5xf32>)->tensor<2x5xf32>}”

[0075] Among them, the dot_general operator, convert operator, broadcast operator, negate operator, convert operator, and exponential operator are all target operators. A target operator can indicate how its input parameters are partitioned across different computing nodes, and how the computation results of the sub-computation parts on different computing nodes are combined. For example, the dot_general operator (used to compute the variable "%0") indicates how its input parameters (e.g., arg0, arg1) are partitioned across different computing nodes, and how the computation results of the dot_general operator's sub-computation parts on different computing nodes are combined. For example, arg0 is partitioned into two vectors (or tensors representing vectors) tensor<2x1xi32> and tensor<2x3xi32> on two computing nodes, and arg1 is partitioned into two vectors (or tensors representing vectors) tensor<1x5xi32> and tensor<3x5xi32> on two computing nodes. The `dot_general` operator has two sub-computational parts on the two computational nodes: `tensor<2x1xi32>·tensor<1x5xi32>` (e.g., indicated by `(tensor<2x1xi32>,tensor<1x5xi32>)->tensor<2x5xi32>`) and `tensor<2x3xi32>·tensor<3x5xi32>` (e.g., indicated by `(tensor<2x3xi32>,tensor<3x5xi32>)->tensor<2x5xi32>`). The results of the two sub-computational parts are combined element-wise by adding "+". Furthermore, when the input data of the target operator can participate in the computation on different computational nodes as full data, the computational results of its sub-computational parts on different computational nodes can also be calculated based on the full data. These target operators may also not explicitly indicate how their input parameters are partitioned on different computational nodes, or how the computational results of the sub-computational parts on different computational nodes are combined. For example, the `convert` operator (used to calculate the variable "%1") can be used because its input data can be used in calculations on different computing nodes as full data. The results of its sub-calculations on different computing nodes can also be calculated based on the full data. Therefore, it is not necessary to explicitly specify how the input parameters are partitioned on different computing nodes, or how the results of the sub-calculations on different computing nodes are combined.

[0076] To improve the efficiency of generating the second and third codes, in one embodiment, a tree structure consisting of multiple tree nodes and directed edges between the tree nodes can be generated based on the second intermediate representation. The tree nodes represent operators in the second intermediate representation, and the directed edges represent data transfer relationships between operators in the second intermediate representation. The first tree node corresponding to the merging operator in the tree structure is found, and the k second codes are generated based on the subtree of the first tree node. The third code is generated based on the other nodes in the tree structure besides the first tree node and its subtree.

[0077] In the example above where the second intermediate representation is generated based on the first intermediate representation of StableHLO, the second intermediate representation can be used to generate... Figure 8 The tree structure is shown. Each tree node corresponds to an operator in the second intermediate representation. In some examples, some tree nodes may also correspond to the input data of the target model (e.g., nodes corresponding to arg0 and arg1) or data defined within the model (e.g., the cst node). Directed edges between tree nodes can also represent the data transfer relationship between the input data or data defined within the target model and the operators. The directed edges between tree nodes respectively represent the data transfer relationship between operators in the second intermediate representation. Subsequently, second, third, and fourth codes can be generated based on this tree structure. Specifically, for example, starting from the root node of the tree structure (e.g., the tree node corresponding to the operator that determines the model output), the tree structure can be traversed to find the tree node corresponding to the merging operator. Multiple second codes are generated based on the subtree of that tree node. Third codes are generated based on the tree node and other nodes outside its subtree in the tree structure. The specific algorithm for traversing the tree structure can differ in different implementations. In one implementation, for example, a depth-first traversal algorithm can be used to traverse the tree nodes of the tree structure. The specific algorithm for traversing the tree structure can differ in different implementations. In one implementation, the tree nodes of the tree structure can be traversed, for example, using a depth-first traversal algorithm.

[0078] In different specific examples, such as depending on the underlying machine learning framework, the generated second code can be different. Similarly, the generated third or fourth code can also be different. In a specific example, the generated second, third, and fourth codes could be...

[0079]

[0080] return result_dict”

[0081] def server(param_aggregate_result):

[0082] param_intermediate1=param_aggregate_result["param_intermediate1"]

[0083] param_cst = 1.000000e+00

[0084] param_8=jnp.add(param_cst,param_intermediate1)

[0085] param_cst_0 = 1.000000e+00

[0086] param_10=jnp.divide(param_cst_0,param_8)

[0087] return param_10”

[0088] In this implementation, the function `client` is the second type of code, used to run on multiple (e.g., k) computing nodes. The function `server` is the third type of code, used to execute on one or more computing nodes that execute the second code. The function `aggregate` is the fourth type of code, used to execute on one or more computing nodes that execute the second code. In different implementations, the computing nodes executing the third and fourth codes can be the same or different. The multiple second codes, as well as the third and fourth codes, can be executed separately on their respective computing nodes to run the target model, or in other words, to complete the computation process of the target model.

[0089] According to another embodiment, an apparatus for generating code that is executed on multiple nodes to run a target model is also provided. Figure 9 This is an architectural diagram of an apparatus for generating code that executes on multiple nodes to run a target model, as described in an embodiment of this specification. Figure 9 As shown, the device 900 includes:

[0090] The acquisition unit 91 is configured to acquire first code for running a target model on a single computing node, convert the first code into a first intermediate representation including a plurality of operators calculated sequentially; the target data of the input target model includes at least the first data that has been split, and based on the splitting method of the first data, the type of the plurality of operators and the calculation order of the plurality of operators, determine n target operators that can be decomposed and calculated sequentially on k computing nodes respectively.

[0091] The generation unit 92 is configured to determine k first computation parts based on the n target operators; and generate k second codes based on the k first computation parts, wherein the multiple second codes are executed on the k computation nodes respectively to run the target model.

[0092] This specification also provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform any of the methods described above.

[0093] This specification also provides a computing device, including a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements any of the methods described above.

[0094] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0095] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0096] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. A typical implementation device is a server system. Of course, this application does not exclude the possibility that, with the future development of computer technology, the computer implementing the functions of the above embodiments can be, for example, a personal computer, a laptop computer, an in-vehicle human-machine interaction device, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0097] While one or more embodiments of this specification provide the steps of operation for the methods described in the embodiments or flowcharts, more or fewer steps may be included based on conventional or non-inventive means. The order of steps listed in the embodiments is merely one possible order of execution among many steps and does not represent the only possible order. In actual device or end product execution, the methods shown in the embodiments or drawings may be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment, or even a distributed data processing environment). The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitations, the presence of other identical or equivalent elements in a process, method, product, or apparatus that includes said elements is not excluded. For example, the use of terms such as "first," "second," etc., is to denote names and does not indicate any particular order.

[0098] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, when implementing one or more of these specifications, the functions of each module can be implemented in one or more software and / or hardware components, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.

[0099] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0100] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0101] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0102] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0103] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0104] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage, graphene storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0105] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0106] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0107] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, system embodiments are basically similar to method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. In the description of this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0108] The above description is merely an embodiment of one or more embodiments of this specification and is not intended to limit the scope of these embodiments. Various modifications and variations can be made to these embodiments by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims.

Claims

1. A method for generating code that executes across multiple nodes to run a target model, comprising: Obtain first code for running the target model on a single computing node, and transform the first code into a first intermediate representation including multiple operators computed sequentially; The target data input to the target model includes at least the first data that has been split. Based on the splitting method of the first data, the type of the plurality of operators, and the calculation order of the plurality of operators, n target operators that can be decomposed and calculated sequentially on k computing nodes are determined from the plurality of operators. Based on the n target operators, k first computational parts are determined; based on the k first computational parts, k second codes are generated, and the multiple second codes are executed on the k computational nodes respectively to run the target model.

2. The method according to claim 1, wherein, Based on the n target operators, k first computational parts are determined, including: Based on the n target operators, k first computation parts are determined; and based on the other operators besides the target operators among the plurality of operators, second computation parts are determined; Based on the k first computational parts, k second codes are generated. These second codes are executed on multiple computational nodes, including: Based on the k first calculation parts, generate k second codes; Furthermore, based on the second calculation part, a third code is generated, wherein the k second codes are executed on the k computing nodes respectively, and the third code is executed on a first node among or outside the k computing nodes.

3. The method according to claim 2, further comprising: A fourth code is generated, which merges the calculation results of the k first calculation parts and inputs the merged result into the second calculation part; The fourth code is to be executed on a second node, either among or outside the k computing nodes.

4. The method according to claim 1, wherein, Based on the splitting method of the first data, the types of the plurality of operators, and the calculation order of the plurality of operators, n target operators that can be sequentially decomposed and calculated are determined from the plurality of operators, including: According to the calculation order of the plurality of operators, it is determined whether each of the plurality of operators is a target operator. The input data of at least one of the plurality of operators includes the first data. Determining whether any first operator among the plurality of operators is a target operator includes: If the first operator does not have a preceding operator, the output of the preceding operator is used to determine the input data of the first operator. Based on whether the first operator has a first equivalent method for calculating the divided input data on k computing nodes, it is determined whether the first operator is a target operator. If the first equivalent method exists, the first operator is determined to be a target operator, and multiple sub-computation parts corresponding to the first operator on the k computing nodes are determined based on the first equivalent method. If the first equivalent method does not exist, the first operator is determined not to be a target operator. If the first operator has a preceding operator, and the preceding operator of the first operator does not include a non-target operator, based on the combination of the calculation results of the multiple sub-computation parts corresponding to the preceding operator of the first operator on the k computation nodes, and the type of the first operator, it is determined whether the first operator has a second equivalent method for calculation on the k computation nodes according to the combination method; if the second equivalent method exists, the first operator is determined to be a target operator, and the k sub-computation parts corresponding to the first operator on the k computation nodes are determined according to the second equivalent method; if the second equivalent method does not exist, the first operator is not a target operator. If the first operator has a preceding operator and the preceding operator of the first operator includes a non-target operator, then the first operator is not a target operator.

5. The method according to claim 4, wherein, Based on the n target operators, k first computational parts are determined, including: Based on the k sub-computation parts corresponding to the n target operators, k first computation parts are determined.

6. The method according to claim 4, wherein, The target data also includes second data input to the first operator; Whether the first operator is a target operator is determined based on whether the first operator has a first equivalent method for computing on k computing nodes according to the divided input data; If the first equivalent method exists, then the first operator is determined to be the target operator, and the k sub-computation parts corresponding to the first operator on the k computation nodes are determined according to the first equivalent method, including: Based on the splitting method of the first data, the type of the second data, and the type of the first operator, determine whether the first equivalent method exists; If the first equivalent method exists, the splitting method of the second data, the k sub-computation parts corresponding to the first operator on the k computing nodes, and the combination method of the output results of the k sub-computation parts corresponding to the first operator on the k computing nodes are determined according to the first equivalent method.

7. The method according to claim 6, wherein, Based on the splitting method of the first data, the type of the second data, and the type of the first operator, the first equivalent method is determined, including: If the first operator is a dot product operator, the first data is a first matrix, the second data is a second matrix, and the first data is split in a way that splits the first matrix into k first sub-matrices according to the dimension, then it is determined that there exists a first equivalent method corresponding to the first operator; Determining the splitting method of the second data according to the first equivalent method includes: The second matrix is ​​split into k second sub-matrices according to the first equivalent method, such that the k first sub-matrices correspond one-to-one with the k second sub-matrices, and the sum of the dot product results between the k first sub-matrices and their corresponding second sub-matrices is equal to the dot product result between the first matrix and the second matrix.

8. The method according to claim 6, wherein, Based on the splitting method of the first data, the type of the second data, and the type of the first operator, the first equivalent method is determined, including: If the first operator is a dot product operator, the first data is a first matrix, the second data is a second matrix, and the first data is split in such a way that the first matrix is ​​split into k first sub-matrices with the same dimension as the first matrix, and the sum of the values ​​of the elements at the same position in the k first sub-matrices is equal to the value of the elements at the same position in the first matrix, then it is determined that there exists a first equivalent method corresponding to the first operator.

9. The method according to claim 4, wherein, Based on the combination of the calculation results of the k sub-computation parts corresponding to the preceding operator of the first operator on the k computation nodes, and the type of the first operator, determine whether the first operator has a second equivalent method for calculation on the k computation nodes according to the combination method, including: If the first operator is a vector element-wise addition operator, and the combination of the calculation results of the k sub-computation parts of the first operator's predecessor is vector element-wise multiplication, then it is determined that the first operator does not have the second equivalent method.

10. The method according to claim 2, wherein, Based on the k first calculation parts, generate k second codes; And, based on the second calculation part, generate third code, including: After determining k first computation parts and second computation parts, a second intermediate representation is generated that indicates the k first computation parts and the second computation parts. The second intermediate representation also includes a merging operator that merges the computation results of the k first computation parts and inputs the merged result into the second computation part. Based on the second intermediate representation, the k second codes and the third code are generated.

11. The method according to claim 10, wherein, Based on the second intermediate representation, the generation of the k second codes and the third code includes: Based on the second intermediate representation, a tree structure consisting of multiple tree nodes and directed edges between the tree nodes is generated. The tree nodes represent operators in the second intermediate representation, and the directed edges represent data transfer relationships between operators in the second intermediate representation. Find the first tree node corresponding to the merge operator in the tree structure, generate the k second codes based on the subtree of the first tree node, and generate the third code based on the other nodes in the tree structure other than the first tree node and its subtree.

12. The method according to claim 1, wherein, The multiple operators include various types of operators such as dot product, vector element-wise multiplication, vector element-wise division, vector element-wise addition, vector element type conversion, natural exponentiation, vector element negation, vector element aggregation, and vector maximum element value taking operator.

13. The method according to claim 1, wherein, The k computing nodes belong to the k participating parties respectively.

14. The method according to claim 1, wherein, The first intermediate representation is an intermediate representation based on a stable set of higher-order operations.

15. The method according to claim 1, wherein, The target model is one of the following: logistic regression model, Transformer model, or deep factorization machine.

16. An apparatus for generating code that executes on multiple nodes to run a target model, comprising: The acquisition unit is configured to acquire first code for running a target model on a single computing node, convert the first code into a first intermediate representation including a plurality of operators calculated sequentially; input the target data of the target model, which includes at least the first data that has been split, and determine n target operators that can be decomposed and calculated sequentially on k computing nodes respectively, based on the splitting method of the first data, the type of the plurality of operators and the calculation order of the plurality of operators. The generation unit is configured to determine k first computation parts based on the n target operators; and generate k second codes based on the k first computation parts, wherein the multiple second codes are executed on the k computation nodes respectively to run the target model.

17. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1-14.

18. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-15.

19. A computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-15.

Images