Method and device for generating code executed at multiple nodes to run target model

Abstract

The embodiment of the invention provides a method and device for generating a code executed at multiple nodes to run a target model, and the method comprises the steps: obtaining a first code used for running the target model on a single calculation node, and converting the first code into a first intermediate representation comprising a plurality of operators which are calculated in sequence; target data of the input target model at least comprises split first data, and n target operators which can be decomposed and calculated on the k calculation nodes in sequence are determined from the multiple operators according to a splitting mode of the first data, types of the multiple operators and a calculation sequence of the multiple operators; determining k first calculation parts according to the n target operators; based on the k first calculation parts, k second codes are generated, and the multiple second codes are used for being executed at the k calculation nodes respectively to run the target model.

Description

Technical Field

[0001] The embodiments of this specification belong to the field of machine learning, and more particularly to a method and apparatus for generating code that is executed on multiple nodes to run a target model. Background Art

[0002] A machine learning model is a computing model that automatically learns and improves performance through data training. It can extract patterns, rules, and relationships from large amounts of data in order to perform tasks such as prediction, classification, and clustering of new data. It is usually easier to run a machine learning model on a single computing node, but due to the limited computing resources (such as memory) on a single computing node and the privacy protection of data from different participants, it is often desirable to run the machine learning model with the support of computing power or data on multiple computing nodes. However, existing solutions for obtaining machine learning models on multiple computing nodes still have problems such as requiring a large amount of manpower or insufficient protection of data privacy. Summary of the invention

[0003] The purpose of the present invention is to provide a method and device for generating code for execution on multiple nodes to run a target model, which can automatically obtain code for execution on computing nodes of multiple parties to run the target model and protect the privacy data of multiple parties.

[0004] To achieve the above-mentioned purpose, the first aspect of this specification provides a method for generating code to be executed on multiple nodes to run a target model, including: obtaining a first code for running the target model on a single computing node, and converting the first code into a first intermediate representation including multiple operators calculated in sequence; the target data of the input target model at least includes split first data, and according to the splitting method of the first data, the types of the multiple operators and the calculation order of the multiple operators, n target operators that can be decomposed and calculated on k computing nodes in sequence are determined from the multiple operators; according to the n target operators, k first calculation parts are determined; based on the k first calculation parts, k second codes are generated, and the multiple second codes are used to be executed separately on the k computing nodes to run the target model.

[0005] According to a second aspect of the present specification, there is provided an apparatus for generating codes for execution on multiple nodes to run a target model, comprising: an acquisition unit, configured to acquire a first code for running the target model on a single computing node, and convert the first code into a first intermediate representation including a plurality of operators that are calculated sequentially; the target data of the input target model at least includes split first data, and according to the splitting method of the first data, the types of the plurality of operators and the calculation order of the plurality of operators, n target operators that can be decomposed and calculated sequentially on k computing nodes are determined from the plurality of operators; a generation unit, configured to determine k first calculation parts according to the n target operators; and based on the k first calculation parts, generate k second codes, and the plurality of second codes are used to be executed separately on the k computing nodes to run the target model.

[0006] A third aspect of the present specification provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method described in the first aspect.

[0007] A fourth aspect of the specification provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in the first aspect is implemented.

[0008] The fifth aspect of this specification provides a computer program product, including a computer program / instruction, which implements the steps of the method described in the first aspect when executed by a processor.

[0009] In the scheme for generating codes for executing on multiple nodes to run a target model provided in the embodiments of this specification, a first code for running a target model on a single computing node can be obtained, and the first code can be converted into a first intermediate representation including multiple operators that are calculated in sequence; the target data of the input target model is obtained, and the target data at least includes the split first data, and n target operators that can be decomposed and calculated on k computing nodes in sequence are determined from multiple operators according to the first data, the types of multiple operators, and the calculation order of multiple operators. According to the n target operators, k first calculation parts are determined; based on the k first calculation parts, n second codes are generated, and the n second codes are used to execute on multiple computing nodes respectively to run the target model. Through this method, it is possible to automatically obtain codes for executing on computing nodes of multiple parties to run the target model and protecting the privacy data of multiple parties during the running process. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0011] Figure 1 is a schematic diagram of a scheme for running machine learning models on multiple nodes;

[0012] Figure 2 This is a diagram of another approach to running machine learning models on multiple nodes.

[0013] Figure 3 is a schematic diagram of a method for generating codes executed on multiple nodes to run a target model in an embodiment of this specification;

[0014] Figure 4 is a flowchart of a method for generating codes executed on multiple nodes to run a target model in an embodiment of this specification;

[0015] Figure 5 is a schematic diagram of splitting a point product operator in an embodiment of this specification;

[0016] Figure 6 is a schematic diagram of another type of splitting of a dot product operator in an embodiment of this specification;

[0017] Figure 7 is a schematic diagram of obtaining multi-node execution code through a second intermediate representation in an embodiment of this specification;

[0018] Figure 8 is a schematic diagram of determining a second code according to a structure tree in an embodiment of this specification;

[0019] Fig. 9 It is an architectural diagram of an apparatus for generating codes to be executed on multiple nodes to run a target model in one embodiment of the present specification. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0021] As mentioned above, a machine learning model is a computing model that automatically learns and improves performance through data training. It can extract patterns, rules and relationships from a large amount of data in order to perform tasks such as prediction, classification and clustering of new data. It is usually easier to run a machine learning model on a single computing node, but due to the limited computing resources (such as memory) on a single computing node and the privacy protection of data from different participants, it is often hoped to run the machine learning model with the support of computing power or data on multiple computing nodes. However, the existing solutions for obtaining machine learning models on multiple computing nodes still have problems such as consuming a large amount of manpower or insufficient protection of data privacy.

[0022] Figure 1 is a schematic diagram of a scheme for running machine learning models on multiple nodes. Figure 1 As shown, for example, if there is only limited local memory on a single working node, and the scale of the training data set is too large to be stored in the local memory of a single working node, the data set needs to be divided and allocated to each working node, and then the working node trains the model based on the complete model and the local data allocated to each node. However, the problem with this solution is that each computing node saves all model parameters, wasting a lot of storage resources. Moreover, in the calculation process of each computing node, more computing resources will be consumed because of the need to process all model parameters, such as the computing time of the central processing unit (CPU) or the graphics processing unit (GPU). Especially when the hardware storage resources or computing resources of the computing node are limited, the operation efficiency of the machine learning model will be low or even difficult to run.

[0023] Figure 2 is a diagram of another approach to running machine learning models on multiple nodes. Figure 2 As shown, the network parameters of each layer of the neural network of the machine learning model are saved to different computing nodes respectively. For example, the network parameters of each layer of the neural network are stored in a computing node, and each computing node processes the calculation process of the parameters of the neural network layer of this node respectively, and sends the processing results to the computing node corresponding to the next layer of the neural network layer of the neural network layer of this node, until the calculation of each layer of the neural network layer of the machine learning model is completed at the computing node corresponding to each layer. However, the problem with this solution is that when the data involved in the model calculation includes private data from multiple participants, the security of the private data of multiple participants is usually not guaranteed. For example, the working node with the first layer of the neural network usually receives data from all parameter parties.

[0024] There is another solution, which is to divide the parameters of each neural network layer into multiple computing nodes. For example, the parameters of each layer are equally divided into several parts, and each computing node stores a part of the parameters of all layers equally divided into several parts. The calculation of each neural network layer is completed through the interaction between multiple computing nodes. This division scheme is also called the tensor division scheme. However, the problem with this scheme is that it only splits the parameters of the neural network layer, and when the data involved in the model calculation includes private data from multiple participants, the security of the private data of multiple participants is usually not guaranteed. In addition, the tensor division scheme cannot automatically generate the code for the split calculation based on the code of running the machine learning model on a single computing node.

[0025] Another solution is to split the machine learning model by manual calculation, and then place different parts of the split model on different computing nodes to run the machine learning model. This solution requires a lot of manual work and time resources. When the deep learning model is updated, it is often necessary to manually split the model again, which is inefficient.

[0026] In order to solve the above technical problems, an embodiment of this specification proposes a method for generating codes that are executed on multiple nodes to run a target model. Figure 3 FIG. 1 is a schematic diagram of a method for generating code to be executed on multiple nodes to run a target model in an embodiment of this specification. Figure 3 As shown, the code used to run a machine learning model (e.g., a target model) on a single computing node can be converted into an intermediate representation (IR). Since the machine learning model is run on a single computing node, all model parameters and all input data of the target model are usually concentrated on a single computing node. Therefore, this code is also referred to as a centralized computing code in this specification. The intermediate representation of the machine learning model is mainly used to convert models trained by different machine learning frameworks into a unified format for deployment and optimization in different environments. Since the machine learning model can usually have a representation form of a mathematical model, the intermediate representation obtained by conversion usually also contains multiple operators in the mathematical model. Then, according to preset rules, it is determined whether there is an operator (e.g., a target operator) that can be decomposed among the multiple operators. If it exists, the target operator is decomposed into multiple parts, and based on the multiple parts, multiple codes that are executed on multiple computing nodes to run the target model are obtained.

[0027] The advantages of this method are: first, the machine learning model can be automatically split according to the code of the machine learning model that is centrally calculated on a single computing node, and the code running on multiple computing nodes can be generated. Compared with various existing model splitting schemes, the efficiency of obtaining the running code of multiple computing nodes is greatly improved. Second, the model splitting method based on the atomic operator enables this method to support the range of various machine learning models. Third, since the calculation part divided to each computing node usually includes the decomposed sub-computation parts of multiple decomposable operators that can be calculated in sequence, and the sub-computation parts of these decomposable operators on each computing node are calculated independently on each computing node according to the calculation order. When the calculation results of each computing node are merged, it is much more difficult for an attacker to reversely confirm the privacy data of the parameters of each computing node based on the calculation results. Therefore, through this method, when the data participating in the model calculation includes privacy data from multiple participants, the security of privacy data on multiple computing nodes belonging to multiple participants can be protected.

[0028] The following further describes the method provided in the embodiments of this specification for generating code that is executed on multiple nodes to run the target model. Figure 4 1 is a flowchart of a method for generating code to be executed on multiple nodes to run a target model in one embodiment of this specification. Figure 4 As shown, the method at least comprises the following steps:

[0029] Step S401: obtaining a first code for running a target model on a single computing node, converting the first code into a first intermediate representation including a plurality of operators to be calculated sequentially; the target data input into the target model at least includes the split first data, and according to the splitting method of the first data, the types of the plurality of operators and the calculation order of the plurality of operators, determining from the plurality of operators n target operators that can be decomposed and calculated sequentially on k computing nodes respectively;

[0030] Step S403: Determine k first calculation parts according to the n target operators; generate k second codes based on the k first calculation parts, and the multiple second codes are used to be executed respectively on k computing nodes to run the target model.

[0031] First, in step S401, a first code for running a target model on a single computing node is obtained. According to different implementations, the target model may be a different machine learning model. The first code is a code for running a target model on a single computing node. According to different implementations, the first code may be a code written in a different programming language, which is not limited in this specification. For example, in one embodiment, the first code may be a code written in Python. In another embodiment, the first code may also be a code written in other languages. According to different implementations, the target model may also be a different machine learning model. For example, in one embodiment, the target model may be one of a logistic regression model, a Transformer model, and a deep factor decomposition machine.

[0032] After obtaining the first code, the first code can be converted into a first intermediate representation including multiple operators calculated in sequence. In the prior art solution, the code used to run the machine learning model is usually run based on a specific machine learning framework, such as TensorFlow, PyTorch, Caffe (Convolutional Architeure for Fast Feature Embedding, a convolutional architecture for fast feature embedding), JAX (Just-in-Time Automatic Differentiation for High-Performance Machine Learning, real-time automatic differentiation for high-performance machine learning), etc. Because the code is based on a specific machine learning framework, it is often difficult to port it to an environment configured with other machine learning frameworks for operation. The intermediate representation of the machine learning model is usually determined based on the mathematical calculation model of the machine learning model and has nothing to do with the specific machine learning framework. Therefore, the machine learning models based on different machine learning frameworks can be converted into a unified format through the intermediate representation, so that they can be converted into running codes based on the local machine learning framework in the operating environment configured with different machine learning frameworks, so as to deploy the machine learning model in the operating environment configured with different machine learning frameworks. By converting the first code into a first intermediate representation including multiple operators calculated in sequence, the subsequent steps can determine the decomposable operator (target operator) from the multiple operators.

[0033] According to different implementations, the first intermediate representation may be based on different preset intermediate representation instruction sets, and different intermediate representation instructions preset in the intermediate representation instruction set may correspond to different preset operators. In one embodiment, for example, it may be an intermediate representation based on a stable set of high-level operations (StableHLO, Stable High-Level Operations). According to different implementations, the operator types of the operators included in the multiple operators in the first intermediate representation may be different. In one embodiment, the multiple operators may include a point multiplication operator, a vector element-by-element multiplication operator, a vector element-by-element division operator, a vector element-by-element addition operator, a vector element type conversion operator, a natural exponential operator, a vector element negation operator, a vector element aggregation operator, and a vector maximum element value operator.

[0034] According to different implementations, the specific method of generating the first intermediate representation according to the first code may be different. As described above, according to different implementations, the machine learning framework on which the first code is based may be different. Therefore, according to different implementations, the first intermediate representation can be obtained according to the intermediate representation generation tool for different machine learning frameworks. For example, in one embodiment, the machine learning framework on which the first code is based may be TensorFlow or Keras, and the intermediate representation based on StableHLO corresponding to the first code can be obtained through the MLIR-EmitC tool. In another embodiment, the machine learning framework on which the first code is based may be PyTorch, and then the intermediate representation based on StableHLO corresponding to the first code can be obtained through the torch_xla.stable_hlo module.

[0035] In one example, the first code may be a JAX-based Python code for running a logistic regression model, specifically,

[0036]

[0037]

[0038] Among them, dot_general is the dot multiplication operator, convert is the vector element type conversion operator, broadcast is the vector extension operator, negate is the vector element negation operator, add is the vector element-by-element addition operator, divide is the vector element-by-element division operator, broadcast is the vector extension operator, and const is the constant definition instruction.

[0039] After obtaining the first intermediate representation, the target data of the input target model can be obtained, and the target data at least includes the split first data. According to the splitting method of the first data, the types of multiple operators and the calculation order of multiple operators, multiple (for example, n, in different specific examples, the value of n can be different) target operators that can be decomposed and calculated on multiple (for example, k, in different specific examples, the value of k can be different) computing nodes can be determined from multiple operators in sequence. According to different implementations, the split first data can be data from different participants and specific data for different specific businesses, and this specification does not limit this. In one implementation, the different parts of the first data that are split can be, for example, private data of different participants. According to different implementations, the splitting method of the first data can also be different, for example, it can be split according to dimensions or split according to element values. In one example, for example, the complete first data is a vector with a dimension of 2×4, of which a part (for example, part A, whose dimension is 2×1) comes from participant A, for example, and another part (for example, part B, whose dimension is 2×3) comes from participant B, for example. In another example, the complete first data is a vector with a dimension of 2×4, a part of which (for example, part A with a dimension of 2×4) comes from participant A, and another part (for example, part B with a dimension of 2×4) comes from participant B. The value of each element of the first data is equal to the sum or product of the corresponding element of the element in part A and the corresponding element of the element in part B.

[0040] According to different implementations, the specific method of determining n target operators for sequential decomposition calculation from multiple operators may be different. In one embodiment, according to the calculation order of the multiple operators, it is determined whether each operator in the multiple operators is the target operator, and the input data of at least one operator in the multiple operators includes the first data. Wherein, determining whether any first operator in the multiple operators is the target operator includes: if the first operator does not have a predecessor operator, the output result of the predecessor operator is used to determine the input data of the first operator, and determining whether the first operator is the target operator according to whether there is a first equivalent method for calculating the first operator on k computing nodes based on the divided input data; if the first equivalent method exists, determining that the first operator is the target operator, and determining the multiple sub-computation parts corresponding to the first operator on the k computing nodes according to the first equivalent method; if the first equivalent method does not exist, determining that the first operator is not the target operator; if the first operator has the predecessor operator and the first operator's The predecessor operator does not include a non-target operator. According to the combination method of the calculation results of multiple sub-computation parts corresponding to the predecessor operator of the first operator on k computing nodes and the type of the first operator, determine whether the first operator has a second equivalent method of calculating on k computing nodes according to the combination method; if the second equivalent method exists, determine that the first operator is the target operator, and determine the k sub-computation parts corresponding to the first operator on the k computing nodes according to the second equivalent method; if the second equivalent method does not exist, the first operator is not the target operator; if the first operator has the predecessor operator and the predecessor operator of the first operator includes a non-target operator, the first operator is not the target operator.

[0041] For example, in the above example of obtaining the first intermediate representation based on StableHLO according to the Python code based on JAX, the dot multiplication operator (dot_general) that is the first among the multiple operators can be first determined, and according to the splitting method of the first data (for example, arg0) and the second data (for example, arg1), it can be determined that there is an equivalent calculation method (first equivalent method) for the dot multiplication operator, and the combination method of the multiple sub-computation parts corresponding to the dot multiplication operator and the output results of the multiple sub-computation parts can be determined. Then, it is determined in turn whether each operator after the dot multiplication operator is the target operator. For each operator, if the preceding operator of the operator is the target operator, according to the combination method of the output results of the multiple sub-computation parts of the preceding operator and the type of the operator, it is determined whether the operator has an equivalent calculation method (second equivalent method) for operation on multiple nodes. If so, it is determined whether the operator is the target operator. If so, the operator is not the target operator; if the preceding operator of the operator is not the target operator, the operator is not the target operator. For example, for the convert operator after the point multiplication operator, since its preceding point multiplication operator is the target operator, and the output results of the multiple sub-computation parts of the point multiplication operator are combined in a vector element-by-element addition manner, the calculation of the convert operator can be continued on different computing nodes according to the output results of the sub-computation parts corresponding to the preceding operator in the computing node without merging the output results of the multiple sub-computation parts. Specifically, different computing nodes can respectively perform convert calculations (i.e., the sub-computation parts corresponding to the convert operator in the computing node) according to the output results of the sub-computation parts corresponding to the preceding point multiplication operator in the computing node, and the results of the convert calculations performed by different computing nodes can be combined (for example, by a vector element-by-element addition manner) into the calculation results of the complete convert operator. Therefore, the convert operator can also be identified as the target operator. Similarly, for the subsequent broadcast operator, add operator, negate operator, convert operator, and exponential operator, since they can all execute the sub-computation part corresponding to the operator in the current computing node according to the output result of the sub-computation part corresponding to the previous operator in the current computing node, and the results of executing the sub-computation part corresponding to the operator in the current computing node on different computing nodes can be combined into the complete calculation result of the operator, these operators can also be identified as target operators. It should be noted that the results of executing the sub-computation part corresponding to the operator in the current computing node on different computing nodes can be combined into the complete calculation result of the operator, which is only a way to determine the target operator, and does not mean that the results of the sub-computation parts corresponding to different computing nodes of each operator determined as the target operator are all merged.

[0042] In actual production scenarios, in addition to the first data, the target model often has other input data for the calculation of each operator. The initial state of other input data may be unsplit. In order to split the calculation of each operator on multiple nodes, other input data may also be split during the calculation process. Therefore, in one embodiment, the target data also includes the second data input to the first operator. According to the splitting method of the first data, the type of the second data and the first operator, determine whether the first equivalent method exists; if the first equivalent method exists, determine the splitting method of the second data, the k sub-computation parts corresponding to the first operator on the k computing nodes, and the combination method of the output results of the k sub-computation parts corresponding to the first operator on the k computing nodes according to the first equivalent method.

[0043] According to different implementations, the specific method of determining whether there is a first equivalent method may be different. In a specific implementation, for example, if the first operator is a point multiplication operator, the first data is a first matrix, the second data is a second matrix, and the first data is split in a way that the first matrix is ​​split into k first sub-matrices according to the dimension, then it is determined that there is a first equivalent method corresponding to the first operator. Furthermore, the second matrix can be split into k second sub-matrices according to the first equivalent method, so that the k first sub-matrices correspond to the multiple second sub-matrices one by one, respectively, and the sum of the point multiplication results between each first sub-matrix and the corresponding second sub-matrix is ​​equal to the point multiplication result between the first matrix and the second matrix.

[0044] In the above example of obtaining the first intermediate representation based on StableHLO according to the Python code based on JAX, for example, the parameter of the dot_general operator is arg0, which is the vector X. Figure 5 As shown, for example, vector X1 and vector X2 come from participant A and participant B, respectively, and X1 and X2 can be combined into a completed vector X. Vector W is the second data, and according to the splitting method of the first data (split by dimension), W can be split into W1 and W2 according to the dimension, so that X1·W1+X2·W2, where (·) is the dot product. Thus, for example, X1·W1 is calculated on computing node 1 (for example, belonging to parameter party A) (i.e., the sub-computation part of the dot product operator on computing node 1 is X1·W1), and X2·W2 is calculated on computing node 2 (for example, belonging to parameter party B) (i.e., the sub-computation part of the dot product operator on computing node 2 is X2·W2), and the sub-computation results of computing node 1 and computing node 2 are collected in a manner of element-by-element addition.

[0045] In another specific implementation, if the first operator is a point multiplication operator, the first data is a first matrix, the second data is a second matrix, and the first data is split in such a way that the first matrix is ​​split into k first sub-matrices of the same dimension as the first matrix, and the sum of the values ​​of the elements at the same position of the k first sub-matrices is equal to the value of the elements at the same position of the first matrix, then it is determined that there is a first equivalent method corresponding to the first operator. Figure 6 As shown, for example, vector X3 and vector X4 come from participant A and participant B respectively, and X3 and X4 can be combined into a completed vector X (first data). Specifically, X3 and X4 have the same dimension as X, and the sum of corresponding elements in X3 and X4 is equal to the corresponding element in X. For example, the sum of elements x11 and x21 of corresponding elements (same position) in X3 and X4 is equal to the value of corresponding element x1 (same position) in X. Vector W is the initial unsplit full data (second data). According to the splitting method of X (element splitting) and W, the equivalent calculation method of the point multiplication operator on multiple computing nodes can be determined, for example, X·W=X3·W+X4·W. Thus, X3·W can be calculated on computing node 1 (for example, belonging to parameter party A), and X4·W can be calculated on computing node 2 (for example, belonging to parameter party B), and the combination method of the calculation results of the two can be element-by-element addition.

[0046] In different examples, it is determined whether the first operator has a second equivalent method of calculating on k computing nodes according to the combination method, and the specific method may also be different. In one example, if the type of the first operator is a vector element-by-element addition operator, and the combination method of the calculation results of the k sub-calculation parts of the predecessor operator of the first operator is vector element-by-element multiplication, it is determined that the first operator does not have the second equivalent method. In the above example of obtaining the first intermediate representation based on StableHLO according to the JAX-based Python code, for example, the add operator after the exponential operator, since the exponential operator is the predecessor operator of the add operator (the input data of the add operator includes the output result "%6" of the exponential operator), and the calculation method of the exponential operator can be expressed as e Y =e Y1 e Y2 , where Y is the input data, Y = Y1 + Y2, Y1 and Y2 are the divisions of the input data in different computing nodes, e Y1 and e Y2are the sub-computation parts of the exponential operator on different computing nodes. However, since the calculation results of the different sub-computation parts of the exponential operator are combined in an element-by-element multiplication manner, it can be determined that there is no equivalent calculation method for the add operator to calculate on multiple computing nodes according to this combination method and obtain the correct calculation result.

[0047] Then, in step S403, k first calculation parts are determined according to the n target operators obtained in step S401; and k second codes are generated based on the k first calculation parts. The k second codes are used to be executed respectively on the k computing nodes to run the target model. In one embodiment, the k computing nodes may belong to k participants respectively. In different specific embodiments, the value of k may be different, and the k participants may be different specific participants respectively, for example, they may be k participants with their own privacy data, and this specification does not limit this.

[0048] According to different implementations, the specific manner of determining the k first calculation parts may be different. In the above embodiment of determining multiple (for example, k) sub-calculation parts corresponding to multiple (for example,) target operators, the k first calculation parts are determined according to the k sub-calculation parts corresponding to the n target operators. The k first calculation parts may correspond to k computing nodes, respectively, and the first calculation part corresponding to each computing node may include the sub-calculation parts of the n target operators corresponding to the computing node.

[0049] Among the multiple operators, other operators other than the target operator cannot be divided into different computing nodes for calculation. The calculations corresponding to these other operators can be completed on one computing node, which can be a computing node among the multiple computing nodes that execute multiple first computing parts, or a computing node outside the multiple computing nodes. Therefore, in one embodiment, k first computing parts can be determined based on the n target operators; and the second computing part can be determined based on the other operators other than the target operator among the multiple operators. Furthermore, multiple second codes can be generated based on the k first computing parts; and a third code can be generated based on the second computing part, the k second codes are used to be executed respectively on the k computing nodes, and the third code is used to be executed on the first node among or outside the k computing nodes.

[0050] In one embodiment, a fourth code may be generated, the fourth code merging the calculation results of the k first calculation parts and inputting the combined result into the second calculation part; the fourth code is used to be executed at a second node among or outside the k calculation nodes. In different specific embodiments, the second node executing the fourth code may be the same as or different from the first node executing the third code in the previous embodiment.

[0051] In one embodiment, after determining k first calculation parts and second calculation parts, a second intermediate representation indicating the k first calculation parts and the second calculation parts may be generated, wherein the second intermediate representation also includes a merging operator for merging the calculation results of the k first calculation parts and inputting the merging results into the second calculation part; and the k second codes and the third code may be generated according to the second intermediate representation. With respect to the first intermediate representation, since the second intermediate representation indicates the first calculation part and the second calculation part, the second code for executing multiple (for example, k) first calculation parts (which may be executed on k computing nodes respectively) and the third code for executing the second calculation part may be generated according to the second intermediate representation, such as Figure 7 shown.

[0052] According to different implementations, the specific form of the generated second intermediate representation may be different. In one example, a parameter may be added to the operator of the first intermediate representation to indicate the operator partitioning. In the above example of obtaining the first intermediate representation based on StableHLO according to the Python code based on JAX, the generated second intermediate representation may be specifically "

[0053] %arg0:{tensor<2x1xi32>,tensor<2x3xi32>}

[0054] %arg1:

[0055] %arg2:

[0056] #According to the dotgeneral splitting rule, one of the inputs is concat split

[0057] %0=dot_general{{%arg0,%arg1},(tensor<2x1xi32>,tensor<1x5xi32>)->tensor<2x5xi32>,(tensor<2x3xi32>,tensor<3x5xi32>)->tensor<2x5xi32>,+}

[0058] %1=convert{{%arg2},(tensor <i32>)->tensor <i32>}

[0059] %2=broadcast{{%1},(tensor <i32>)->tensor<2x5xi32>}

[0060] #Since %0 is in the add split form, adding %0 requires dividing the other operand by the number of splits

[0061] %intermediate0=divide{{%2,%split_cnt},(tensor<2x5xi32>,tensor <2> )->tensor<2x5xi32>}

[0062] %3=add{{%0,%intermediate0},(tensor<2x5xi32>,tensor<2x5xi32>)->tensor<2x5xi32>,(tensor<2x5xi32>,tensor<2x5xi32>)->tensor<2x5xi32>,+}

[0063] #3, 4, 5 are all in add split form. Each split part performs negate, convert, and exponential operations respectively. After the exponential operation, it is converted into multiplication split form according to the splitting rules.

[0064] %4=negate{{%3},(tensor<2x5xi32>)->tensor<2x5xi32>,(tensor<2x5xi32>)->tensor<2x5xi32>,+}

[0065] %5=convert{{%4}(tensor<2x5xi32>)->tensor<2x5xf32>,(tensor<2x5xi32>)->tensor<2x5xf32>,+}

[0066] %6=exponential{{%5},(tensor<2x5xf32>)->tensor<2x5xf32>,(tensor<2x5xf32>)->te nsor<2x5xf32>,x}

[0067] %cst=constant{dense<1.000000e+00>,tensor <f32>}

[0068] %7=broadcast_in_dim{{%cst},(tensor <f32>)->tensor<2x5xf32>}

[0069] #The multiplication split variable 6 cannot continue to perform addition calculations in the split form, so add a multiplication merge operator multiply_merge to merge the split variables, and then perform normal calculations

[0070] %intermediate1=multiply_merge{{%6},(tensor<2x5xf32>,tensor<2x5xf32>)->tensor<2x5xf32>}

[0071] %8=add{{%7,%intermediate1},(tensor<2x5xf32>,tensor<2x5xf32>)->tensor2x5xf32}

[0072] %cst_0=constant{dense<1.000000e+00>,tensor <f32>}

[0073] %9=broadcast_in_dim{{%cst_0},(tensor <f32>)->tensor<2x5xf32>}

[0074] %10=divide{{%9,%8},(tensor<2x5xf32>,tensor<2x5xf32>)->tensor<2x5xf32>}”

[0075] Among them, the dot_general operator, convert operator, broadcast operator, negate operator, convert operator, and exponential operator are all target operators. The target operator can indicate how its input parameters are divided on different computing nodes, and how the calculation results of the sub-computation parts of different computing nodes are combined. For example, the dot_general operator (used to calculate the variable "%0") indicates how its input parameters (such as arg0, arg1) are divided on different computing nodes, and how the calculation results of the sub-computation parts of the dot_general operator are combined on different computing nodes. For example, arg0 is divided into two vectors (or tensors representing vectors) tensor<2x1xi32>, tensor<2x3xi32> input on two computing nodes, and arg1 is divided into two vectors (or tensors representing vectors) tensor<1x5xi32>, tensor<3x5xi32> input on two computing nodes. The sub-computation parts of the dot_general operator on the two computing nodes are respectively tensor<2x1xi32>·tensor<1x5xi32> (for example, indicated by (tensor<2x1xi32>, tensor<1x5xi32>)->tensor<2x5xi32>), and tensor<2x3xi32>·tensor<3x5xi32> (for example, indicated by (tensor<2x3xi32>, tensor<3x5xi32>)->tensor<2x5xi32>), and the calculation results of the two sub-computation parts are combined by adding "+" to the element-by-element vector. In addition, when the input data of the target operator can be calculated in the form of full data on different computing nodes, the calculation results of the sub-computation parts on different computing nodes can also be calculated based on the full data. These target operators may not explicitly indicate how their input parameters are divided on different computing nodes, and how the calculation results of the sub-computation parts on different computing nodes are combined. For example, the convert operator (used to calculate the variable "%1"), since the input data of the operator can be calculated in different computing nodes in the form of full data, the calculation results of the sub-computation parts in different computing nodes can also be calculated based on the full data. Therefore, it is not necessary to explicitly indicate how the input parameters are divided on different computing nodes and how the calculation results of the sub-computation parts on different computing nodes are combined.

[0076] In order to improve the efficiency of generating the second code and the third code, in one embodiment, a tree structure consisting of multiple tree nodes and directed edges between the tree nodes can also be generated based on the second intermediate representation, the tree nodes represent the operators in the second intermediate representation, and the directed edges represent the data transfer relationship between the operators in the second intermediate representation; the first tree node corresponding to the merge operator in the tree structure is searched, and the k second codes are generated based on the subtree of the first tree node; the third code is generated based on the first tree node in the tree structure and other nodes other than the subtree of the first tree node.

[0077] In the above example of generating a second intermediate representation based on the first intermediate representation of StableHLO, it is possible to generate Figure 8 The structure tree shown. Wherein, each tree node corresponds to each operator in the second intermediate representation. In some examples, some tree nodes may also correspond to the input data of the target model (e.g., the nodes corresponding to arg0 and arg1) or the data defined in the model (e.g., cst nodes). The directed edges between the tree nodes may also represent the input data of the target model or the data defined in the target model, and the data transfer relationship between each operator. The directed edges between the tree nodes correspond to the data transfer relationship between the operators in the second intermediate representation. Thereafter, the second code, the third code, and the fourth code may be generated according to the structure tree. Specifically, the structure tree may be traversed from the root node of the structure tree (e.g., the tree node corresponding to the operator that determines the output result of the model), and the tree node corresponding to the merge operator in the tree structure may be searched. According to the subtree of the tree node, a plurality of second codes may be generated. According to the tree node described in the tree structure and other nodes outside the subtree of the tree node, the third code may be generated. In different embodiments, the specific algorithm for traversing the structure tree may be different. In one embodiment, for example, the tree nodes of the structure tree may be traversed by a depth traversal algorithm. In different embodiments, the specific algorithm for traversing the structure tree may be different. In one implementation, for example, the tree nodes of the structure tree may be traversed by a depth traversal algorithm.

[0078] In different specific examples, for example, the generated second code may be different depending on the machine learning framework based on it. Similarly, the generated third code or fourth code may also be different. In a specific example, the generated second code, third code and fourth code. For example, it may be specifically "

[0079]

[0080] return result_dict"

[0081] def server(param_aggregate_result):

[0082] param_intermediate1=param_aggregate_result["param_intermediate1"]

[0083] param_cst = 1.000000e+00

[0084] param_8=jnp.add(param_cst,param_intermediate1)

[0085] param_cst_0 = 1.000000e+00

[0086] param_10=jnp.divide(param_cst_0,param_8)

[0087] return param_10"

[0088] Among them, the function client is a second code, which is used to run on multiple (for example, k) computing nodes. The function server is a third code, which is used to execute on a computing node among or outside the multiple computing nodes that respectively execute the second code. The function aggregate is a fourth code, which is used to execute on a computing node among or outside the multiple computing nodes that respectively execute the second code. In different embodiments, the computing nodes that execute the third code and the fourth code may be the same or different. Multiple second codes, as well as the third code and the fourth code can be executed separately on the computing nodes to which they belong, to run the target model, or to complete the calculation process of the target model.

[0089] According to an embodiment of another aspect, a device for generating codes executed on multiple nodes to run a target model is also provided. Fig. 9 is an architectural diagram of a device for generating codes to be executed on multiple nodes to run a target model in an embodiment of this specification, such as Fig. 9 As shown, the device 900 includes:

[0090] The acquisition unit 91 is configured to acquire a first code for running a target model on a single computing node, and convert the first code into a first intermediate representation including a plurality of operators that are calculated sequentially; the target data input into the target model at least includes the split first data, and according to the splitting method of the first data, the types of the plurality of operators, and the calculation order of the plurality of operators, n target operators that can be decomposed and calculated sequentially on k computing nodes are determined from the plurality of operators;

[0091] The generation unit 92 is configured to determine k first calculation parts according to the n target operators; and generate k second codes based on the k first calculation parts, wherein the multiple second codes are used to be executed respectively on k computing nodes to run the target model.

[0092] On another aspect, the present specification provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute any one of the above methods.

[0093] On another aspect, the present specification provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, any one of the above methods is implemented.

[0094] In the 1990s, it was very clear whether the improvement of a technology was a hardware improvement (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or a software improvement (improvement of the method flow). However, with the development of technology, many improvements in the method flow today can be regarded as direct improvements in the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs. The original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0095] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0096] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a server system. Of course, the present application does not exclude that with the development of computer technology in the future, the computer that implements the functions of the above embodiments may be, for example, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0097] Although one or more embodiments of the present specification provide method operation steps as described in the embodiments or flow charts, more or less operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiments is only one way of executing the order of many steps, and does not represent the only execution order. When the device or terminal product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment). The term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or equipment including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such process, method, product or equipment. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or equipment including the elements. For example, if the words first, second, etc. are used to represent the name, they do not represent any specific order.

[0098] For the convenience of description, the above devices are described in various modules according to their functions. Of course, when implementing one or more of the present specification, the functions of each module can be implemented in the same or more software and / or hardware, or the module implementing the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0099] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0100] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0101] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0102] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0103] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0104] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage, graphene storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0105] It should be understood by those skilled in the art that one or more embodiments of the present specification may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0106] One or more embodiments of the present specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0107] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. In the description of this specification, the description of the reference term "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of this specification. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, in the absence of mutual contradiction, a person skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0108] The above description is only an example of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. For those skilled in the art, one or more embodiments of the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims.

Claims

1. A method for generating code to be executed on multiple nodes to run a target model, comprising: Obtaining a first code for running a target model on a single computing node, and converting the first code into a first intermediate representation including a plurality of operators that are calculated in sequence; The target data of the input target model at least includes split first data, and according to the splitting method of the first data, the types of the multiple operators and the calculation order of the multiple operators, n target operators that can be decomposed and calculated on k computing nodes in sequence are determined from the multiple operators; According to the n target operators, k first calculation parts are determined; based on the k first calculation parts, k second codes are generated, and the multiple second codes are used to be executed respectively on k computing nodes to run the target model.

2. The method according to claim 1, wherein: According to the n target operators, k first calculation parts are determined, including: Determine k first calculation parts according to the n target operators; and determine second calculation parts according to other operators other than the target operators among the multiple operators; Based on the k first computing parts, k second codes are generated, and the plurality of second codes are used to be respectively executed on a plurality of computing nodes, including: Based on the k first calculation parts, generate k second codes; And, according to the second computing part, a third code is generated, the k second codes are used to be executed respectively on the k computing nodes, and the third code is used to be executed on a first node among or outside the k computing nodes.

3. The method according to claim 2, further comprising: generating a fourth code, the fourth code merging the calculation results of the k first calculation parts and inputting the combined result into the second calculation part; The fourth code is configured to be executed at a second node among or outside the k computing nodes.

4. The method according to claim 1, wherein: According to the splitting method of the first data, the types of the multiple operators, and the calculation order of the multiple operators, n target operators that can be decomposed and calculated in sequence are determined from the multiple operators, including: Determining whether each of the multiple operators is a target operator according to a calculation order of the multiple operators, input data of at least one of the multiple operators includes the first data, wherein determining whether any first operator of the multiple operators is a target operator includes: If the first operator does not have a predecessor operator, the output result of the predecessor operator is used to determine the input data of the first operator, and whether the first operator is a target operator is determined according to whether there is a first equivalent method for calculating the first operator on k computing nodes according to the divided input data; if the first equivalent method exists, the first operator is determined to be the target operator, and multiple sub-computation parts corresponding to the first operator on the k computing nodes are determined according to the first equivalent method; if the first equivalent method does not exist, the first operator is determined not to be the target operator; If the first operator has the preceding operator, and the preceding operator of the first operator does not include a non-target operator, determine whether there is a second equivalent method of calculating the first operator on the k computing nodes according to the combination method according to the combination method of the calculation results of multiple sub-computation parts corresponding to the preceding operator of the first operator on the k computing nodes, and the type of the first operator; if the second equivalent method exists, determine that the first operator is the target operator, and determine the k sub-computation parts corresponding to the first operator on the k computing nodes according to the second equivalent method; if the second equivalent method does not exist, the first operator is not the target operator; If the first operator has the preceding operator, and the preceding operator of the first operator includes a non-target operator, then the first operator is not the target operator.

5. The method according to claim 4, wherein: According to the n target operators, k first calculation parts are determined, including: Determine k first calculation parts according to the k sub-calculation parts corresponding to the n target operators respectively.

6. The method according to claim 4, wherein: The target data also includes second data input into the first operator; Determining whether the first operator is a target operator according to whether there is a first equivalent manner for the first operator to be calculated on k computing nodes according to the divided input data; If the first equivalent method exists, determining the first operator as the target operator, and determining k sub-computation parts corresponding to the first operator on k computing nodes according to the first equivalent method, including: Determining whether the first equivalent method exists according to the splitting method of the first data, the second data, and the type of the first operator; If the first equivalent method exists, determine the splitting method of the second data, the k sub-computation parts corresponding to the first operator on the k computing nodes, and the combination method of the output results of the k sub-computation parts corresponding to the first operator on the k computing nodes according to the first equivalent method.

7. The method according to claim 6, wherein: Determining the first equivalent manner according to the splitting manner of the first data, the second data, and the type of the first operator includes: If the first operator is a point multiplication operator, the first data is a first matrix, the second data is a second matrix, and the first data is split into k first sub-matrices according to the dimension, then it is determined that there is a first equivalent method corresponding to the first operator; Determining a splitting method of the second data according to the first equivalent method includes: According to the first equivalent method, the second matrix is ​​split into k second sub-matrices, so that the k first sub-matrices correspond to the k second sub-matrices one-to-one, and the sum of the dot product operation results between the k first sub-matrices and the corresponding second sub-matrices is equal to the dot product operation result between the first matrix and the second matrix.

8. The method according to claim 6, wherein: Determining the first equivalent manner according to the splitting manner of the first data, the second data, and the type of the first operator includes: If the first operator is a point multiplication operator, the first data is a first matrix, the second data is a second matrix, and the first data is split into k first sub-matrices with the same dimension as the first matrix, and the sum of the values ​​of the elements at the same position of the k first sub-matrices is equal to the value of the elements at the same position of the first matrix, then it is determined that there is a first equivalent method corresponding to the first operator.

9. The method according to claim 4, wherein: Determining, according to a combination method of calculation results of k sub-computation parts corresponding to a preceding operator of a first operator on k computing nodes and a type of the first operator, whether there is a second equivalent method of calculating the first operator on the k computing nodes according to the combination method, comprising: If the type of the first operator is a vector element-by-element addition operator, and the combination method of the calculation results of the k sub-calculation parts of the preceding operator of the first operator is vector element-by-element multiplication, it is determined that the first operator does not have the second equivalent method.

10. The method according to claim 2, wherein: Based on the k first calculation parts, generate k second codes; And, according to the second calculation part, generating a third code, including: After determining k first calculation parts and a second calculation part, generating a second intermediate representation indicating the k first calculation parts and the second calculation part, wherein the second intermediate representation also includes a merging operator for merging calculation results of the k first calculation parts and inputting the merging result into the second calculation part; The k second codes and the third code are generated according to the second intermediate representation.

11. The method according to claim 10, wherein: Generating the k second codes and the third code according to the second intermediate representation comprises: According to the second intermediate representation, generating a tree structure consisting of a plurality of tree nodes and directed edges between the tree nodes, wherein the tree nodes represent operators in the second intermediate representation, and the directed edges represent data transfer relationships between the operators in the second intermediate representation; Search the first tree node corresponding to the merge operator in the tree structure, generate the k second codes according to the subtree of the first tree node, and generate the third code according to the first tree node and other nodes other than the subtree of the first tree node in the tree structure.

12. The method according to claim 1, wherein: The multiple operators include a point multiplication operator, a vector element-by-element multiplication operator, a vector element-by-element division operator, a vector element-by-element addition operator, a vector element type conversion operator, a natural exponential operator, a vector element negation operator, a vector element aggregation operator, and a vector maximum element value operator.

13. The method according to claim 1, wherein: The k computing nodes belong to k participants respectively.

14. The method according to claim 1, wherein: The first intermediate representation is an intermediate representation based on a stable set of high-order operations.

15. The method according to claim 1, wherein: The target model is one of a logistic regression model, a Transformer model, and a deep factorization machine.

16. A device for generating code to be executed on multiple nodes to run a target model, comprising: The acquisition unit is configured to acquire a first code for running a target model on a single computing node, and convert the first code into a first intermediate representation including a plurality of operators that are calculated sequentially; the target data input into the target model at least includes the split first data, and according to the splitting method of the first data, the types of the plurality of operators, and the calculation order of the plurality of operators, n target operators that can be decomposed and calculated sequentially on k computing nodes are determined from the plurality of operators; The generation unit is configured to determine k first calculation parts according to the n target operators; generate k second codes based on the k first calculation parts, and the multiple second codes are used to be executed respectively on k computing nodes to run the target model.

17. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 14.

18. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 15.

19. A computing device comprising a memory and a processor, wherein the memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 15 is implemented.

Images