A data processing method and a data processing device

Abstract

This application discloses a data processing method in the field of artificial intelligence, applicable to distributed parallel model training, such as distributed training of text translation models, speech recognition models, face recognition models, 3D reconstruction models, and virtual reality models. This method supports hybrid parallelism in distributed clusters. The method includes: based on the tensor arrangement of each operator in the deep neural network model, inserting rearrangement operators between operators with input-output dependencies to achieve conversion between different tensor arrangements; inserting rearrangement operators in the slice computation graph; and determining the updated slice computation graph to achieve parallel model training of the deep neural network.

Description

[0001] This application is a divisional application. The original application has the application number 202010231450.7 and the original application date is March 27, 2020. The entire contents of the original application are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence, and more particularly to a data processing method and a data processing device. Background Technology

[0003] Deploying deep learning models across multiple computing devices is one way to train large-scale, complex models. Data parallelism is the most widely used parallel strategy, but as datasets and models grow larger, single-card memory becomes limited, and the number of training devices increases, leading to increased communication overhead. Data parallelism encounters a bottleneck, necessitating hybrid parallelism of data and models.

[0004] Parallelism in deep learning models can be represented by the tensor arrangement of all operators in the model. The tensor arrangement includes the device matrix, tensor shape, and tensor mapping. In the existing Mesh-Tensorflow scheme, tensors can be partitioned along any dimension, but the device matrix of all tensors must be identical.

[0005] Because the Mesh-Tensorflow scheme requires all tensors to have the same device matrix, there are constraints on tensor mapping. For example, the sample dimension of a tensor must be mapped to the same dimension of the device matrix of all tensors, which limits the conversion of various parallel methods. For example, hybrid parallelism of data parallelism and model parallelism cannot be achieved. Summary of the Invention

[0006] This application provides a data processing method applied to a distributed cluster for training a deep neural network model, enabling hybrid parallelism composed of different parallel modes to be implemented in the distributed cluster.

[0007] This application provides a data processing method, which is typically applied to a distributed cluster. The distributed cluster includes multiple data processing devices. This method can be used in one or more data processing devices within the distributed cluster. The method includes: acquiring a deep neural network model, and a tensor arrangement of the input tensors and the output tensors of each operator in the deep neural network model. The tensor arrangement includes a device matrix, a tensor mapping, and a tensor shape. Each element in the device matrix represents a data processing device in the distributed cluster. Multiple data processing devices corresponding to all elements in the device matrix are used to execute the deep neural network model in parallel. The deep neural network model includes a first operator and a second operator, wherein the first operator and the second operator are... In a deep neural network model, two consecutive operators are used, where the output tensor of the first operator is the input tensor of the second operator. The arrangement of the first tensor and the arrangement of the second tensor are inconsistent, wherein the arrangement of the first tensor is the arrangement of the output tensor of the first operator, and the arrangement of the second tensor is the arrangement of the input tensor of the second operator. Based on the arrangement of the input tensor and the arrangement of the output tensor of each operator, a slice computation graph of the data processing device is determined. A reordering operator is determined between the first operator and the second operator, which is used to convert the arrangement of the first tensor to the arrangement of the second tensor. The reordering operator is inserted into the slice computation graph to determine an updated slice computation graph, which is used to indicate the part of the deep neural network model being executed.

[0008] The data processing method provided in this application involves a data processing device acquiring a deep neural network model and the tensor arrangements of all operators therein, including the tensor arrangements of input tensors and output tensors. If two consecutive operators exist where the tensor arrangement of the output tensor of the first operator is inconsistent with the tensor arrangement of the input tensor of the second operator, the slice computation graph obtained based on this tensor arrangement cannot be executed. In this application, the data processing device determines a rearrangement operator between the first and second operators to convert the tensor arrangement of the output tensor of the first operator into the tensor arrangement of the input tensor of the second operator. Thus, a rearrangement operator is inserted into the slice computation graph, and the determined updated slice computation graph can be executed. This scheme can be applied to cases where each operator in a deep neural network model has arbitrary tensor arrangements. By using a rearrangement operator to achieve parallelism conversion, various hybrid parallelism methods can be implemented in a distributed cluster.

[0009] In one possible implementation of the first aspect, the device matrix of the first tensor arrangement is inconsistent with the device matrix of the second tensor arrangement, and / or, the tensor mapping of the first tensor arrangement is inconsistent; determining the rearrangement operator between the first operator and the second operator includes: determining an intermediate tensor arrangement based on the first tensor arrangement and the second tensor arrangement; determining a first reshaping operator based on the first tensor arrangement and the intermediate tensor arrangement, and the tensor arrangement of the input tensor of the first reshaping operator and the tensor arrangement of the output tensor of the first reshaping operator, the first reshaping operator being used to implement the transformation from the first tensor arrangement to the intermediate tensor arrangement, and / or determining a second reshaping operator based on the second tensor arrangement and the intermediate tensor arrangement, and the tensor arrangement of the input tensor of the second reshaping operator and the tensor arrangement of the output tensor of the second reshaping operator, the second reshaping operator being located between the first reshaping operator and the second operator, the second reshaping operator being used to implement the transformation from the intermediate tensor arrangement to the second tensor arrangement.

[0010] The data processing method provided in this application addresses a scenario where, in tensor arrangements, the tensor shapes of the first and second tensor arrangements are identical, but at least one of the device matrix and tensor mappings is inconsistent. Specifically, it determines the re-arrangement operator by generating at least one reshaping operator from an intermediate tensor arrangement. Inserting the reshaping operator between the first and second operators transforms the first tensor arrangement into the second tensor arrangement. Updating the slice computation graph using the generated reshaping operator reduces user workload and improves the efficiency of parallel model training.

[0011] In one possible implementation of the first aspect, the device matrix of the first tensor arrangement is inconsistent with the device matrix of the second tensor arrangement; determining the intermediate tensor arrangement based on the first tensor arrangement and the second tensor arrangement includes: determining an extended device matrix based on the device matrix of the first tensor arrangement and the device matrix of the second tensor arrangement, wherein the element-wise product of the extended device matrix is ​​the same as the element-wise product of the device matrix of the output tensor of the first operator, and the element-wise product of the second device matrix of the input tensor of the second operator, and any element in the device matrix of the first tensor arrangement and the device matrix of the second tensor arrangement... The value is equal to an element in the extended device matrix or the product of at least two elements in the extended device matrix; based on the extended device matrix, a first equivalent tensor arrangement equivalent to the first tensor arrangement and a second equivalent tensor arrangement equivalent to the second tensor arrangement are determined, wherein the device matrix of the first equivalent tensor arrangement is consistent with the device matrix of the second equivalent tensor arrangement; when the tensor shape of the first equivalent tensor arrangement is consistent with the tensor shape of the second equivalent tensor arrangement, the intermediate tensor arrangement includes the first equivalent tensor arrangement and the second equivalent tensor arrangement, wherein the tensor shape is the number of elements in each dimension of the tensor.

[0012] The data processing method provided in this application specifically describes how to determine the reshaping operator when the device matrix of the first tensor arrangement is inconsistent with the device matrix of the second tensor arrangement. Specifically, it involves determining the extended device matrix, finding a first equivalent tensor arrangement equivalent to the first tensor arrangement, and a second equivalent tensor arrangement equivalent to the second tensor arrangement. The device matrix of the first equivalent tensor arrangement is consistent with the device matrix of the second equivalent tensor arrangement. Then, a reshaping operator is generated based on the first and second equivalent tensor arrangements. This scheme automatically generates the reshaping operator to update the slice computation graph, reducing the workload of users designing transformations between tensor arrangements and improving the efficiency of parallel model training.

[0013] The tensor shape of the first equivalent tensor arrangement is consistent with the tensor shape of the second equivalent tensor arrangement, and the tensor mapping of the first equivalent tensor arrangement is inconsistent with the tensor mapping of the second equivalent tensor arrangement; the method further includes: determining one or more tensor mapping transformation operators, the tensor mapping transformation operators including splitting operators, merging operators or communication operators, the tensor arrangement of the input tensors of the one or more tensor mapping transformation operators is consistent with the first equivalent tensor arrangement, the tensor arrangement of the output tensors of the one or more tensor mapping transformation operators is consistent with the second equivalent tensor arrangement, and the one or more tensor mapping transformation operators are used to determine the updated slice computation graph.

[0014] The data processing method provided in this application specifically describes the following situation after device matrix normalization: if the tensor shapes are consistent but the tensor mappings are inconsistent, the data processing device determines a first tensor mapping transformation operator sequence, including one or more tensor mapping transformation operators, to realize the transformation from the tensor mapping of the first equivalent tensor arrangement to the tensor mapping of the second equivalent tensor arrangement. By updating the slice computation graph through the generated one or more tensor mapping transformation operators, the workload of users in designing transformations between tensor arrangements can be reduced, and the efficiency of parallel training of models can be improved.

[0015] In one possible implementation of the first aspect, when the tensor shape of the first equivalent tensor arrangement is inconsistent with the tensor shape of the second equivalent tensor arrangement, the method further includes: normalizing the tensor shape based on the first equivalent tensor arrangement and the second equivalent tensor arrangement to determine a third equivalent tensor arrangement equivalent to the first equivalent tensor arrangement and a fourth equivalent tensor arrangement equivalent to the second equivalent tensor arrangement, wherein the device matrix of the third equivalent tensor arrangement is consistent with the device matrix of the fourth equivalent tensor arrangement, and the tensor shape of the third equivalent tensor arrangement is consistent with the tensor shape of the fourth equivalent tensor arrangement; the intermediate tensor arrangement includes the third equivalent tensor arrangement and the fourth equivalent tensor arrangement.

[0016] The data processing method provided in this application specifically describes a scenario where the tensor shapes are inconsistent after the device matrices of the first and second tensor arrangements are normalized. Tensor shape normalization is required to determine a third equivalent tensor arrangement, which is equivalent to the first equivalent tensor arrangement, and is used to determine the reshaping operator. By generating the reshaping operator to update the slice computation graph, the workload of users designing transformations between tensor arrangements can be reduced, improving the efficiency of parallel model training.

[0017] In one possible implementation of the first aspect, the tensor mapping of the third equivalent tensor arrangement is inconsistent with the tensor mapping of the fourth equivalent tensor arrangement; the method further includes: determining one or more tensor mapping transformation operators, the tensor mapping transformation operators including a splitting operator, a merging operator, or a communication operator, the tensor arrangement of the input tensors of the one or more tensor mapping transformation operators being consistent with the third equivalent tensor arrangement, the tensor arrangement of the output tensors of the one or more tensor mapping transformation operators being consistent with the fourth equivalent tensor arrangement, and the one or more tensor mapping transformation operators being used to determine the updated slice computation graph.

[0018] The data processing method provided in this application specifically describes a scenario where, after the device matrix and tensor shape of the first and second tensor arrangements are normalized, the tensor mappings are still inconsistent. In this case, it is necessary to determine a second tensor mapping transformation operator sequence, including one or more tensor mapping transformation operators, to implement the transformation from a third equivalent tensor arrangement to a fourth equivalent tensor arrangement. Updating the slice computation graph by generating one or more tensor mapping transformation operators can reduce the workload of users designing transformations between tensor arrangements and improve the efficiency of parallel model training.

[0019] In one possible implementation of the first aspect, the device matrix of the first tensor arrangement is consistent with the device matrix of the second tensor arrangement, and the tensor shape of the first tensor arrangement is inconsistent with the tensor shape of the second tensor arrangement; determining the intermediate tensor arrangement based on the first tensor arrangement and the second tensor arrangement includes: determining a fifth equivalent tensor arrangement equivalent to the first tensor arrangement and a sixth equivalent tensor arrangement equivalent to the second equivalent tensor arrangement, wherein the device matrix of the fifth equivalent tensor arrangement is consistent with the device matrix of the sixth equivalent tensor arrangement, and the tensor shape of the fifth equivalent tensor arrangement is consistent with the tensor shape of the sixth equivalent tensor arrangement; the intermediate tensor arrangement includes the fifth equivalent tensor arrangement and the sixth equivalent tensor arrangement.

[0020] The data processing method provided in this application specifically describes a scenario where the device matrices of the first tensor arrangement and the second tensor arrangement are the same, but the tensor shapes are different. To perform tensor shape normalization, it is necessary to determine a fifth equivalent tensor arrangement that is equivalent to the first tensor arrangement and a sixth equivalent tensor arrangement that is equivalent to the second equivalent tensor arrangement. The fifth equivalent tensor arrangement and the sixth equivalent tensor arrangement are used to generate a reshaping operator. By generating the reshaping operator to update the slice computation graph, the workload of users in designing the transformation between tensor arrangements can be reduced, and the efficiency of parallel training of the model can be improved.

[0021] In one possible implementation of the first aspect, the tensor mapping of the fifth equivalent tensor arrangement is inconsistent with the tensor mapping of the sixth equivalent tensor arrangement; the method further includes: determining one or more tensor mapping transformation operators, the tensor mapping transformation operators including a splitting operator, a merging operator, or a communication operator, the tensor arrangement of the input tensors of the one or more tensor mapping transformation operators being consistent with the fifth equivalent tensor arrangement, the tensor arrangement of the output tensors of the one or more tensor mapping transformation operators being consistent with the sixth equivalent tensor arrangement, and the one or more tensor mapping transformation operators being used to determine the updated slice computation graph.

[0022] The data processing method provided in this application specifically describes a scenario where, when the device matrices of the first and second tensor arrangements are the same but the tensor shapes are different, the tensor mappings remain inconsistent after tensor shape normalization. The data processing device further determines a third tensor mapping transformation operator sequence, including one or more tensor mapping transformation operators, to implement the transformation from a fourth equivalent tensor arrangement to a fifth equivalent tensor arrangement. By updating the slice computation graph through the generated one or more tensor mapping transformation operators, the workload of designing transformations between tensor arrangements can be reduced, improving the efficiency of parallel model training.

[0023] In one possible implementation of the first aspect, the device matrix of the first tensor arrangement and the device matrix of the second tensor arrangement are consistent, the tensor shape of the first tensor arrangement is consistent with the tensor shape of the second tensor arrangement, and the tensor mapping of the first tensor arrangement is inconsistent with the tensor mapping of the second tensor arrangement; determining the rearrangement operator between the first operator and the second operator includes: determining one or more tensor mapping transformation operators, the tensor mapping transformation operators including splitting operators, merging operators or communication operators, the one or more tensor mapping transformation operators being used to input the output tensor of the first operator and output the input tensor of the second operator.

[0024] The data processing method provided in this application specifically describes a scenario where the device matrices of the first tensor arrangement and the second tensor arrangement are consistent, the tensor shapes are consistent, but the tensor mappings are inconsistent. The data processing device further determines a fourth tensor mapping transformation operator sequence, including one or more tensor mapping transformation operators, to realize the transformation from the first tensor arrangement to the second tensor arrangement. Since this solution can ultimately generate one or more tensor mapping transformation operators from the first tensor arrangement and the second tensor arrangement to update the slice computation graph, it can reduce the workload of users designing transformations between tensor arrangements and improve the efficiency of parallel training of the model.

[0025] In one possible implementation of the first aspect, obtaining a deep neural network model, and the tensor arrangement of the input tensors of each operator in the deep neural network model and the tensor arrangement of the output tensors of each operator, includes: obtaining a deep neural network model and a segmentation strategy, the segmentation strategy including the number of segments of the tensors of the deep neural network model in each dimension; and determining the tensor arrangement of the input tensors of each operator in the deep neural network model and the tensor arrangement of the output tensors of each operator according to the deep neural network model and the segmentation strategy.

[0026] The data processing method provided in this application embodiment, and the specific method for the data processing device to obtain tensor arrangement, may involve first obtaining a segmentation strategy, and then generating the tensor arrangement of each operator based on the deep neural network model and the segmentation strategy. This provides another way to determine the tensor arrangement of operators and increases the flexibility of the scheme implementation.

[0027] In one possible implementation of the first aspect, the segmentation strategy includes a first segmentation strategy and a second segmentation strategy; determining the tensor arrangement of the input tensors of each operator and the tensor arrangement of the output tensors of each operator in the deep neural network model according to the deep neural network model and the segmentation strategy includes: determining a first overall tensor arrangement corresponding to the first segmentation strategy and a second overall tensor arrangement corresponding to the second segmentation strategy, wherein the first overall tensor arrangement is the tensor arrangement of the input tensors of each operator and the tensor arrangement of the output tensors of each operator in the deep neural network model determined based on the first segmentation strategy, and the second overall tensor arrangement is based on... The method further includes: determining, from the first overall tensor arrangement and the second overall tensor arrangement, the tensor arrangement of the input tensors and the tensor arrangement of the output tensors of each operator in the deep neural network model as determined by the second segmentation strategy; the method further includes: determining, from the first overall tensor arrangement and the second overall tensor arrangement, the first overall tensor arrangement being the tensor arrangement of the input tensors and the tensor arrangement of the output tensors of each operator in the deep neural network model, wherein the sum of the communication time and computation time required to train the deep neural network model based on the first overall tensor arrangement is less than the sum of the communication time and computation time required to train the deep neural network model based on the second overall tensor arrangement.

[0028] The data processing method provided in this application embodiment allows the data processing device to acquire at least two segmentation strategies. Based on these strategies, the overall tensor arrangement for each segmentation strategy is determined. It should be noted that one segmentation strategy can correspond to multiple overall tensor arrangements. Different overall tensor arrangements are compared to determine a first overall tensor arrangement with lower overhead. Overhead refers to the sum of communication and computation time for training a deep neural network model based on this overall tensor arrangement. It should be noted that the process of training a deep neural network model based on this overall tensor arrangement requires determining the rearrangement operator to be inserted based on this overall tensor arrangement. A slice computation graph is then determined based on the first overall tensor arrangement. This scheme can consider overall tensor arrangements corresponding to multiple segmentation strategies and select the one with lower overhead for slicing, thereby reducing the overhead during deep neural network model training.

[0029] In one possible implementation of the first aspect, the cost model of the first overall tensor arrangement is less than the cost model of the second overall tensor arrangement. The cost model of the first overall tensor arrangement is a value obtained by weighted summation based on the size of the data tensor, the size of the communication tensor, and the size of the parameter tensor in the first overall tensor arrangement, as well as the weight coefficients of the size of the data tensor, the size of the communication tensor, and the size of the parameter tensor. The cost model of the second overall tensor arrangement is a value obtained by weighted summation based on the size of the data tensor, the size of the communication tensor, and the size of the parameter tensor in the second overall tensor arrangement, as well as the weight coefficients of the size of the data tensor, the size of the communication tensor, and the size of the parameter tensor.

[0030] The data processing method provided in this application embodiment, when screening and determining the first overall tensor arrangement, can compare the overhead of different overall tensor arrangements through a cost model, and approximate the storage and computation overhead through tensors, providing a specific implementation method for comparing the overhead of different overall tensor arrangements. In addition, the weighting coefficients of the size of the data tensor, the size of the communication tensor, and the size of the parameter tensor can be flexibly set according to different data processing device types, improving the flexibility of the scheme implementation.

[0031] In one possible implementation of the first aspect, the segmentation strategy is a user-specified segmentation strategy.

[0032] The data processing method provided in this application embodiment can also be applied to scenarios where the user specifies the segmentation strategy.

[0033] In one possible implementation of the first aspect, the input tensor of each operator in the deep neural network model includes a training dataset, which includes a text dataset, an image dataset, or an audio dataset.

[0034] The data processing method provided in this application embodiment includes a training dataset as the input tensor for each operator in the deep neural network model. The training dataset includes text datasets, image datasets, or audio datasets, and can be used for distributed training processes of text translation models, speech recognition models, face recognition models, 3D reconstruction models, and virtual reality models. For example, if a text dataset is input, the corresponding deep neural network model can be used to achieve automatic translation; if an image dataset is input, the corresponding deep neural network model can be used to achieve image recognition, face recognition, or 3D modeling.

[0035] A second aspect of this application provides a data processing apparatus, characterized in that the apparatus comprises:

[0036] An acquisition unit is used to acquire a deep neural network model, and the tensor arrangement of the input tensors and the tensor arrangement of the output tensors of each operator in the deep neural network model. The tensor arrangement includes a device matrix, tensor mapping, and tensor shape. The deep neural network model includes a first operator and a second operator, which are two consecutive operators in the deep neural network model. The output tensor of the first operator is the input tensor of the second operator. The first tensor arrangement and the second tensor arrangement are not the same, wherein the first tensor arrangement is the tensor arrangement of the output tensor of the first operator. The second tensor arrangement is the tensor arrangement of the input tensors of the second operator; the determining unit is configured to determine a slice computation graph based on the tensor arrangement of the input tensors of each operator and the tensor arrangement of the output tensors of each operator; the determining unit is further configured to determine a reordering operator between the first operator and the second operator, the reordering operator being used to convert the first tensor arrangement into the second tensor arrangement; the determining unit is further configured to insert the reordering operator into the slice computation graph to determine an updated slice computation graph, the updated slice computation graph being used to indicate the part of the deep neural network model being executed.

[0037] In one possible implementation of the second aspect, the device matrix of the first tensor arrangement is inconsistent with the device matrix of the second tensor arrangement, and / or the tensor mapping of the first tensor arrangement is inconsistent.

[0038] The determining unit is specifically used for:

[0039] Determine the intermediate tensor layout based on the first tensor layout and the second tensor layout;

[0040] A first reshaping operator is determined based on the first tensor arrangement and the intermediate tensor arrangement, along with the tensor arrangements of the input tensors and the output tensors of the first reshaping operator. The first reshaping operator is used to implement the transformation from the first tensor arrangement to the intermediate tensor arrangement, and / or

[0041] The second reshaping operator is determined based on the second tensor arrangement and the intermediate tensor arrangement, as well as the tensor arrangement of the input tensor and the tensor arrangement of the output tensor of the second reshaping operator. The second reshaping operator is located between the first reshaping operator and the second operator, and the second reshaping operator is used to realize the transformation from the intermediate tensor arrangement to the second tensor arrangement.

[0042] In one possible implementation of the second aspect, the device matrix of the first tensor arrangement is inconsistent with the device matrix of the second tensor arrangement;

[0043] The determining unit is specifically used for:

[0044] Based on the device matrix arranged in the first tensor and the device matrix arranged in the second tensor, an extended device matrix is ​​determined. The element-wise product of the extended device matrix is ​​the same as the element-wise product of the device matrix of the output tensor of the first operator and the element-wise product of the second device matrix of the input tensor of the second operator. Any element in the device matrix arranged in the first tensor and the device matrix arranged in the second tensor is equal to an element in the extended device matrix or the product of at least two elements of the extended device matrix.

[0045] Based on the extended device matrix, a first equivalent tensor arrangement equivalent to the first tensor arrangement and a second equivalent tensor arrangement equivalent to the second tensor arrangement are determined, wherein the device matrix of the first equivalent tensor arrangement is consistent with the device matrix of the second equivalent tensor arrangement.

[0046] When the tensor shape of the first equivalent tensor arrangement is consistent with the tensor shape of the second equivalent tensor arrangement, the intermediate tensor arrangement includes the first equivalent tensor arrangement and the second equivalent tensor arrangement, and the tensor shape is the number of elements in each dimension of the tensor.

[0047] In one possible implementation of the second aspect, the tensor shape of the first equivalent tensor arrangement is consistent with the tensor shape of the second equivalent tensor arrangement, and the tensor mapping of the first equivalent tensor arrangement is inconsistent with the tensor mapping of the second equivalent tensor arrangement.

[0048] The determining unit is further configured to determine one or more tensor mapping transformation operators, the tensor mapping transformation operators including a splitting operator, a merging operator, or a communication operator, wherein the tensor arrangement of the input tensors of the one or more tensor mapping transformation operators is consistent with the first equivalent tensor arrangement, and the tensor arrangement of the output tensors of the one or more tensor mapping transformation operators is consistent with the second equivalent tensor arrangement, and the one or more tensor mapping transformation operators are used to determine the updated slice computation graph.

[0049] In one possible implementation of the second aspect, when the tensor shape of the first equivalent tensor arrangement is inconsistent with the tensor shape of the second equivalent tensor arrangement, the determining unit is further configured to perform tensor shape normalization based on the first equivalent tensor arrangement and the second equivalent tensor arrangement, determine a third equivalent tensor arrangement equivalent to the first equivalent tensor arrangement, and a fourth equivalent tensor arrangement equivalent to the second equivalent tensor arrangement, wherein the device matrix of the third equivalent tensor arrangement is consistent with the device matrix of the fourth equivalent tensor arrangement, and the tensor shape of the third equivalent tensor arrangement is consistent with the tensor shape of the fourth equivalent tensor arrangement; the intermediate tensor arrangement includes the third equivalent tensor arrangement and the fourth equivalent tensor arrangement.

[0050] In one possible implementation of the second aspect, the tensor mapping of the third equivalent tensor arrangement is inconsistent with the tensor mapping of the fourth equivalent tensor arrangement.

[0051] The determining unit is further configured to: determine one or more tensor mapping transformation operators, the tensor mapping transformation operators including a splitting operator, a merging operator, or a communication operator, wherein the tensor arrangement of the input tensors of the one or more tensor mapping transformation operators is consistent with the third equivalent tensor arrangement, and the tensor arrangement of the output tensors of the one or more tensor mapping transformation operators is consistent with the fourth equivalent tensor arrangement, and the one or more tensor mapping transformation operators are used to determine the updated slice computation graph.

[0052] In one possible implementation of the second aspect, the device matrix of the first tensor arrangement is consistent with the device matrix of the second tensor arrangement, and the tensor shape of the first tensor arrangement is inconsistent with the tensor shape of the second tensor arrangement.

[0053] The determining unit is specifically configured to: determine a fifth equivalent tensor arrangement that is equivalent to the first tensor arrangement and a sixth equivalent tensor arrangement that is equivalent to the second equivalent tensor arrangement based on the first tensor arrangement and the second tensor arrangement, wherein the device matrix of the fifth equivalent tensor arrangement is consistent with the device matrix of the sixth equivalent tensor arrangement, and the tensor shape of the fifth equivalent tensor arrangement is consistent with the tensor shape of the sixth equivalent tensor arrangement; the intermediate tensor arrangement includes the fifth equivalent tensor arrangement and the sixth equivalent tensor arrangement.

[0054] In one possible implementation of the second aspect, the tensor mapping of the fifth equivalent tensor arrangement is inconsistent with the tensor mapping of the sixth equivalent tensor arrangement.

[0055] The determining unit is further configured to determine one or more tensor mapping transformation operators, the tensor mapping transformation operators including a splitting operator, a merging operator, or a communication operator, wherein the tensor arrangement of the input tensors of the one or more tensor mapping transformation operators is consistent with the fifth equivalent tensor arrangement, and the tensor arrangement of the output tensors of the one or more tensor mapping transformation operators is consistent with the sixth equivalent tensor arrangement, and the one or more tensor mapping transformation operators are used to determine the updated slice computation graph.

[0056] In one possible implementation of the second aspect, the device matrix of the first tensor arrangement is consistent with the device matrix of the second tensor arrangement, the tensor shape of the first tensor arrangement is consistent with the tensor shape of the second tensor arrangement, and the tensor mapping of the first tensor arrangement is inconsistent with the tensor mapping of the second tensor arrangement.

[0057] The determining unit is specifically used for:

[0058] One or more tensor mapping transformation operators are determined, including splitting operators, merging operators, or communication operators. The one or more tensor mapping transformation operators are used to input the output tensor of the first operator and output the input tensor of the second operator.

[0059] In one possible implementation of the second aspect, the acquiring unit is specifically used for:

[0060] Obtain a deep neural network model and a segmentation strategy, wherein the segmentation strategy includes the number of segments of the tensor of the deep neural network model in each dimension;

[0061] The determining module is specifically used to determine the tensor arrangement of the input tensors of each operator in the deep neural network model and the tensor arrangement of the output tensors of each operator based on the deep neural network model and the segmentation strategy.

[0062] In one possible implementation of the second aspect,

[0063] The segmentation strategy includes a first segmentation strategy and a second segmentation strategy;

[0064] The determining unit is specifically used for:

[0065] A first overall tensor arrangement corresponding to the first segmentation strategy and a second overall tensor arrangement corresponding to the second segmentation strategy are determined. The first overall tensor arrangement is the tensor arrangement of the input tensor of each operator and the tensor arrangement of the output tensor of each operator in the deep neural network model determined based on the first segmentation strategy. The second overall tensor arrangement is the tensor arrangement of the input tensor of each operator and the tensor arrangement of the output tensor of each operator in the deep neural network model determined based on the second segmentation strategy.

[0066] The first overall tensor arrangement is determined from the first overall tensor arrangement and the second overall tensor arrangement as the tensor arrangement of the input tensor and the tensor arrangement of the output tensor of each operator in the deep neural network model. The sum of the communication time and computation time required for the training of the deep neural network model based on the first overall tensor arrangement is less than the sum of the communication time and computation time required for the training of the deep neural network model based on the second overall tensor arrangement.

[0067] In one possible implementation of the second aspect, the cost model of the first overall tensor arrangement is smaller than the cost model of the second overall tensor arrangement. The cost model of the first overall tensor arrangement is a value obtained by weighted summation based on the size of the data tensor, the size of the communication tensor, and the size of the parameter tensor in the first overall tensor arrangement, as well as the weight coefficients of the size of the data tensor, the weight coefficients of the size of the communication tensor, and the weight coefficients of the size of the parameter tensor.

[0068] The cost model for the second overall tensor arrangement is a value obtained by weighted summation based on the size of the data tensor, the size of the communication tensor, and the size of the parameter tensor in the second overall tensor arrangement, as well as the weight coefficients of the size of the data tensor, the size of the communication tensor, and the size of the parameter tensor.

[0069] In one possible implementation of the second aspect, the segmentation strategy is a user-specified segmentation strategy.

[0070] In one possible implementation of the second aspect, the input tensor of each operator in the deep neural network model includes a training dataset, which includes a text dataset, an image dataset, or an audio dataset.

[0071] A third aspect of this application provides a data processing device, including a processor and a memory, the processor and the memory being interconnected, wherein the memory is used to store a computer program, the computer program including program instructions, and the processor is used to invoke the program instructions to execute the method as described in any one of the first aspects and various possible implementations above.

[0072] A fourth aspect of this application provides a computer program product containing instructions, characterized in that, when run on a computer, it causes the computer to perform the method as described in any one of the first aspects and various possible implementations above.

[0073] A fifth aspect of this application provides a computer-readable storage medium including instructions, characterized in that, when the instructions are executed on a computer, the computer performs the method as described in any one of the first aspects and various possible implementations above.

[0074] A sixth aspect of this application provides a chip including a processor. The processor is used to read and execute a computer program stored in a memory to perform a method in any possible implementation of any of the above aspects. Optionally, the chip includes a memory, and the memory is connected to the processor via a circuit or wire. Further optionally, the chip also includes a communication interface, to which the processor is connected. The communication interface is used to receive data and / or information to be processed, the processor obtains the data and / or information from the communication interface, processes the data and / or information, and outputs the processing result through the communication interface. The communication interface may be an input / output interface.

[0075] A seventh aspect of this application provides a distributed cluster, characterized in that the distributed cluster includes one or more data processing devices as described in any of the second aspects above and various possible implementations.

[0076] The technical effects of any of the implementation methods in the second, third, fourth, fifth, sixth, and seventh aspects can be found in the technical effects of the corresponding implementation methods in the first aspect, and will not be repeated here.

[0077] The data processing method provided in this application involves a data processing device acquiring a deep neural network model and the tensor arrangements of all operators therein, including the tensor arrangements of input tensors and output tensors. If two consecutive operators exist where the tensor arrangement of the output tensor of the first operator is inconsistent with the tensor arrangement of the input tensor of the second operator, the slice computation graph obtained based on this tensor arrangement cannot be executed. In this application, the data processing device determines a rearrangement operator between the first and second operators to convert the tensor arrangement of the output tensor of the first operator into the tensor arrangement of the input tensor of the second operator. Thus, a rearrangement operator is inserted into the slice computation graph, and the determined updated slice computation graph can be executed. This scheme can be applied to cases where each operator in a deep neural network model has arbitrary tensor arrangements. By using a rearrangement operator to achieve parallelism conversion, various hybrid parallelism methods can be implemented in a distributed cluster. Attached Figure Description

[0078] Figure 1 A schematic diagram of an artificial intelligence main framework provided in an embodiment of this application;

[0079] Figure 2 A schematic diagram of an application environment provided for an embodiment of this application;

[0080] Figure 3 A schematic diagram of a computational graph provided for an embodiment of this application;

[0081] Figure 4 This is a schematic diagram of a distributed cluster topology provided in an embodiment of this application;

[0082] Figure 5 This is a schematic diagram of an application scenario in an embodiment of this application;

[0083] Figure 6 This is a schematic diagram of one embodiment of the data processing method in this application;

[0084] Figure 7 This is a schematic diagram of one embodiment of tensor arrangement in this application;

[0085] Figure 8 This is a schematic diagram of another embodiment of the data processing method in this application;

[0086] Figure 9 This is a schematic diagram of one embodiment of the rearrangement operator in this application.

[0087] Figure 10 This is a schematic diagram of another embodiment of the rearrangement operator in this application;

[0088] Figure 11 This is a schematic diagram of an embodiment for determining the overall tensor arrangement in this application.

[0089] Figure 12 This is a schematic diagram of another embodiment for determining the overall tensor arrangement in this application.

[0090] Figure 13 This is a schematic diagram of one embodiment of the data processing device in this application;

[0091] Figure 14 This is a schematic diagram of another embodiment of the data processing device in this application;

[0092] Figure 15 This is a chip hardware structure diagram provided for an embodiment of this application. Detailed Implementation

[0093] This application provides a data processing method for a distributed cluster used in training deep neural network models, which can achieve parallelism in the training process of deep neural network models through tensor arrangement of each operator.

[0094] For ease of understanding, the following is a brief introduction to some of the technical terms used in the embodiments of this application:

[0095] 1. The deep neural network model, also referred to as the model, network, or algorithm in this embodiment, is divided into a forward algorithm part and a backward computation part; forward propagation, or forward computation part, is the process of model computation, which can give a corresponding output for a set of inputs; backward propagation, or backward computation part, is the training of model parameters, using gradient descent on all parameters to minimize the loss function of the model on the training data.

[0096] 2. Computation Graph: Also called a data flow graph. Each computation in a neural network is a node on the computation graph, and the edges between nodes represent the dependencies between data inputs and outputs. A sliced ​​computation graph, compared to the full computation graph, retains the original nodes, but the data volume corresponding to the edges between nodes is a portion of the complete data volume. Furthermore, in this embodiment, nodes for rearrangement can be added to the sliced ​​computation graph.

[0097] 3. Tensor: An n-dimensional array, it is an n-dimensional generalization of scalars, 1-dimensional vectors, and 2-dimensional matrices. In machine learning model training, both training data and intermediate computation results can be viewed as tensors.

[0098] 4. Operators: Operations on the properties of tensors, such as matrix multiplication, tensor addition, convolution, etc. In this paper, they can be equated with layers in neural networks.

[0099] 5. Tensor shape: A one-dimensional array consisting of the number of elements in each dimension of a tensor.

[0100] 6. Device Matrix: A one-dimensional array. Represents the arrangement of devices. The number of array elements indicates the dimension of the device arrangement. The product of the array elements equals the total number of devices.

[0101] 7. Tensor Mapping: A one-dimensional array. The number of elements is equal to the number of elements in the tensor shape. The numerical value represents the mapping of the tensor's corresponding dimension partition onto the device matrix.

[0102] 8. Tensor Arrangement: The arrangement of distributed tensors across various devices. It is expressed by tensor shape, device matrix, and tensor mapping.

[0103] 9. Distributed tensor layout refers to the arrangement of elements in a distributed tensor across various devices. In this embodiment, it is also simply referred to as tensor layout.

[0104] Tensor shape: The complete shape, the number of elements in each dimension.

[0105] Device matrix: Each operator designs a device matrix.

[0106] Tensor maps are vectors with the same dimensions as the shape of a tensor.

[0107] The embodiments of this application are described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. As those skilled in the art will recognize, with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0108] The term "and / or" appearing in this application can describe a relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, or B exists alone, where A and B can be singular or plural. Additionally, the character " / " in this application generally indicates that the preceding and following related objects are in an "or" relationship. In this application, "at least one" means one or more, and "multiple" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0109] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0110] Figure 1 A schematic diagram of an artificial intelligence framework is shown, which describes the overall workflow of an artificial intelligence system and is applicable to general artificial intelligence domain needs.

[0111] The above-mentioned artificial intelligence framework will be elaborated from two dimensions: "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis).

[0112] The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it could be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom."

[0113] The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence, information (provided and processed by technology) to the industrial ecosystem of systems.

[0114] (1) Infrastructure

[0115] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0116] (2) Data

[0117] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0118] (3) Data processing

[0119] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0120] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training of data by symbolizing and formalizing it.

[0121] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0122] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0123] (4) General ability

[0124] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0125] (5) Smart Products and Industry Applications

[0126] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They encapsulate overall artificial intelligence solutions, productize intelligent information decision-making, and realize practical applications. Their application areas mainly include: intelligent manufacturing, intelligent transportation, smart home, intelligent healthcare, intelligent security, autonomous driving, safe city, and intelligent terminals.

[0127] The data processing method provided in this application can be applied to the parallel training of deep neural network models in various distributed cluster scenarios. It can independently determine the segmentation strategy for each operator, generate tensor arrangement, and obtain a slice map of the deep neural network model executable by a single data processing device by inserting rearrangement operators.

[0128] See appendix Figure 2 This application provides a system architecture 200. Optionally, a data acquisition device 260 is used to collect data and store it in a database 230, and a training device 220 generates a target model / rule 201 based on the data maintained in the database 230. Optionally, the data can be text data, audio data, or image data, and images include pictures and videos; the specific data type is not limited here. The following will describe in more detail how the training device 220 obtains the target model / rule 201 based on the data. The target model / rule 201 can be used in application scenarios such as text translation, speech recognition, face recognition, 3D reconstruction, and virtual reality.

[0129] The target model / rule 201 can be obtained based on a deep neural network, which will be introduced below.

[0130] The function of each layer in a deep neural network can be expressed mathematically. To describe it: From a physical perspective, the work of each layer in a deep neural network can be understood as transforming the input space (the set of input vectors) to the output space (i.e., from the row space to the column space of a matrix) through five operations on the input space. These five operations include: 1. Dimensionality increase / decrease; 2. Magnification / scaling; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are... The operation 4 is completed using +b, and the operation 5 is implemented using a(). The term "space" is used here because the objects being classified are not individual things, but a class of things; space refers to the set of all individuals within this class of things. Here, W is the weight vector, where each value represents the weight of a neuron in that layer of the neural network. This vector W determines the spatial transformation from the input space to the output space, as described above; that is, the weights W of each layer control how the space is transformed. The purpose of training a deep neural network is to ultimately obtain the weight matrix of all layers of the trained neural network (a weight matrix formed by the vectors W from many layers). Therefore, the training process of a neural network is essentially learning how to control spatial transformation, more specifically, learning the weight matrix.

[0131] Because the goal is for the output of a deep neural network to be as close as possible to the actual predicted value, we can compare the current network's prediction with the desired target value and update the weight vector of each layer based on the difference. (Of course, there's usually an initialization process before the first update, where parameters are pre-configured for each layer in the deep neural network). For example, if the network's prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the neural network can predict the actual target value. Therefore, it's necessary to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, so training a deep neural network becomes a process of minimizing this loss as much as possible.

[0132] The target model / rules obtained from training device 220 can be applied to different systems or devices. (See attached...) Figure 2 In the process, the execution device 210 is equipped with an I / O interface 212 for data interaction with external devices. The "user" can input data to the I / O interface 212 through the client device 240.

[0133] The execution device 210 can call data, code, etc. in the data storage system 250, and can also store data, instructions, etc. in the data storage system 250.

[0134] The calculation module 211 uses the target model / rule 201 to process the input data. Taking 3D modeling as an example, the calculation module 211 can parse the input image or image sequence to recover the depth information of the target.

[0135] The associated function module 213 can preprocess the image data in the calculation module 211.

[0136] The associated function module 214 can preprocess the image data in the calculation module 211.

[0137] Finally, I / O interface 212 returns the processing result to client device 240 for use by the user.

[0138] At a deeper level, the training device 220 can generate corresponding target models / rules 201 based on different data for different objectives, in order to provide users with better results.

[0139] In the appendix Figure 2 In the scenario shown, the user can manually specify the data input to execution device 210, for example, by operating through the interface provided by I / O interface 212. Alternatively, client device 240 can automatically input data to I / O interface 212 and obtain results. If client device 240 requires user authorization to automatically input data, the user can set appropriate permissions on client device 240. The user can view the results output by execution device 210 on client device 240, which can be presented in various ways such as display, sound, or animation. Client device 240 can also act as a data acquisition terminal, storing the collected training data into database 230.

[0140] It is worth noting that, attached Figure 2 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in the attached diagram... Figure 2 In this context, the data storage system 250 is an external memory relative to the execution device 210. However, in other cases, the data storage system 250 can also be placed within the execution device 210. For example, in the attached... Figure 2 In this embodiment, the training device 220, the execution device 210, and the client device 240 are each independent devices. In other cases, the training device 220 and the execution device 210 can be the same physical device, which can implement all the functions of the training device 220 and the execution device 210. Optionally, the execution device 210 and the client device 240 can also be the same physical device, which can implement all the functions of the execution device 210 and the client device 240. Optionally, the training device 220, the execution device 210, and the client device 240 are all the same physical device, which can implement all the functions of the training device 220, the execution device 210, and the client device 240. For the specific scenario architecture of this application embodiment, no limitation is made here.

[0141] Existing parallelism schemes for deep learning models, such as the Mesh-Tensorflow scheme, allow for partitioning of all tensors in the model based on any dimension, but all tensors must have identical device matrices. Because the Mesh-Tensorflow scheme requires identical device matrices for all tensors, it imposes constraints on tensor mapping. For example, the sample dimension of a tensor must map to the same dimension of the device matrix for all tensors, limiting the conversion between various parallelism methods; for instance, hybrid parallelism combining data parallelism and model parallelism cannot be achieved. Furthermore, it's impossible to partition tensors individually for each operator, resulting in low overall communication efficiency for the parallel scheme.

[0142] Please see Figure 3 This is a schematic diagram of a computation graph provided in an embodiment of this application;

[0143] In a deep neural network model, a layer can be viewed as an operator, and a deep neural network model includes multiple operators. Figure 3 This section uses the first and second operators as examples to illustrate a local part of the computational graph of a deep neural network. The input tensor of the first operator includes a data tensor X and a parameter tensor W. The output tensor of the first operator is the input tensor of the second operator. The second operator also has one input tensor, namely the parameter tensor V. The output tensor of the second operator is tensor Z. It can be understood that the tensors of an operator include input and output tensors. The number of tensors in an operator is not limited. An operator can have input parameter tensors or no input parameter tensors; this is not specifically limited here. Figure 3 This is only one possible computational graph.

[0144] Figure 4 This is a schematic diagram of a distributed cluster topology provided in an embodiment of this application;

[0145] A distributed cluster typically consists of multiple servers, and each server may include multiple data processing devices. These data processing devices can be CPUs, GPUs, or other types of processors, such as Ascend chips, etc. The specifics are not limited here. Figure 4 This diagram illustrates a possible distributed cluster topology, comprising n servers, each with eight data processing devices deployed (often simply referred to as one machine with eight processing cards). The servers communicate via a switching network; it's understood that the communication latency between servers is greater than the communication latency between the data processing devices within each server. Deep neural networks can be deployed on this distributed cluster, with multiple data processing devices across multiple servers training the deep neural network model in parallel.

[0146] Figure 5 This is a schematic diagram of an application scenario in an embodiment of this application;

[0147] The deep neural network model can be converted into a full computation graph through graph compilation using the Python front-end. In this application, the full computation graph for single-machine execution is obtained. Based on the full computation graph, a parallel scheme is generated according to the data processing method provided in the embodiments of this application to obtain a sliced ​​computation graph. The sliced ​​computation graph is used for the part of the deep neural network model training executed by a single data processing device. The sliced ​​computation graph is compiled into an execution graph through graph optimization using automatic differential graph optimization.

[0148] The data processing method provided in this application is mainly used to generate parallel schemes, specifically including an operator parallel segmentation modeling module, a cost model module, a parallel segmentation strategy module, and a graph segmentation module. The operator parallel segmentation modeling module is used to generate multiple candidate segmentation strategies and the corresponding overall operator tensor arrangement of the deep neural network model. The cost model module determines the target segmentation strategy from multiple candidate segmentation strategies based on the cost. The parallel segmentation strategy module is used to insert rearrangement operators. The graph segmentation module performs graph segmentation based on the deep neural network model updated by the inserted rearrangement operators and the tensor arrangement of each operator, thereby obtaining a slice computation graph.

[0149] Please see Figure 6 This is a schematic diagram of an embodiment of the data processing method in this application.

[0150] 601. Data processing equipment acquires deep neural network models and segmentation strategies;

[0151] The data processing device acquires a deep neural network model, also referred to as a model, network, or algorithm in this embodiment of the application, and determines the full forward graph of the model executed by a single machine based on the acquired model.

[0152] The data processing device acquires the segmentation strategy, which refers to the way each tensor in the model is segmented, including the number of segments in each dimension. It should be noted that this segmentation strategy can be specified by the user or generated by the data processing device; no specific limitation is made here. Optionally, the data processing device generates multiple candidate segmentation strategies based on the model and device topology information, and determines the optimal segmentation strategy from among these candidates. For specific implementation methods, please refer to subsequent embodiments.

[0153] It should be noted that step 601, which obtains the segmentation strategy, is an optional operation. If the data processing device generates multiple candidate segmentation strategies based on the model and device topology information, and then obtains multiple overall tensor arrangements, and determines the target overall tensor arrangement from them, step 603 can be executed directly.

[0154] 602. The data processing equipment determines the tensor arrangement of each operator based on the deep neural network model and the segmentation strategy;

[0155] Distributed tensor layout refers to the arrangement of elements in a distributed tensor across various devices. It consists of a device matrix, a tensor map, and a tensor shape. In this embodiment and subsequent embodiments, distributed tensor layout will be simply referred to as tensor layout. The tensor shape is a one-dimensional array representing the number of elements in each dimension of the tensor. The device matrix, a one-dimensional array, expresses the device layout; its number of elements represents the dimension of the device arrangement, and the product of the array elements equals the total number of devices. The tensor map is also a one-dimensional array with the same number of elements as the tensor shape. The value of each element represents the mapping of each dimension of the tensor onto the device matrix. Tensor layout can support the tensor layout needs of operators in various parallel modes.

[0156] Once the data processing device determines the tensor arrangement of each operator, it can obtain tensor slices of the data processing device and thus obtain a slice computation graph.

[0157] Based on the full forward graph of the deep neural network model and the segmentation strategy, the data processing equipment can determine the tensor arrangement of each operator. Specifically, it can determine the tensors of all operators based on the deep neural network model, including the number of elements in each dimension of the tensor, i.e., the tensor shape; and determine the device matrix and tensor mapping of the tensor of each operator based on the segmentation strategy, i.e., the number of segments of the tensor in each dimension, and the device topology of the distributed cluster, including the number of data processing devices in each server and the connection relationship between servers.

[0158] Optionally, if the segmentation strategy in step 601 is specified by the user, the data processing device can generate tensor arrangements for each operator according to the user-specified segmentation strategy and preset rules. Alternatively, it can generate multiple overall tensor arrangements according to the user-specified segmentation strategy. The overall tensor arrangement is the tensor arrangement of the tensors of each operator in the deep neural network model, and then an optimal overall tensor arrangement is determined from the multiple overall tensor arrangements.

[0159] Optionally, if in step 601 the segmentation strategy is determined by the data processing device from multiple candidate segmentation strategies, the data processing device can determine the optimal overall tensor arrangement corresponding to that segmentation strategy. Please refer to subsequent embodiments for details.

[0160] For example, please refer to Figure 7 This is a schematic diagram of an embodiment of tensor arrangement in this application.

[0161] The matrix multiplication operator (matmul) takes tensors A and B as inputs and outputs tensor C.

[0162] Figure 7 Table 1 shows the tensor arrangement of tensors A, B, and C. It can be seen that the device matrices of all three tensors are [3, 2, 4]. The tensor shape of tensor A is [h...]. A w A The tensor map is [2, 1], representing the 0th dimension h of tensor A. A Mapped to the second dimension of the device matrix, the first dimension w A Mapped to the first dimension of the device matrix. Similarly, tensor B has a tensor shape of [hB, wB] and a tensor map of [1, 0], while tensor C has a tensor shape of [hC, wC] and a tensor map of [2, 0].

[0163] It should be noted that steps 601 to 602 are optional. The data processing device can directly acquire the deep neural network model, as well as the tensor arrangement of the input tensors and the tensor arrangement of the output tensors of each operator in the deep neural network model, before step 603. Based on the deep neural network model, and the tensor arrangement of the input tensors and the tensor arrangement of the output tensors of each operator in the deep neural network model, the tensor slice of the data processing device can be obtained, resulting in a slice computation graph.

[0164] 603. Data processing equipment inserts rearrangement operators between consecutive operators with different tensor arrangements;

[0165] Assuming that the first operator and the second operator are two consecutive operators in the full forward graph of the deep neural network model, and the tensor layout (tensor_layout_from) of the output tensor of the first operator is inconsistent with the tensor layout (tensor_layout_to) of the input tensor of the second operator, the data processing device cannot execute the tensor slice determined in step 602. In this embodiment of the application, a redistribution operator is inserted between the first operator and the second operator.

[0166] Inconsistent tensor arrangement refers to a difference in at least one of the device matrix, tensor mapping, and tensor shape.

[0167] If the device matrices of tensor_layout_from and tensor_layout_to are inconsistent, the data processing device performs device matrix normalization to determine the first equivalent tensor layout of tensor_layout_from (tensor_layout_from2) and the second equivalent tensor layout of tensor_layout_to (tensor_layout_to2). The device layout of tensor_layout_from2 is consistent with the device layout of tensor_layout_to2. The conversion between equivalent tensor layouts can be achieved through the reshape operator. For example, tensor_layout_from is converted to tensor_layout_from2 through the reshape operator.

[0168] If the device matrices of tensor_layout_from and tensor_layout_to are the same, but their tensor shapes are different; or, if the tensor shape of tensor_layout_from2 is different from the tensor shape of tensor_layout_to2, then the data processing device performs tensor shape normalization to determine the third equivalent tensor arrangement (tensor_layout_from3) of tensor_layout_from and the fourth equivalent tensor arrangement (tensor_layout_to3) of tensor_layout_to. The tensor shape of tensor_layout_from3 is the same as that of tensor_layout_to3. The conversion between equivalent tensor arrangements can be achieved by the reshape operator. For example, tensor_layout_from2 is converted to tensor_layout_from3 by the reshape operator.

[0169] If `tensor_layout_from` and `tensor_layout_to` have the same device matrix and tensor shape, but different tensor mappings; or if `tensor_layout_from2` and `tensor_layout_to2` have the same device layout and tensor shape, but different tensor mappings; or if `tensor_layout_from3` and `tensor_layout_to3` have the same device layout and tensor shape, but different tensor mappings, then tensor mapping transformation operators such as slice operators, concat operators, or communication operators are inserted between tensors with different shapes. Communication operators include `alltoall` and `allgather` operators. Tensor mapping transformation operators can be used to convert between tensors with different mappings, such as the conversion from `tensor_layout_from3` to `tensor_layout_to3`. It's understandable that the tensor layout of the operator can be obtained when generating communication or slice operators.

[0170] Depending on the specific discrepancy between `tensor_layout_from` and `tensor_layout_to`, the data processing device will generate one or more of the following operators: `reshape`, `slice`, `concat`, `alltoall`, and `allgather`. In other words, redistribution operators can include a single operator or a sequence of redistribution operators; the specific number and type of operators are not limited here. Through redistribution operators, the transformation from `tensor_layout_from` to `tensor_layout_to` can be achieved.

[0171] For the specific calculation method of generating the rearrangement operator, please refer to the following embodiments.

[0172] It should be noted that there may be multiple sets of first operators and second operators in a deep neural network model. By inserting rearrangement operators between each set of first operators and second operators, an executable updated deep neural network model can be determined.

[0173] Optionally, if in step 601 the segmentation strategy is determined by the data processing device from multiple candidate segmentation strategies, the data processing device can directly obtain the rearrangement operator that needs to be inserted between the first operator and the second operator. In this step, the rearrangement operator is inserted between the first operator and the second operator in sequence.

[0174] 604. The data processing equipment updates the slice calculation graph according to the rearrangement operator and determines the updated slice calculation graph;

[0175] The data processing equipment updates the slice computation graph according to the rearrangement operator, and then generates the slice execution graph according to the graph compilation process, through automatic differentiation, graph optimization and other processes. A single data processing equipment executes a part of the deep neural network model training process based on the corresponding slice execution graph.

[0176] Similarly, each data processing device in a distributed cluster can obtain its own slice computation graph, generate an execution graph from the slice computation graph, and execute a part of the deep neural network model training process.

[0177] The data processing method provided in this application involves a data processing device acquiring a deep neural network model and the tensor arrangements of all operators therein, including the tensor arrangements of input tensors and output tensors. If two consecutive operators exist where the tensor arrangement of the output tensor of the first operator is inconsistent with the tensor arrangement of the input tensor of the second operator, the slice computation graph obtained based on this tensor arrangement cannot be executed. In this application, the data processing device determines a rearrangement operator between the first and second operators to convert the tensor arrangement of the output tensor of the first operator into the tensor arrangement of the input tensor of the second operator. Thus, by inserting the rearrangement operator into the slice computation graph, the determined updated slice computation graph can be executed. This scheme can be applied to any given operator tensor arrangement, achieving parallelism conversion through the rearrangement operator, enabling various hybrid parallelism methods to be implemented in a distributed cluster.

[0178] The data processing method provided in this application supports flexible tensor partitioning and parallelization. Each operator is modeled independently and can be partitioned across all dimensions of the tensor. The tensor arrangement is transformed by rearranging the operator sequence. Existing Mesh-Tensorflow methods model the entire deep neural network and cannot support the transformation of parallel modes between operators.

[0179] For example, Hybrid Parallelism Method 1: Channel model parallelism to batch data parallelism is a common parallel training method for deep learning recommendation models (DLRM). For this hybrid parallelism, the Mesh-Tensorflow method cannot support the free switching of parallel modes between operators. However, this application, through a rearrangement operator generation device, can generate rearrangement operators required for arbitrary tensor arrangement transformation, and supports flexible splitting strategy configuration at the operator level.

[0180] For example, hybrid parallelism method 2: data parallelism superimposed on model parallelism, is a commonly used parallelism method in Transformer networks. Existing solutions cannot support full-dimensional partitioning of tensors, and therefore cannot support this partitioning scenario. This application, however, can support partitioning of tensors in each dimension.

[0181] The parallelism provided in this application supports any specified parallelism strategy, such as data parallelism to model parallelism.

[0182] Please see Figure 8 This is a schematic diagram of another embodiment of the data processing method in this application.

[0183] Figure 8 The diagram illustrates the overall flow of the data processing method in this embodiment. The network model is searched using a cost model and a parallel strategy, where the parallel strategy can combine data parallelism and model parallelism. After obtaining multiple slice computation graphs based on the parallel strategy, they are executed by multiple data processing devices in a distributed cluster via a communication network.

[0184] Please see Figure 9 This is a schematic diagram of an embodiment of generating rearrangement operators in this application.

[0185] Assuming that the first operator and the second operator are two consecutive operators in the full forward graph of the deep neural network model, and the tensor layout (tensor_layout_from) of the output tensor of the first operator is inconsistent with the tensor layout (tensor_layout_to) of the input tensor of the second operator, the deep neural network model cannot be segmented and executed according to the tensor layout determined in step 602. In this embodiment, a redistribution operator is inserted between the first operator and the second operator to update the slice computation graph, so that the updated slice computation graph can be executed.

[0186] Inconsistent tensor arrangement refers to a difference in at least one of the device matrix, tensor mapping, and tensor shape.

[0187] If the device matrices of tensor_layout_from and tensor_layout_to are inconsistent, then proceed to steps 901 to 904.

[0188] If the device matrix of tensor_layout_from and tensor_layout_to is the same, but the tensor shapes are different, then proceed to steps 902 to 904.

[0189] If the device matrix of tensor_layout_from and tensor_layout_to are the same, the tensor shape is the same, but the tensor mapping is different, then proceed to steps 903 to 904.

[0190] If the device matrix, tensor mapping, and tensor shape of tensor_layout_from and tensor_layout_to are all the same, then there is no need to insert a rearrangement step operator between the first and second tensors.

[0191] 901. The data processing device determines the equivalent layout of tensor_layout_from2 under the extended device matrix, and the equivalent layout of tensor_layout_to2 under the extended device matrix.

[0192] The data processing device determines a first equivalent tensor layout tensor_layout_from2, which is equivalent to tensor_layout_from, and a second equivalent tensor layout tensor_layout_to2, which is equivalent to tensor_layout_to, based on the device matrix of tensor_layout_from and tensor_layout_to.

[0193] Optionally, one implementation for determining tensor_layout_from2 and tensor_layout_to2 is as follows:

[0194] 1) The device matrix based on tensor_layout_from is different from the device matrix based on tensor_layout_to. The data processing device is based on the first device matrix (device_matrix_from) [A0, A1, ..., A1] of tensor_layout_from. n The second device matrix (device_matrix_to) of tensor_layout_to [B0, B1, ..., B] n Determine the cumulative device matrix of the first device matrix and the cumulative device matrix of the second device matrix. A0, A1, ..., A n , where n is a positive integer greater than or equal to 1; B0, B1, ..., B m , where m is an element in the second device matrix, and m is a positive integer greater than or equal to 1.

[0195] First device matrix [A0, A1, ..., A n The cumulative device matrix is ​​[A0A1…A]n A1…A n A n ];

[0196] Second device matrix [B0, B1, ..., B] m The cumulative device matrix is ​​[B0B1…B n B1…B n B m ].

[0197] For example, device_matrix_from = [2, 16], device_matrix_to = [4, 4, 2], the cumulative device matrix of device_matrix_from = [2*16, 16] = [32, 16], and the cumulative device matrix of device_matrix_to = [4*4*2, 4*2, 2] = [32, 8, 2].

[0198] 2) Determine the minimum cumulative normalized extended device matrix device_matrix_equal2_accum based on the cumulative device matrix of the first device matrix and the cumulative device matrix of the second device matrix;

[0199] Based on the union of the cumulative device matrices of the first device matrix and the cumulative device matrices of the second device matrix, we obtain the minimum cumulative normalized extended device matrix [C0C1…C k C1…C k C k ]; k is a positive integer greater than or equal to 1, C0C1…C k C1…C k C k The minimum cumulative normalized extended device matrix contains elements, with the number of elements equal to k+1. Specifically, it is the number of elements in the cumulative device matrix of the first device matrix and the cumulative device matrix of the second device matrix after removing duplicate elements.

[0200] For example, the union of the cumulative device matrix of device_matrix_from = [32, 16] and the cumulative device matrix of device_matrix_to = [32, 8, 2] is obtained as device_matrix_equal2_accum = [32, 16, 8, 2].

[0201] 3) Determine the minimum normalized extended device matrix based on the minimum cumulative normalized extended device matrix;

[0202] Extend the device matrix based on minimum cumulative normalization [C0C1…C k C1…C k C k ], thus obtaining the minimum normalized extended device matrix [C0, C1, ..., C k ].

[0203] For example, based on device_matrix_equal2_accum = [32, 16, 8, 2], the minimum normalized extended device matrix device_matrix_equal2 = [2, 2, 4, 2] is obtained.

[0204] 4) Determine the first equivalent tensor layout (tensor_layout_from2) of the first tensor and the second equivalent tensor layout (tensor_layout_to2) of the second tensor based on the minimum normalized extended device matrix;

[0205] Specifically, based on the minimum normalized extended device matrix, the equivalent tensor layout of the first tensor layout (tensor_layout_from) is determined as follows:

[0206] Assuming tensor_layout_from is determined to be:

[0207] tensor_shape = [s[N-1], s[N-2], ..., s[0]]; s[N-1], s[N-2], ..., s[0] are the elements in the tensor shape.

[0208] device_matrix = [d[D-1], d[D-2], ..., d[0]]; d[D-1], d[D-2], ..., d[0] are the elements in the device matrix.

[0209] tensor_map = [m[N-1], m[N-2], ..., m[0]]; m[N-1], m[N-2], ..., m[0] are the elements in the tensor map.

[0210] Suppose that d[i] = mn is expanded in device_matrix = [d[D-1], d[D-2], ..., d[0]], the expanded device matrix is: device_matrix_e = [d[D-1], d[D-2], ..., d[i+1], m, n, ..., d[0]].

[0211] Determine if m[k] = i exists in the tensor_map;

[0212] If m[k] = i exists in tensor_map, then the tensor mapping of equivalent tensor arrangement is:

[0213] tensor_map_e=[me[N], me[N-1],…, me[k+1], me[k],…, me[0]];

[0214]

[0215] Tensor shapes of equivalent tensor arrangements:

[0216] tensor_shape_e=[se[N-1], se[N-2],…, se[k+1], se[k],…, se[0]]; se[N-1], se[N-2],…, se[k+1], se[k],…, se[0] are elements in tensor_shape_e.

[0217] Where se[k+1]=m;

[0218] se[k] = s[k] / m;

[0219]

[0220] If m[k] = i does not exist in tensor_map, then the tensor mapping and tensor shape of the equivalent tensor arrangement remain unchanged.

[0221] For example:

[0222] Example 1: Original tensor arrangement: tensor_shape = [512, 1024], device_matrix = [8, 4], tensor_map = [1, 0]. Based on the minimum normalization expansion of the device matrix, device_matrix_e = [4, 2, 4]. In the equivalent tensor arrangement: tensor_map_e = [2, 1, 0], tensor_shape_e = [4, 128, 1024].

[0223] Example 2: Original tensor arrangement: tensor_shape = [512, 1024], device_matrix = [8, 4], tensor_map = [0, 1]. Based on the minimum normalization expansion of the device matrix, device_matrix_e = [4, 2, 4]. In the equivalent tensor arrangement: tensor_map_e = [0, 2, 1], tensor_shape_e = [512, 4, 256].

[0224] Example 3: Original tensor arrangement: tensor_shape = [512, 1024], device_matrix = [8, 4], tensor_map = [-1, 0]. Based on the minimum normalization expansion of the device matrix, device_matrix_e = [4, 2, 4]. In the equivalent tensor arrangement: tensor_map_e = [-1, 0], tensor_shape_e = [512, 1024].

[0225] Example 4. Original tensor arrangement: tensor_shape = [512, 1024], device_matrix = [8, 4], tensor_map = [-1, 1]. Based on the minimum normalization expansion of the device matrix, device_matrix_e = [4, 2, 4]. In the equivalent tensor arrangement: tensor_map_e = [-1, 2, 1], tensor_shape_e = [512, 4, 256].

[0226] It should be noted that the conversion between tensor_layout_from and tensor_layout_from2 can be achieved using the reshape operator, as can the conversion between tensor_layout_to and tensor_layout_to2.

[0227] 902. The data processing device normalizes the tensor shape based on tensor_layout_from2 and tensor_layout_to2, and determines the third equivalent tensor layout (tensor_layout_from3) which is equivalent to tensor_layout_from2, and the fourth equivalent tensor layout (tensor_layout_to3) which is equivalent to tensor_layout_to2.

[0228] If the device matrix of the first equivalent tensor layout (tensor_layout_from2) and the device matrix of the second equivalent tensor layout (tensor_layout_to2) are the same but the tensor shapes are different, then this step is executed. If the device matrix of tensor_layout_from2 and the device matrix of tensor_layout_to2 are the same and the tensor shapes are the same, then step 903 is executed directly. Optionally, if the device matrix of tensor_layout_from and tensor_layout_to are the same but the tensor shapes are different, then step 901 can be skipped and step 902 can be executed directly. It can be assumed that tensor_layout_from2 is equal to tensor_layout_from and tensor_layout_to2 is equal to tensor_layout_to.

[0229] Suppose that s[i] = mn is expanded in tensor_shape = [s[N-1], s[N-2], ..., s[0]], the expanded tensor shape is: tensor_shape_e = [s[N-1], s[N-2], ..., s[i+1], m, n, s[i-1], ..., s[0]].

[0230] Determine if d[m[i]] is greater than m:

[0231] If d[m[i]]>m;

[0232] The device matrix with equivalent tensor arrangement:

[0233] device_matrix_e=[de[D],de[D-1],…,de[0]];

[0234]

[0235] Tensor mappings of equivalent tensor arrangements:

[0236] tensor_map_e=[me[N],me[N-1],…,me[i+1],me[i],…,me[0]]

[0237]

[0238] If d[m[i]]>m;

[0239] The device matrix of equivalent tensor arrangement remains unchanged;

[0240] The tensor mapping of equivalent tensor arrangements is:

[0241]

[0242] For example:

[0243] Example 1: Tensor arrangement before expansion: tensor_shape = [512, 1024], device_matrix = [8, 4], tensor_map = [1, 0], tensor shape after expansion: tensor_shape_e = [512, 2, 512]; in determining the equivalent tensor arrangement: device_matrix_e = [8, 2, 2], tensor_map_e = [2, 1, 0].

[0244] Example 2, Tensor arrangement before expansion: tensor_shape = [512, 1024], device_matrix = [8, 4], tensor_map = [1, 0], Tensor shape after expansion: tensor_shape_e = [128, 4, 1024]; Determine equivalent tensor arrangement: device_matrix_e = [8, 4], tensor_map_e = [1, -1, 0].

[0245] It should be noted that the conversion between tensor_layout_from2 and tensor_layout_from3 can be achieved using the reshape operator, as can the conversion between tensor_layout_to2 and tensor_layout_to3.

[0246] Before executing step 903, first determine whether the tensor mappings of tensor_layout_from3 and tensor_layout_to3 are consistent. If they are consistent, it means that the shapes of the tensor slices are the same, so step 903 is not executed. If they are inconsistent, it means that the shapes of the tensor slices are different, and the tensor mappings need to be converted, that is, the tensor mapping conversion operator is inserted, that is, step 903 is executed.

[0247] 903. Based on the third equivalent tensor layout (tensor_layout_from3) of the first tensor and the fourth equivalent tensor layout (tensor_layout_to3) of the second tensor, transform the tensor mapping to determine the communication operator, splitting operator, or merging operator;

[0248] Since tensor_layout_from3 and tensor_layout_to3 have the same device layout and tensor shape, but different tensor mappings, tensor mapping transformation operators need to be inserted between tensors with different shapes. These operators include communication operators, slice operators, or merge operators. Communication operators include alltoall operators and allgather operators. Optionally, if tensor_layout_from2 and tensor_layout_to2 have the same device matrix, tensor shape, but different tensor mappings, step 902 can be skipped, and step 903 can be executed directly. In this case, tensor_layout_from3 can be considered equal to tensor_layout_from2, and tensor_layout_to3 can be considered equal to tensor_layout_to2. Optionally, if tensor_layout_from and tensor_layout_to have the same device matrix, the same tensor shape, and different tensor mappings, then steps 901 to 902 can be skipped, and step 903 can be executed directly. It can be assumed that tensor_layout_from3 is equal to tensor_layout_from, and tensor_layout_to3 is equal to tensor_layout_to.

[0249] Tensor mapping transformation operators can be used to convert between different tensors. For example, the conversion between `tensor_layout_from3` and `tensor_layout_to3`. It's understandable that the tensor arrangement of the operators can be obtained when generating communication, splitting, and merging operators.

[0250] Depending on the specific circumstances of the tensor mapping inconsistency, the data processing device will generate one or more of the reshape operator, slice operator, alltoall operator, and allgather operator.

[0251] Optionally, one tensor map conversion process in this application embodiment is as follows:

[0252] 1) First, determine if there is a transformation from no splitting to splitting between `tensor_map_from3` and `tensor_map_to3`. Elements in `tensor_map` represent the mapping of each dimension of the tensor to the device matrix. If an element in `tensor_map` is greater than or equal to 0, it represents the corresponding dimension of the device matrix. If an element in `tensor_map` is -1, it means that the tensor dimension is split within the device matrix. Therefore, if there is a transformation from -1 to greater than or equal to 0 for corresponding elements between `tensor_map_from3` and `tensor_map_to3`, insert the slice operator.

[0253] For example: when tensor_map_from3 is (-1, -1, -1, 3, 2, 1, 0) and is converted to tensor_map_to3 (4, -1, -1, 3, 2, 1, 0), the 0th dimension element changes from -1 to 4. Then, the slice operator is inserted to slice the 0th dimension of the tensor.

[0254] 2) If there is an element position transformation between tensor_map_from3 and tensor_map_to3, then insert the communication operator (alltoall operator).

[0255] For example, a tensor map is transformed from (4, -1, -1, 3, 2, 1, 0) to (4, 3, -1, -1, 2, 1, 0), and the elements of the first and third dimensions are swapped, then the alltoall operator is inserted.

[0256] 3) Finally, perform the conversion between splitting and not splitting. If there is a conversion between elements with values ​​greater than or equal to 0 and -1, then insert the communication operator (allgather operator).

[0257] Based on a series of tensor mapping transformation operators, tensor_layout_from3 can be converted to tensor_layout_to3.

[0258] 904. Determine the reshape operator;

[0259] Based on tensor_layout_from3 and tensor_layout_to3 obtained in step 902, determine the reshape operator.

[0260] Determine whether the slice shapes determined by tensor_layout_from and tensor_layout_from3 are consistent. If they are inconsistent, the first reshape operator needs to be inserted. The input of the first reshape operator is tensor_layout_from, and the output is tensor_layout_from3.

[0261] Determine whether the slice shapes determined by tensor_layout_to and tensor_layout_to3 are consistent. If they are inconsistent, a second reshape operator needs to be inserted. The input of the second reshape operator is tensor_layout_to, and the output is tensor_layout_to3.

[0262] Example 1: tensor_shape_from = [1024, 512], tensor_shape_from3 = [2, 256, 2, 2, 2, 2, 64], have different shapes. According to tensor_layout_from3, the slice shape tensor_shape_from_slice3 = [2, 256, 2, 1, 1, 1, 32]. Therefore, when the reshape operator is inserted at the beginning of the rearrangement sequence, the shape after reshape is tensor_shape_from_slice3.

[0263] Example 2: tensor_shape_to = [512, 1024], tensor_shape_to3 = [2, 256, 2, 2, 2, 2, 64] have different shapes. According to tensor_layout_to, the slice shape tensor_shape_from_slice = [128, 256]. Therefore, a reshape operator is inserted at the end of the rearrangement sequence, and the shape after reshaping is tensor_shape_from_slice.

[0264] Finally, the rearrangement sequence can be determined; the rearrangement sequence is a sequence of operators inserted sequentially between the first and second operators, and the rearrangement sequence includes one or more operators, the specific number of which is not limited here. The rearrangement sequence includes all rearrangement step operators determined according to steps 901 to 904, including one or more of the following: splitting operator, merging operator, reshape operator, and communication operator.

[0265] The first operator and the second operator are two consecutive operators in the full forward graph of the deep neural network model. The tensor layout (tensor_layout_from) of the output tensor of the first operator is inconsistent with the tensor layout (tensor_layout_to) of the input tensor of the second operator. According to the rearrangement sequence, the tensor layout (tensor_layout_from) of the output tensor of the first operator can be transformed into the tensor layout (tensor_layout_to) of the input tensor of the second operator.

[0266] The rearrangement operator generation apparatus provided in this application embodiment can generate the required rearrangement operator from any tensor layout transformation. Based on tensor layout representation, it generates rearrangement operators from any tensor layout transformation. The generated operator sequence has minimal overhead.

[0267] Figure 10 This is a schematic diagram of another embodiment of the rearrangement operator in this application;

[0268] Assuming the first and second operators are two consecutive operators in the full forward graph of a deep neural network model, and the tensor layout (tensor_layout_from) of the output tensor of the first operator is inconsistent with the tensor layout (tensor_layout_to) of the input tensor of the second operator, the deep neural network model cannot execute based on this tensor layout. This embodiment generates a redistribution operator based on tensor_layout_from and tensor_layout_to. By inserting a redistribution operator between the first and second operators, the deep neural network model is updated, making the updated deep neural network model executable. Redistribution operators include reshape operators, communication operators, splitting operators, and merging operators, etc. The redistribution operator inserted between the first and second operators can include one or more of these types, without specific limitations.

[0269] The data processing device determines the optimal segmentation strategy from multiple candidate segmentation strategies, and uses the overall tensor arrangement corresponding to the optimal segmentation strategy to generate the slice computation graph. Alternatively, based on multiple candidate segmentation strategies, it obtains the overall tensor arrangement corresponding to each candidate strategy, resulting in multiple overall tensor arrangements. From these multiple overall tensor arrangements, it determines the optimal overall tensor arrangement to generate the slice computation graph. Please refer to [link to relevant documentation]. Figure 11 This is a schematic diagram of an embodiment for determining the overall tensor arrangement in this application.

[0270] 1101. Determine the arrangement of multiple global tensors of the deep neural network model based on the candidate segmentation strategy;

[0271] Data processing equipment can determine all tensors in a deep neural network model, and then determine the shape of the tensor based on the number of elements in each dimension.

[0272] The data processing device can also acquire device topology information of the distributed cluster for the cluster resource distribution of the deep neural network model, including the number of servers, the number of data processing devices on each server, and the connection relationships between servers. The data processing device can be, for example, a GPU, CPU, or other types of processors; the specific type is not limited here. The topology information yields the total number of data processing devices in the distributed cluster. This total number can be used to constrain the device matrix in the generated tensor arrangement. The data processing device determines the tensor of the first operator based on the full forward graph. The tensor of the first operator includes an input tensor and an output tensor. Based on the number of devices, the data processing device determines the same device matrix for the input and output tensors of each operator according to a preset rule.

[0273] Based on the tensor shape of each tensor and the device topology information, different candidate segmentation strategies can be determined by traversal, i.e., the segmentation method of each tensor, including the number of segments in each dimension of the tensor, to determine the mapping of slices of all tensors in each dimension in the device cluster in the deep neural network model. That is, to determine the device matrix and tensor mapping of the tensor under the segmentation strategy, thereby obtaining multiple overall tensor arrangements of the deep neural network model. The overall tensor arrangement refers to the tensor arrangement of the input tensors of all operators and the tensor arrangement of the output tensors of all operators in the deep neural network model.

[0274] Similarly, iterate through all candidate segmentation strategies to determine the overall tensor arrangement corresponding to each candidate segmentation strategy.

[0275] Optionally, different preset rules are determined based on the operator type to determine the device matrix of the appropriate operator tensor. Different types of operators include matrix multiplication operators, tensor addition operators, convolution operators, and softmax operators.

[0276] Optionally, since data is parallelized during deep neural network model training, meaning the tensor batch dimension is relatively insensitive to communication latency and bandwidth compared to other dimensions, and considering the high bandwidth and low latency of network communication between multiple GPUs within a server, while the high latency of communication between servers, the batch dimension is preferentially split into parallel distributions between nodes, while the model dimension is split into distributions within nodes. Specifically, assume the device matrix is ​​[D2, D1, D0]. 1) When constructing the device matrix based on the device topology information of the distributed cluster: prioritize arranging data processing devices within the same server along the D1 / D2 axis, and then arrange data processing devices between servers or within servers along the D0 axis. 2) When mapping tensors to the device matrix, identify the dimensions of the tensors, and map the batch dimension of the tensor to the inter-node, i.e., the device matrix D0 dimension, i.e., the non-batch dimension of the tensor, or the model dimension, to the D1 or D2 dimension.

[0277] For example, please refer to Figure 12 This is a schematic diagram of another embodiment for determining the overall tensor arrangement in this application.

[0278] Figure 12 This paper illustrates a parallel modeling process for a two-dimensional matrix multiplication operator. There are 16 devices in total, with 4 cards per machine. The input tensors of the operator are two-dimensional tensors M and N, and the output tensor of the operator is a two-dimensional tensor Q.

[0279] As shown in the figure, the first server has four data processing devices: A1, A2, A3, and A4; the second server has four data processing devices: B1, B2, B3, and B4; the third server has four data processing devices: C1, C2, C3, and C4; and the fourth server has four data processing devices: E1, E2, E3, and E4. Each element in the device matrix, i.e., each cube in the figure, corresponds to one data processing device. All elements in the device matrix represent all data processing devices in the distributed cluster used to execute the deep neural network model in this application. The device matrix [D2, D1, D0] in the figure is [2, 2, 4].

[0280] Tensor M is mapped to the D0 and D1 planes. Assuming the rows of tensor M represent the sample dimension, these rows are mapped to the D0 dimension. The sample dimension has lower communication requirements. Correspondingly, data processing devices between servers are arranged in the D0 dimension, while devices within the servers are arranged in the other dimensions. For example, A1, B1, C1, and E1 are data processing devices in different servers, therefore arranged along the D0 axis of the device matrix. A1 and A2 are two data processing devices within the same server, therefore arranged along either the D1 or D2 axis.

[0281] Therefore, this solution is designed to address the different bandwidths of devices, expressing the cluster topology through a device matrix, which can flexibly adapt to various hierarchical network topologies and reduce communication latency.

[0282] 1102. Determine the target tensor arrangement from multiple global tensor arrangements using a cost model;

[0283] The segmentation strategy is determined from the plurality of candidate segmentation strategies. Based on the candidate segmentation strategy, the tensor arrangement of each operator in the deep neural network model is obtained. A rearrangement operator is generated between consecutive tensors with different tensor arrangements. Thus, the tensor arrangements of all operators in the deep neural network model can be obtained, hereinafter referred to as the overall tensor arrangement. It is understood that multiple overall tensor arrangements can be obtained based on the segmentation strategy.

[0284] Based on each global tensor arrangement, a cost model is calculated. This cost model considers the overhead of both operators and rearrangement operators, specifically measured by the storage, computation, and communication overhead of the tensor shape approximation operator. Weighting coefficients control the proportion of computation and communication overhead in the loss function, adapting to different device platforms. The candidate segmentation strategy with the minimum cost model is selected from multiple candidate segmentation strategies as the target segmentation strategy, used for graph segmentation.

[0285] Optionally, the optimal global tensor arrangement is determined from multiple global tensor arrangements for execution. Figure 6 Corresponding implementation examples.

[0286] Optionally, an optimal segmentation strategy is determined based on multiple global tensor arrangements. Based on the optimal segmentation strategy, a global tensor arrangement is generated according to preset rules for execution. Figure 6 Corresponding implementation examples.

[0287] Optionally, a complete stand-alone forward computation graph is input, and a sliced ​​forward computation graph is output. A rearrangement operator is inserted into the sliced ​​forward computation graph. The output sliced ​​forward computation graph is then used to generate a reverse computation graph through automatic differentiation.

[0288] Optionally, the candidate segmentation strategy with the minimum loss among the multiple candidate segmentation strategies is determined by a cost model. The value is obtained by weighting and summing the data tensor size, communication tensor size, and parameter tensor size in the overall tensor arrangement, as well as the weight coefficients of the data tensor size, communication tensor size, and parameter tensor size. The data tensor size, communication tensor size, and parameter tensor size are respectively the storage space of the data tensor, communication tensor, and parameter tensor required to execute the deep neural network model based on the first candidate segmentation strategy.

[0289] The cost model defines the loss function as: alpha * size of the input operator data tensor + beta * size of the communication tensor + gamma * size of the operator parameter tensor.

[0290] Wherein, alpha represents the weighting factor for the size of the data tensor; beta represents the weighting factor for the size of the communication tensor; and gamma represents the weighting factor for the size of the parameter tensor. The specific values ​​of alpha, beta, and gamma are not limited here. Optionally, the values ​​of alpha, beta, and gamma can be flexibly set according to the type of data processing equipment.

[0291] The operators in the cost model include a sequence of rearranged operators. Different partitioning strategies have different operator costs, and the costs of rearranging operator sequences among operators also vary. These costs include storage and communication overhead. A storage overhead limit is set based on the actual device's memory size, and the combination of operator partitioning strategies with the lowest computational cost is sought.

[0292] Optionally, the shape of the input tensors for the forward and reverse operators is used to approximate the storage and computational overhead. Adjusting the ratio of alpha, beta, and gamma can adapt to different hardware platforms. For example, increasing beta means increasing the proportion of communication overhead, thus reducing the communication required for the search strategy.

[0293] The method for determining the overall tensor arrangement provided in this application embodiment can independently model operators and configure the tensor arrangements of operator input tensors and output tensors. The tensor arrangement transformation required between operators is generated by a tensor arrangement transformation module. The cost model considers both operator and tensor arrangement operator costs, approximating the storage, computation, and communication costs of operators using tensor shapes; weighting coefficients control the proportion of computation and communication costs in the loss function to adapt to different device platforms. Optionally, this solution can input a complete single-machine forward computation graph and output a sliced ​​forward computation graph, inserting other operators such as tensor arrangement and AllReduce into the sliced ​​forward computation graph. The output sliced ​​forward computation graph is automatically differentiated to generate a reverse computation graph.

[0294] The method for determining the overall tensor arrangement provided in this application embodiment, combined with Figure 6 and Figure 9The corresponding embodiments provide a flexible parallel mode configuration for the overall automatic parallel process. Each operator is modeled independently, and rearranged operator sequences are inserted between operators, adapting to the hybrid parallel requirements of various common networks and overcoming the shortcomings of existing technologies in supporting network types. The cost model considers both operator and rearrangement costs, and can search for the parallel scheme with the minimum overall cost. Tensor shape is used to approximate the computational cost, providing a good approximation of the actual cost without requiring extensive testing, achieving platform independence. The proportion of communication computation is controlled by weighting coefficients to adapt to different device platforms. There is no need to generate a reverse computation network; automatic differentiation and graph optimization functions are utilized. Tensor rearrangement supports arbitrary tensor arrangement transformations with low transformation overhead. Support for arbitrary tensor arrangement transformations allows operators to be modeled independently and flexibly. Besides being used for arrangement transformations between operators, tensor rearrangement can also be used for the distributed implementation of reshape operators. The topology-aware scheduling involved in this application, by adjusting the device matrix in the Tensor Layout, allows batch-dimensional parallelism, which is insensitive to communication latency and bandwidth, to be placed between servers, while model splitting is placed within the servers. Through device matrix configuration, it is simple and flexible to adapt to different cluster topologies.

[0295] Please see Figure 13 This is a schematic diagram of one embodiment of the data processing device in this application.

[0296] The data processing device provided in this application embodiment is applied to a distributed cluster, and the device includes:

[0297] The acquisition unit 1301 is used to acquire a deep neural network model, and the tensor arrangement of the input tensors of each operator in the deep neural network model and the tensor arrangement of the output tensors of each operator. The tensor arrangement includes a device matrix, tensor mapping, and tensor shape. The deep neural network model includes a first operator and a second operator. The first operator and the second operator are two consecutive operators in the deep neural network model, and the output tensor of the first operator is the input tensor of the second operator. The first tensor arrangement and the second tensor arrangement are inconsistent. Specifically, the first tensor arrangement is the tensor arrangement of the output tensor of the first operator, and the second tensor arrangement is the tensor arrangement of the input tensor of the second operator.

[0298] The determining unit 1302 is used to determine the slice calculation graph based on the tensor arrangement of the input tensors of each operator and the tensor arrangement of the output tensors of each operator;

[0299] The determining unit 1302 is further configured to determine a rearrangement operator between the first operator and the second operator, wherein the rearrangement operator is configured to convert the first tensor arrangement into the second tensor arrangement;

[0300] The determining unit 1302 is further configured to insert the rearrangement operator into the slice computation graph to determine the updated slice computation graph, which is used to indicate the part of the deep neural network model being executed.

[0301] Optionally, the device matrix of the first tensor arrangement is inconsistent with the device matrix of the second tensor arrangement, and / or the tensor mapping of the first tensor arrangement is inconsistent;

[0302] The determining unit 1302 is specifically used for:

[0303] Determine the intermediate tensor layout based on the first tensor layout and the second tensor layout;

[0304] A first reshaping operator is determined based on the first tensor arrangement and the intermediate tensor arrangement, along with the tensor arrangements of the input tensors and the output tensors of the first reshaping operator. The first reshaping operator is used to implement the transformation from the first tensor arrangement to the intermediate tensor arrangement; and / or,

[0305] The second reshaping operator is determined based on the second tensor arrangement and the intermediate tensor arrangement, as well as the tensor arrangement of the input tensor and the tensor arrangement of the output tensor of the second reshaping operator. The second reshaping operator is located between the first reshaping operator and the second operator, and the second reshaping operator is used to realize the transformation from the intermediate tensor arrangement to the second tensor arrangement.

[0306] Optionally, the device matrix arranged in the first tensor is different from the device matrix arranged in the second tensor;

[0307] The determining unit 1302 is specifically used for:

[0308] Based on the device matrix arranged in the first tensor and the device matrix arranged in the second tensor, an extended device matrix is ​​determined. The element-wise product of the extended device matrix is ​​the same as the element-wise product of the device matrix of the output tensor of the first operator and the element-wise product of the second device matrix of the input tensor of the second operator. Any element in the device matrix arranged in the first tensor and the device matrix arranged in the second tensor is equal to an element in the extended device matrix or the product of at least two elements of the extended device matrix.

[0309] Based on the extended device matrix, a first equivalent tensor arrangement equivalent to the first tensor arrangement and a second equivalent tensor arrangement equivalent to the second tensor arrangement are determined, wherein the device matrix of the first equivalent tensor arrangement is consistent with the device matrix of the second equivalent tensor arrangement.

[0310] When the tensor shape of the first equivalent tensor arrangement is consistent with the tensor shape of the second equivalent tensor arrangement, the intermediate tensor arrangement includes the first equivalent tensor arrangement and the second equivalent tensor arrangement, and the tensor shape is the number of elements in each dimension of the tensor.

[0311] Optionally, the tensor shape of the first equivalent tensor arrangement is consistent with the tensor shape of the second equivalent tensor arrangement, and the tensor mapping of the first equivalent tensor arrangement is inconsistent with the tensor mapping of the second equivalent tensor arrangement.

[0312] The determining unit 1302 is further configured to determine one or more tensor mapping transformation operators, the tensor mapping transformation operators including a splitting operator, a merging operator, or a communication operator, wherein the tensor arrangement of the input tensors of the one or more tensor mapping transformation operators is consistent with the first equivalent tensor arrangement, and the tensor arrangement of the output tensors of the one or more tensor mapping transformation operators is consistent with the second equivalent tensor arrangement, and the one or more tensor mapping transformation operators are used to determine the updated slice computation graph.

[0313] Optionally, when the tensor shape of the first equivalent tensor arrangement is inconsistent with the tensor shape of the second equivalent tensor arrangement, the determining unit 1302 is further configured to perform tensor shape normalization based on the first equivalent tensor arrangement and the second equivalent tensor arrangement, determine a third equivalent tensor arrangement equivalent to the first equivalent tensor arrangement, and a fourth equivalent tensor arrangement equivalent to the second equivalent tensor arrangement, wherein the device matrix of the third equivalent tensor arrangement is consistent with the device matrix of the fourth equivalent tensor arrangement, and the tensor shape of the third equivalent tensor arrangement is consistent with the tensor shape of the fourth equivalent tensor arrangement; the intermediate tensor arrangement includes the third equivalent tensor arrangement and the fourth equivalent tensor arrangement.

[0314] Optionally, the tensor mapping of the third equivalent tensor arrangement is inconsistent with the tensor mapping of the fourth equivalent tensor arrangement;

[0315] The determining unit 1302 is further configured to: determine one or more tensor mapping transformation operators, the tensor mapping transformation operators including a splitting operator, a merging operator, or a communication operator, wherein the tensor arrangement of the input tensors of the one or more tensor mapping transformation operators is consistent with the third equivalent tensor arrangement, and the tensor arrangement of the output tensors of the one or more tensor mapping transformation operators is consistent with the fourth equivalent tensor arrangement, and the one or more tensor mapping transformation operators are used to determine the updated slice computation graph.

[0316] Optionally, the device matrix of the first tensor arrangement is the same as the device matrix of the second tensor arrangement, and the tensor shape of the first tensor arrangement is different from the tensor shape of the second tensor arrangement.

[0317] The determining unit 1302 is specifically configured to: determine a fifth equivalent tensor arrangement that is equivalent to the first tensor arrangement and a sixth equivalent tensor arrangement that is equivalent to the second equivalent tensor arrangement based on the first tensor arrangement and the second tensor arrangement, wherein the device matrix of the fifth equivalent tensor arrangement is consistent with the device matrix of the sixth equivalent tensor arrangement, and the tensor shape of the fifth equivalent tensor arrangement is consistent with the tensor shape of the sixth equivalent tensor arrangement; the intermediate tensor arrangement includes the fifth equivalent tensor arrangement and the sixth equivalent tensor arrangement.

[0318] Optionally, the tensor mapping of the fifth equivalent tensor arrangement is inconsistent with the tensor mapping of the sixth equivalent tensor arrangement;

[0319] The determining unit 1302 is further configured to determine one or more tensor mapping transformation operators, the tensor mapping transformation operators including a splitting operator, a merging operator, or a communication operator, the tensor arrangement of the input tensors of the one or more tensor mapping transformation operators being consistent with the fifth equivalent tensor arrangement, the tensor arrangement of the output tensors of the one or more tensor mapping transformation operators being consistent with the sixth equivalent tensor arrangement, and the one or more tensor mapping transformation operators being used to determine the updated slice computation graph.

[0320] Optionally, the device matrix of the first tensor arrangement is the same as the device matrix of the second tensor arrangement, the tensor shape of the first tensor arrangement is the same as the tensor shape of the second tensor arrangement, and the tensor mapping of the first tensor arrangement is different from the tensor mapping of the second tensor arrangement.

[0321] The determining unit 1302 is specifically used for:

[0322] One or more tensor mapping transformation operators are determined, including splitting operators, merging operators, or communication operators. The one or more tensor mapping transformation operators are used to input the output tensor of the first operator and output the input tensor of the second operator.

[0323] Optionally, the acquisition unit 1301 is specifically used for:

[0324] Obtain a deep neural network model and a segmentation strategy, wherein the segmentation strategy includes the number of segments of the tensor of the deep neural network model in each dimension;

[0325] The determining module is specifically used to determine the tensor arrangement of the input tensors of each operator in the deep neural network model and the tensor arrangement of the output tensors of each operator based on the deep neural network model and the segmentation strategy.

[0326] Optionally, the segmentation strategy includes a first segmentation strategy and a second segmentation strategy;

[0327] The determining unit 1302 is specifically used for:

[0328] A first overall tensor arrangement corresponding to the first segmentation strategy and a second overall tensor arrangement corresponding to the second segmentation strategy are determined. The first overall tensor arrangement is the tensor arrangement of the input tensor of each operator and the tensor arrangement of the output tensor of each operator in the deep neural network model determined based on the first segmentation strategy. The second overall tensor arrangement is the tensor arrangement of the input tensor of each operator and the tensor arrangement of the output tensor of each operator in the deep neural network model determined based on the second segmentation strategy.

[0329] The first overall tensor arrangement is determined from the first overall tensor arrangement and the second overall tensor arrangement as the tensor arrangement of the input tensor and the tensor arrangement of the output tensor of each operator in the deep neural network model. The sum of the communication time and computation time required for the training of the deep neural network model based on the first overall tensor arrangement is less than the sum of the communication time and computation time required for the training of the deep neural network model based on the second overall tensor arrangement.

[0330] Optionally, the cost model of the first overall tensor arrangement is smaller than the cost model of the second overall tensor arrangement. The cost model of the first overall tensor arrangement is a value obtained by weighted summation based on the size of the data tensor, the size of the communication tensor, and the size of the parameter tensor in the first overall tensor arrangement, as well as the weight coefficients of the size of the data tensor, the weight coefficients of the size of the communication tensor, and the weight coefficients of the size of the parameter tensor.

[0331] The cost model for the second overall tensor arrangement is a value obtained by weighted summation based on the size of the data tensor, the size of the communication tensor, and the size of the parameter tensor in the second overall tensor arrangement, as well as the weight coefficients of the size of the data tensor, the size of the communication tensor, and the size of the parameter tensor.

[0332] Optionally, the segmentation strategy is a user-specified segmentation strategy.

[0333] Optionally, the input tensor of each operator in the deep neural network model includes a training dataset, which may include a text dataset, an image dataset, or an audio dataset.

[0334] Please see Figure 14 This is a schematic diagram of another embodiment of the data processing device in this application;

[0335] The data processing device provided in this embodiment can be a processor, a server, or a dedicated data processing device, etc. The specific form of the device is not limited in this embodiment.

[0336] The data processing device 1400 may vary considerably due to different configurations or performance, and may include one or more processors 1401 and a memory 1402, in which programs or data are stored.

[0337] The memory 1402 can be volatile or non-volatile memory. Optionally, the processor 1401 is one or more central processing units (CPUs), graphics processing units (GPUs), or other dedicated processors, such as Ascend. The CPU can be a single-core CPU or a multi-core CPU. The processor 1401 can communicate with the memory 1402 and execute a series of instructions stored in the memory 1402 on the data processing device 1400.

[0338] The data processing device 1400 also includes one or more wired or wireless network interfaces 1403, such as Ethernet interfaces.

[0339] Optionally, although Figure 14 As not shown in the diagram, the data processing device 1400 may also include one or more power supplies; one or more input / output interfaces, which can be used to connect to a monitor, mouse, keyboard, touch screen device or sensing device, etc. The input / output interfaces are optional components and may or may not be present, and are not limited here.

[0340] The process executed by the processor 1401 in the data processing device 1400 in this embodiment can refer to the method flow described in the foregoing method embodiment, and will not be repeated here.

[0341] Please refer to Figure 15 This is a chip hardware structure diagram provided in an embodiment of this application.

[0342] The deep neural network algorithm involved in the embodiments of this application can be used in... Figure 15 It is executed in the NPU chip shown.

[0343] The Neural Processing Unit (NPU) 50 is mounted as a coprocessor on the host CPU, which allocates tasks to it. The core of the NPU is the arithmetic circuit 503, which is controlled by the controller 504 to retrieve matrix data from memory and perform multiplication operations.

[0344] In some implementations, the arithmetic circuit 503 internally includes multiple process engines (PEs). In some implementations, the arithmetic circuit 503 is a two-dimensional pulsating array. The arithmetic circuit 503 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 503 is a general-purpose matrix processor.

[0345] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 502 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 501 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 508.

[0346] Unified memory 506 is used to store input and output data. Weight data is directly transferred to weight memory 502 via direct memory access controller 505 (DMAC). Input data is also transferred to unified memory 506 via DMAC.

[0347] BIU stands for Bus Interface Unit 510, which is used for interaction between the AXI bus and the DMAC and the instruction fetch buffer 509.

[0348] The bus interface unit 510 (BIU) is used by the instruction fetch memory 509 to fetch instructions from external memory, and also by the memory access controller 505 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0349] The DMAC is mainly used to move input data from external memory DDR to unified memory 506, or to weight data to weight memory 502, or to input data to input memory 501.

[0350] The vector computation unit 507 may include multiple computation processing units, which, when necessary, further process the output of the computation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for computation in non-convolutional / FC layer networks of neural networks, such as pooling, batch normalization, and local response normalization.

[0351] In some implementations, vector computation unit 507 stores the processed output vector in unified memory 506. For example, vector computation unit 507 can apply a nonlinear function to the output of arithmetic circuit 503, such as a vector of accumulated values, to generate activation values. In some implementations, vector computation unit 507 generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to arithmetic circuit 503, for example, for use in subsequent layers of a neural network.

[0352] The instruction fetch buffer 509 connected to the controller 504 is used to store the instructions used by the controller 504;

[0353] Unified memory 506, input memory 501, weighted memory 502, and instruction fetch memory 509 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0354] The operations of each layer in the deep neural network, i.e. the operators in this embodiment, can be executed by the matrix calculation unit or the vector calculation unit 507.

[0355] The methods described in this application can be applied to a processor, or the steps of the methods described in the above application can be implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the methods described in the above application can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and an NP, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in this application. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory, and the processor reads the information from the memory and, in conjunction with its hardware, completes the steps of the above method. Although only one processor is shown in the figure, the device may include multiple processors or the processor may include multiple processing units. Specifically, the processor may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor.

[0356] Memory is used to store computer instructions executed by the processor. Memory can be a storage circuit or a memory. Memory can be volatile memory or non-volatile memory, or it can include both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. Memory can be independent of the processor or it can be a storage unit within the processor; this is not limited here. Although only one memory is shown in the figure, the device may include multiple memories or the memory may include multiple storage units.

[0357] A transceiver is used to enable content interaction between the processor and other units or network elements. Specifically, the transceiver can be a communication interface of the device, a transceiver circuit or communication unit, or a transceiver transceiver. The transceiver can also be the processor's communication interface or transceiver circuit. In one possible implementation, the transceiver can be a transceiver chip. The transceiver can also include a transmitting unit and / or a receiving unit. In one possible implementation, the transceiver can include at least one communication interface. In another possible implementation, the transceiver can also be a unit implemented in software. In various embodiments of this application, the processor can interact with other units or network elements through the transceiver. For example, the processor can acquire or receive content from other network elements through the transceiver. If the processor and the transceiver are physically separate components, the processor can interact with other units of the device without going through the transceiver.

[0358] In one possible implementation, the processor, memory, and transceiver can be interconnected via a bus. This bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be categorized as an address bus, data bus, control bus, etc.

[0359] In this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being better or more advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0360] In the various embodiments of this application, numerous examples have been provided for ease of understanding. However, these examples are merely illustrative and do not imply the best way to implement this application.

[0361] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, they can be implemented in whole or in part in the form of a computer program product.

[0362] The computer program product includes one or more computer instructions. When the computer execution instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0363] The technical solutions provided in this application have been described in detail above. Specific examples have been used in this application to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data processing method, characterized by, The method, applied to a data processing device in a distributed cluster, includes: A slice computation graph of the data processing device is obtained. The slice computation graph is used to represent the deep neural network model. The slice computation graph includes a first tensor arrangement, a second tensor arrangement, and a rearrangement operator. The first tensor arrangement and the second tensor arrangement are not the same. The first tensor arrangement is the tensor arrangement of the output tensor of the first operator, and the second tensor arrangement is the tensor arrangement of the input tensor of the second operator. The output tensor of the first operator is the input tensor of the second operator. The rearrangement operator is used to convert the first tensor arrangement into the second tensor arrangement. The first tensor arrangement and the second tensor arrangement include a device matrix, a tensor mapping, and a tensor shape. Multiple data processing devices corresponding to all elements in the device matrix are used to execute the deep neural network model in parallel. The tensor shape indicates the number of elements in each dimension of the tensor, and the tensor mapping indicates the mapping of each dimension of the tensor on the device matrix. The slice computation graph is processed to train the deep neural network model.

2. The method according to claim 1, characterized in that, The process of obtaining the slice calculation map of the data processing device includes: The rearrangement operator is inserted between the first tensor arrangement and the second tensor arrangement.

3. The method according to claim 1 or 2, characterized in that, The device matrix of the first tensor arrangement is inconsistent with the device matrix of the second tensor arrangement, and / or the tensor mapping of the first tensor arrangement is inconsistent; The method further includes: Determine the intermediate tensor layout based on the first tensor layout and the second tensor layout; A first reshaping operator is determined based on the first tensor arrangement and the intermediate tensor arrangement, along with the tensor arrangements of the input tensors and the output tensors of the first reshaping operator. The first reshaping operator is used to implement the transformation from the first tensor arrangement to the intermediate tensor arrangement; and / or, The second reshaping operator is determined based on the second tensor arrangement and the intermediate tensor arrangement, as well as the tensor arrangement of the input tensor and the tensor arrangement of the output tensor of the second reshaping operator. The second reshaping operator is located between the first reshaping operator and the second operator, and the second reshaping operator is used to realize the transformation from the intermediate tensor arrangement to the second tensor arrangement.

4. The method according to claim 3, characterized in that, The device matrix of the first tensor arrangement is inconsistent with the device matrix of the second tensor arrangement; The step of determining the intermediate tensor arrangement based on the first tensor arrangement and the second tensor arrangement includes: Based on the device matrix arranged in the first tensor and the device matrix arranged in the second tensor, an extended device matrix is ​​determined. The element-wise product of the extended device matrix is ​​the same as the element-wise product of the device matrix of the output tensor of the first operator and the element-wise product of the second device matrix of the input tensor of the second operator. Any element in the device matrix arranged in the first tensor and the device matrix arranged in the second tensor is equal to an element in the extended device matrix or the product of at least two elements of the extended device matrix. Based on the extended device matrix, a first equivalent tensor arrangement equivalent to the first tensor arrangement and a second equivalent tensor arrangement equivalent to the second tensor arrangement are determined, wherein the device matrix of the first equivalent tensor arrangement is consistent with the device matrix of the second equivalent tensor arrangement. When the tensor shape of the first equivalent tensor arrangement is consistent with the tensor shape of the second equivalent tensor arrangement, the intermediate tensor arrangement includes the first equivalent tensor arrangement and the second equivalent tensor arrangement, and the tensor shape is the number of elements in each dimension of the tensor.

5. The method according to claim 4, characterized in that, The tensor shape of the first equivalent tensor arrangement is the same as that of the second equivalent tensor arrangement, and the tensor mapping of the first equivalent tensor arrangement is different from that of the second equivalent tensor arrangement. The method further includes: One or more tensor mapping transformation operators are determined, including a splitting operator, a merging operator, or a communication operator. The tensor arrangement of the input tensors of the one or more tensor mapping transformation operators is consistent with the first equivalent tensor arrangement, and the tensor arrangement of the output tensors of the one or more tensor mapping transformation operators is consistent with the second equivalent tensor arrangement. The one or more tensor mapping transformation operators are used to determine the updated slice computation graph.

6. The method according to claim 4, characterized in that, When the tensor shape of the first equivalent tensor arrangement is inconsistent with the tensor shape of the second equivalent tensor arrangement, the method further includes: Tensor shape normalization is performed based on the first and second equivalent tensor arrangements to determine a third equivalent tensor arrangement that is equivalent to the first equivalent tensor arrangement and a fourth equivalent tensor arrangement that is equivalent to the second equivalent tensor arrangement. The device matrix of the third equivalent tensor arrangement is consistent with the device matrix of the fourth equivalent tensor arrangement, and the tensor shape of the third equivalent tensor arrangement is consistent with the tensor shape of the fourth equivalent tensor arrangement. The intermediate tensor arrangement includes the third and fourth equivalent tensor arrangements.

7. The method according to claim 6, characterized in that, The tensor mapping of the third equivalent tensor arrangement is inconsistent with the tensor mapping of the fourth equivalent tensor arrangement. The method further includes: One or more tensor mapping transformation operators are determined, including a splitting operator, a merging operator, or a communication operator. The tensor arrangement of the input tensors of the one or more tensor mapping transformation operators is consistent with the third equivalent tensor arrangement, and the tensor arrangement of the output tensors of the one or more tensor mapping transformation operators is consistent with the fourth equivalent tensor arrangement. The one or more tensor mapping transformation operators are used to determine the updated slice computation graph.

8. The method according to claim 3, characterized in that, The device matrix of the first tensor arrangement is the same as the device matrix of the second tensor arrangement, and the tensor shape of the first tensor arrangement is different from the tensor shape of the second tensor arrangement. The step of determining the intermediate tensor arrangement based on the first tensor arrangement and the second tensor arrangement includes: Based on the first tensor arrangement and the second tensor arrangement, a fifth equivalent tensor arrangement equivalent to the first tensor arrangement and a sixth equivalent tensor arrangement equivalent to the second tensor arrangement are determined. The device matrix of the fifth equivalent tensor arrangement is consistent with the device matrix of the sixth equivalent tensor arrangement, and the tensor shape of the fifth equivalent tensor arrangement is consistent with the tensor shape of the sixth equivalent tensor arrangement. The intermediate tensor arrangement includes the fifth equivalent tensor arrangement and the sixth equivalent tensor arrangement.

9. The method according to claim 8, characterized in that, The tensor mapping of the fifth equivalent tensor arrangement is inconsistent with the tensor mapping of the sixth equivalent tensor arrangement. The method further includes: One or more tensor mapping transformation operators are determined, including a splitting operator, a merging operator, or a communication operator. The tensor arrangement of the input tensors of the one or more tensor mapping transformation operators is consistent with the fifth equivalent tensor arrangement, and the tensor arrangement of the output tensors of the one or more tensor mapping transformation operators is consistent with the sixth equivalent tensor arrangement. The one or more tensor mapping transformation operators are used to determine the updated slice computation graph.

10. The method according to claim 1 or 2, characterized in that, The tensor arrangement includes a device matrix, a tensor mapping, and a tensor shape. The device matrix of the first tensor arrangement is the same as that of the second tensor arrangement. The tensor shape of the first tensor arrangement is the same as that of the second tensor arrangement. However, the tensor mapping of the first tensor arrangement is different from that of the second tensor arrangement. The method further includes: One or more tensor mapping transformation operators are determined, including splitting operators, merging operators, or communication operators. The one or more tensor mapping transformation operators are used to input the output tensor of the first operator and output the input tensor of the second operator.

11. The method according to claim 1 or 2, characterized in that, The deep neural network model includes a tensor arrangement of multiple operators; The tensor arrangement of the multiple operators is determined based on the deep neural network model and the segmentation strategy. The segmentation strategy includes the number of segments of the tensor of the deep neural network model in each dimension.

12. The method according to claim 11, characterized in that, The segmentation strategy includes a first segmentation strategy and a second segmentation strategy; The tensor arrangement of the plurality of operators includes: The first overall tensor arrangement corresponding to the first segmentation strategy and the second overall tensor arrangement corresponding to the second segmentation strategy, wherein the first overall tensor arrangement is a tensor arrangement of multiple operators in the deep neural network model determined based on the first segmentation strategy, and the second overall tensor arrangement is a tensor arrangement of multiple operators in the deep neural network model determined based on the second segmentation strategy. The method further includes: The first overall tensor arrangement is determined to be a tensor arrangement of multiple operators in the deep neural network model, wherein the sum of the communication time and computation time required for the first overall tensor arrangement to perform training of the deep neural network model is less than the sum of the communication time and computation time required for the second overall tensor arrangement to perform training of the deep neural network model.

13. The method according to claim 12, characterized in that, The cost model of the first overall tensor arrangement is smaller than the cost model of the second overall tensor arrangement. The cost model of the first overall tensor arrangement is a value obtained by weighted summation based on the size of the data tensor, the size of the communication tensor, and the size of the parameter tensor in the first overall tensor arrangement, as well as the weight coefficients of the size of the data tensor, the weight coefficient of the size of the communication tensor, and the weight coefficient of the size of the parameter tensor. The cost model for the second overall tensor arrangement is a value obtained by weighted summation based on the size of the data tensor, the size of the communication tensor, and the size of the parameter tensor in the second overall tensor arrangement, as well as the weight coefficients of the size of the data tensor, the size of the communication tensor, and the size of the parameter tensor.

14. The method according to claim 11, characterized in that, The segmentation strategy is a user-specified segmentation strategy.

15. The method according to claim 1 or 2, characterized in that, The inputs to multiple operators in the deep neural network model include a training dataset, which may include a text dataset, an image dataset, or an audio dataset.

16. The method according to claim 1 or 2, characterized in that, The process of obtaining the slice calculation map of the data processing device includes: Receive the slice calculation graph.

17. A data processing system, characterized in that, The system, used in distributed clusters, includes: An acquisition unit is used to acquire a deep neural network model and a tensor arrangement of multiple operators in the deep neural network model. The tensor arrangement includes a device matrix, a tensor mapping, and a tensor shape. The deep neural network model includes a first operator and a second operator, and the output tensor of the first operator is the input tensor of the second operator. The first tensor arrangement and the second tensor arrangement are not the same, wherein the first tensor arrangement is the tensor arrangement of the output tensor of the first operator, and the second tensor arrangement is the tensor arrangement of the input tensor of the second operator. Multiple data processing devices corresponding to all elements in the device matrix are used to execute the deep neural network model in parallel. The device matrix is ​​related to the number and connection relationship of multiple data processing devices in the distributed cluster. The tensor shape indicates the number of elements in each dimension of the tensor, and the tensor mapping indicates the mapping of each dimension of the tensor onto the device matrix. A determining unit is used to generate a slice computation graph corresponding to the deep neural network model. The slice computation graph includes a first tensor arrangement, a second tensor arrangement, and a rearrangement operator between the first tensor arrangement and the second tensor arrangement. The rearrangement operator is used to convert the first tensor arrangement into the second tensor arrangement.

18. The system according to claim 17, characterized in that, The system is also used to perform the method of any one of claims 2 to 16.

19. A chip used in a data processing device in a distributed cluster, comprising a processor and a memory; The memory is connected to the processor; The processor is configured to perform the following steps based on the program stored in the memory: A slice computation graph of the data processing device is obtained. The slice computation graph is used to represent the deep neural network model. The slice computation graph includes a first tensor arrangement, a second tensor arrangement, and a rearrangement operator. The first tensor arrangement and the second tensor arrangement are not the same. The first tensor arrangement is the tensor arrangement of the output tensor of the first operator, and the second tensor arrangement is the tensor arrangement of the input tensor of the second operator. The output tensor of the first operator is the input tensor of the second operator. The rearrangement operator is used to convert the first tensor arrangement into the second tensor arrangement. The first tensor arrangement and the second tensor arrangement include a device matrix, a tensor mapping, and a tensor shape. Multiple data processing devices corresponding to all elements in the device matrix are used to execute the deep neural network model in parallel. The tensor shape indicates the number of elements in each dimension of the tensor, and the tensor mapping indicates the mapping of each dimension of the tensor on the device matrix. The slice computation graph is processed to train the deep neural network model.

20. A data processing system, characterized in that, The device includes a processor and a memory interconnected thereto, wherein the memory is used to store a computer program, the computer program including program instructions, and the processor is used to invoke the program instructions to perform the method as described in any one of claims 1 to 16.

21. A computer program product containing instructions, characterized in that, When it is run on a computer, it causes the computer to perform the method as described in any one of claims 1 to 16.

22. A computer-readable storage medium comprising instructions, characterized in that, When the instructions are executed on a computer, the computer performs the method as described in any one of claims 1 to 16.

23. A distributed cluster, characterized in that, The distributed cluster includes one or more data processing systems as described in claim 17 or 18.

Images