A data processing method and device thereof

Abstract

The present application discloses a data processing method, which relates to the field of artificial intelligence, and includes: determining a processing method corresponding to the neural network model according to a data flow graph, wherein the processing method includes, in addition to a method for dividing the input tensor of an operator, a method for rearranging the tensors between operators. By determining the method for rearranging the operators, the present application can increase the selection of methods for dividing the input tensors of the operators, thereby improving the processing efficiency of the data flow graph.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a data processing method and device thereof. Background Art

[0002] Deep neural networks have made great progress in recent years, showing high accuracy in tasks such as vision and translation. However, neural networks with high accuracy require huge training costs, which come from a large amount of training data and model parameters.

[0003] In order to solve the challenge of large amounts of training data, one solution is data parallelism. Each computing device has a full copy of the model to be trained, and each computing device is only responsible for training a portion of the slice of the tensor. During the training process, the device needs to synchronize the local model parameters after each iteration. When the parameters of the model to be trained are small and the model is composed of operators with high computational overhead (such as convolution operations), data parallelism can achieve better performance.

[0004] However, in the prior art, after a computing device performs an operation corresponding to a certain operator, in order to ensure that subsequent operators can continue to perform operations, the tensors in all layers of the entire network are only split according to the same pattern, which makes the splitting strategy of the prior art have great limitations. Summary of the invention

[0005] In a first aspect, the present application provides a data processing method, which rearranges the tensors obtained by performing operations corresponding to operators, so that there are more options for splitting the input tensors of subsequent operators, and accordingly, the processing efficiency of the data flow graph can be improved. The data processing method includes:

[0006] Obtaining a data flow graph of a neural network model, wherein the data flow graph includes a first operator and a second operator of the neural network model;

[0007] Determining a processing method corresponding to the neural network model according to the data flow graph, the processing method comprising a first segmentation method for segmenting an input tensor of the first operator, a second segmentation method for segmenting an input tensor of the second operator, and a rearrangement method for rearranging tensors between the first operator and the second operator;

[0008] Among them, the first splitting method is used to split the input tensor of the first operator to obtain M first sub-tensors, and the rearrangement method is used to rearrange the M second sub-tensors to obtain the first tensor, the M second sub-tensors are the outputs of the M first sub-tensors after being processed by the first operator, and the first tensor is the input tensor of the second operator, and M is a positive integer greater than 1.

[0009] Since the number of parameters included in each dimension of the sub-tensor obtained by the operation corresponding to the operator after the split tensor is obtained, the number of parameters included in each dimension of the sub-tensor obtained by the operation corresponding to the operator after the unsplit tensor is obtained, therefore, the selection of the splitting method corresponding to the input tensor of the subsequent operator for the sub-tensor is very limited. In this embodiment, since the sub-tensor obtained by the operation corresponding to the operator is rearranged, it is equivalent to restoring the tensor obtained by the operation corresponding to the operator after the unsplit tensor is obtained. At this time, compared with the selection of the splitting method corresponding to the input tensor of the subsequent operator, the selection of the splitting method corresponding to the input tensor of the rearranged tensor by the subsequent operator is more, that is, through the definition of rearrangement, the splitting method of the input tensor of the operator can be more selected, and accordingly, there are more processing methods that can be used as candidates, so that different operators can use different corresponding splitting methods to split the input tensor, that is, each uses a splitting method suitable for itself. This is very helpful for improving the processing efficiency of the data flow graph, and can also reduce the overhead of the device in calculating the data flow graph. Because a data flow graph often includes different operators, and different operators may be suitable for different methods of splitting input data. An operator can only guarantee its computational efficiency if it splits input data in the most suitable way. However, if different operators split input data in different ways, the resulting output data formats will be different. During the execution of the data flow graph, the output data of one operator is often the input data of another operator. Therefore, in order to execute the entire data flow graph, the existing technology chooses to let all operators in the data flow graph use the same method for splitting.

[0010] In a possible implementation of the first aspect, each second sub-tensor of the M second sub-tensors is obtained by performing an operation corresponding to the first operator on a corresponding computing device on a corresponding first sub-tensor of the M first sub-tensors.

[0011] In a possible implementation of the first aspect, the first segmentation method and the second segmentation method are different segmentation methods.

[0012] In a possible implementation of the first aspect, the tensor rearrangement method is used to obtain the first tensor by transferring the M second sub-tensors to the same computing device for rearrangement.

[0013] In a possible implementation of the first aspect, a result obtained after the operation corresponding to the first operator is performed on an operating device on the input tensor of the first operator is the same as the first tensor.

[0014] It should be understood that the result obtained after the operation corresponding to the first operator is performed on the input tensor of the first operator on a computing device can be exactly the same as the first tensor, or, only the arrangement of data in each dimension in the tensor is different, but the overall data content included is the same.

[0015] In a possible implementation of the first aspect, the second sub-tensor is a P-dimensional tensor, the sub-tensor obtained after splitting the first tensor by the second splitting method is a Q-dimensional tensor, and P is not an integer multiple of Q.

[0016] In a possible implementation of the first aspect, the neural network model includes N operators, the N operators include the first operator and the second operator, and determining the processing method corresponding to the neural network model according to the data flow graph includes:

[0017] Determine multiple candidate processing methods corresponding to the neural network model according to the data flow graph; wherein each of the multiple candidate processing methods includes a tensor splitting method for splitting the input tensor of each operator in the N operators, and a rearrangement method for rearranging the tensors between at least two operators in the N operators, the tensor splitting method is used to split the input tensor of the corresponding operator, and the tensor rearrangement method is used to transmit the operation result obtained after performing the operation corresponding to the operator to a computing device for rearrangement;

[0018] The processing method is determined from the multiple candidate processing methods according to the overhead value corresponding to each candidate processing method in the multiple candidate processing methods, wherein the overhead value includes a memory overhead value corresponding to the tensor splitting method of each operator in the N operators and a first communication overhead value corresponding to the tensor rearrangement method included in each candidate processing method, the memory overhead value represents the memory overhead generated when performing the operation corresponding to the operator, the first communication overhead value represents the communication overhead generated in the process of transmitting the operation result to a computing device, and the processing method is the candidate processing method with the smallest overhead value among the at least one candidate processing method.

[0019] In an embodiment of the present application, a constraint condition may be added to the process of determining the processing method: the memory overhead of the determined processing method is less than the upper limit of the memory overhead that each computing device can bear.

[0020] This application defines a rearrangement method that allows the front and rear operators to have different ways of splitting the input tensor. This makes it possible to convert model parallelism to data parallelism, or data parallelism to model parallelism in a hybrid parallel way, making the strategy search space larger. In addition, this application implements the function of calculating memory cost and communication cost. Given a processing method, the memory overhead and communication overhead consumed by the operator and rearrangement can be calculated, and the definition of the overhead value includes both memory overhead and communication overhead. It is possible to model the optimal memory overhead and communication overhead.

[0021] In a possible implementation of the first aspect, the N operators include a third operator, each candidate processing method includes a target splitting method for splitting the input tensor of the third operator, the third operator is used to perform a first operation on a sub-tensor obtained by splitting the input tensor of the third operator by the target splitting method, and transmit the operation result obtained by the operation to a computing device for a second operation, and the overhead value also includes a second communication overhead, which represents the communication overhead generated in the process of transmitting the operation result to a computing device when running the third operator.

[0022] In a possible implementation of the first aspect, the memory overhead value is related to the number and type of parameters included in the M first sub-tensors.

[0023] In a possible implementation of the first aspect, the first communication overhead value is related to the number and type of parameters included in the M second sub-tensors.

[0024] In a possible implementation of the first aspect, the second communication overhead value is related to the number and type of parameters included in the M second sub-tensors.

[0025] In a possible implementation of the first aspect, the overhead value is an average value of the memory overhead value, the first communication overhead value, and the second communication overhead.

[0026] In a possible implementation of the first aspect, the method further includes:

[0027] Determine multiple processing methods corresponding to the neural network model according to the data flow graph;

[0028] The plurality of candidate processing methods are determined from the plurality of processing methods according to a memory overhead value or a first communication overhead value corresponding to each processing method in the plurality of processing methods, wherein the plurality of candidate processing methods are part of the plurality of processing methods.

[0029] In the embodiment of the present application, after the processing methods are generated, they can be arranged in a non-decreasing order of memory usage, and a strategy can be selected according to a fixed step size. Such a selection method can effectively reduce the search time of the processing method while maintaining accuracy to a certain extent.

[0030] In a possible implementation of the first aspect, the data flow graph further includes a third operator of the neural network model, an output tensor of the third operator is an input tensor of the second operator, and the method further includes:

[0031] The first operator is merged into the second operator so that the second segmentation method corresponding to the second operator is associated with the first segmentation method and the tensor rearrangement method.

[0032] In an embodiment of the present application, the first operator is merged into the second operator. At the same time, the first segmentation method corresponding to the input tensor of the first operator and the rearrangement method between the first operator and the second operator are also merged into the second operator, and a piece of data including the first segmentation method, the tensor rearrangement method and the second segmentation method is formed (i.e., an association between the first segmentation method, the tensor rearrangement method and the second segmentation method is established).

[0033] In a possible implementation of the first aspect, the data flow graph further includes a fourth operator of the neural network model, an output tensor of the first operator is an input tensor of the fourth operator, and the method further includes:

[0034] The second operator is merged into the first operator, so that the first segmentation method corresponding to the first operator is associated with the tensor rearrangement method and the second segmentation method.

[0035] In an embodiment of the present application, the second operator is merged into the first operator. At the same time, the second segmentation method corresponding to the input tensor of the second operator and the rearrangement method between the first operator and the second operator are also merged into the first operator, and a piece of data including the first segmentation method, the tensor rearrangement method and the second segmentation method is formed (i.e., an association between the first segmentation method, the tensor rearrangement method and the second segmentation method is established).

[0036] In a possible implementation of the first aspect, the data flow graph further includes a fifth operator and a sixth operator of the neural network model, wherein the first tensor is a first input tensor of the second operator and a first input tensor of the fifth operator, and the output tensor of the sixth operator is a second input tensor of the second operator and a second input tensor of the fifth operator, and the method further includes:

[0037] Acquire a third slicing method for slicing the input tensor of the sixth operator, a fourth slicing method for slicing the second input tensor of the second operator, and a first tensor rearrangement method for rearranging the tensors between the fourth operator and the second operator; the third slicing method is used to slice the second input tensor of the sixth operator to obtain a plurality of third sub-tensors, the first tensor rearrangement method is used to rearrange a plurality of fourth sub-tensors obtained after performing operations corresponding to the sixth operator on the plurality of third sub-tensors to obtain a third tensor, and the fourth slicing method is used to slice the third tensor;

[0038] The sixth operator is merged into the first operator, so that the first segmentation method corresponding to the first operator establishes a first association relationship with the third segmentation method, the fourth segmentation method and the first tensor rearrangement method.

[0039] In a possible implementation of the first aspect, the method further includes:

[0040] The sixth operator is merged into the first operator, so that the first segmentation method corresponding to the first operator establishes a second association relationship with the third segmentation method and the second segmentation method.

[0041] This application adds three new graph contraction operations to support more complex neural network strategy search. It adds MergeElimination, Contract Elimination, and Star Elimination operations, and provides a method for calculating the cost value after contraction.

[0042] The Merge Elimination operation may represent the above-mentioned merging of the first operator into the second operator, so that the second segmentation method corresponding to the second operator establishes an association relationship with the first segmentation method and the tensor rearrangement method;

[0043] The Contract Elimination operation may represent the above-mentioned merging of the second operator into the first operator, so that the first segmentation method corresponding to the first operator is associated with the tensor rearrangement method and the second segmentation method;

[0044] The Star Elimination operation may represent the above-mentioned merging of the sixth operator into the first operator, so that the first segmentation method corresponding to the first operator establishes a first association relationship with the third segmentation method, the fourth segmentation method and the first tensor rearrangement method.

[0045] In a possible implementation of the first aspect, the neural network model is used to process image data, audio data, video data, or text data;

[0046] The input tensor of the operator included in the neural network model is the image data, the audio data, the video data or the text data; or,

[0047] The input tensor of the operator included in the neural network model is obtained by processing the image data, the audio data, the video data or the text data using at least one operator included in the neural network model.

[0048] In a second aspect, the present application provides a data processing method, the method comprising:

[0049] Acquire a neural network model and a segmentation method corresponding to the neural network model, wherein the neural network model includes a first operator and a second operator; the segmentation method includes a first segmentation method for segmenting an input tensor of the first operator, a second segmentation method for segmenting an input tensor of the second operator, and a rearrangement method for rearranging tensors between the first operator and the second operator;

[0050] Splitting the input tensor of the first operator by the first splitting method to obtain M first sub-tensors, wherein the M first sub-tensors include a first target sub-tensor;

[0051] Processing the first target sub-tensor using the first operator to obtain a second target sub-tensor;

[0052] In an embodiment of the present application, the computing device may call the code corresponding to the first operator and process the first target sub-tensor based on the called code.

[0053] Receive at least one second sub-tensor sent by at least one computing device; wherein each second sub-tensor is obtained by performing an operation corresponding to the first operator on a first sub-tensor other than the first target sub-tensor among the M first sub-tensors by the corresponding computing device;

[0054] Rearranging the second target sub-tensor and the at least one second sub-tensor by the rearrangement method to obtain a first tensor;

[0055] The first tensor is split using the second splitting method.

[0056] In a possible implementation of the second aspect, each second sub-tensor of the M second sub-tensors is transmitted to a corresponding computing device for a corresponding first sub-tensor of the M first sub-tensors, and is obtained by performing an operation corresponding to the first operator on the corresponding computing device.

[0057] In a possible implementation of the second aspect, the tensor rearrangement method is used to obtain the first tensor by transferring the M second sub-tensors to the same computing device for rearrangement.

[0058] In a possible implementation of the second aspect, a result obtained after the operation corresponding to the first operator is performed on an operating device on the input tensor of the first operator is the same as the first tensor.

[0059] In a possible implementation of the second aspect, the second sub-tensor is a P-dimensional tensor, the sub-tensor obtained after splitting the first tensor by the second splitting method is a Q-dimensional tensor, and P is not an integer multiple of Q.

[0060] In a possible implementation of the second aspect, the neural network model is used to process image data, audio data, video data, or text data;

[0061] The input tensor of the operator included in the neural network model is the image data, the audio data, the video data or the text data; or,

[0062] The input tensor of the operator included in the neural network model is obtained by processing the image data, the audio data, the video data or the text data using at least one operator included in the neural network model.

[0063] In a third aspect, the present application provides a data processing method, characterized in that the method comprises:

[0064] Obtaining a neural network model and a processing method corresponding to the neural network model, wherein the neural network model includes a first operator and a second operator; the processing method includes a first segmentation method for segmenting an input tensor of the first operator and a second segmentation method for segmenting an input tensor of the second operator;

[0065] Splitting the input tensor of the first operator by the first splitting method to obtain M first sub-tensors, wherein the M first sub-tensors include a first target sub-tensor;

[0066] Processing the first target sub-tensor using the first operator to obtain a second target sub-tensor;

[0067] Sending the second target sub-tensor to a computing device;

[0068] receiving a first tensor sent by the computing device, where the first tensor is obtained by the computing device by rearranging a plurality of second sub-tensors through a rearrangement method, where the plurality of second sub-tensors include the second target sub-tensor;

[0069] The first tensor is split using the second splitting method.

[0070] In a possible implementation of the third aspect, a result obtained after the operation corresponding to the first operator is performed on an operation device on the input tensor of the first operator is the same as the first tensor.

[0071] In a possible implementation of the third aspect, the second sub-tensor is a P-dimensional tensor, the sub-tensor obtained after splitting the first tensor by the second splitting method is a Q-dimensional tensor, and P is not an integer multiple of Q.

[0072] In a possible implementation of the third aspect, the neural network model is used to process image data, audio data, video data, or text data;

[0073] The input tensor of the operator included in the neural network model is the image data, the audio data, the video data or the text data; or,

[0074] The input tensor of the operator included in the neural network model is obtained by processing the image data, the audio data, the video data or the text data using at least one operator included in the neural network model.

[0075] In a fourth aspect, the present application provides a data processing device, the device comprising:

[0076] An acquisition module, used to acquire a data flow graph of a neural network model, wherein the data flow graph includes a first operator and a second operator of the neural network model;

[0077] a determination module, configured to determine a processing method corresponding to the neural network model according to the data flow graph, the processing method comprising a first segmentation method for segmenting an input tensor of the first operator, a second segmentation method for segmenting an input tensor of the second operator, and a rearrangement method for rearranging tensors between the first operator and the second operator;

[0078] Among them, the first splitting method is used to split the input tensor of the first operator to obtain M first sub-tensors, and the rearrangement method is used to rearrange the M second sub-tensors to obtain the first tensor, the M second sub-tensors are the outputs of the M first sub-tensors after being processed by the first operator, and the first tensor is the input tensor of the second operator, and M is a positive integer greater than 1.

[0079] In a possible implementation of the fourth aspect, each second sub-tensor of the M second sub-tensors is transmitted to a corresponding computing device for a corresponding first sub-tensor of the M first sub-tensors, and is obtained by performing an operation corresponding to the first operator on the corresponding computing device.

[0080] In a possible implementation of the fourth aspect, the tensor rearrangement method is used to obtain the first tensor by transferring the M second sub-tensors to the same computing device for splicing.

[0081] In a possible implementation of the fourth aspect, a result obtained after the input tensor of the first operator is operated on a computing device and the operation corresponding to the first operator is performed is the same as the first tensor.

[0082] In a possible implementation of the fourth aspect, the second sub-tensor is a P-dimensional tensor, the sub-tensor obtained after splitting the first tensor by the second splitting method is a Q-dimensional tensor, and P is not an integer multiple of Q.

[0083] In a possible implementation of the fourth aspect, the neural network model includes N operators, the N operators include the first operator and the second operator, and the determining module is specifically configured to:

[0084] Determine multiple candidate processing methods corresponding to the neural network model according to the data flow graph; wherein each of the multiple candidate processing methods includes a tensor splitting method for splitting the input tensor of each operator in the N operators, and a rearrangement method for rearranging the tensors between at least two operators in the N operators, the tensor splitting method is used to split the input tensor of the corresponding operator, and the tensor rearrangement method is used to transmit the operation result obtained after performing the operation corresponding to the operator to a computing device for rearrangement;

[0085] The processing method is determined from the multiple candidate processing methods according to the overhead value corresponding to each candidate processing method in the multiple candidate processing methods, wherein the overhead value includes a memory overhead value corresponding to the tensor splitting method of each operator in the N operators and a first communication overhead value corresponding to the tensor rearrangement method included in each candidate processing method, the memory overhead value represents the memory overhead generated when performing the operation corresponding to the operator, the first communication overhead value represents the communication overhead generated in the process of transmitting the operation result to a computing device, and the processing method is the candidate processing method with the smallest overhead value among the at least one candidate processing method.

[0086] In a possible implementation of the fourth aspect, the N operators include a third operator, each candidate processing method includes a target splitting method for splitting the input tensor of the third operator, the third operator is used to perform a first operation on a sub-tensor obtained by splitting the input tensor of the third operator by the target splitting method, and transmit the operation result obtained by the operation to a computing device for a second operation, and the overhead value also includes a second communication overhead, which represents the communication overhead generated in the process of transmitting the operation result to a computing device when running the third operator.

[0087] In a possible implementation of the fourth aspect, the memory overhead value is related to the number and type of parameters included in the M first sub-tensors.

[0088] In a possible implementation of the fourth aspect, the first communication overhead value is related to the number and type of parameters included in the M second sub-tensors.

[0089] In a possible implementation of the fourth aspect, the second communication overhead value is related to the number and type of parameters included in the M second sub-tensors.

[0090] In a possible implementation of the fourth aspect, the overhead value is an average of the memory overhead value, the first communication overhead value, and the second communication overhead.

[0091] In a possible implementation of the fourth aspect, the determining module is further configured to:

[0092] Determine multiple processing methods corresponding to the neural network model according to the data flow graph;

[0093] The plurality of candidate processing methods are determined from the plurality of processing methods according to the memory overhead value or the first communication overhead value corresponding to each processing method in the plurality of segmentation methods, wherein the plurality of candidate processing methods are part of the plurality of processing methods.

[0094] In a possible implementation of the fourth aspect, the data flow graph further includes a third operator of the neural network model, an output tensor of the third operator is an input tensor of the second operator, and the device further includes:

[0095] A merging module is used to merge the first operator into the second operator so that the second segmentation method corresponding to the second operator establishes an association relationship with the first segmentation method and the tensor rearrangement method.

[0096] In a possible implementation of the fourth aspect, the data flow graph further includes a fourth operator of the neural network model, the output tensor of the first operator is an input tensor of the fourth operator, and the device further includes:

[0097] A merging module is used to merge the second operator into the first operator so that the first segmentation method corresponding to the first operator is associated with the tensor rearrangement method and the second segmentation method.

[0098] In a possible implementation of the fourth aspect, the data flow graph further includes a fifth operator and a sixth operator of the neural network model, wherein the first tensor is a first input tensor of the second operator and a first input tensor of the fifth operator, and the output tensor of the sixth operator is a second input tensor of the second operator and a second input tensor of the fifth operator, and the acquisition module is used to:

[0099] Acquire a third slicing device for slicing the input tensor of the sixth operator, a fourth slicing device for slicing the second input tensor of the second operator, and a first tensor rearrangement device for rearranging the tensors between the fourth operator and the second operator; the third slicing device is used to slice the second input tensor of the sixth operator to obtain a plurality of third sub-tensors, the first tensor rearrangement device is used to rearrange a plurality of fourth sub-tensors obtained after performing operations corresponding to the sixth operator on the plurality of third sub-tensors to obtain a third tensor, and the fourth slicing device is used to slice the third tensor;

[0100] The device also includes:

[0101] A merging module is used to merge the sixth operator into the first operator so that the first segmentation method corresponding to the first operator establishes a first association relationship with the third segmentation method, the fourth segmentation method and the first tensor rearrangement method.

[0102] In a possible implementation of the fourth aspect, the apparatus further includes:

[0103] A merging module is used to merge the sixth operator into the first operator, so that the first segmentation method corresponding to the first operator establishes a second association relationship with the third segmentation method and the second segmentation method.

[0104] In a possible implementation of the fourth aspect, the neural network model is used to process image data, audio data, video data, or text data;

[0105] The input tensor of the operator included in the neural network model is the image data, the audio data, the video data or the text data; or,

[0106] The input tensor of the operator included in the neural network model is obtained by processing the image data, the audio data, the video data or the text data using at least one operator included in the neural network model.

[0107] In a fifth aspect, the present application provides a data processing device, the device comprising:

[0108] An acquisition module, used to acquire a neural network model and a processing method corresponding to the neural network model, wherein the neural network model includes a first operator and a second operator; the processing method includes a first segmentation method for segmenting an input tensor of the first operator, a second segmentation method for segmenting an input tensor of the second operator, and a rearrangement method for rearranging tensors between the first operator and the second operator;

[0109] A first segmentation module, configured to segment the input tensor of the first operator by using the first segmentation method to obtain M first sub-tensors, wherein the M first sub-tensors include a first target sub-tensor;

[0110] A computing module, configured to process the first target sub-tensor using the first operator to obtain a second target sub-tensor;

[0111] A receiving module, configured to receive at least one second sub-tensor sent by at least one computing device; wherein each second sub-tensor is obtained by performing an operation corresponding to the first operator on a first sub-tensor other than the first target sub-tensor among the M first sub-tensors by the corresponding computing device;

[0112] a rearrangement module, configured to rearrange the second target sub-tensor and the at least one second sub-tensor by using the rearrangement method to obtain a first tensor;

[0113] The second segmentation module is used to segment the first tensor using the second segmentation method.

[0114] In a possible implementation of the fifth aspect, each second sub-tensor of the M second sub-tensors is transmitted to a corresponding computing device for a corresponding first sub-tensor of the M first sub-tensors, and is obtained by performing an operation corresponding to the first operator on the corresponding computing device.

[0115] In a possible implementation of the fifth aspect, the tensor rearrangement method is used to obtain the first tensor by transferring the M second sub-tensors to the same computing device for rearrangement.

[0116] In a possible implementation of the fifth aspect, a result obtained after the input tensor of the first operator is operated on a computing device and the operation corresponding to the first operator is performed is the same as the first tensor.

[0117] In a possible implementation of the fifth aspect, the second sub-tensor is a P-dimensional tensor, the sub-tensor obtained after splitting the first tensor by the second splitting method is a Q-dimensional tensor, and P is not an integer multiple of Q.

[0118] In a possible implementation of the fifth aspect, the neural network model is used to process image data, audio data, video data, or text data;

[0119] The input tensor of the operator included in the neural network model is the image data, the audio data, the video data or the text data; or,

[0120] The input tensor of the operator included in the neural network model is obtained by processing the image data, the audio data, the video data or the text data using at least one operator included in the neural network model.

[0121] In a sixth aspect, an embodiment of the present application provides a data processing device, the device comprising:

[0122] An acquisition module, used to acquire a neural network model and a processing method corresponding to the neural network model, wherein the neural network model includes a first operator and a second operator; the processing method includes a first segmentation method for segmenting an input tensor of the first operator and a second segmentation method for segmenting an input tensor of the second operator;

[0123] A first segmentation module, configured to segment the input tensor of the first operator by using the first segmentation method to obtain M first sub-tensors, wherein the M first sub-tensors include a first target sub-tensor;

[0124] A computing module, configured to process the first target sub-tensor using the first operator to obtain a second target sub-tensor;

[0125] A sending module, configured to send the second target sub-tensor to a computing device;

[0126] a receiving module, configured to receive a first tensor sent by the computing device, wherein the first tensor is obtained by the computing device by rearranging a plurality of second sub-tensors through a rearrangement method, wherein the plurality of second sub-tensors include the second target sub-tensor;

[0127] The second segmentation module is used to segment the first tensor using the second segmentation method.

[0128] In a possible implementation of the sixth aspect, a result obtained after the input tensor of the first operator is operated on a computing device and the operation corresponding to the first operator is performed is the same as the first tensor.

[0129] In a possible implementation of the sixth aspect, the second sub-tensor is a P-dimensional tensor, the sub-tensor obtained after splitting the first tensor by the second splitting method is a Q-dimensional tensor, and P is not an integer multiple of Q.

[0130] In a possible implementation of the sixth aspect, the neural network model is used to process image data, audio data, video data, or text data;

[0131] The input tensor of the operator included in the neural network model is the image data, the audio data, the video data or the text data; or,

[0132] The input tensor of the operator included in the neural network model is obtained by processing the image data, the audio data, the video data or the text data using at least one operator included in the neural network model.

[0133] In the seventh aspect, an embodiment of the present application provides a data processing device, which may include a memory, a processor, and a bus system, wherein the memory is used to store programs, and the processor is used to execute the programs in the memory to execute the first aspect and any optional method of the first aspect mentioned above.

[0134] In an eighth aspect, an embodiment of the present application provides a data processing device, which may include a memory, a processor, and a bus system, wherein the memory is used to store programs, and the processor is used to execute the programs in the memory to execute the second aspect and any optional method of the second aspect mentioned above.

[0135] In the ninth aspect, an embodiment of the present application provides a data processing device, which may include a memory, a processor, and a bus system, wherein the memory is used to store programs, and the processor is used to execute the programs in the memory to execute the third aspect and any optional method of the third aspect mentioned above.

[0136] In the tenth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the above-mentioned first aspect or any optional method thereof.

[0137] In the eleventh aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the above-mentioned second aspect or any optional method of the second aspect.

[0138] In the twelfth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the above-mentioned third aspect or the third aspect and any optional method thereof.

[0139] In the thirteenth aspect, an embodiment of the present application provides a computer program, which, when executed on a computer, enables the computer to execute the above-mentioned first aspect and any optional method thereof.

[0140] In a fourteenth aspect, an embodiment of the present application provides a computer program which, when executed on a computer, enables the computer to execute the above-mentioned second aspect and any optional method thereof.

[0141] In a fifteenth aspect, an embodiment of the present application provides a computer program which, when executed on a computer, enables the computer to execute the third aspect and any optional method thereof.

[0142] In a sixteenth aspect, the present application provides a chip system, which includes a processor for supporting an execution device to implement the functions involved in the above aspects, such as sending or processing the data involved in the above methods; or information. In a possible design, the chip system also includes a memory, which is used to store program instructions and data necessary for the execution device or training device. The chip system can be composed of a chip, or it can include a chip and other discrete devices.

[0143] The present application provides a data processing method, characterized in that the method includes: obtaining a data flow graph of a neural network model, the data flow graph including a first operator and a second operator of the neural network model; determining a processing method corresponding to the neural network model according to the data flow graph, the processing method including a rearrangement method for rearranging tensors between the first operator and the second operator. Since the number of parameters in each dimension of the sub-tensor obtained by the operation corresponding to the operator after the split tensor is obtained may be much smaller than the tensor obtained by the operation corresponding to the operator after the unsplit tensor is obtained, the choice of the splitting method corresponding to the input tensor of the subsequent operator is very limited. In this embodiment, since the sub-tensor obtained by the operation corresponding to the operator is rearranged, it is equivalent to restoring the tensor obtained by the operation corresponding to the operator after the unsplit tensor is obtained. At this time, compared with the choice of the splitting method corresponding to the input tensor of the subsequent operator, the splitting method corresponding to the input tensor of the rearranged tensor by the subsequent operator is more. Through the definition of rearrangement, the splitting method of the input tensor of the operator can be selected more, so that different operators can use different corresponding splitting methods to split the input tensor, that is, each uses a splitting method suitable for itself. This is very helpful for improving the processing efficiency of the data flow graph, and can also reduce the overhead of the device in calculating the data flow graph. Because a data flow graph often includes different operators, and different operators may be suitable for different methods of splitting input data. An operator can only guarantee its computational efficiency if it splits input data in the most suitable way. However, if different operators split input data in different ways, the resulting output data formats will be different. During the execution of the data flow graph, the output data of one operator is often the input data of another operator. Therefore, in order to execute the entire data flow graph, the existing technology chooses to let all operators in the data flow graph use the same method for splitting. BRIEF DESCRIPTION OF THE DRAWINGS

[0144] Figure 1 A structural diagram of the main framework of artificial intelligence;

[0145] Figure 2 This is a schematic diagram of a tensor segmentation method;

[0146] Figure 3 This is a schematic diagram of a deep learning framework software stack;

[0147] Figure 4a An example of an application architecture provided in an embodiment of the present application;

[0148] Figure 4b An application architecture diagram provided for an embodiment of the present application;

[0149] Figure 5A schematic diagram of a system architecture provided for an embodiment of the present application;

[0150] Figure 6 This is a schematic diagram of a convolutional layer / pooling layer;

[0151] Figure 7 Schematic diagram of a convolutional neural network;

[0152] Figure 8 A hardware structure of a chip provided in an embodiment of the present application;

[0153] Fig. 9 A flowchart of a data processing method provided in an embodiment of the present application;

[0154] Fig.10 A flowchart of a data processing method in an embodiment of the present application is shown;

[0155] Fig.11 A flowchart of a data processing method in an embodiment of the present application is shown;

[0156] Fig.12 A schematic diagram of a rearrangement method in an embodiment of the present application;

[0157] Fig.13 This is a schematic diagram of a shrinkage in an embodiment of the present application;

[0158] Fig.14 This is a schematic diagram of a shrinkage in an embodiment of the present application;

[0159] Fig.15 This is a schematic diagram of a shrinkage in an embodiment of the present application;

[0160] Fig.16 This is a schematic diagram of a shrinkage in an embodiment of the present application;

[0161] Fig.17 This is a schematic diagram of a shrinkage in an embodiment of the present application;

[0162] Fig.18 A flowchart of a data processing method provided in an embodiment of the present application;

[0163] Fig.19 A flowchart of a data processing method provided in an embodiment of the present application;

[0164] Fig. 20 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0165] Fig.21 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0166] Fig. 22A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0167] Fig.23 A schematic diagram of the structure of an execution device provided in an embodiment of the present application;

[0168] Fig.24 It is a structural schematic diagram of a training device provided in an embodiment of the present application;

[0169] Fig.25 A schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0170] The embodiments of the present invention are described below in conjunction with the accompanying drawings in the embodiments of the present invention. The terms used in the implementation mode of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention.

[0171] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0172] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0173] First, the overall workflow of the artificial intelligence system is described. Figure 1 , Figure 1 The figure shows a structural diagram of the main framework of artificial intelligence. The following is an explanation of the above artificial intelligence theme framework from the two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has undergone a condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the industrial ecology process of the system.

[0174] (1) Infrastructure

[0175] The infrastructure provides computing power support for the AI ​​system, enables communication with the outside world, and supports it through the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips (CPU, NPU, GPU, ASIC, FPGA and other hardware acceleration chips); the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

[0176] (2) Data

[0177] The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.

[0178] (3) Data processing

[0179] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

[0180] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0181] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0182] Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

[0183] (4) General ability

[0184] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0185] (5) Smart products and industry applications

[0186] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical applications. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, safe cities, etc.

[0187] When processing data, data parallelism can be seen as splitting the operator's input tensor in batch dimensions. The split tensors are then operated on by the corresponding computing devices, organizing the devices into a high-dimensional mesh structure. Each dimension has a corresponding name, and accordingly, the tensors are also named to facilitate the mapping of tensors to devices. Figure 2 In the example, MatMul(x,w)=h, MatMul(h,v)=y, MatMul represents matrix multiplication, x, w, h, v, y are matrices (two-dimensional tensors); the dimension of x can be expressed as (b,d_io), the dimension of w can be expressed as (d_io,d_h); the four computing devices can also be expressed as matrices (r,c). The tensor-to-device splitting rule can be expressed as [(b,r),(d_h,c)]. If the total dimension of the tensor to be split is N, and the dimension of the grid composed of devices is M, then there are M^N possible total mapping methods. Mesh-TensorFlow adopts the integer programming modeling method, and then uses the existing solver to obtain the optimal mapping.

[0188] The above embodiment can realize the normalization processing of data parallelism and model parallelism, realize the simultaneous segmentation of training data and model parameters, and can actually train the neural network on a specific hardware device. However, the tensors in all layers of the entire network can only be segmented according to the same mode, and hybrid parallelism cannot be achieved. Figure 2 In the example, due to the mapping relationship (d_h,c), the second dimension of tensor w, the second dimension of tensor h, and the first dimension of tensor v are all mapped to the second dimension of the device matrix. This mapping method has limitations and does not allow the corresponding dimensions of tensors of different operators to be mapped to different device dimensions. It is impossible to achieve mixed splitting from data parallelism to model parallelism, or from model parallelism to data parallelism, that is, it does not support the rearrangement of tensors. And when using integer programming to solve the mapping relationship with the goal of minimizing memory usage, it may lead to poor training performance. Because one of the factors that slow down the training process is the large amount of network communication data, optimizing memory usage alone may lead to an increase in communication volume. For example, for the matrix multiplication operator, splitting the last dimension of the first tensor will cause the forward calculation to introduce AllReduce, thereby triggering communication overhead.

[0189] This application can be applied to the field of deep learning. The deep learning framework software stack can be as follows Figure 3As shown in the figure, the user submits the script program to the framework. After parsing the front-end expression, the framework obtains a data flow graph composed of operators. After searching the parallel strategy, the data flow graph marks the parallel strategy for each operator. The subsequent graph segmentation process divides the entire data flow graph according to the parallel strategy and assigns a computing subgraph to each device. The subgraph will be sent to the computing device for execution.

[0190] Next, an application architecture of an embodiment of the present application is described. Figure 4a , Figure 4a An application architecture diagram is provided for an embodiment of the present application, such as Figure 4a As shown, the computer system may be a multi-core system including a CPU and multiple GPUs, in which the CPU or GPU may call and execute the code in the corresponding memory, wherein the CPU and the multiple GPUs may be interconnected via a bus.

[0191] In the embodiment of the present application, each GPU can obtain a data flow graph corresponding to the neural network model from the corresponding memory, and the data flow graph includes each operator in the neural network model and the segmentation method corresponding to the neural network model, and each GPU can perform data processing based on the data flow graph. How each GPU performs data processing based on the data flow graph will be described in the subsequent embodiments and will not be repeated here.

[0192] In the following text, processing modules such as GPU and / or CPU may be described as computing devices.

[0193] Next, another application architecture of the embodiment of the present application is described. Figure 4b , Figure 4b An application architecture diagram is provided for an embodiment of the present application, such as Figure 4b As shown, the computer system may include multiple computing devices, which may be independent computing devices, such as independent computers or terminal devices, etc. In the system, multiple computing devices may be interconnected via a bus.

[0194] In the embodiment of the present application, each computing device can obtain a data flow graph corresponding to the neural network model, and the data flow graph includes each operator in the neural network model and a segmentation method corresponding to the neural network model, and each computing device can perform data processing based on the data flow graph. How each computing device performs data processing based on the data flow graph will be described in subsequent embodiments and will not be repeated here.

[0195] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.

[0196] (1) Neural Network

[0197] A neural network may be composed of neural units, and a neural unit may refer to an operational unit that takes xs and an intercept 1 as input.

[0198] (2) Deep Neural Networks

[0199] A deep neural network (DNN) can be understood as a neural network with many hidden layers. There is no special metric for "many" here. The multi-layer neural network and deep neural network we often talk about are essentially the same thing. According to the position of different layers of DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer.

[0200] (3) Convolutional Neural Network (CNN) is a deep neural network with a convolutional structure.

[0201] (4) Back propagation algorithm

[0202] Convolutional neural networks can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model during the training process, so that the reconstruction error loss of the super-resolution model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the error loss information is back-propagated to update the parameters in the initial super-resolution model, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal parameters of the super-resolution model, such as the weight matrix.

[0203] (5) Recurrent neural networks (RNN) are used to process sequence data.

[0204] RNN aims to enable machines to have memory capabilities like humans. Therefore, the output of RNN needs to rely on current input information and historical memory information.

[0205] (6) Loss Function

[0206] In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0207] (7) Back propagation algorithm

[0208] Neural networks can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the parameters in the initial neural network model are updated by back propagating the error loss information, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, which aims to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0209] The following describes the system architecture provided by the embodiments of the present application.

[0210] See also Figure 5, the embodiment of the present application provides a system architecture 100. As shown in the system architecture 100, the data acquisition device 160 is used to collect training data. In the embodiment of the present application, the training data includes: an image or an image block of an object and a category of the object; and the training data is stored in the database 130. The training device 120 obtains a CNN feature extraction model based on the training data maintained in the database 130 (Explanation: The feature extraction model here is the model obtained by training in the training phase introduced above, which can be a neural network for feature extraction, etc.). The following will describe in more detail how the training device 120 obtains a CNN feature extraction model based on training data with the first embodiment. The CNN feature extraction model can be used to implement the neural network provided in the embodiment of the present application, that is, the image or image block to be identified is input into the CNN feature extraction model after relevant preprocessing, and the 2D, 3D, Mask, key points and other information of the object of interest of the image or image block to be identified can be obtained. The CNN feature extraction model in the embodiment of the present application can be specifically a CNN convolutional neural network. It should be noted that in actual applications, the training data maintained in the database 130 may not all come from the collection of the data acquisition device 160, but may also be received from other devices. It should also be noted that the training device 120 does not necessarily train the CNN feature extraction model entirely based on the training data maintained by the database 130, and it is also possible to obtain training data from the cloud or other places for model training. The above description should not be regarded as a limitation on the embodiments of the present application.

[0211] The target model / rule obtained by training the training device 120 can be applied to different systems or devices, such as Figure 5 The execution device 110 shown in the figure may be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR) / virtual reality (VR), a vehicle terminal, etc., or a server or a cloud. Figure 5 In the embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with an external device. A user may input data to the I / O interface 112 through a client device 140. The input data may include: an image to be identified or an image block or an image in the embodiment of the application.

[0212] When the execution device 120 preprocesses the input data, or when the computing module 111 of the execution device 120 performs calculation and other related processing (such as implementing the function of the neural network in the present application), the execution device 120 can call the data, code, etc. in the data storage system 150 for the corresponding processing, and can also store the data, instructions, etc. obtained by the corresponding processing in the data storage system 150.

[0213] Finally, the I / O interface 112 returns the processing result, such as the image or image block obtained above or the 2D, 3D, Mask, key point and other information of the object of interest in the image, to the client device 140 for providing to the user.

[0214] Optionally, the client device 140 may be a planning control unit in an autonomous driving system or a beautification algorithm module in a mobile phone terminal.

[0215] It is worth noting that the training device 120 can generate corresponding target models / rules based on different training data for different goals or different tasks, and the corresponding target models / rules can be used to achieve the above goals or complete the above tasks, thereby providing users with the desired results.

[0216] exist Figure 5 In the case shown in the figure, the user can manually give input data, and the manual giving can be operated through the interface provided by the I / O interface 112. In another case, the client device 140 can automatically send input data to the I / O interface 112. If the client device 140 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 140. The user can view the results output by the execution device 110 on the client device 140, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 140 can also be used as a data acquisition terminal to collect the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as shown in the figure as new sample data, and store them in the database 130. Of course, it is also possible not to collect through the client device 140, but the I / O interface 112 directly stores the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as new sample data in the database 130.

[0217] It is worth noting that Figure 5 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 5 In the embodiment, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed in the execution device 110.

[0218] like Figure 5 As shown, a CNN feature extraction model is obtained by training with the training device 120. The CNN feature extraction model in the embodiment of the present application can be a CNN convolutional neural network or a neural network to be introduced in the following embodiment.

[0219] Since CNN is a very common neural network, Figure 5 The focus is on a detailed introduction to the structure of CNN. As mentioned in the basic concept introduction above, a convolutional neural network is a deep neural network with a convolutional structure and a deep learning architecture. A deep learning architecture refers to multiple levels of learning at different levels of abstraction through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network in which each neuron can respond to the image input into it.

[0220] The structure of the convolutional neural network used in the embodiment of the present application can be as follows: Figure 6 As shown. Figure 6 In the example, the convolutional neural network (CNN) 200 may include an input layer 210, a convolutional layer / pooling layer 220 (where the pooling layer is optional), and a neural network layer 230. The input layer 210 may obtain an image to be processed, and the obtained image to be processed is passed to the convolutional layer / pooling layer 220 and the subsequent neural network layer 230 for processing, and the image processing result may be obtained. Figure 6 The internal layer structure of CNN200 is introduced in detail.

[0221] An operator in this application may represent an entire layer structure, or a partial operation in the layer architecture.

[0222] Convolutional layer / pooling layer 220:

[0223] Convolutional Layer:

[0224] like Figure 6 The convolution layer / pooling layer 220 shown may include layers 221-226, for example: in one implementation, layer 221 is a convolution layer, layer 222 is a pooling layer, layer 223 is a convolution layer, layer 224 is a pooling layer, layer 225 is a convolution layer, and layer 226 is a pooling layer; in another implementation, layers 221 and 222 are convolution layers, layer 223 is a pooling layer, layer 224 and 225 are convolution layers, and layer 226 is a pooling layer. That is, the output of a convolution layer can be used as the input of a subsequent pooling layer, or as the input of another convolution layer to continue the convolution operation.

[0225] Pooling layer:

[0226] Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer. Figure 6 The layers 221-226 illustrated in 220 may be a convolutional layer followed by a pooling layer, or multiple convolutional layers may be followed by one or more pooling layers.

[0227] Neural Network Layer 230:

[0228] After being processed by the convolution layer / pooling layer 220, the convolution neural network 200 is not sufficient to output the required output information. As mentioned above, the convolution layer / pooling layer 220 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network 200 needs to use the neural network layer 230 to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer 230 may include multiple hidden layers (such as Figure 6 231, 232 to 23n) and the output layer 240 shown, the parameters contained in the multi-layer hidden layer can be pre-trained according to relevant training data of a specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

[0229] It should be noted that if Figure 2 The convolutional neural network 210 shown is only an example of a convolutional neural network. In specific applications, the convolutional neural network may also exist in the form of other network models.

[0230] The structure of the convolutional neural network in the embodiment of the present application can be as follows: Figure 7 As shown. Figure 7 In the embodiment of the present invention, the convolutional neural network (CNN) 200 may include an input layer 110, a convolutional layer / pooling layer 120 (wherein the pooling layer is optional), and a neural network layer 130. Figure 6 compared to, Figure 7 The multiple convolutional layers / pooling layers in the convolutional layer / pooling layer 120 are operated in parallel, and the features extracted respectively are input to the full neural network layer 130 for processing.

[0231] Figure 8 The hardware structure of a chip provided in the embodiment of the present application includes a neural network processor NPU50. The chip can be set as follows Figure 5 The execution device 110 shown in FIG. 1 is used to complete the calculation work of the calculation module 111. The chip can also be set in Figure 5 The training device 120 shown in FIG. 1 is used to complete the training work of the training device 120 and output the target model / rule. Figure 6 and Figure 7 The algorithms of each layer in the convolutional neural network shown in the figure can be used in Figure 8 is implemented in the chip shown.

[0232] The neural network processor NPU 50 is mounted on the host central processing unit (CPU) as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 503, and the controller 504 controls the operation circuit 503 to extract data from the memory (weight memory or input memory) and perform operations.

[0233] The vector calculation unit 507 may further process the output of the operation circuit.

[0234] The unified memory 506 is used to store input data and output data.

[0235] The weight data is directly transferred from the external memory to the input memory 501 and / or the unified memory 506 through the direct memory access controller 505 (DMAC), the weight data in the external memory is stored in the weight memory 502, and the data in the unified memory 506 is stored in the external memory.

[0236] The bus interface unit (BIU) 510 is used to implement the interaction between the main CPU, DMAC and instruction fetch memory 509 through the bus.

[0237] An instruction fetch buffer 509 connected to the controller 504 and used to store instructions used by the controller 504;

[0238] The controller 504 is used to call the instructions cached in the memory 509 to control the working process of the computing accelerator.

[0239] See also Fig. 9 , Fig. 9 A data processing method according to an embodiment of the present invention is shown in FIG. Fig. 9 As shown in the figure, the data processing method provided in the embodiment of the present application includes:

[0240] 901. Obtain a data flow graph of a neural network model, wherein the data flow graph includes a first operator and a second operator of the neural network model.

[0241] In an embodiment of the present application, a data flow graph of a neural network model can be obtained, and the data flow graph includes a first operator and a second operator of the neural network model.

[0242] In an embodiment of the present application, the data flow graph may include multiple operators, and the multiple operators include a first operator and a second operator.

[0243] In this embodiment of the present application, the output tensor of the first operator is the input tensor of the second operator.

[0244] 902. Determine a processing method corresponding to the neural network model according to the data flow graph, the processing method comprising a first segmentation method for segmenting an input tensor of the first operator, a second segmentation method for segmenting an input tensor of the second operator, and a rearrangement method for rearranging tensors between the first operator and the second operator;

[0245] Among them, the first splitting method is used to split the input tensor of the first operator to obtain M first sub-tensors, and the rearrangement method is used to rearrange the M second sub-tensors to obtain the first tensor, the M second sub-tensors are the outputs of the M first sub-tensors after being processed by the first operator, and the first tensor is the input tensor of the second operator, and M is a positive integer greater than 1.

[0246] In an embodiment of the present application, after obtaining the data flow graph, the processing method corresponding to the neural network model can be determined according to the data flow graph.

[0247] In an embodiment of the present application, a data flow graph includes multiple operators, and a possible splitting method of the input tensor of each operator in the multiple operators included in the data flow graph can be determined. The splitting method includes the number of shares of each dimension of the tensor, which is related to the number of computing devices, the size of each dimension in the operator, and the type of operator. Among them, the number of computing devices determines the upper limit of the number of shares of each dimension, that is, the number of shares does not exceed the number of computing devices; the size of each dimension in the operator determines the number of shares of each dimension, for example, the number of shares of each dimension can be an integer multiple of the tensor size of the corresponding dimension; the type of operator determines the splitting rule of the number of shares of each dimension, for example, for the MatMul operator, there are two input tensors: B tensor and C tensor, the size of the second dimension of each sub-tensor after the B tensor is split is the same as the size of the first dimension of each sub-tensor after the C tensor is split.

[0248] It should be noted that in this embodiment, the size of a tensor in one dimension can be understood as the specification of the tensor in this dimension, which can indicate how many elements the tensor includes in this dimension.

[0249] For example, taking the operator type of MatMul as an example, A=MatMul(B, C), where the shape of the B tensor is [8, 32], the shape of the C tensor is [32, 16], and the number of computing devices is 4, the following segmentation methods can be generated:

[0250] [1,1],[1,1], which means that the first dimension of the B tensor is cut into 1 part, that is, the first dimension of the B tensor is not cut, the first dimension of the C tensor is cut into 1 part, that is, the first dimension of the C tensor is not cut, the second dimension of the B tensor is cut into 1 part, that is, the second dimension of the B tensor is not cut, and the second dimension of the C tensor is cut into 1 part, that is, the second dimension of the C tensor is not cut;

[0251] [1,1],[1,2], which means that the first dimension of the B tensor is split into 1 part, that is, the first dimension of the B tensor is not split, the first dimension of the C tensor is split into 1 part, that is, the first dimension of the C tensor is not split, the second dimension of the B tensor is split into 1 part, that is, the second dimension of the B tensor is not split, and the second dimension of the C tensor is split into 2 parts; after the C tensor is split, two sub-tensors are obtained, and the size of each sub-tensor is [32, 8];

[0252] [1,1],[1,4], which means that the first dimension of the B tensor is split into 1 part, that is, the first dimension of the B tensor is not split, the first dimension of the C tensor is split into 1 part, that is, the first dimension of the C tensor is not split, the second dimension of the B tensor is split into 1 part, that is, the second dimension of the B tensor is not split, and the second dimension of the C tensor is split into 4 parts; after the C tensor is split, four sub-tensors are obtained, and the size of each sub-tensor is [32, 4];

[0253] [1,2],[2,1], which means that the first dimension of the B tensor is split into 1 part, that is, the first dimension of the B tensor is not split, the first dimension of the C tensor is split into 2 parts, the second dimension of the B tensor is split into 2 parts, and the second dimension of the C tensor is split into 1 part, that is, the second dimension of the C tensor is not split; after the B tensor is split, two sub-tensors are obtained, and the size of each sub-tensor is [8, 16]; after the C tensor is split, two sub-tensors are obtained, and the size of each sub-tensor is [16, 16];

[0254] [1,2],[2,2], which means that the first dimension of the B tensor is split into 1 part, that is, the first dimension of the B tensor is not split, the first dimension of the C tensor is split into 2 parts, the second dimension of the B tensor is split into 2 parts, and the second dimension of the C tensor is split into 2 parts; after the B tensor is split, two sub-tensors are obtained, and the size of each sub-tensor is [8, 16]; after the C tensor is split, two sub-tensors are obtained, and the size of each sub-tensor is [16, 8];

[0255] [1,4],[4,1], which means that the first dimension of the B tensor is split into 1 part, that is, the first dimension of the B tensor is not split, the first dimension of the C tensor is split into 4 parts, the second dimension of the B tensor is split into 4 parts, and the second dimension of the C tensor is split into 1 part, that is, the second dimension of the C tensor is not split; after the B tensor is split, two sub-tensors are obtained, and the size of each sub-tensor is [8, 8]; after the C tensor is split, two sub-tensors are obtained, and the size of each sub-tensor is [8, 16];

[0256] [2,1],[1,1], which means that the first dimension of the B tensor is split into 2 parts, the first dimension of the C tensor is split into 1 part, that is, the first dimension of the C tensor is not split, the second dimension of the B tensor is split into 1 part, that is, the second dimension of the B tensor is not split, and the second dimension of the C tensor is split into 1 part, that is, the second dimension of the C tensor is not split; after the B tensor is split, two sub-tensors are obtained, and the size of each sub-tensor is [4,32];

[0257] [2,1],[1,2], which means that the first dimension of the B tensor is split into 2 parts, the first dimension of the C tensor is split into 1 part, that is, the first dimension of the C tensor is not split, and the second dimension of the B tensor is split into 1 part, that is, the second dimension of the B tensor is not split, and the second dimension of the C tensor is split into 2 parts; after the B tensor is split, two sub-tensors are obtained, and the size of each sub-tensor is [4, 32]; after the C tensor is split, two sub-tensors are obtained, and the size of each sub-tensor is [32, 8];

[0258] [2,2],[2,1], which means that the first dimension of the B tensor is split into 2 parts, the first dimension of the C tensor is split into 2 parts, the second dimension of the B tensor is split into 2 parts, and the second dimension of the C tensor is split into 1 part, that is, the second dimension of the C tensor is not split; after the B tensor is split, two sub-tensors are obtained, and the size of each sub-tensor is [4, 16]; after the C tensor is split, two sub-tensors are obtained, and the size of each sub-tensor is [16, 16];

[0259] [4,1],[1,1], which means that the first dimension of the B tensor is split into 4 parts, the first dimension of the C tensor is split into 1 part, that is, the first dimension of the C tensor is not split, the second dimension of the B tensor is split into 1 part, that is, the second dimension of the B tensor is not split, and the second dimension of the C tensor is split into 1 part, that is, the second dimension of the C tensor is not split; after the B tensor is split, two sub-tensors are obtained, and the size of each sub-tensor is [2, 32];

[0260] In an embodiment of the present application, a data flow graph may include multiple operators and a connection relationship between the operators, and the connection relationship may represent the flow direction of a tensor, wherein the data flow graph includes a first operator and a second operator, the first operator and the second operator are connected, and the output tensor of the first operator is the input tensor of the second operator, that is, after the first operator is used to operate on the input tensor, the output tensor obtained can be used as the input tensor of the second operator.

[0261] In an embodiment of the present application, each operator included in the data flow graph can be traversed, and a processing method corresponding to the neural network model can be determined, wherein the processing method can include a first segmentation method for segmenting the input tensor of the first operator, wherein the first segmentation method is a segmentation method for the input tensor of the first operator. The processing method can include a second segmentation method for segmenting the input tensor of the second operator, wherein the third segmentation method is a segmentation method for the input tensor of the third operator.

[0262] In an embodiment of the present application, the processing method may further include a rearrangement method for rearranging the tensors between the first operator and the second operator; wherein the first segmentation method is used to segment the input tensor of the first operator to obtain M first sub-tensors, and the rearrangement method is used to rearrange the M second sub-tensors to obtain a first tensor, the M second sub-tensors are the outputs of the M first sub-tensors after being processed by the first operator, and the first tensor is the input tensor of the second operator, and M is a positive integer greater than 1.

[0263] In an embodiment of the present application, each of the M second sub-tensors is obtained by performing an operation corresponding to the first operator on the corresponding computing device on the corresponding first sub-tensor of the M first sub-tensors. Specifically, the first operator can be used to split the input tensor of the first operator to obtain M first sub-tensors, and each of the M first sub-tensors can be processed by the first operator to obtain a second sub-tensor, that is, the M first sub-tensors are processed by the first operator to obtain M second sub-tensors, and then the M second sub-tensors can be rearranged by a rearrangement method, wherein the rearrangement can be understood as splicing the M second sub-tensors to obtain the first tensor, wherein the result obtained after the input tensor of the first operator performs the operation corresponding to the first operator on a computing device is the same as the first tensor. Specifically, the tensor rearrangement method can be used to transfer the M second sub-tensors to the same computing device for splicing to obtain the first tensor, and the second splitting method is used to split the first tensor.

[0264] In the embodiment of the present application, Figure 4aIn the architecture shown, M GPUs can obtain the input tensor of the first operator from the memory, and each GPU can use the first splitting method to split the input tensor of the first operator to obtain M first sub-tensors. Each GPU can select one of the first sub-tensors to perform the operation corresponding to the first operator to obtain a second sub-tensor. At this time, each GPU in the M GPUs can obtain a second sub-tensor. Afterwards, M-1 GPUs can send their respective second sub-tensors to a GPU through a rearrangement method. The GPU can rearrange the received M-1 first sub-tensors and the second sub-tensors calculated by itself, wherein the rearrangement can be understood as splicing the M second sub-tensors to obtain a first tensor, wherein the result obtained after the input tensor of the first operator is operated on a computing device corresponding to the first operator is the same as the first tensor. Afterwards, the GPU can send the rearranged first tensor to the remaining M-1 GPUs, and each GPU can use the second splitting method to split the first tensor.

[0265] In the embodiment of the present application, Figure 4b In the architecture shown, M computing devices can obtain the input tensor of the first operator, and each computing device can use the first splitting method to split the input tensor of the first operator to obtain M first sub-tensors. Each computing device can select one of the first sub-tensors to perform the operation corresponding to the first operator to obtain a second sub-tensor. At this time, each computing device in the M computing devices can obtain a second sub-tensor. Afterwards, the M-1 computing devices can send their respective second sub-tensors to a computing device through a rearrangement method. The computing device can rearrange the received M-1 first sub-tensors and the second sub-tensor calculated by itself, wherein the rearrangement can be understood as splicing the M second sub-tensors to obtain a first tensor, wherein the result obtained after the input tensor of the first operator is operated on a computing device corresponding to the first operator is the same as the first tensor. Afterwards, the computing device can send the rearranged first tensor to the remaining M-1 computing devices, and each computing device can use the second splitting method to split the first tensor.

[0266] In an embodiment of the present application, the second sub-tensor is a P-dimensional tensor, and the sub-tensor obtained after the first tensor is divided by the second segmentation method is a Q-dimensional tensor, and P is not an integer multiple of Q. That is, each of the M first sub-tensors can be processed by the first operator to obtain a second sub-tensor, and on a GPU, the second sub-tensor cannot be divided to obtain a sub-tensor obtained after the first tensor is divided by the second segmentation method. At this time, if multiple second sub-tensors are rearranged to obtain a first tensor, the first tensor can be segmented according to the second segmentation method on a GPU.

[0267] Reference Fig.10 , Fig.10 The following is a flow chart of a data processing method in an embodiment of the present application. Fig.10 As shown in , the computer system includes GPU1, GPU2, GPU3 and GPU4. GPU1 can obtain two tensors (tensor B and tensor C) and perform a rectangular multiplication operation on the obtained tensors: A=MatMul(B, C). The shape of tensor B is [8, 32], the shape of tensor C is [32, 16], and the number of computing devices is 4.

[0268] It should be noted that, in this embodiment, the shape of the tensor can be understood as the specification of the tensor, which can indicate how many elements each dimension includes.

[0269] Specifically, GPU1 can split the first dimension of the B tensor into 2 parts, split the first dimension of the C tensor into 1 part, do not split the first dimension of the C tensor, do not split the second dimension of the B tensor, and split the second dimension of the C tensor into 2 parts; after the B tensor is split, two sub-tensors are obtained, and the shape of each sub-tensor is [4, 32]; after the C tensor is split, two sub-tensors are obtained, and the shape of each sub-tensor is [32, 8]; GPU1 can perform matrix multiplication operations on a sub-tensor obtained by splitting the B tensor and a sub-tensor obtained by splitting the C tensor to obtain a result tensor, and the shape of the result tensor is [4, 8]. GPU2 can split the first dimension of the B tensor into 2 parts, split the first dimension of the C tensor into 1 part, do not split the first dimension of the C tensor, do not split the second dimension of the B tensor, and split the second dimension of the C tensor into 2 parts; after splitting the B tensor, two sub-tensors are obtained, and the shape of each sub-tensor is [4, 32]; after splitting the C tensor, two sub-tensors are obtained, and the shape of each sub-tensor is [32, 8]; GPU2 can perform matrix multiplication operations on a sub-tensor obtained by splitting the B tensor and a sub-tensor obtained by splitting the C tensor to obtain a result tensor, and the shape of the result tensor is [4, 8]. GPU3 can split the first dimension of the B tensor into 2 parts, split the first dimension of the C tensor into 1 part, do not split the first dimension of the C tensor, do not split the second dimension of the B tensor, and split the second dimension of the C tensor into 2 parts; after splitting the B tensor, two sub-tensors are obtained, and the shape of each sub-tensor is [4, 32]; after splitting the C tensor, two sub-tensors are obtained, and the shape of each sub-tensor is [32, 8]; GPU3 can perform matrix multiplication operations on a sub-tensor obtained by splitting a B tensor and a sub-tensor obtained by splitting a C tensor to obtain a result tensor, and the shape of the result tensor is [4, 8]. GPU4 can split the first dimension of the B tensor into 2 parts, split the first dimension of the C tensor into 1 part, do not split the first dimension of the C tensor, do not split the second dimension of the B tensor, and split the second dimension of the C tensor into 2 parts; after the B tensor is split, two sub-tensors are obtained, and the shape of each sub-tensor is [4, 32]; after the C tensor is split, two sub-tensors are obtained, and the shape of each sub-tensor is [32, 8]; GPU4 can perform matrix multiplication operations on a sub-tensor obtained by splitting a B tensor and a sub-tensor obtained by splitting a C tensor to obtain a result tensor, and the shape of the result tensor is [4, 8].

[0270] Afterwards, GPU2, GPU3 and GPU4 can send their respective result tensors to GPU1, GPU1 can receive them, and GPU2, GPU3 and GPU4 can rearrange their respective result tensors with their own result tensors, that is, concatenate the four result tensors together to obtain a first tensor, the shape of the first tensor is [8, 16], and the first tensor is the same as the result obtained by directly performing matrix multiplication operation on tensor B and tensor C.

[0271] Afterwards, GPU1 can send the first tensor to GPU2, GPU3 and GPU4. Correspondingly, after receiving the first tensor, GPU2, GPU3 and GPU4 can perform a splitting operation on the first tensor.

[0272] It should be noted that the data may be aggregated to any one of GPU1, GPU2, GPU3 and GPU4, or aggregated to other GPUs, which is not limited in this application.

[0273] In one embodiment, the computer system may include computing device 1, computing device 2, computing device 3 and computing device 4. Computing device 1 can obtain two tensors (tensor B and tensor C) and perform a rectangular multiplication operation on the obtained tensors: A=MatMul(B, C), the shape of tensor B is [8, 32], the shape of tensor C is [32, 16], and the number of computing devices is 4.

[0274] Specifically, the computing device 1 can split the first dimension of the B tensor into 2 parts, split the first dimension of the C tensor into 1 part, do not split the first dimension of the C tensor, do not split the second dimension of the B tensor, and split the second dimension of the C tensor into 2 parts; after the B tensor is split, two sub-tensors are obtained, and the shape of each sub-tensor is [4, 32]; after the C tensor is split, two sub-tensors are obtained, and the shape of each sub-tensor is [32, 8]; the computing device 1 can perform matrix multiplication operations on a sub-tensor obtained by splitting a B tensor and a sub-tensor obtained by splitting a C tensor to obtain a result tensor, and the shape of the result tensor is [4, 8]. The computing device 2 can split the first dimension of the B tensor into 2 parts, split the first dimension of the C tensor into 1 part, do not split the first dimension of the C tensor, do not split the second dimension of the B tensor, and split the second dimension of the C tensor into 2 parts; after the B tensor is split, two sub-tensors are obtained, and the shape of each sub-tensor is [4, 32]; after the C tensor is split, two sub-tensors are obtained, and the shape of each sub-tensor is [32, 8]; the computing device 2 can perform matrix multiplication operations on a sub-tensor obtained by splitting the B tensor and a sub-tensor obtained by splitting the C tensor to obtain a result tensor, and the shape of the result tensor is [4, 8]. The computing device 3 can split the first dimension of the B tensor into 2 parts, split the first dimension of the C tensor into 1 part, do not split the first dimension of the C tensor, do not split the second dimension of the B tensor, and split the second dimension of the C tensor into 2 parts; after the B tensor is split, two sub-tensors are obtained, and the shape of each sub-tensor is [4, 32]; after the C tensor is split, two sub-tensors are obtained, and the shape of each sub-tensor is [32, 8]; the computing device 3 can perform matrix multiplication operation on a sub-tensor obtained by splitting the B tensor and a sub-tensor obtained by splitting the C tensor to obtain a result tensor, and the shape of the result tensor is [4, 8]. The computing device 4 can split the first dimension of the B tensor into 2 parts, split the first dimension of the C tensor into 1 part, do not split the first dimension of the C tensor, do not split the second dimension of the B tensor, and split the second dimension of the C tensor into 2 parts; after the B tensor is split, two sub-tensors are obtained, and the shape of each sub-tensor is [4, 32]; after the C tensor is split, two sub-tensors are obtained, and the shape of each sub-tensor is [32, 8]; the computing device 4 can perform matrix multiplication operations on a sub-tensor obtained by splitting the B tensor and a sub-tensor obtained by splitting the C tensor to obtain a result tensor, and the shape of the result tensor is [4, 8].

[0275] Afterwards, computing device 2, computing device 3 and computing device 4 can send their respective result tensors to computing device 1, computing device 1 can receive computing device 2, computing device 3 and computing device 4, and computing device 2, computing device 3 and computing device 4 can rearrange their respective result tensors and their own result tensors, that is, splice the four result tensors together to obtain a first tensor, the shape of the first tensor is [8, 16], and the first tensor is the same as the result obtained by directly performing matrix multiplication operation on tensor B and tensor C.

[0276] Afterwards, computing device 1 may send the first tensor to computing device 2, computing device 3, and computing device 4. Accordingly, after computing device 2, computing device 3, and computing device 4 receive the first tensor, they may perform a splitting operation on the first tensor.

[0277] It should be noted that the data may be aggregated to any one of computing device 1, computing device 2, computing device 3 and computing device 4, or aggregated to other computing devices, and the present application does not limit this.

[0278] Next, it is described how to determine the processing method corresponding to the neural network model according to the data flow graph:

[0279] In an embodiment of the present application, multiple candidate processing methods corresponding to the neural network model can be determined according to the data flow graph; wherein each of the multiple candidate processing methods includes a tensor splitting method for splitting the input tensor of each of the N operators, and a rearrangement method for rearranging the tensors between at least two of the N operators, the tensor splitting method is used to split the input tensor of the corresponding operator, and the tensor rearrangement method is used to transmit the operation result obtained after performing the operation corresponding to the operator to a computing device for rearrangement, and the processing method is determined from the multiple candidate processing methods according to the overhead value corresponding to each of the multiple candidate processing methods, wherein the overhead value includes a memory overhead value corresponding to the tensor splitting method of each of the N operators and a first communication overhead value corresponding to the tensor rearrangement method included in each candidate processing method, the memory overhead value represents the memory overhead generated when performing the operation corresponding to the operator, and the first communication overhead value represents the communication overhead generated in the process of transmitting the operation result to a computing device, and the processing method is the candidate processing method with the smallest overhead value among the at least one candidate processing method.

[0280] In an embodiment of the present application, the input tensor of each operator may correspond to multiple segmentation methods. When the above-mentioned situation where segmentation cannot be performed occurs, a tensor rearrangement method may also be included between operators.

[0281] In an embodiment of the present application, a data flow graph can be traversed to determine multiple candidate processing methods corresponding to the neural network model, wherein each candidate processing method includes a tensor splitting method for splitting the input tensor of each of the N operators, and a rearrangement method for rearranging the tensors between at least two of the N operators.

[0282] In order to select a better processing method from multiple candidate processing methods, it is necessary to determine the overhead value of each candidate processing method among the multiple candidate processing methods, wherein the overhead value includes a memory overhead value corresponding to the tensor splitting method of each operator in the N operators and a first communication overhead value corresponding to the tensor rearrangement method included in each candidate processing method, wherein the memory overhead value represents the memory overhead generated when performing the operation corresponding to the operator, and the first communication overhead value represents the communication overhead generated in the process of transmitting the operation result to a computing device.

[0283] In an embodiment of the present application, the N operators include a third operator, and each candidate processing method includes a target splitting method for splitting the input tensor of the third operator. The third operator is used to perform a first operation on a sub-tensor obtained by splitting the input tensor of the third operator by the target splitting method, and transmit the operation result obtained by the operation to a computing device for a second operation. The overhead value also includes a second communication overhead, and the second communication overhead represents the communication overhead generated in the process of transmitting the operation result to a computing device when running the third operator.

[0284] Reference Fig.11 , Fig.11 The following is a flow chart of a data processing method in an embodiment of the present application. Fig.11 As shown in , the computer system includes GPU1, GPU2, GPU3 and GPU4. GPU1 can obtain two tensors (tensor B and tensor C) and perform a rectangular multiplication operation on the obtained tensors: A=MatMul(B, C). The shape of tensor B is [8, 32], the shape of tensor C is [32, 16], and the number of computing devices is 4.

[0285] Specifically, GPU1 can split the second dimension of the B tensor into 2 parts, and split the first dimension of the C tensor into 2 parts; after the B tensor is split, two sub-tensors are obtained, and the size of each sub-tensor is [8, 16]; after the C tensor is split, two sub-tensors are obtained, and the size of each sub-tensor is [16, 16]; GPU1 can perform matrix multiplication operations on a sub-tensor obtained by splitting the B tensor and a sub-tensor obtained by splitting the C tensor to obtain a result tensor, and the size of the result tensor is [8, 16]. GPU2 can split the second dimension of the B tensor into 2 parts, and split the first dimension of the C tensor into 2 parts; after the B tensor is split, two sub-tensors are obtained, and the size of each sub-tensor is [8, 16]; after the C tensor is split, two sub-tensors are obtained, and the size of each sub-tensor is [16, 16]; GPU2 can perform matrix multiplication operations on a sub-tensor obtained by splitting the B tensor and a sub-tensor obtained by splitting the C tensor to obtain a result tensor, and the size of the result tensor is [8, 16]. GPU3 can split the second dimension of the B tensor into 2 parts, and split the first dimension of the C tensor into 2 parts; after the B tensor is split, two sub-tensors are obtained, and the size of each sub-tensor is [8, 16]; after the C tensor is split, two sub-tensors are obtained, and the size of each sub-tensor is [16, 16]; GPU3 can perform matrix multiplication operations on a sub-tensor obtained by splitting the B tensor and a sub-tensor obtained by splitting the C tensor to obtain a result tensor, and the size of the result tensor is [8, 16]. GPU4 can split the second dimension of the B tensor into 2 parts, and split the first dimension of the C tensor into 2 parts; after the B tensor is split, two sub-tensors are obtained, and the size of each sub-tensor is [8, 16]; after the C tensor is split, two sub-tensors are obtained, and the size of each sub-tensor is [16, 16]; GPU4 can perform matrix multiplication operations on a sub-tensor obtained by splitting the B tensor and a sub-tensor obtained by splitting the C tensor to obtain a result tensor, and the size of the result tensor is [8, 16].

[0286] Afterwards, GPU2, GPU3 and GPU4 can send their respective result tensors to GPU1, GPU1 can receive them, and GPU2, GPU3 and GPU4 can add their respective result tensors and their own result tensors element by element to obtain a first tensor. The size of the first tensor is [8, 16], and the first tensor is the same as the result obtained by directly performing matrix multiplication operation on tensor B and tensor C.

[0287] Afterwards, GPU1 can send the first tensor to GPU2, GPU3 and GPU4. Correspondingly, after receiving the first tensor, GPU2, GPU3 and GPU4 can perform a splitting operation on the first tensor.

[0288] It should be noted that the data may be aggregated to any one of GPU1, GPU2, GPU3 and GPU4, or aggregated to other GPUs, which is not limited in this application.

[0289] In an embodiment of the present application, the overhead value also includes a second communication overhead, and the second communication overhead represents the communication overhead generated in the process of transmitting the operation result to a computing device when running the third operator. Taking the above example, the second communication overhead can represent the communication overhead generated in the process that the above-mentioned GPU2, GPU3 and GPU4 can send their respective result tensors to GPU1.

[0290] In the embodiment of the present application, the second communication overhead value is related to the number of parameters and parameter types included in the M second sub-tensors. Taking the above example, the second communication overhead value is related to the number of parameters (the number of elements included in the tensor) and parameter type (such as integer or floating point) that GPU2, GPU3 and GPU4 can include in their respective result tensors.

[0291] In an embodiment of the present application, the memory overhead value is related to the number of parameters and parameter types included in the M first sub-tensors. The first communication overhead value is related to the number of parameters and parameter types included in the M second sub-tensors. The overhead value is the average of the memory overhead value, the first communication overhead value, and the second communication overhead. The average value can be an arithmetic mean, a geometric mean, a square mean (root mean square mean, rms), a harmonic mean, a weighted mean, etc., which is not limited in this application.

[0292] In the embodiment of the present application, the specific calculation formula of the cost value can be:

[0293] Memory cost = (0 or 1) * (the number of parameters included in the M first sub-tensors) * parameter type, communication cost = (0 or 1) * (the number of parameters included in the M second sub-tensors) * parameter type. Take A = MatMul(B,C) as an example, the shape of the B matrix is ​​[8,32], the shape of the C matrix is ​​[32,16], and the number of devices is 4; if one of B or C is a parameter that needs gradient update, when the strategy is [[4,1], [1,1]], memory cost = (2*32+32*16) * parameter type weight = 768 * parameter type, communication cost = 0, when the strategy is [[1,4], [4,1]], memory cost = (8*8+8*16+8*16) * parameter type = 321 * parameter type weight, communication cost = (8*16) * parameter type weight = 128 * parameter type.

[0294] It should be noted that in this application, if the input tensor of the operator does not need to be updated with a gradient during the training process of the neural network model, then no memory overhead will be incurred regardless of the corresponding segmentation method.

[0295] After the operator strategy is generated and sparse, it is necessary to generate a strategy for rearrangement. The rearrangement strategy is defined as the edge strategy, which is a tuple consisting of the previous operator strategy and the current operator strategy, such as Fig.12 As shown in the figure. If there are N available strategies for node 1 and M available strategies for node 2, then there are N*M strategies for the edge. Each rearrangement strategy will also have a cost, which is calculated in a similar way to the cost calculation of the operator.

[0296] In an embodiment of the present application, multiple processing methods corresponding to the neural network model can also be determined based on the data flow graph; the multiple candidate processing methods can be determined from the multiple processing methods based on the memory overhead value or the first communication overhead value corresponding to each processing method in the multiple processing methods, wherein the multiple candidate processing methods are part of the multiple processing methods.

[0297] In the embodiment of the present application, the processing methods can be streamlined after they are generated: for example, the processing methods can be arranged in non-decreasing order of their memory usage, and then the step size is calculated according to the goal of retaining 1 / epsilon, and 1 / epsilon processing methods are selected from them. The advantage of this is that after sparsification, the number of processing methods retained is independent of the tensor shape and the number of devices. When the shape is large or the number of devices is large, this is a great simplification; on the other hand, epsilon is a user-configurable option. If the user wants to quickly search for a strategy with acceptable performance, then epsilon can be increased. If the user wants to search for the processing method with the best performance, then epsilon needs to be reduced.

[0298] In the embodiment of the present application, after the processing methods are generated, they can be arranged in a non-decreasing order of memory usage, and a strategy can be selected according to a fixed step size. Such a selection method can effectively reduce the search time of the processing method while maintaining accuracy to a certain extent.

[0299] In an embodiment of the present application, the data flow graph also includes a third operator of the neural network model, the output tensor of the third operator is the input tensor of the second operator, and the first operator can be merged into the second operator so that the second segmentation method corresponding to the second operator establishes an association relationship with the first segmentation method and the tensor rearrangement method.

[0300] In the embodiment of the present application, a Merge Elimination operation is proposed, which can process complex graph structures.

[0301] like Fig.13 As shown in the figure, when encountering such a substructure, the first segmentation method N2 of the first operator together with the tensor rearrangement method E2 will be contracted into the second operator, so that the second segmentation method corresponding to the second operator is associated with the first segmentation method and the tensor rearrangement method. When contracting, all strategies in N2 and E2 will be traversed, and the corresponding cost values ​​will be added. The pseudo code can be as follows.

[0302]

[0303] The pseudo code of the entire calculation process can be shown as follows.

[0304]

[0305]

[0306] In an embodiment of the present application, the data flow graph also includes a fourth operator of the neural network model, the output tensor of the first operator is the input tensor of the fourth operator, and the second operator can be merged into the first operator so that the first segmentation method corresponding to the first operator establishes an association relationship with the tensor rearrangement method and the second segmentation method.

[0307] In the embodiment of the present application, a Contract Elimination operation is proposed, which can process complex graph structures.

[0308] like Fig.14 As shown, when encountering such a substructure in the data flow graph, the second segmentation method N2 can be contracted together with the tensor rearrangement method E1 into the first operator. During contraction, all strategies in the second segmentation method N2 and the tensor rearrangement method E1 are traversed, and the corresponding cost values ​​are added. The pseudo code can be as follows.

[0309]

[0310] In an embodiment of the present application, the data flow graph also includes a fifth operator and a sixth operator of the neural network model, wherein the first tensor is the first input tensor of the second operator and the first input tensor of the fifth operator, the output tensor of the sixth operator is the second input tensor of the second operator and the second input tensor of the fifth operator, and a third splitting method for splitting the input tensor of the sixth operator, a fourth splitting method for splitting the second input tensor of the second operator, and a first tensor rearrangement method for rearranging the tensors between the fourth operator and the second operator are obtained. method; the third slicing method is used to slice the second input tensor of the sixth operator to obtain multiple third sub-tensors, the first tensor rearrangement method is used to rearrange the multiple fourth sub-tensors obtained after the multiple third sub-tensors are operated corresponding to the sixth operator to obtain a third tensor, and the fourth slicing method is used to slice the third tensor; the sixth operator is merged into the first operator, so that the first slicing method corresponding to the first operator establishes a first association relationship with the third slicing method, the fourth slicing method and the first tensor rearrangement method.

[0311] like Fig.15 As shown, when encountering such a substructure in the data flow graph, the third segmentation method N4 can be contracted into the first operator N1, and the first tensor rearrangement method E3 is updated to (the first tensor rearrangement method and the first segmentation method) E3'. Similarly, E4 is updated to E4'. When contracting the third segmentation method N4 into the first operator N1, it is necessary to update the various segmentation methods and costs included in the first operator N1 so that the updated code contains the strategy of the third segmentation method N4.

[0312] For example, in Fig.16 In the example, after Star Elimination, the graph undergoes two edge contractions, one Contract Elimination, and one Merge Elimination in sequence to shrink the graph into a final graph with only one point.

[0313] The pseudo code is as follows:

[0314]

[0315] In the embodiment of the present application, after the data flow graph is shrunk into a final graph with only one point, how to select the best one from the candidate cost list, the cost value model f(memory,communication)=alpha*memory(memory cost)+beta*communication(communication cost) is defined; where alpha and beta are configurable parameters. This model is more general than the prior arts one and three because both memory and communication costs are considered. Alpha and beta are not set to fixed values ​​because different hardware and processor speeds and network card transmission speeds are different, resulting in different settings of these two values.

[0316] exist Fig.16 In the example, NodeElimination, EdgeElimination, and MergeElimination are performed in sequence to shrink the graph to a final graph with only one node. After the optimal strategy is selected for the final graph, since the top element of the stack stores the two-point-one-edge graph involved in MergeElimination and a node of the final graph, the strategy for the two-point-one-edge graph will be further set; this operation is repeated until the original graph is restored, so that the strategy of each operator is set.

[0317] like Fig.17 As shown, a small network consisting of a MatMul operator, two Mul operators, a Onehot operator, and an Add operator is taken as an example. The network is a substructure of the ReID neural network. ReID is a technology used to determine whether there is a specific pedestrian in an image or video sequence.

[0318] The dynamic programming algorithm that supports rearrangement is applied on this small network. The specific process is as follows:

[0319] First, we determine the substructure that can be used for Star Elimination, so we shrink the Onehot operator into Matmul. Fig.15The StarEliminationCostCalculation method described updates the processing method cost value of the MatMul operator. Determine two groups of edges that can perform Edge Elimination. E1 and E2 can be merged into a new edge, and E3 and E4 can be merged into another new edge; the cost of the new edge is the sum of the corresponding edges under the same strategy. Determine the two groups of structures that can perform NodeElimination, and shrink the two Mul operators respectively. The candidate strategy of the new edge E5 includes all the candidate strategies of the Mul operator on the left. Similarly, the candidate strategy of the new edge E6 includes all the candidate strategies of the Mul operator on the right. Determine the structure that can perform Edge Elimination, so E5 and E6 can be merged into a new edge. Similarly, the cost of the new edge is the sum of the corresponding edges under the same strategy. Determine the structure that can perform Merge Elimination, and then shrink the MatMul operator into the Add operator. When shrinking, it will traverse all strategies in the MatMul operator and the edge, and add the corresponding cost values, such as Fig.13 The MergeEliminationCostCalculation is shown below.

[0320] In an embodiment of the present application, the neural network model can be used to process image data, audio data, video data or text data; the input tensor of the operator included in the neural network model is the image data, the audio data, the video data or the text data; or, the input tensor of the operator included in the neural network model is obtained by processing the image data, the audio data, the video data or the text data using at least one operator included in the neural network model.

[0321] In the embodiments of the present application, a variety of graph operations are added to solve the strategy search problem of complex graph structures; a new rearrangement strategy is added to solve the problem of not being able to search for the optimal strategy; a new objective function design is added to solve the problem that the derived strategy cannot shorten the end-to-end training time.

[0322] The embodiment of the present application provides a data processing method, the method comprising: obtaining a data flow graph of a neural network model, the data flow graph comprising a first operator and a second operator of the neural network model; determining a processing method corresponding to the neural network model according to the data flow graph, the processing method comprising a first segmentation method for segmenting the input tensor of the first operator, a second segmentation method for segmenting the input tensor of the second operator, and a rearrangement method for rearranging the tensors between the first operator and the second operator; wherein the first segmentation method is used to segment the input tensor of the first operator to obtain M first sub-tensors, and the rearrangement method is used to rearrange the M second sub-tensors to obtain a first tensor, the M second sub-tensors are the outputs obtained after the M first sub-tensors are processed by the first operator, the first tensor is the input tensor of the second operator, and M is a positive integer greater than 1. The present application defines a rearrangement method, which allows the front and rear operators to have different input tensor segmentation methods. This makes it possible to convert model parallelism to data parallelism, or data parallelism to model parallelism in a hybrid parallel mode, making the strategy search space larger. Since the number of parameters in each dimension of the sub-tensor obtained by the operation corresponding to the operator after the split tensor is obtained may be much smaller than the tensor obtained by the operation corresponding to the operator after the unsplit tensor is obtained, the choice of the splitting method corresponding to the input tensor of the subsequent operator is very limited. In this embodiment, since the sub-tensor obtained by the operation corresponding to the operator is rearranged, it is equivalent to restoring the tensor obtained by the operation corresponding to the operator after the unsplit tensor is obtained. At this time, compared with the choice of the splitting method corresponding to the input tensor of the subsequent operator, the splitting method corresponding to the input tensor of the rearranged tensor by the subsequent operator is more. Through the definition of rearrangement, the splitting method of the input tensor of the operator can be selected more, so that different operators can use different corresponding splitting methods to split the input tensor, that is, each uses a splitting method suitable for itself. This is very helpful for improving the processing efficiency of the data flow graph, and can also reduce the overhead of the device in calculating the data flow graph. Because a data flow graph often includes different operators, and different operators may be suitable for different methods of splitting input data. An operator can only guarantee its computational efficiency if it splits input data in the most suitable way. However, if different operators split input data in different ways, the resulting output data formats will be different. During the execution of the data flow graph, the output data of one operator is often the input data of another operator. Therefore, in order to execute the entire data flow graph, the existing technology chooses to let all operators in the data flow graph use the same method for splitting.

[0323] In addition, this application adds three new graph contraction operations to support strategy search for more complex neural networks. Merge Elimination, Contract Elimination, and Star Elimination operations are added, and a method for calculating the cost value after contraction is given.

[0324] In addition, the present application implements the function of calculating memory cost and communication cost. Given a processing method, the memory cost and communication cost consumed by operators and rearrangements can be calculated.

[0325] In addition, after the processing methods are generated, the present application can arrange them in a non-decreasing order of memory usage and select the processing method according to a fixed step size. Such a selection method can effectively reduce the strategy search time while maintaining accuracy to a certain extent.

[0326] In addition, the definition of the overhead value of this application includes both memory overhead and communication overhead, and can model the optimal memory overhead and communication overhead.

[0327] Reference Fig.18 , Fig.18 A data processing method according to an embodiment of the present invention is shown in FIG. Fig.18 As shown in the figure, the data processing method provided in the embodiment of the present application includes:

[0328] 1801. Obtain a neural network model and a processing method corresponding to the neural network model, wherein the neural network model includes a first operator and a second operator, and the processing method includes a first splitting method for splitting an input tensor of the first operator, a second splitting method for splitting an input tensor of the second operator, and a rearrangement method for rearranging tensors between the first operator and the second operator.

[0329] In an embodiment of the present application, a user submits a written script program to the framework, and after parsing the front-end expression, the framework obtains a data flow graph composed of operators; the data flow graph is searched for parallel strategies and the parallel strategies of each operator are marked; and the subsequent graph segmentation process segments the entire data flow graph according to the parallel strategy and allocates a computing subgraph to each device; the subgraph will be sent to the computing device for execution, and each subgraph may include the processing method corresponding to the above-mentioned neural network model.

[0330] 1802. Split the input tensor of the first operator by using the first splitting method to obtain M first sub-tensors, where the M first sub-tensors include a first target sub-tensor.

[0331] The specific description of step 1802 can refer to the description related to the first segmentation method in the above embodiment, which will not be repeated here.

[0332] 1803. Use the first operator to process the first target sub-tensor to obtain a second target sub-tensor.

[0333] 1804. Receive at least one second sub-tensor sent by at least one computing device; wherein each second sub-tensor is obtained by the corresponding computing device performing an operation corresponding to the first operator on a first sub-tensor other than the first target sub-tensor among the M first sub-tensors.

[0334] 1805. Concatenate the second target sub-tensor and the at least one second sub-tensor by using the rearrangement method to obtain a first tensor;

[0335] The specific description of step 1805 can refer to the description related to the rearrangement method in the above embodiment, which will not be repeated here.

[0336] 1806. Split the first tensor using the second splitting method.

[0337] The specific description of step 1806 can refer to the description related to the second segmentation method in the above embodiment, which will not be repeated here.

[0338] Optionally, each second sub-tensor of the M second sub-tensors is obtained by transmitting a corresponding first sub-tensor of the M first sub-tensors to a corresponding computing device, and performing an operation corresponding to the first operator on the corresponding computing device.

[0339] Optionally, the tensor rearrangement method is used to obtain the first tensor by transferring the M second sub-tensors to the same computing device for rearrangement.

[0340] Optionally, a result obtained after the input tensor of the first operator is operated on a computing device and the operation corresponding to the first operator is performed is the same as the first tensor.

[0341] Optionally, the second sub-tensor is a P-dimensional tensor, the sub-tensor obtained after splitting the first tensor by the second splitting method is a Q-dimensional tensor, and P is not an integer multiple of Q.

[0342] Optionally, the neural network model is used to process image data, audio data, video data or text data; the input tensor of the operator included in the neural network model is the image data, the audio data, the video data or the text data; or, the input tensor of the operator included in the neural network model is obtained by processing the image data, the audio data, the video data or the text data using at least one operator included in the neural network model.

[0343] The embodiment of the present application provides a data processing method, the method comprising: obtaining a neural network model and a processing method corresponding to the neural network model, the neural network model comprising a first operator and a second operator, the processing method comprising a first slicing method for slicing the input tensor of the first operator, a second slicing method for slicing the input tensor of the second operator, and a rearrangement method for rearranging the tensor between the first operator and the second operator; slicing the input tensor of the first operator by the first slicing method to obtain M first sub-tensors; splicing the M second sub-tensors obtained after performing the operation corresponding to the first operator on the M first sub-tensors by the rearrangement method to obtain a first tensor, wherein M is a positive integer greater than 1; slicing the first tensor by the second slicing method. In the above manner, after the slicing tensor performs the operation corresponding to the operator, the rearrangement of the tensor can be performed once, so that there are more options for the slicing method of the input tensor of the subsequent operator.

[0344] Reference Fig.19 , Fig.19 A data processing method according to an embodiment of the present invention is shown in FIG. Fig.19 As shown in the figure, the data processing method provided in the embodiment of the present application includes:

[0345] 1901. Obtain a neural network model and a processing method corresponding to the neural network model, wherein the neural network model includes a first operator and a second operator; the processing method includes a first segmentation method for segmenting an input tensor of the first operator and a second segmentation method for segmenting an input tensor of the second operator;

[0346] In an embodiment of the present application, a user submits a written script program to the framework, and after parsing the front-end expression, the framework obtains a data flow graph composed of operators; the data flow graph is searched for parallel strategies and the parallel strategies of each operator are marked; and the subsequent graph segmentation process segments the entire data flow graph according to the parallel strategy and allocates a computing subgraph to each device; the subgraph will be sent to the computing device for execution, and each subgraph may include the processing method corresponding to the above-mentioned neural network model.

[0347] 1902. Split the input tensor of the first operator by using the first splitting method to obtain M first sub-tensors, where the M first sub-tensors include a first target sub-tensor;

[0348] The specific description of step 1902 can refer to the description related to the first segmentation method in the above embodiment, which will not be repeated here.

[0349] 1903. Perform an operation corresponding to the first operator on the first target sub-tensor to obtain a second target sub-tensor;

[0350] 1904. Send the second target sub-tensor to a computing device;

[0351] 1905. Receive a first tensor sent by the computing device, where the first tensor is obtained by splicing a plurality of second sub-tensors by the computing device through a rearrangement method, and the plurality of second sub-tensors include the second target sub-tensor;

[0352] 1906. Split the first tensor using the second splitting method.

[0353] The specific description of step 1906 can refer to the description related to the second segmentation method in the above embodiment, which will not be repeated here.

[0354] Optionally, a result obtained after the input tensor of the first operator is operated on a computing device and the operation corresponding to the first operator is performed is the same as the first tensor.

[0355] Optionally, the second sub-tensor is a P-dimensional tensor, the sub-tensor obtained after splitting the first tensor by the second splitting method is a Q-dimensional tensor, and P is not an integer multiple of Q.

[0356] Optionally, the neural network model is used to process image data, audio data, video data or text data; the input tensor of the operator included in the neural network model is the image data, the audio data, the video data or the text data; or, the input tensor of the operator included in the neural network model is obtained by processing the image data, the audio data, the video data or the text data using at least one operator included in the neural network model.

[0357] The embodiment of the present application provides a data processing method, the method comprising: obtaining a neural network model and a processing method corresponding to the neural network model, the neural network model comprising a first operator and a second operator; the processing method comprising a first splitting method for splitting the input tensor of the first operator and a second splitting method for splitting the input tensor of the second operator; splitting the input tensor of the first operator by the first splitting method to obtain M first sub-tensors, the M first sub-tensors including a first target sub-tensor; using the first operator to process the first target sub-tensor to obtain a second target sub-tensor; sending the second target sub-tensor to a computing device; receiving the first tensor sent by the computing device, the first tensor being obtained by the computing device by rearranging multiple second sub-tensors by a rearrangement method, the multiple second sub-tensors including the second target sub-tensor; splitting the first tensor by the second splitting method. In the above manner, after the split tensor performs the operation corresponding to the operator, the tensor can be rearranged once, so that there are more options for the splitting method of the input tensor of the subsequent operator.

[0358] Reference Fig. 20 , Fig. 20 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application is shown in FIG. Fig. 20 As shown in FIG. 1 , the data processing device 2000 provided in the embodiment of the present application includes:

[0359] An acquisition module 2001 is used to acquire a data flow graph of a neural network model, wherein the data flow graph includes a first operator and a second operator of the neural network model;

[0360] A determination module 2002 is used to determine a processing method corresponding to the neural network model according to the data flow graph, wherein the processing method includes a first segmentation method for segmenting an input tensor of the first operator, a second segmentation method for segmenting an input tensor of the second operator, and a rearrangement method for rearranging tensors between the first operator and the second operator;

[0361] Among them, the first splitting method is used to split the input tensor of the first operator to obtain M first sub-tensors, and the rearrangement method is used to rearrange the M second sub-tensors to obtain the first tensor, the M second sub-tensors are the outputs of the M first sub-tensors after being processed by the first operator, and the first tensor is the input tensor of the second operator, and M is a positive integer greater than 1.

[0362] Optionally, each second sub-tensor of the M second sub-tensors is obtained by transmitting a corresponding first sub-tensor of the M first sub-tensors to a corresponding computing device, and performing an operation corresponding to the first operator on the corresponding computing device.

[0363] Optionally, the tensor rearrangement method is used to obtain the first tensor by transferring the M second sub-tensors to the same computing device for splicing.

[0364] Optionally, a result obtained after performing an operation corresponding to the first operator on the input tensor of the first operator on a computing device is the same as the first tensor.

[0365] Optionally, the second sub-tensor is a P-dimensional tensor, the sub-tensor obtained after splitting the first tensor by the second splitting method is a Q-dimensional tensor, and P is not an integer multiple of Q.

[0366] Optionally, the neural network model includes N operators, the N operators include the first operator and the second operator, and the determination module is specifically used to:

[0367] Determine multiple candidate processing methods corresponding to the neural network model according to the data flow graph; wherein each of the multiple candidate processing methods includes a tensor splitting method for splitting the input tensor of each operator in the N operators, and a rearrangement method for rearranging the tensors between at least two operators in the N operators, the tensor splitting method is used to split the input tensor of the corresponding operator, and the tensor rearrangement method is used to transmit the operation result obtained after performing the operation corresponding to the operator to a computing device for rearrangement;

[0368] The processing method is determined from the multiple candidate processing methods according to the overhead value corresponding to each candidate processing method in the multiple candidate processing methods, wherein the overhead value includes a memory overhead value corresponding to the tensor splitting method of each operator in the N operators and a first communication overhead value corresponding to the tensor rearrangement method included in each candidate processing method, the memory overhead value represents the memory overhead generated when performing the operation corresponding to the operator, the first communication overhead value represents the communication overhead generated in the process of transmitting the operation result to a computing device, and the processing method is the candidate processing method with the smallest overhead value among the at least one candidate processing method.

[0369] Optionally, the N operators include a third operator, and each candidate processing method includes a target splitting method for splitting the input tensor of the third operator. The third operator is used to perform a first operation on a sub-tensor obtained by splitting the input tensor of the third operator by the target splitting method, and transmit the operation result obtained by the operation to a computing device for a second operation. The overhead value also includes a second communication overhead, and the second communication overhead represents the communication overhead generated in the process of transmitting the operation result to a computing device when running the third operator.

[0370] Optionally, the memory overhead value is related to the number and type of parameters included in the M first sub-tensors.

[0371] Optionally, the first communication overhead value is related to the number and type of parameters included in the M second sub-tensors.

[0372] Optionally, the second communication overhead value is related to the number and type of parameters included in the M second sub-tensors.

[0373] Optionally, the overhead value is an average of the memory overhead value, the first communication overhead value, and the second communication overhead.

[0374] Optionally, the determining module is further used to:

[0375] Determine multiple processing methods corresponding to the neural network model according to the data flow graph;

[0376] The plurality of candidate processing methods are determined from the plurality of processing methods according to the memory overhead value or the first communication overhead value corresponding to each processing method in the plurality of segmentation methods, wherein the plurality of candidate processing methods are part of the plurality of processing methods.

[0377] Optionally, the data flow graph further includes a third operator of the neural network model, the output tensor of the third operator is the input tensor of the second operator, and the device further includes:

[0378] A merging module is used to merge the first operator into the second operator so that the second segmentation method corresponding to the second operator establishes an association relationship with the first segmentation method and the tensor rearrangement method.

[0379] Optionally, the data flow graph further includes a fourth operator of the neural network model, the output tensor of the first operator is the input tensor of the fourth operator, and the device further includes:

[0380] A merging module is used to merge the second operator into the first operator so that the first segmentation method corresponding to the first operator is associated with the tensor rearrangement method and the second segmentation method.

[0381] Optionally, the data flow graph further includes a fifth operator and a sixth operator of the neural network model, wherein the first tensor is a first input tensor of the second operator and a first input tensor of the fifth operator, and the output tensor of the sixth operator is a second input tensor of the second operator and a second input tensor of the fifth operator, and the acquisition module is used to:

[0382] Acquire a third slicing device for slicing the input tensor of the sixth operator, a fourth slicing device for slicing the second input tensor of the second operator, and a first tensor rearrangement device for rearranging the tensors between the fourth operator and the second operator; the third slicing device is used to slice the second input tensor of the sixth operator to obtain a plurality of third sub-tensors, the first tensor rearrangement device is used to rearrange a plurality of fourth sub-tensors obtained after performing operations corresponding to the sixth operator on the plurality of third sub-tensors to obtain a third tensor, and the fourth slicing device is used to slice the third tensor;

[0383] The device also includes:

[0384] A merging module is used to merge the sixth operator into the first operator so that the first segmentation method corresponding to the first operator establishes a first association relationship with the third segmentation method, the fourth segmentation method and the first tensor rearrangement method.

[0385] Optionally, the device further comprises:

[0386] A merging module is used to merge the sixth operator into the first operator, so that the first segmentation method corresponding to the first operator establishes a second association relationship with the third segmentation method and the second segmentation method.

[0387] Optionally, the neural network model is used to process image data, audio data, video data or text data;

[0388] The input tensor of the operator included in the neural network model is the image data, the audio data, the video data or the text data; or,

[0389] The input tensor of the operator included in the neural network model is obtained by processing the image data, the audio data, the video data or the text data using at least one operator included in the neural network model.

[0390] Reference Fig.21 , Fig.21 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application is shown in FIG. Fig.21 As shown in FIG. 1 , the data processing device 2100 provided in the embodiment of the present application includes:

[0391] An acquisition module 2101 is used to acquire a neural network model and a processing method corresponding to the neural network model, wherein the neural network model includes a first operator and a second operator; the processing method includes a first segmentation method for segmenting an input tensor of the first operator, a second segmentation method for segmenting an input tensor of the second operator, and a rearrangement method for rearranging tensors between the first operator and the second operator;

[0392] A first segmentation module 2102 is used to segment the input tensor of the first operator by using the first segmentation method to obtain M first sub-tensors, wherein the M first sub-tensors include a first target sub-tensor;

[0393] An operation module 2103 is used to process the first target sub-tensor using the first operator to obtain a second target sub-tensor;

[0394] The receiving module 2104 is configured to receive at least one second sub-tensor sent by at least one computing device; wherein each second sub-tensor is obtained by performing an operation corresponding to the first operator on a first sub-tensor other than the first target sub-tensor among the M first sub-tensors by the corresponding computing device;

[0395] A rearrangement module 2105, configured to rearrange the second target sub-tensor and the at least one second sub-tensor by using the rearrangement method to obtain a first tensor;

[0396] The second segmentation module 2106 is used to segment the first tensor using the second segmentation method.

[0397] Optionally, each second sub-tensor of the M second sub-tensors is obtained by transmitting a corresponding first sub-tensor of the M first sub-tensors to a corresponding computing device, and performing an operation corresponding to the first operator on the corresponding computing device.

[0398] Optionally, the tensor rearrangement method is used to obtain the first tensor by transferring the M second sub-tensors to the same computing device for rearrangement.

[0399] Optionally, a result obtained after the input tensor of the first operator is operated on a computing device and the operation corresponding to the first operator is performed is the same as the first tensor.

[0400] Optionally, the second sub-tensor is a P-dimensional tensor, the sub-tensor obtained after splitting the first tensor by the second splitting method is a Q-dimensional tensor, and P is not an integer multiple of Q.

[0401] Optionally, the neural network model is used to process image data, audio data, video data or text data;

[0402] The input tensor of the operator included in the neural network model is the image data, the audio data, the video data or the text data; or,

[0403] The input tensor of the operator included in the neural network model is obtained by processing the image data, the audio data, the video data or the text data using at least one operator included in the neural network model.

[0404] Reference Fig. 22 , Fig. 22 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application is shown in FIG. Fig. 22 As shown in FIG. 2 , the data processing device 2200 provided in the embodiment of the present application includes:

[0405] An acquisition module 2201 is used to acquire a neural network model and a processing method corresponding to the neural network model, wherein the neural network model includes a first operator and a second operator; the processing method includes a first segmentation method for segmenting an input tensor of the first operator and a second segmentation method for segmenting an input tensor of the second operator;

[0406] A first segmentation module 2202 is used to segment the input tensor of the first operator by using the first segmentation method to obtain M first sub-tensors, wherein the M first sub-tensors include a first target sub-tensor;

[0407] An operation module 2203 is used to process the first target sub-tensor using the first operator to obtain a second target sub-tensor;

[0408] A sending module 2204 is used to send the second target sub-tensor to a computing device;

[0409] A receiving module 2205 is configured to receive a first tensor sent by the computing device, where the first tensor is obtained by the computing device rearranging a plurality of second sub-tensors by a rearrangement method, where the plurality of second sub-tensors include the second target sub-tensor;

[0410] The second segmentation module 2206 is used to segment the first tensor using the second segmentation method.

[0411] Optionally, a result obtained after the input tensor of the first operator is operated on a computing device and the operation corresponding to the first operator is performed is the same as the first tensor.

[0412] Optionally, the second sub-tensor is a P-dimensional tensor, the sub-tensor obtained after splitting the first tensor by the second splitting method is a Q-dimensional tensor, and P is not an integer multiple of Q.

[0413] Optionally, the neural network model is used to process image data, audio data, video data or text data; the input tensor of the operator included in the neural network model is the image data, the audio data, the video data or the text data; or, the input tensor of the operator included in the neural network model is obtained by processing the image data, the audio data, the video data or the text data using at least one operator included in the neural network model.

[0414] Next, an execution device provided by an embodiment of the present application is introduced. Fig.23 , Fig.23 This is a schematic diagram of the structure of an execution device provided in an embodiment of the present application. The execution device 2300 can be specifically a mobile phone, a tablet, a laptop, a smart wearable device, a server, etc., which is not limited here. Specifically, the execution device 2300 includes: a receiver 2301, a transmitter 2302, a processor 2303 and a memory 2304 (wherein the number of processors 2303 in the execution device 2300 can be one or more, Fig.23 In the example of FIG. 2301 , a processor 2303 may include an application processor 23031 and a communication processor 23032. In some embodiments of the present application, the receiver 2301, the transmitter 2302, the processor 2303 and the memory 2304 may be connected via a bus or other means.

[0415] The memory 2304 may include a read-only memory and a random access memory, and provides instructions and data to the processor 2303. A portion of the memory 2304 may also include a non-volatile random access memory (NVRAM). The memory 2304 stores processor and operation instructions, executable modules or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0416] The processor 2303 controls the operation of the execution device. In a specific application, the various components of the execution device are coupled together through a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus, and a status signal bus, etc. However, for the sake of clarity, various buses are referred to as bus systems in the figure.

[0417] The method disclosed in the above embodiment of the present application can be applied to the processor 2303, or implemented by the processor 2303. The processor 2303 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 2303. The above processor 2303 can be a general processor, a digital signal processor (digital signal processing, DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (application specific integrated circuit, ASIC), a field programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The processor 2303 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor can be combined and executed. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 2304, and the processor 2303 reads the information in the memory 2304 and completes the steps of the above method in combination with its hardware.

[0418] The receiver 2301 can be used to receive input digital or character information and generate signal input related to the relevant settings and function control of the execution device. The transmitter 2302 can be used to output digital or character information through the first interface; the transmitter 2302 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 2302 can also include a display device such as a display screen.

[0419] In an embodiment of the present application, in one case, the processor 2303 is used to execute the data processing method in the above embodiment.

[0420] The present application also provides a training device. Fig.24 , Fig.24It is a structural diagram of a training device provided in an embodiment of the present application. Specifically, the training device 2400 is implemented by one or more servers. The training device 2400 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 2424 (for example, one or more processors) and memory 2432, and one or more storage media 2430 (for example, one or more massive storage devices) storing application programs 2442 or data 2444. Among them, the memory 2432 and the storage medium 2430 can be short-term storage or persistent storage. The program stored in the storage medium 2430 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the training device. Furthermore, the central processing unit 2424 can be configured to communicate with the storage medium 2430 to execute a series of instruction operations in the storage medium 2430 on the training device 2400.

[0421] The training device 2400 may also include one or more power supplies 2426, one or more wired or wireless network interfaces 2450, one or more input and output interfaces 2458; or, one or more operating systems 2441, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0422] In the embodiment of the present application, the central processor 2424 is used to execute the steps related to the perceptual network training method in the above embodiment.

[0423] Also provided in an embodiment of the present application is a computer program product which, when executed on a computer, enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0424] A computer-readable storage medium is also provided in an embodiment of the present application, which stores a program for signal processing. When the computer-readable storage medium is run on a computer, it enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0425] The execution device, training device or terminal device provided in the embodiments of the present application may specifically be a chip, and the chip includes: a processing unit and a communication unit, wherein the processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin or a circuit, etc. The processing unit may execute the computer execution instructions stored in the storage unit so that the chip in the execution device executes the data processing method described in the above embodiment, or so that the chip in the training device executes the data processing method described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0426] For details, please refer to Fig.25 , Fig.25 A schematic diagram of the structure of a chip provided in an embodiment of the present application, the chip can be expressed as a neural network processor NPU 2500, NPU 2500 is mounted on the host CPU (Host CPU) as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 2503, which is controlled by the controller 2504 to extract matrix data from the memory and perform multiplication operations.

[0427] In some implementations, the operation circuit 2503 includes multiple processing units (Process Engine, PE) inside. In some implementations, the operation circuit 2503 is a two-dimensional systolic array. The operation circuit 2503 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 2503 is a general-purpose matrix processor.

[0428] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 2502 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 2501 and performs matrix operations with matrix B. The partial results or final results of the obtained matrix are stored in the accumulator 2508.

[0429] The unified memory 2506 is used to store input data and output data. The weight data is directly transferred to the weight memory 2502 through the direct memory access controller (DMAC) 2505. The input data is also transferred to the unified memory 2506 through the DMAC.

[0430] BIU stands for Bus Interface Unit, i.e., bus interface unit 2510 , which is used for the interaction between AXI bus and DMAC and instruction fetch buffer (IFB) 2509 .

[0431] The bus interface unit 2510 (Bus Interface Unit, BIU for short) is used for the instruction fetch memory 2509 to obtain instructions from the external memory, and is also used for the storage unit access controller 2505 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0432] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 2506 or to transfer weight data to the weight memory 2502 or to transfer input data to the input memory 2501.

[0433] The vector calculation unit 2507 includes multiple operation processing units, and further processes the output of the operation circuit 2503 when necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as Batch Normalization, pixel-level summation, upsampling of feature planes, etc.

[0434] In some implementations, the vector calculation unit 2507 can store the processed output vector to the unified memory 2506. For example, the vector calculation unit 2507 can apply a linear function; or a nonlinear function to the output of the operation circuit 2503, such as linear interpolation of the feature plane extracted by the convolution layer, and then, for example, a vector of accumulated values ​​to generate an activation value. In some implementations, the vector calculation unit 2507 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 2503, for example, for use in a subsequent layer in a neural network.

[0435] An instruction fetch buffer 2509 connected to the controller 2504 is used to store instructions used by the controller 2504;

[0436] Unified memory 2506, input memory 2501, weight memory 2502 and instruction fetch memory 2509 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0437] The processor mentioned in any of the above places may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above programs.

[0438] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0439] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0440] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0441] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A data processing method, characterized in that: The method comprises: Obtaining a data flow graph of a neural network model, the data flow graph comprising a first operator and a second operator of the neural network model, wherein the neural network model is used to process image data, audio data, video data or text data; Determining a processing method corresponding to the neural network model according to the data flow graph, the processing method comprising a first segmentation method for segmenting an input tensor of the first operator, a second segmentation method for segmenting an input tensor of the second operator, and a rearrangement method for rearranging tensors between the first operator and the second operator; The first segmentation method is used to segment the input tensor of the first operator to obtain M first sub-tensors, and the rearrangement method is used to rearrange the M second sub-tensors to obtain a first tensor, the M second sub-tensors are outputs of the M first sub-tensors after being processed by the first operator, the first tensor is the input tensor of the second operator, and M is a positive integer greater than 1; The processing method is a candidate processing method with the smallest overhead value among at least one candidate processing method, and the overhead value represents the memory overhead generated when performing the operation corresponding to the operator, and / or the communication overhead generated in the process of transmitting the operation result, and the operation result is obtained by performing the operation corresponding to the operator.

2. The method according to claim 1, characterized in that A result obtained after the operation corresponding to the first operator is performed on a computing device on the input tensor of the first operator is the same as the first tensor.

3. The method according to claim 1 or 2, characterized in that: The second sub-tensor is a P-dimensional tensor, and the sub-tensor obtained after the first tensor is split by the second splitting method is a Q-dimensional tensor, and P is not an integer multiple of Q.

4. The method according to any one of claims 1 to 2, characterized in that: The neural network model includes N operators, the N operators include the first operator and the second operator, and the processing method corresponding to the neural network model is determined according to the data flow graph, including: Determine multiple candidate processing methods corresponding to the neural network model according to the data flow graph; wherein each of the multiple candidate processing methods includes a tensor splitting method for splitting the input tensor of each of the N operators, and a rearrangement method for rearranging the tensors between at least two of the N operators, the tensor splitting method is used to split the input tensor of the corresponding operator, and the rearrangement method is used to transmit the operation result obtained after performing the operation corresponding to the operator to a computing device for rearrangement; The processing method is determined from the multiple candidate processing methods according to the overhead value corresponding to each candidate processing method in the multiple candidate processing methods, wherein the overhead value includes a memory overhead value corresponding to the tensor splitting method of each operator in the N operators and a first communication overhead value corresponding to the rearrangement method included in each candidate processing method, the memory overhead value represents the memory overhead generated when performing the operation corresponding to the operator, and the first communication overhead value represents the communication overhead generated in the process of transmitting the operation result to a computing device.

5. The method according to claim 4, characterized in that The N operators include a third operator, and each candidate processing method includes a target splitting method for splitting an input tensor of the third operator. The third operator is used to perform a first operation on a sub-tensor obtained by splitting the input tensor of the third operator by the target splitting method, and transmit the operation result obtained by the operation to a computing device for performing a second operation. The overhead value also includes a second communication overhead, and the second communication overhead represents the communication overhead generated in the process of transmitting the operation result to a computing device when running the third operator.

6. The method according to any one of claims 1 to 2, characterized in that: The data flow graph further includes a third operator of the neural network model, the output tensor of the third operator is the input tensor of the second operator, and the method further includes: The first operator is merged into the second operator, so that the second segmentation method corresponding to the second operator is associated with the first segmentation method and the rearrangement method.

7. The method according to any one of claims 1 to 2, characterized in that: The data flow graph further includes a fourth operator of the neural network model, the output tensor of the first operator is the input tensor of the fourth operator, and the method further includes: The second operator is merged into the first operator, so that the first segmentation method corresponding to the first operator is associated with the rearrangement method and the second segmentation method.

8. The method according to claim 7, characterized in that The data flow graph further includes a fifth operator and a sixth operator of the neural network model, wherein the first tensor is a first input tensor of the second operator and a first input tensor of the fifth operator, and an output tensor of the sixth operator is a second input tensor of the second operator and a second input tensor of the fifth operator, and the method further includes: Acquire a third slicing method for slicing the input tensor of the sixth operator, a fourth slicing method for slicing the second input tensor of the second operator, and a first tensor rearrangement method for rearranging the tensors between the fourth operator and the second operator; the third slicing method is used to slice the second input tensor of the sixth operator to obtain a plurality of third sub-tensors, the first tensor rearrangement method is used to rearrange a plurality of fourth sub-tensors obtained after performing operations corresponding to the sixth operator on the plurality of third sub-tensors to obtain a third tensor, and the fourth slicing method is used to slice the third tensor; The sixth operator is merged into the first operator, so that the first segmentation method corresponding to the first operator establishes a first association relationship with the third segmentation method, the fourth segmentation method and the first tensor rearrangement method.

9. The method according to any one of claims 1 to 2, characterized in that: The input tensor of the operator included in the neural network model is the image data, the audio data, the video data or the text data; or, The input tensor of the operator included in the neural network model is obtained by processing the image data, the audio data, the video data or the text data using at least one operator included in the neural network model.

10. A data processing method, characterized in that: The method comprises: Obtaining a neural network model and a processing method corresponding to the neural network model, wherein the neural network model includes a first operator and a second operator; the processing method includes a first segmentation method for segmenting an input tensor of the first operator, a second segmentation method for segmenting an input tensor of the second operator, and a rearrangement method for rearranging tensors between the first operator and the second operator, and the neural network model is used to process image data, audio data, video data, or text data; Splitting the input tensor of the first operator by the first splitting method to obtain M first sub-tensors, wherein the M first sub-tensors include a first target sub-tensor; Processing the first target sub-tensor using the first operator to obtain a second target sub-tensor; Receive at least one second sub-tensor sent by at least one computing device; wherein each second sub-tensor is obtained by performing an operation corresponding to the first operator on a first sub-tensor other than the first target sub-tensor among the M first sub-tensors by the corresponding computing device; Rearranging the second target sub-tensor and the at least one second sub-tensor by the rearrangement method to obtain a first tensor; Splitting the first tensor using the second splitting method; The processing method is a candidate processing method with the smallest overhead value among at least one candidate processing method, and the overhead value represents the memory overhead generated when performing the operation corresponding to the operator, and / or the communication overhead generated in the process of transmitting the operation result, and the operation result is obtained by performing the operation corresponding to the operator.

11. The method according to claim 10, characterized in that A result obtained after the operation corresponding to the first operator is performed on a computing device on the input tensor of the first operator is the same as the first tensor.

12. The method according to claim 10 or 11, characterized in that: The second sub-tensor is a P-dimensional tensor, and the sub-tensor obtained after the first tensor is split by the second splitting method is a Q-dimensional tensor, and P is not an integer multiple of Q.

13. The method according to any one of claims 10 to 11, characterized in that: The input tensor of the operator included in the neural network model is the image data, the audio data, the video data or the text data; or, The input tensor of the operator included in the neural network model is obtained by processing the image data, the audio data, the video data or the text data using at least one operator included in the neural network model.

14. A data processing method, characterized in that: The method comprises: Obtaining a neural network model and a processing method corresponding to the neural network model, wherein the neural network model includes a first operator and a second operator; the processing method includes a first segmentation method for segmenting an input tensor of the first operator, and a second segmentation method for segmenting an input tensor of the second operator, and the neural network model is used to process image data, audio data, video data or text data; Splitting the input tensor of the first operator by the first splitting method to obtain M first sub-tensors, wherein the M first sub-tensors include a first target sub-tensor; Processing the first target sub-tensor using the first operator to obtain a second target sub-tensor; Sending the second target sub-tensor to a computing device; receiving a first tensor sent by the computing device, where the first tensor is obtained by the computing device by rearranging a plurality of second sub-tensors through a rearrangement method, where the plurality of second sub-tensors include the second target sub-tensor; Splitting the first tensor using the second splitting method; The processing method is a candidate processing method with the smallest overhead value among at least one candidate processing method, and the overhead value represents the memory overhead generated when performing the operation corresponding to the operator, and / or the communication overhead generated in the process of transmitting the operation result, and the operation result is obtained by performing the operation corresponding to the operator.

15. The method according to claim 14, characterized in that A result obtained after the operation corresponding to the first operator is performed on a computing device on the input tensor of the first operator is the same as the first tensor.

16. The method according to claim 14 or 15, characterized in that The second sub-tensor is a P-dimensional tensor, and the sub-tensor obtained after the first tensor is split by the second splitting method is a Q-dimensional tensor, and P is not an integer multiple of Q.

17. The method according to any one of claims 14 to 15, characterized in that: The input tensor of the operator included in the neural network model is the image data, the audio data, the video data or the text data; or, The input tensor of the operator included in the neural network model is obtained by processing the image data, the audio data, the video data or the text data using at least one operator included in the neural network model.

18. A data processing device, characterized in that: The device comprises: An acquisition module, used to acquire a data flow graph of a neural network model, wherein the data flow graph includes a first operator and a second operator of the neural network model, and the neural network model is used to process image data, audio data, video data or text data; a determination module, configured to determine a processing method corresponding to the neural network model according to the data flow graph, the processing method comprising a first segmentation method for segmenting an input tensor of the first operator, a second segmentation method for segmenting an input tensor of the second operator, and a rearrangement method for rearranging tensors between the first operator and the second operator; The first segmentation method is used to segment the input tensor of the first operator to obtain M first sub-tensors, and the rearrangement method is used to rearrange the M second sub-tensors to obtain a first tensor, the M second sub-tensors are outputs of the M first sub-tensors after being processed by the first operator, the first tensor is the input tensor of the second operator, and M is a positive integer greater than 1; The processing method is a candidate processing method with the smallest overhead value among at least one candidate processing method, and the overhead value represents the memory overhead generated when performing the operation corresponding to the operator, and / or the communication overhead generated in the process of transmitting the operation result, and the operation result is obtained by performing the operation corresponding to the operator.

19. The device according to claim 18, characterized in that A result obtained after the operation corresponding to the first operator is performed on a computing device on the input tensor of the first operator is the same as the first tensor.

20. The device according to claim 18 or 19, characterized in that The second sub-tensor is a P-dimensional tensor, and the sub-tensor obtained after the first tensor is split by the second splitting method is a Q-dimensional tensor, and P is not an integer multiple of Q.

21. The device according to any one of claims 18 to 19, characterized in that The neural network model includes N operators, the N operators include the first operator and the second operator, and the determination module is specifically used to: Determine multiple candidate processing methods corresponding to the neural network model according to the data flow graph; wherein each of the multiple candidate processing methods includes a tensor splitting method for splitting the input tensor of each of the N operators, and a rearrangement method for rearranging the tensors between at least two of the N operators, the tensor splitting method is used to split the input tensor of the corresponding operator, and the rearrangement method is used to transmit the operation result obtained after performing the operation corresponding to the operator to a computing device for rearrangement; The processing method is determined from the multiple candidate processing methods according to the overhead value corresponding to each candidate processing method in the multiple candidate processing methods, wherein the overhead value includes a memory overhead value corresponding to the tensor splitting method of each operator in the N operators and a first communication overhead value corresponding to the rearrangement method included in each candidate processing method, the memory overhead value represents the memory overhead generated when performing the operation corresponding to the operator, and the first communication overhead value represents the communication overhead generated in the process of transmitting the operation result to a computing device.

22. The device according to claim 21, characterized in that The N operators include a third operator, and each candidate processing method includes a target splitting method for splitting an input tensor of the third operator. The third operator is used to perform a first operation on a sub-tensor obtained by splitting the input tensor of the third operator by the target splitting method, and transmit the operation result obtained by the operation to a computing device for performing a second operation. The overhead value also includes a second communication overhead, and the second communication overhead represents the communication overhead generated in the process of transmitting the operation result to a computing device when running the third operator.

23. The device according to claim 18 or 22, characterized in that The data flow graph further includes a third operator of the neural network model, the output tensor of the third operator is the input tensor of the second operator, and the device further includes: A merging module is used to merge the first operator into the second operator so that the second segmentation method corresponding to the second operator is associated with the first segmentation method and the rearrangement method.

24. The device according to any one of claims 18 to 19, characterized in that The data flow graph further includes a fourth operator of the neural network model, the output tensor of the first operator is the input tensor of the fourth operator, and the device further includes: A merging module is used to merge the second operator into the first operator so that the first segmentation method corresponding to the first operator is associated with the rearrangement method and the second segmentation method.

25. The device according to claim 24, characterized in that The data flow graph also includes a fifth operator and a sixth operator of the neural network model, wherein the first tensor is a first input tensor of the second operator and a first input tensor of the fifth operator, and the output tensor of the sixth operator is a second input tensor of the second operator and a second input tensor of the fifth operator, and the acquisition module is used to: Acquire a third slicing method for slicing the input tensor of the sixth operator, a fourth slicing method for slicing the second input tensor of the second operator, and a first tensor rearrangement method for rearranging the tensors between the fourth operator and the second operator; the third slicing method is used to slice the second input tensor of the sixth operator to obtain a plurality of third sub-tensors, the first tensor rearrangement method is used to rearrange a plurality of fourth sub-tensors obtained after performing operations corresponding to the sixth operator on the plurality of third sub-tensors to obtain a third tensor, and the fourth slicing method is used to slice the third tensor; The device also includes: A merging module is used to merge the sixth operator into the first operator so that the first segmentation method corresponding to the first operator establishes a first association relationship with the third segmentation method, the fourth segmentation method and the first tensor rearrangement method.

26. The device according to any one of claims 18 to 19, characterized in that The input tensor of the operator included in the neural network model is the image data, the audio data, the video data or the text data; or, The input tensor of the operator included in the neural network model is obtained by processing the image data, the audio data, the video data or the text data using at least one operator included in the neural network model.

27. A data processing device, characterized in that: The device comprises: An acquisition module, used to acquire a neural network model and a processing method corresponding to the neural network model, wherein the neural network model includes a first operator and a second operator; the processing method includes a first segmentation method for segmenting an input tensor of the first operator, a second segmentation method for segmenting an input tensor of the second operator, and a rearrangement method for rearranging tensors between the first operator and the second operator, and the neural network model is used to process image data, audio data, video data or text data; A first segmentation module, configured to segment the input tensor of the first operator by using the first segmentation method to obtain M first sub-tensors, wherein the M first sub-tensors include a first target sub-tensor; A computing module, configured to process the first target sub-tensor using the first operator to obtain a second target sub-tensor; A receiving module, configured to receive at least one second sub-tensor sent by at least one computing device; wherein each second sub-tensor is obtained by performing an operation corresponding to the first operator on a first sub-tensor other than the first target sub-tensor among the M first sub-tensors by the corresponding computing device; a rearrangement module, configured to rearrange the second target sub-tensor and the at least one second sub-tensor by using the rearrangement method to obtain a first tensor; A second segmentation module, configured to segment the first tensor using the second segmentation method; The processing method is a candidate processing method with the smallest overhead value among at least one candidate processing method, and the overhead value represents the memory overhead generated when performing the operation corresponding to the operator, and / or the communication overhead generated in the process of transmitting the operation result, and the operation result is obtained by performing the operation corresponding to the operator.

28. The device according to claim 27, characterized in that A result obtained after the operation corresponding to the first operator is performed on a computing device on the input tensor of the first operator is the same as the first tensor.

29. The device according to claim 27 or 28, characterized in that The second sub-tensor is a P-dimensional tensor, and the sub-tensor obtained after the first tensor is split by the second splitting method is a Q-dimensional tensor, and P is not an integer multiple of Q.

30. The device according to any one of claims 27 to 28, characterized in that The input tensor of the operator included in the neural network model is the image data, the audio data, the video data or the text data; or, The input tensor of the operator included in the neural network model is obtained by processing the image data, the audio data, the video data or the text data using at least one operator included in the neural network model.

31. A data processing device, characterized in that: The device comprises: An acquisition module, used to acquire a neural network model and a processing method corresponding to the neural network model, wherein the neural network model includes a first operator and a second operator; the processing method includes a first segmentation method for segmenting an input tensor of the first operator and a second segmentation method for segmenting an input tensor of the second operator, and the neural network model is used to process image data, audio data, video data or text data; A first segmentation module, configured to segment the input tensor of the first operator by using the first segmentation method to obtain M first sub-tensors, wherein the M first sub-tensors include a first target sub-tensor; A computing module, configured to process the first target sub-tensor using the first operator to obtain a second target sub-tensor; A sending module, configured to send the second target sub-tensor to a computing device; a receiving module, configured to receive a first tensor sent by the computing device, wherein the first tensor is obtained by the computing device by rearranging a plurality of second sub-tensors through a rearrangement method, wherein the plurality of second sub-tensors include the second target sub-tensor; A second segmentation module, configured to segment the first tensor using the second segmentation method; The processing method is a candidate processing method with the smallest overhead value among at least one candidate processing method, and the overhead value represents the memory overhead generated when performing the operation corresponding to the operator, and / or the communication overhead generated in the process of transmitting the operation result, and the operation result is obtained by performing the operation corresponding to the operator.

32. The device according to claim 31, characterized in that A result obtained after the operation corresponding to the first operator is performed on a computing device on the input tensor of the first operator is the same as the first tensor.

33. The device according to claim 31 or 32, characterized in that The second sub-tensor is a P-dimensional tensor, and the sub-tensor obtained after the first tensor is split by the second splitting method is a Q-dimensional tensor, and P is not an integer multiple of Q.

34. The device according to any one of claims 31 to 32, characterized in that The input tensor of the operator included in the neural network model is the image data, the audio data, the video data or the text data; or, The input tensor of the operator included in the neural network model is obtained by processing the image data, the audio data, the video data or the text data using at least one operator included in the neural network model.

35. An electronic device comprising a memory, a processor and a bus system, wherein: The memory is used to store programs, and the processor is used to execute the programs in the memory to perform the data processing method described in any one of claims 1 to 17.

36. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer is caused to execute the data processing method according to any one of claims 1 to 17.

Images