Parallel processing method and device of model, electronic equipment and readable storage medium

Abstract

The present disclosure provides a parallel processing method of a model, and relates to the technical field of artificial intelligence such as deep learning, natural language processing, image processing, and large language model. The method is applied to a first computing device in N computing devices, and comprises: obtaining a target first data sub-matrix and a target second data sub-matrix; starting a calculation process of matrix multiplication, copying a first candidate data sub-matrix in other N-1 computing devices while processing the target first data sub-matrix and the target second data sub-matrix; in response to obtaining a first processing result between the target first data sub-matrix and the target second data sub-matrix, processing the copied first candidate data sub-matrix and a corresponding target data sub-matrix; and in response to obtaining a second processing result between the first candidate data sub-matrix and the corresponding target data sub-matrix, obtaining a target processing result of the first computing device according to the first processing result and the second processing result.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to the fields of artificial intelligence, such as deep learning, natural language processing, image processing, and large language models. It provides a method, apparatus, electronic device, and readable storage medium for parallel processing of models. Background Technology

[0002] With the successful application of deep learning models in various fields, people have begun to focus on how to scale these models to a larger scale to improve their data processing capabilities, accuracy, and performance. Based on this, ultra-large-scale deep learning models have emerged. However, ultra-large-scale deep learning models face pressure from memory and training speed limitations; however, the memory of a single computing device is very limited. Therefore, how to utilize the limited memory of various computing devices to train larger models is a pressing technical problem that needs to be solved. Summary of the Invention

[0003] According to a first aspect of this disclosure, a parallel training method for a model is provided, applied to a first computing device among N computing devices, comprising: acquiring a target first data submatrix from N first data submatrixes and a target second data submatrix from N second data submatrixes; wherein the N first data submatrixes are obtained by partitioning the first data matrix according to a first partitioning method, and the N second data submatrixes are obtained by partitioning the second data matrix according to a second partitioning method; wherein N is a positive integer greater than or equal to 2; initiating a matrix multiplication calculation process, processing the target first data submatrix and the target second data submatrix simultaneously, and training the other N-1 computing devices. The first candidate data submatrix is ​​copied; in response to obtaining a first processing result between the target first data submatrix and the target second data submatrix, the copied first candidate data submatrix and its corresponding target data submatrix are processed; in response to obtaining a second processing result between the first candidate data submatrix and its corresponding target data submatrix, the target processing result of the first computing device is obtained based on the first processing result and the second processing result; wherein, the target processing result of the first computing device is used to concatenate with the N-1 target processing results of the other N-1 computing devices to obtain the target processing result between the first data matrix and the second data matrix.

[0004] According to a second aspect of this disclosure, a parallel processing apparatus for a model is provided, comprising a first computing device located among N computing devices, including: an acquisition unit configured to acquire a target first data submatrix from N first data submatrices and a target second data submatrix from N second data submatrices; wherein the N first data submatrices are obtained by partitioning a first data matrix according to a first partitioning method, and the N second data submatrices are obtained by partitioning a second data matrix according to a second partitioning method; wherein N is a positive integer greater than or equal to 2; and a first processing unit configured to initiate a matrix multiplication calculation process, processing the target first data submatrix and the target second data submatrix while simultaneously processing the first data submatrix from the other N-1 computing devices. A candidate data submatrix is ​​copied; a second processing unit is configured to process the copied first candidate data submatrix and its corresponding target data submatrix in response to obtaining a first processing result between the target first data submatrix and the target second data submatrix; a third processing unit is configured to obtain a target processing result of the first computing device based on the first processing result and the second processing result in response to obtaining a second processing result between the first candidate data submatrix and its corresponding target data submatrix; wherein, the target processing result of the first computing device is used to concatenate with the N-1 target processing results of the other N-1 computing devices to obtain a target processing result between the first data matrix and the second data matrix.

[0005] According to a third aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0006] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method described above.

[0007] According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method described above.

[0008] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0010] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure;

[0011] Figure 2 This is a schematic diagram according to the second embodiment of the present disclosure;

[0012] Figure 3 This is a schematic diagram according to the third embodiment of the present disclosure;

[0013] Figure 4 This is a schematic diagram according to the fourth embodiment of the present disclosure;

[0014] Figure 5 This is a schematic diagram according to the fifth embodiment of the present disclosure;

[0015] Figure 6 This is a schematic diagram according to the sixth embodiment of the present disclosure;

[0016] Figure 7 This is a schematic diagram according to the seventh embodiment of the present disclosure;

[0017] Figure 8 This is a schematic diagram according to the eighth embodiment of the present disclosure;

[0018] Figure 9 This is a schematic diagram according to the ninth embodiment of the present disclosure;

[0019] Figure 10 This is a schematic diagram according to the tenth embodiment of the present disclosure;

[0020] Figure 11 This is a schematic diagram according to the eleventh embodiment of the present disclosure;

[0021] Figure 12 This is a schematic diagram according to the twelfth embodiment of the present disclosure;

[0022] Figure 13 This is a schematic diagram according to the thirteenth embodiment of this disclosure;

[0023] Figure 14 This is a schematic diagram according to the fourteenth embodiment of this disclosure;

[0024] Figure 15 This is a schematic diagram according to the fifteenth embodiment of this disclosure;

[0025] Figure 16 This is a schematic diagram according to the sixteenth embodiment of this disclosure;

[0026] Figure 17 This is a schematic diagram according to the seventeenth embodiment of this disclosure;

[0027] Figure 18 This is a schematic diagram according to the eighteenth embodiment of this disclosure;

[0028] Figure 19 This is a schematic diagram according to the nineteenth embodiment of this disclosure;

[0029] Figure 20 This is a schematic diagram according to the twentieth embodiment of the present disclosure;

[0030] Figure 21 This is a schematic diagram according to the twenty-first embodiment of this disclosure;

[0031] Figure 22 This is a schematic diagram according to the twenty-second embodiment of this disclosure;

[0032] Figure 23 This is a schematic diagram according to the twenty-third embodiment of this disclosure;

[0033] Figure 24 This is a schematic diagram according to the twenty-fourth embodiment of this disclosure;

[0034] Figure 25 This is a schematic diagram according to the twenty-fifth embodiment of this disclosure;

[0035] Figure 26 This is a schematic diagram according to the twenty-sixth embodiment of this disclosure;

[0036] Figure 27 This is a schematic diagram according to the twenty-seventh embodiment of this disclosure;

[0037] Figure 28 This is a block diagram of an electronic device used to implement the parallel processing method of the model in the embodiments of this disclosure. Detailed Implementation

[0038] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and mechanisms are omitted in the following description.

[0039] Figure 1 This is a schematic diagram based on the first embodiment of this disclosure. (See diagram below.) Figure 1 As shown, the parallel processing method of the model in this embodiment is applied to the first computing device among N computing devices, and specifically includes the following steps:

[0040] S101. Obtain the target first data submatrix from N first data submatrixes and the target second data submatrix from N second data submatrixes; wherein, the N first data submatrixes are obtained by dividing the first data matrix according to a first division method, and the N second data submatrixes are obtained by dividing the second data matrix according to a second division method; and N is a positive integer greater than or equal to 2.

[0041] S102. Initiate the matrix multiplication calculation process, while processing the target first data submatrix and the target second data submatrix, copy the first candidate data submatrix in the other N-1 computing devices;

[0042] S103. In response to obtaining the first processing result between the target first data submatrix and the target second data submatrix, process the copied first candidate data submatrix and its corresponding target data submatrix.

[0043] S104. In response to obtaining the second processing result between the first candidate data submatrix and its corresponding target data submatrix, the target processing result of the first computing device is obtained based on the first processing result and the second processing result; wherein, the target processing result of the first computing device is used to concatenate with the N-1 target processing results of the other N-1 computing devices to obtain the target processing result between the first data matrix and the second data matrix.

[0044] In this embodiment, the first data matrix can be either the feature matrix corresponding to the input data or the weight matrix corresponding to the target model; the second data matrix can be either the feature matrix corresponding to the input data or the weight matrix corresponding to the target model.

[0045] In this embodiment, the target model is a deep learning model, and the elements in the weight matrix of the target model are the parameters in the deep learning model. It can be understood that the weight matrix in this embodiment can be the weight matrix of some network layers in the target model.

[0046] In this embodiment, if the target model is an image processing model and the input data is an image, then the feature matrix corresponding to the input data can be a matrix composed of the pixel values ​​of each pixel in the image; if the target model is a natural language processing model and the input data is text, then the feature matrix corresponding to the input data can be a matrix composed of the word vectors of each word in the text.

[0047] In this embodiment, the computing device can be a device with parallel computing capabilities, such as a GPU (Graphics Processing Unit), an NPU (Neural Processing Unit), a GPU-like device, or an XPU. This embodiment does not limit the specific device to this type.

[0048] In this embodiment, the first computing device is one of N computing devices, and the N computing devices correspond one-to-one with the N first data sub-matrices and the N second data sub-matrices, where N is a positive integer greater than or equal to 2; wherein, the data sub-matrices corresponding to different computing devices can be stored in the memory of the computing devices.

[0049] In this embodiment, any segmentation method can be used to segment the first data matrix and the second data matrix; wherein, the first segmentation method corresponding to the first data matrix and the second segmentation method corresponding to the second data matrix can be the same or different.

[0050] In other words, this embodiment does not limit the way the data submatrix obtained by the computing device is divided, so that the computing device can perform parallel processing on the data submatrix obtained by any division method, thereby expanding the application scenarios and realizing the purpose of true distributed parallel matrix multiplication of computing devices such as GPUs, NPUs, GPU-like devices or XPUs.

[0051] In this embodiment, the first segmentation method can be row segmentation (i.e., segmenting the first data matrix in the row direction) or column segmentation (i.e., segmenting the first data matrix in the column direction); the second segmentation method can be row segmentation (i.e., segmenting the second data matrix in the row direction) or column segmentation (i.e., segmenting the second data matrix in the column direction).

[0052] Row cutting refers to dividing an M-row, K-column data matrix into N M / N-row, K-column data submatrices; column cutting refers to dividing an M-row, K-column data matrix into N M-row, K / N-column data submatrices.

[0053] In this embodiment, each of the N first data sub-matrixes and each of the N second data sub-matrixes are distributed to different computing devices. When the first computing device in this embodiment executes S101, it will use the distributed first data sub-matrix as the target first data sub-matrix and the distributed second data sub-matrix as the target second data sub-matrix.

[0054] In this embodiment, after the first computing device receives the target first data submatrix and the target second data submatrix in S101, it executes S102 to start the matrix multiplication calculation process, processes the received target first data submatrix and target second data submatrix, and simultaneously copies the first candidate data submatrix in the other N-1 computing devices.

[0055] In this embodiment, when the first computing device executes S102, it can initiate the matrix multiplication calculation process by calling the General Matrix Multiply (GEMM) kernel.

[0056] In this embodiment, after the first computing device executes S102 to complete the matrix multiplication calculation process, it can simultaneously perform matrix multiplication of the target first data submatrix and the target second data submatrix, and copy the first candidate data submatrix from the other N-1 computing devices.

[0057] In other words, in this embodiment, the first computing device communicates with the other N-1 computing devices while performing matrix multiplication on the existing data submatrix, thereby copying the first candidate data submatrix from the other N-1 computing devices. This enables the overlap between computation and communication when the first computing device, such as GPU, NPU, GPU-like or XPU, is performing parallel processing.

[0058] In this embodiment, when the first computing device performs S102 to process the received target first data submatrix and target second data submatrix, the following implementation method can be adopted: according to the first preset block size, the target first data submatrix is ​​divided into multiple target first matrix blocks and the target second data submatrix is ​​divided into multiple target second matrix blocks; based on the obtained multiple target first matrix blocks and multiple target second matrix blocks, the processing result between the target first data submatrix and the target second data submatrix is ​​obtained, and the obtained processing result is the matrix multiplication result.

[0059] In this embodiment, the first preset block size can be a block size that matches the size of a Warp (Warp is the basic unit of scheduling and execution in a GPU).

[0060] In other words, this embodiment achieves the goal of performing Warp-level calculations within the invoked general matrix multiplication kernel by dividing the data submatrix and then performing matrix multiplication calculations between the data submatrixes based on the divided matrix blocks. This can improve the computational efficiency of the first computing device, such as GPU, NPU, GPU-like or XPU, when performing matrix multiplication.

[0061] In this embodiment, the first candidate data submatrix that the first computing device wants to copy from the other N-1 computing devices can be all or part of the first data submatrix corresponding to the other N-1 computing devices, or it can be all or part of the second data submatrix corresponding to the other N-1 computing devices.

[0062] In this embodiment, when the first computing device performs S102 to copy the first candidate data sub-matrix from the other N-1 computing devices, the following implementation method can be adopted: construct a set of partitioning methods based on the first partitioning method, the second partitioning method, and the third partitioning method corresponding to the output data matrix; determine the first candidate data sub-matrix based on the constructed set of partitioning methods; and copy the first candidate data sub-matrix from the other N-1 computing devices.

[0063] In this embodiment, the first computing device copies the first candidate data submatrix of the other N-1 computing devices by accessing the memory of the other N-1 computing devices.

[0064] In this embodiment, the output data matrix is ​​the target processing result between the first data matrix and the second data matrix; the third segmentation method can be row segmentation (i.e., segmenting the output data matrix in the row direction) or column segmentation (i.e., segmenting the output data matrix in the column direction).

[0065] Since this embodiment supports arbitrary partitioning of the input and output data matrices (usually by average partitioning the matrix), the partitioning method set constructed in this embodiment will have 8 cases based on the partitioning method of the data matrix, specifically (row partitioning, row partitioning, row partitioning), (row partitioning, row partitioning, column partitioning), (row partitioning, column partitioning, row partitioning), (row partitioning, column partitioning, column partitioning), (column partitioning, row partitioning, row partitioning), (column partitioning, row partitioning, column partitioning), (column partitioning, column partitioning, row partitioning) and (column partitioning, column partitioning, column partitioning). In this embodiment, different partitioning method sets correspond to different types of first candidate data submatrices. Therefore, based on the constructed partitioning method set, the first computing device determines what type of first candidate data submatrix to be copied from the other N-1 computing devices.

[0066] In this embodiment, the first computing device obtains the first candidate data submatrix required for matrix multiplication calculation from the other N-1 computing devices by copying it. This avoids the steps of sending and receiving submatrixes between computing devices, which can reduce the time required for the first computing device, such as GPU, NPU, GPU-like or XPU, to obtain the first candidate data submatrix from the other N-1 computing devices, thereby improving the efficiency of subsequent matrix multiplication calculation based on the first candidate data submatrix.

[0067] In this embodiment, the first computing device can store the copied first candidate data submatrix, which corresponds to different computing devices, into the memory of the first computing device for subsequent processing.

[0068] In this embodiment, when the first computing device performs S102 to copy the first candidate data sub-matrix in the other N-1 computing devices, it can copy the first candidate data sub-matrix in the other N-1 computing devices multiple times according to the second preset block size. That is, each time it copies a matrix block corresponding to the second preset block size from the other N-1 computing devices. In this embodiment, the second preset block size can be the same as or different from the first preset block size. In this embodiment, the second preset block size can be set according to actual needs.

[0069] In other words, in this embodiment, the first computing device can copy the first candidate data sub-matrix in the other N-1 computing devices multiple times in smaller blocks. After completing the processing between the target first data sub-matrix and the target second data sub-matrix, the first computing device can perform matrix multiplication calculations more quickly based on the copied first candidate data sub-matrix (or the matrix blocks corresponding to the first candidate data sub-matrix). This can further improve the overlap efficiency between the computing communication of the first computing devices such as GPU, NPU, GPU-like or XPU.

[0070] In this embodiment, after executing S102, the first computing device executes S103 in response to obtaining the first processing result between the target first data submatrix and the target second data submatrix, and processes the copied first candidate data submatrix and its corresponding target data submatrix.

[0071] In this embodiment, if the first candidate data submatrix is ​​a first data submatrix in the other N-1 computing devices, then the target data submatrix corresponding to the first candidate data submatrix is ​​the target second data submatrix in the first computing device; if the first candidate data submatrix is ​​a second data submatrix in the other N-1 computing devices, then the target data submatrix corresponding to the first candidate data submatrix is ​​the target first data submatrix in the first computing device.

[0072] In this embodiment, after the first computing device executes S103 to determine that the matrix multiplication calculation between the target first data submatrix and the target second data submatrix has been completed, it can immediately perform matrix multiplication calculation on the first candidate data submatrix copied from the other N-1 computing devices and its corresponding target data submatrix.

[0073] It is understood that when the first computing device in this embodiment executes S103, it may also include the following: obtaining the copy status of the first candidate data submatrix; in response to determining that the copy status is incomplete, while processing the copied first candidate data submatrix and its corresponding target data submatrix, it continues to copy the remaining first candidate data submatrix in the other N-1 computing devices.

[0074] In this embodiment, after executing S103, the first computing device executes S104 in response to obtaining the second processing result between the first candidate data submatrix and its corresponding target data submatrix. Based on the first processing result and the candidate processing result, the target processing result of the first computing device is obtained.

[0075] In this embodiment, the target processing result of the first computing device is concatenated with the target processing results of the other N-1 computing devices to obtain the output data matrix. This output data matrix is ​​the target processing result between the first data matrix and the second data matrix.

[0076] In this embodiment, the second processing result obtained by the first computing device when executing S104 is the processing result between the complete first candidate data submatrix and its corresponding target data submatrix.

[0077] In this embodiment, after executing S104 to obtain the second processing result between the first candidate data submatrix and its corresponding target data submatrix, the first computing device can close the matrix multiplication calculation process, and then obtain the target processing result corresponding to the first computing device based on the first processing result and the second processing result.

[0078] In some special cases, such as when the first candidate data submatrix copied by the first computing device is the first data submatrix in the other N-1 computing devices, the first computing device in this embodiment obtains the target processing result corresponding to the first computing device when executing S104, based on the first calculation result and the second calculation result copied from the other N-1 computing devices.

[0079] In other words, the first computing device in this embodiment can obtain the target processing result based on the first calculation result and the second calculation result calculated by itself, or it can obtain the target processing result based on the first calculation result calculated by itself and the second calculation result calculated by other computing devices, which can further improve the accuracy of the obtained target processing result.

[0080] In this embodiment, the parallel processing method of the above model only requires one call to the general matrix multiplication kernel, avoiding the problem of multiple calls to the general matrix multiplication kernel by the first computing devices such as GPUs, NPUs, GPU-like devices, or XPUs. This improves the computational efficiency of the first computing devices such as GPUs, NPUs, GPU-like devices, or XPUs when performing matrix multiplication. Furthermore, in this embodiment, the first computing devices such as GPUs, NPUs, GPU-like devices, or XPUs can perform matrix multiplication calculations as soon as the data submatrices required for the matrix multiplication calculation are ready, without affecting the copying process of the data submatrices. This allows the computation and communication of the first computing devices such as GPUs, NPUs, GPU-like devices, or XPUs to overlap during the parallel processing of the model without affecting the computational efficiency of matrix multiplication. This significantly improves the efficiency of the overlapping computation and communication of the first computing devices such as GPUs, NPUs, GPU-like devices, or XPUs, thereby achieving the goal of distributed parallel matrix computation of the first computing devices such as GPUs, NPUs, GPU-like devices, or XPUs more efficiently.

[0081] In this embodiment, the weight matrix may include parameters of some network layers in the target model. Accordingly, the processing result between the feature matrix and the weight matrix can be the processing result of some network layers in the target model, i.e., a staged processing result.

[0082] Therefore, in practical applications, when obtaining the target processing results from each computing device, it can be determined whether to concatenate the target processing results from each computing device based on the model structure. For example, one can choose to continue processing the next network layer while maintaining the segmented state, or one can choose to concatenate the processing results from each computing device to obtain the target processing result between the feature matrix and the weight matrix.

[0083] Figure 2 This is a schematic diagram according to the second embodiment of this disclosure. (See diagram below.) Figure 2 As shown in the figure, when executing S104 "obtain the target processing result of the first computing device based on the first processing result and the second processing result", the implementation method that can be adopted in this embodiment is as follows:

[0084] S201. Construct a set of segmentation methods based on the first segmentation method, the second segmentation method, and the third segmentation method corresponding to the output data matrix;

[0085] S202. Determine the second candidate data submatrix based on the set of segmentation methods;

[0086] S203, Copy the second candidate data submatrix from the other N-1 computing devices;

[0087] S204. Based on the first processing result, the second processing result, and the second candidate data submatrix, the target processing result of the first computing device is obtained.

[0088] In other words, the first computing device in this embodiment can also determine the second candidate data sub-matrix to be copied from the other N-1 computing devices based on the set of splitting methods constructed by the splitting method, and then obtain the corresponding target processing result based on the first processing result, the second processing result and the copied second candidate data sub-matrix.

[0089] In this embodiment, different sets of segmentation methods correspond to different second candidate data sub-matrices; wherein, the second candidate data sub-matrices are the second calculation results obtained by the other N-1 computing devices through matrix multiplication.

[0090] Figure 3 This is a schematic diagram according to the third embodiment of this disclosure. (See diagram below.) Figure 3 As shown in the figure, when executing S102 "copying the first candidate data submatrix in the other N-1 computing devices", this embodiment can be implemented in the following way:

[0091] S301. Obtain the preset cycle order among the N computing devices;

[0092] S302. According to the preset cyclic order, the first candidate data submatrix in the computing devices located after the first computing device is copied sequentially.

[0093] In other words, in this embodiment, the first computing device copies the first candidate data submatrix in the computing device following it in a preset cyclic order among the N computing devices, thereby ensuring the orderly execution of the data copying process in the first computing device such as GPU, NPU, GPU-like device or XPU.

[0094] In this embodiment, the preset cycle order refers to the order in which the devices form a cycle. For example, if N computing devices include computing device 0, computing device 1, computing device 2, and computing device 3, then the preset cycle order is the order of the cycle formed by computing device 0 - computing device 1 - computing device 2 - computing device 3 - computing device 0.

[0095] In this embodiment, when the first computing device executes S302, it sequentially copies the first candidate data submatrix in the computing devices located after the first computing device. After completing the copying of the candidate data submatrix in the current computing device, it copies the candidate data submatrix in the next computing device, and so on.

[0096] Figure 4 This is a schematic diagram according to the fourth embodiment of the present disclosure. Figure 4 The diagram illustrates the first method of legally partitioning the first data matrix, the second data matrix, and the output data matrix, wherein the first partitioning method is row partitioning, the second partitioning method is row partitioning, and the third partitioning method is row partitioning.

[0097] Figure 5 This is a schematic diagram according to the fifth embodiment of the present disclosure. Figure 5 The diagram illustrates a second method for legally partitioning the first data matrix, the second data matrix, and the output data matrix. The first partitioning method is row partitioning, the second partitioning method is row partitioning, and the third partitioning method is column partitioning.

[0098] Figure 6 A schematic diagram based on the sixth embodiment of this disclosure. Figure 6 The diagram illustrates a third method for legally partitioning the first data matrix, the second data matrix, and the output data matrix. The first partitioning method is row partitioning, the second partitioning method is column partitioning, and the third partitioning method is row partitioning.

[0099] Figure 7 This is a schematic diagram according to the seventh embodiment of the present disclosure. Figure 7 The diagram illustrates a fourth method for legally partitioning the first data matrix, the second data matrix, and the output data matrix. The first partitioning method is row partitioning, the second partitioning method is column partitioning, and the third partitioning method is column partitioning.

[0100] Figure 8 This is a schematic diagram according to the eighth embodiment of the present disclosure. Figure 8 The diagram illustrates the fifth method of legally partitioning the first data matrix, the second data matrix, and the output data matrix. The first partitioning method is column partitioning, the second partitioning method is row partitioning, and the third partitioning method is row partitioning.

[0101] Figure 9 This is a schematic diagram according to the ninth embodiment of the present disclosure. Figure 9 The diagram illustrates the sixth method of legally partitioning the first data matrix, the second data matrix, and the output data matrix. The first partitioning method is column partitioning, the second partitioning method is row partitioning, and the third partitioning method is column partitioning.

[0102] Figure 10 This is a schematic diagram according to the tenth embodiment of the present disclosure. Figure 10 The diagram illustrates the seventh method of legally partitioning the first data matrix, the second data matrix, and the output data matrix. The first partitioning method is column partitioning, the second partitioning method is column partitioning, and the third partitioning method is row partitioning.

[0103] Figure 11 This is a schematic diagram according to the eleventh embodiment of the present disclosure. Figure 11 The diagram illustrates the eighth method of legally partitioning the first data matrix, the second data matrix, and the output data matrix. The first partitioning method is column partitioning, the second partitioning method is column partitioning, and the third partitioning method is column partitioning.

[0104] Figure 12 This is a schematic diagram according to the twelfth embodiment of this disclosure. Figure 12 The diagram shows a schematic of the segmentation communication module in this embodiment. In this embodiment, the segmentation unit in the segmentation communication module is used to segment the first data matrix, the second data matrix and the output data matrix using any legal segmentation method (such as any segmentation method in embodiments 4 to 11 above). The communication unit in the segmentation communication module is used to select the target communication method from the candidate communication methods in the set of segmentation methods used by the segmentation unit.

[0105] Figure 13 This is a schematic diagram according to the thirteenth embodiment of this disclosure. Figure 13 The diagram shows the selection of the target communication method when using the first method to legally partition the matrix in this embodiment: The communication unit selects the target communication method from multiple first candidate communication methods corresponding to the first partitioning method set, based on the first partitioning method being row partitioning, the second partitioning method being row partitioning, and the third partitioning method being row partitioning.

[0106] Figure 14 This is a schematic diagram according to the fourteenth embodiment of this disclosure. Figure 14 The diagram shows the selection of the target communication method when using the second method to legally partition the matrix in this embodiment: The communication unit selects the target communication method from multiple second candidate communication methods corresponding to the second partitioning method set, which is obtained by the first partitioning method being row partitioning, the second partitioning method being row partitioning, and the third partitioning method being column partitioning.

[0107] Figure 15 This is a schematic diagram according to the fifteenth embodiment of this disclosure. Figure 15 The diagram shows the selection of the target communication method when using the third method to legally partition the matrix in this embodiment: The communication unit selects the target communication method from multiple third candidate communication methods corresponding to the third partitioning method set, based on the third partitioning method set obtained by the first partitioning method being row partitioning, the second partitioning method being column partitioning, and the third partitioning method being row partitioning.

[0108] Figure 16 This is a schematic diagram according to the sixteenth embodiment of this disclosure. Figure 16The diagram shows the selection of the target communication method when using the fourth method to legally partition the matrix in this embodiment: The communication unit selects the target communication method from multiple fourth candidate communication methods corresponding to the fourth partitioning method set obtained by the first partitioning method being row partitioning, the second partitioning method being column partitioning, and the third partitioning method being column partitioning.

[0109] Figure 17 This is a schematic diagram according to the seventeenth embodiment of this disclosure. Figure 17 The diagram shows the selection of the target communication method when using the fifth method to legally partition the matrix in this embodiment: The communication unit selects the target communication method from multiple fifth candidate communication methods corresponding to the fifth partitioning method set obtained by the first partitioning method being column partitioning, the second partitioning method being row partitioning, and the third partitioning method being row partitioning.

[0110] Figure 18 This is a schematic diagram according to the eighteenth embodiment of this disclosure. Figure 18 The diagram shows the selection of the target communication method when using the sixth method to legally partition the matrix in this embodiment: The communication unit selects the target communication method from the set of sixth partitioning methods obtained by the first partitioning method being column partitioning, the second partitioning method being row partitioning, and the third partitioning method being column partitioning.

[0111] Figure 19 This is a schematic diagram according to the nineteenth embodiment of this disclosure. Figure 19 The diagram shows the selection of the target communication method when using the seventh method to legally partition the matrix in this embodiment: The communication unit selects the target communication method from multiple seventh candidate communication methods corresponding to the seventh partitioning method set obtained by the first partitioning method being column partitioning, the second partitioning method being column partitioning, and the third partitioning method being row partitioning.

[0112] Figure 20 This is a schematic diagram according to the twentieth embodiment of the present disclosure. Figure 20 The diagram shows the selection of the target communication method when using the eighth method to legally partition the matrix in this embodiment: The communication unit selects the target communication method from the multiple seventh candidate communication methods corresponding to the seventh partitioning method set obtained by the first partitioning method being column partitioning, the second partitioning method being column partitioning, and the third partitioning method being row partitioning.

[0113] Figure 21 This is a schematic diagram according to the twenty-first embodiment of this disclosure. Figure 21The diagram shows a processing flowchart of the first computing device performing parallel processing of the model based on a target communication method selected in the thirteenth embodiment; in this embodiment, the first segmentation method is row segmentation, the second segmentation method is row segmentation, and the third segmentation method is row segmentation.

[0114] like Figure 21 As shown in the figure, this embodiment includes two computing devices, namely GPU1 and GPU2, wherein GPU1 is the first computing device and GPU2 is another computing device corresponding to GPU1; the target first data sub-matrix obtained by GPU1 is A1 composed of A11 and A12, and the target second data sub-matrix is ​​B1; the target first data sub-matrix obtained by GPU2 is A2 composed of A21 and A22, and the target second data sub-matrix is ​​B2.

[0115] In this embodiment, GPU1 performs matrix multiplication calculation based on the obtained A11 and B1, and simultaneously copies B2 (i.e., the first candidate data submatrix) from GPU2; after determining that the matrix multiplication calculation between A11 and B1 has been completed and obtained (A11*B1=C11), GPU1 continues to perform matrix multiplication calculation based on A12 and the copied B2 to obtain (A12*B2=C22); GPU1 obtains (C11+C12=C1) based on the obtained C11 (i.e., the first calculation result) and C12 (i.e., the second calculation result), and C1 is the target calculation result of the first computing device.

[0116] Figure 22 This is a schematic diagram according to the twenty-second embodiment of this disclosure. Figure 22 The diagram shows a processing flowchart of the first computing device performing parallel processing of the model based on another target communication method selected in the thirteenth embodiment; in this embodiment, the first segmentation method is row segmentation, the second segmentation method is row segmentation, and the third segmentation method is row segmentation.

[0117] like Figure 22 As shown in the figure, this embodiment includes two computing devices, namely GPU1 and GPU2, wherein GPU1 is the first computing device and GPU2 is another computing device corresponding to GPU1; the target first data sub-matrix obtained by GPU1 is A1 composed of A11 and A12, and the target second data sub-matrix is ​​B1; the target first data sub-matrix obtained by GPU2 is A2 composed of A21 and A22, and the target second data sub-matrix is ​​B2.

[0118] In this embodiment, GPU1 performs matrix multiplication calculation based on the obtained A11 and B1, and simultaneously copies A21 (i.e., the first candidate data submatrix) from GPU2; after GPU1 determines that it has completed the matrix multiplication calculation between A11 and B1 and obtained (A11*B1=C11), it performs the calculation of (A21*B1=C22); after GPU1 determines that it has completed the calculation of C22, it copies C12 from GPU2; based on the obtained C11 (i.e., the first calculation result) and C12, GPU1 obtains (C11+C12=C1), and C1 is the target calculation result of the first computing device.

[0119] Figure 23 This is a schematic diagram according to the twenty-third embodiment of this disclosure. Figure 23 The diagram shows a processing flowchart of the first computing device performing parallel processing of the model based on a target communication method selected in the fourteenth embodiment; in this embodiment, the first segmentation method is row segmentation, the second segmentation method is row segmentation, and the third segmentation method is column segmentation.

[0120] like Figure 23 As shown in the figure, this embodiment includes two computing devices, namely GPU1 and GPU2, wherein GPU1 is the first computing device and GPU2 is another computing device corresponding to GPU1; the target first data sub-matrix obtained by GPU1 is A1 composed of A11 and A12, and the target second data sub-matrix is ​​B1 composed of B11 and B12; the target first data sub-matrix obtained by GPU2 is A2 composed of A21 and A22, and the target second data sub-matrix is ​​B2 composed of B21 and B22.

[0121] In this embodiment, GPU1 performs matrix multiplication calculations based on the acquired A11 and B11, and A11 and B12, and simultaneously copies B21 and B22 (i.e., the first candidate data sub-matrices) from GPU2. After determining that the matrix multiplication calculation between A11 and B11 has been completed to obtain (A11*B11 = C11_1), and the matrix multiplication result between A11 and B12 has been obtained to obtain (A11*B12 = C21_1), GPU1 continues to perform matrix multiplication calculations based on A12 and the copied data. The matrix multiplication of B21, A12, and the copied B22 yields (A12*B21=C11_2) and (A12*B22=C21_2) respectively; GPU1 obtains C11 based on C11_1 (i.e., the first calculation result) and C11_2 (i.e., the second calculation result), and (C11+C12=C1) based on C12 (i.e., the second candidate data submatrix) copied from GPU2. C1 is the target calculation result of the first computing device.

[0122] Figure 24 This is a schematic diagram according to the twenty-fourth embodiment of this disclosure. Figure 24 The diagram shows a processing flowchart of the first computing device performing parallel processing of the model based on another target communication method selected in the fourteenth embodiment; in this embodiment, the first segmentation method is row segmentation, the second segmentation method is row segmentation, and the third segmentation method is column segmentation.

[0123] like Figure 24 As shown in the figure, this embodiment includes two computing devices, namely GPU1 and GPU2, wherein GPU1 is the first computing device and GPU2 is another computing device corresponding to GPU1; the target first data sub-matrix obtained by GPU1 is A1 composed of A11 and A12, and the target second data sub-matrix is ​​B1; the target first data sub-matrix obtained by GPU2 is A2 composed of A21 and A22, and the target second data sub-matrix is ​​B2.

[0124] In this embodiment, GPU1 performs matrix multiplication on A11 and B11, and simultaneously copies B2 (i.e., the first candidate data submatrix) from GPU2. After determining that the matrix multiplication between A11 and B11 has been completed and obtained (A11*B11=C11_C21_1), GPU1 continues to perform matrix multiplication on A12 and A12, obtaining (A12*B2=C11_C21_2). Based on C11_C21_2 and C11_C21_1, C11_C21 is obtained. After GPU1 divides C11_C21 into C11 and C21, it obtains (C11+C12=C1) based on C12 copied from GPU2 and the divided C11. C1 is the target calculation result of the first computing device.

[0125] Figure 25 This is a schematic diagram according to the 25th embodiment of this disclosure. Figure 25 The diagram shows the framework of the first computing device performing parallel processing of the model: Figure 25 While performing matrix multiplication, the first computing device also performs operations on the first candidate data submatrix in the other N-1 computing devices, thereby achieving overlap between computation and communication. Furthermore, the first computing device will continuously perform matrix multiplication based on the copied first candidate data submatrix, thereby improving the computational efficiency of matrix multiplication.

[0126] Figure 26This is a schematic diagram according to the 26th embodiment of this disclosure. This embodiment illustrates the overlap of computational communication between computing devices based on the Wrap level. In this embodiment, GPU1 and GPU2 further divide the submatrix to be multiplied into corresponding Wrap-level matrix blocks, and then perform matrix multiplication calculations based on the matrix blocks. During the calculation process, IPC Copy is also executed to copy the corresponding data from other computing devices for subsequent matrix multiplication calculations.

[0127] Figure 27 This is a schematic diagram according to the twenty-seventh embodiment of this disclosure. (See diagram below.) Figure 27 As shown, the parallel processing device 2700 of the model in this embodiment, located in the first computing device among N computing devices, includes:

[0128] Acquisition unit 2701 is used to acquire a target first data submatrix from N first data submatrices and a target second data submatrix from N second data submatrices; wherein, the N first data submatrices are obtained by dividing the first data matrix according to a first division method, and the N second data submatrices are obtained by dividing the second data matrix according to a second division method; wherein N is a positive integer greater than or equal to 2;

[0129] The first processing unit 2702 is used to initiate the matrix multiplication calculation process, and while processing the target first data submatrix and the target second data submatrix, it copies the first candidate data submatrix in the other N-1 computing devices.

[0130] The second processing unit 2703 is configured to process the copied first candidate data submatrix and its corresponding target data submatrix in response to obtaining a first processing result between the target first data submatrix and the target second data submatrix;

[0131] The third processing unit 2704 is configured to, in response to obtaining a second processing result between the first candidate data submatrix and its corresponding target data submatrix, obtain a target processing result of the first computing device based on the first processing result and the second processing result; wherein, the target processing result of the first computing device is used to concatenate with the N-1 target processing results of the other N-1 computing devices to obtain a target processing result between the first data matrix and the second data matrix.

[0132] In this embodiment, each of the N first data sub-matrixes and each of the N second data sub-matrixes are distributed to different computing devices. Then, the acquisition unit 2701 acquires the distributed first data sub-matrix as the target first data sub-matrix and the distributed second data sub-matrix as the target second data sub-matrix.

[0133] In this embodiment, after the first computing device obtains the target first data submatrix and the target second data submatrix by the acquisition unit 2701, the first processing unit 2702 initiates the matrix multiplication calculation process to process the received target first data submatrix and target second data submatrix, and at the same time copies the first candidate data submatrix in the other N-1 computing devices.

[0134] The first processing unit 2702 can initiate the matrix multiplication calculation process by calling the General Matrix Multiply (GEMM) kernel.

[0135] After the first processing unit 2702 completes the matrix multiplication calculation process, it can simultaneously perform matrix multiplication of the target first data submatrix and the target second data submatrix, and copy the first candidate data submatrix from the other N-1 computing devices.

[0136] In other words, while performing matrix multiplication on the existing data submatrix, the first processing unit 2702 communicates with the other N-1 computing devices to copy the first candidate data submatrix from the other N-1 computing devices. This enables the overlap between computation and communication when the first computing devices such as GPU, NPU, GPU-like or XPU are performing parallel processing.

[0137] When the first processing unit 2702 processes the received target first data submatrix and target second data submatrix, it can adopt the following implementation method: according to the first preset block size, the target first data submatrix is ​​divided into multiple target first matrix blocks and the target second data submatrix is ​​divided into multiple target second matrix blocks; based on the obtained multiple target first matrix blocks and multiple target second matrix blocks, the processing result between the target first data submatrix and the target second data submatrix is ​​obtained, and the obtained processing result is the matrix multiplication result.

[0138] In this embodiment, the first preset block size can be a block size that matches the size of a Warp (Warp is the basic unit of scheduling and execution in a GPU).

[0139] In other words, the first processing unit 2702 achieves the purpose of performing Warp-level calculations within the invoked general matrix multiplication core by dividing the data submatrix and then performing matrix multiplication calculations between the data submatrixes based on the divided matrix blocks. This can improve the computational efficiency of the first computing device, such as GPU, NPU, GPU-like or XPU, when performing matrix multiplication.

[0140] In this embodiment, the first candidate data submatrix that the first processing unit 2702 wants to copy from the other N-1 computing devices can be all or part of the first data submatrix corresponding to the other N-1 computing devices, or it can be all or part of the second data submatrix corresponding to the other N-1 computing devices.

[0141] When the first processing unit 2702 copies the first candidate data sub-matrix in the other N-1 computing devices, it can do so by: constructing a set of partitioning methods based on the first partitioning method, the second partitioning method, and the third partitioning method corresponding to the output data matrix; determining the first candidate data sub-matrix based on the constructed set of partitioning methods; and copying the first candidate data sub-matrix in the other N-1 computing devices.

[0142] The first processing unit 2702 copies the first candidate data submatrix of the other N-1 computing devices by accessing the memory of the other N-1 computing devices.

[0143] In this embodiment, the output data matrix is ​​the target processing result between the first data matrix and the second data matrix; the third segmentation method can be row segmentation (i.e., segmenting the output data matrix in the row direction) or column segmentation (i.e., segmenting the output data matrix in the column direction).

[0144] In this embodiment, different sets of segmentation methods correspond to different types of first candidate data submatrices. Therefore, based on the constructed set of segmentation methods, the first computing device determines what type of first candidate data submatrices need to be copied from the other N-1 computing devices.

[0145] In this embodiment, the first processing unit 2702 obtains the first candidate data submatrix required for matrix multiplication calculation from the other N-1 computing devices by copying, avoiding the steps of sending and receiving submatrixes between computing devices. This reduces the time required for the first computing device, such as GPU, NPU, GPU-like or XPU, to obtain the first candidate data submatrix from the other N-1 computing devices, thereby improving the efficiency of subsequent matrix multiplication calculation based on the first candidate data submatrix.

[0146] The first processing unit 2702 can store the copied first candidate data sub-matrix corresponding to different other computing devices into the memory of the first computing device for subsequent processing.

[0147] When the first processing unit 2702 copies the first candidate data sub-matrix in the other N-1 computing devices, it can copy the first candidate data sub-matrix in the other N-1 computing devices multiple times according to the second preset block size, that is, copying a matrix block corresponding to the second preset block size from the other N-1 computing devices each time; wherein, the second preset block size in this embodiment can be the same as or different from the first preset block size, and the second preset block size can be set according to actual needs.

[0148] In other words, the first processing unit 2702 can copy the first candidate data sub-matrix in the other N-1 computing devices multiple times in smaller blocks. After the processing between the target first data sub-matrix and the target second data sub-matrix is ​​completed, the first computing device can perform matrix multiplication calculations more quickly based on the copied first candidate data sub-matrix (or the matrix blocks corresponding to the first candidate data sub-matrix). This can further improve the overlap efficiency between the computing communication of the first computing devices such as GPU, NPU, GPU-like or XPU.

[0149] When the first processing unit 2702 copies the first candidate data submatrix in the other N-1 computing devices, it can do so by: obtaining a preset cyclic order among the N computing devices; and copying the first candidate data submatrix in the computing devices located after the first computing device in sequence according to the preset cyclic order.

[0150] In other words, the first processing unit 2702 copies the first candidate data submatrix in the computing devices located after it in a preset cyclic order among the N computing devices, thereby ensuring that the data copying process is carried out in an orderly manner.

[0151] The first processing unit 2702 sequentially copies the first candidate data submatrix in the computing devices located after the first computing device. After completing the copying of the candidate data submatrix in the current computing device, it copies the candidate data submatrix in the next computing device, and so on.

[0152] In this embodiment, after the first processing unit 2702 completes its execution, the second processing unit 2703 processes the copied first candidate data submatrix and its corresponding target data submatrix in response to the first processing result obtained between the target first data submatrix and the target second data submatrix.

[0153] In this embodiment, if the first candidate data submatrix is ​​a first data submatrix in the other N-1 computing devices, then the target data submatrix corresponding to the first candidate data submatrix is ​​the target second data submatrix in the first computing device; if the first candidate data submatrix is ​​a second data submatrix in the other N-1 computing devices, then the target data submatrix corresponding to the first candidate data submatrix is ​​the target first data submatrix in the first computing device.

[0154] After determining that the matrix multiplication calculation between the target first data submatrix and the target second data submatrix has been completed, the second processing unit 2703 can immediately perform matrix multiplication calculation on the first candidate data submatrix copied from the other N-1 computing devices and its corresponding target data submatrix.

[0155] It is understood that the second processing unit 2703 may also perform the following: obtain the copy status of the first candidate data submatrix; in response to determining that the copy status is incomplete, while processing the copied first candidate data submatrix and its corresponding target data submatrix, continue to copy the remaining first candidate data submatrix in the other N-1 computing devices.

[0156] In this embodiment, after the first computing device completes the execution of the second processing unit 2703, the third processing unit 2704 responds to the second processing result between the first candidate data submatrix and its corresponding target data submatrix. Based on the first processing result and the candidate processing result, the target processing result of the first computing device is obtained.

[0157] In this embodiment, the target processing result of the first computing device is concatenated with the target processing results of the other N-1 computing devices to obtain the output data matrix. This output data matrix is ​​the target processing result between the first data matrix and the second data matrix.

[0158] The second processing result obtained by the third processing unit 2704 is the processing result between the complete first candidate data submatrix and its corresponding target data submatrix.

[0159] After obtaining the second processing result between the first candidate data submatrix and its corresponding target data submatrix, the third processing unit 2704 can close the matrix multiplication calculation process and then obtain the target processing result corresponding to the first computing device based on the first processing result and the second processing result.

[0160] In some special cases, such as when the first candidate data submatrix copied by the first computing device is the first data submatrix in the other N-1 computing devices, the third processing unit 2704 also obtains the target processing result corresponding to the first computing device based on the first calculation result and the second calculation result copied from the other N-1 computing devices.

[0161] In other words, the third processing unit 2704 can obtain the target processing result based on the first calculation result and the second calculation result calculated by itself, or it can obtain the target processing result based on the first calculation result calculated by itself and the second calculation result calculated by other computing devices, which can further improve the accuracy of the obtained target processing result.

[0162] When the third processing unit 2704 obtains the target processing result of the first computing device based on the first processing result and the second processing result, the implementation method may be as follows: construct a segmentation method set based on the first segmentation method, the second segmentation method, and the third segmentation method corresponding to the output data matrix; determine the second candidate data sub-matrix based on the segmentation method set; copy the second candidate data sub-matrix in the other N-1 computing devices; and obtain the target processing result of the first computing device based on the first processing result, the second processing result, and the second candidate data sub-matrix.

[0163] In other words, the third processing unit 2704 can also determine the second candidate data submatrix to be copied from the other N-1 computing devices based on the set of splitting methods constructed by the splitting method, and then obtain the corresponding target processing result based on the first processing result, the second processing result and the copied second candidate data submatrix obtained by itself.

[0164] In this embodiment, different sets of segmentation methods correspond to different second candidate data sub-matrices; wherein, the second candidate data sub-matrices are the second calculation results obtained by the other N-1 computing devices through matrix multiplication.

[0165] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0166] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0167] like Figure 28The diagram shown is a block diagram of an electronic device for a parallel processing method according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0168] like Figure 28 As shown, device 2800 includes a computing unit 2801, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 2802 or a computer program loaded from storage unit 2808 into random access memory (RAM) 2803. RAM 2803 may also store various programs and data required for the operation of device 2800. The computing unit 2801, ROM 2802, and RAM 2803 are interconnected via bus 2804. Input / output (I / O) interface 2805 is also connected to bus 2804.

[0169] Multiple components in device 2800 are connected to I / O interface 2805, including: input unit 2806, such as keyboard, mouse, etc.; output unit 2807, such as various types of displays, speakers, etc.; storage unit 2808, such as disk, optical disk, etc.; and communication unit 2809, such as network card, modem, wireless transceiver, etc. Communication unit 2809 allows device 2800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0170] Computing unit 2801 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of computing unit 2801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Computing unit 2801 performs the various methods and processes described above, such as parallel processing methods for models. For example, in some embodiments, parallel processing methods for models may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 2808.

[0171] In some embodiments, part or all of the computer program may be loaded and / or installed on the device 2800 via ROM 2802 and / or communication unit 2809. When the computer program is loaded into RAM 2803 and executed by computing unit 2801, one or more steps of the parallel processing method of the model described above may be performed. Alternatively, in other embodiments, computing unit 2801 may be configured to execute the parallel processing method of the model by any other suitable means (e.g., by means of firmware).

[0172] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0173] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable parallel processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0174] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0175] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for showing information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0176] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0177] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the management difficulties and weak business scalability inherent in traditional physical hosts and VPS (Virtual Private Server) services. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0178] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0179] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A parallel processing method for a model, applied to the first computing device among N computing devices, comprising: Obtain the target first data submatrix from N first data submatrices, and the target second data submatrix from N second data submatrices; wherein, the N first data submatrices are obtained by partitioning the first data matrix according to a first partitioning method, and the N second data submatrices are obtained by partitioning the second data matrix according to a second partitioning method; the first partitioning method and the second partitioning method are valid partitioning methods that support matrix multiplication of the partitioned data submatrices; and N is a positive integer greater than or equal to 2. Initiate the matrix multiplication calculation process, while processing the target first data submatrix and the target second data submatrix, and simultaneously copy the first candidate data submatrix from the other N-1 computing devices; In response to obtaining a first processing result between the target first data submatrix and the target second data submatrix, the copied first candidate data submatrix and its corresponding target data submatrix are processed; In response to obtaining a second processing result between the first candidate data submatrix and its corresponding target data submatrix, a target processing result of the first computing device is obtained based on the first processing result and the second processing result; wherein, the target processing result of the first computing device is used to concatenate with the N-1 target processing results of the other N-1 computing devices to obtain a target processing result between the first data matrix and the second data matrix.

2. The method according to claim 1, wherein, The processing of the first target data submatrix and the second target data submatrix includes: According to the first preset block size, the target first data submatrix is ​​divided into multiple target first matrix blocks, and the target second data submatrix is ​​divided into multiple target second matrix blocks; Based on the plurality of target first matrix blocks and the plurality of target second matrix blocks, the processing result between the target first data submatrix and the target second data submatrix is ​​obtained.

3. The method according to claim 1, wherein, The copying of the first candidate data submatrix from the other N-1 computing devices includes: Based on the first segmentation method, the second segmentation method, and the third segmentation method corresponding to the output data matrix, construct a segmentation method set; The first candidate data submatrix is ​​determined based on the set of segmentation methods; The first candidate data submatrix is ​​copied from the other N-1 computing devices.

4. The method according to claim 3, wherein, The copying of the first candidate data submatrix in the other N-1 computing devices includes: Based on the second preset block size, the first candidate data submatrix in the other N-1 computing devices is copied multiple times.

5. The method according to claim 4, wherein, The processing of the copied first candidate data submatrix and its corresponding target data submatrix includes: Obtain the copy status of the first candidate data sub-matrix; In response to determining that the copy state is incomplete, while processing the already copied first candidate data submatrix and its corresponding target data submatrix, the remaining first candidate data submatrix in the other N-1 computing devices are copied.

6. The method according to claim 1, wherein, The step of obtaining the target processing result of the first computing device based on the first processing result and the second processing result includes: Based on the first segmentation method, the second segmentation method, and the third segmentation method corresponding to the output data matrix, construct a segmentation method set; Based on the set of segmentation methods, determine the second candidate data submatrix; Copy the second candidate data submatrix from the other N-1 computing devices; Based on the first processing result, the second processing result, and the second candidate data submatrix, the target processing result of the first computing device is obtained.

7. The method according to claim 1, wherein, The copying of the first candidate data submatrix from the other N-1 computing devices includes: Obtain the preset loop order among the N computing devices; According to the preset cyclic order, the first candidate data submatrix in the computing devices located after the first computing device is copied sequentially.

8. A parallel processing device for a model, wherein the first computing device among N computing devices comprises: The acquisition unit is used to acquire a target first data submatrix from N first data submatrices and a target second data submatrix from N second data submatrices; wherein the N first data submatrices are obtained by partitioning the first data matrix according to a first partitioning method, and the N second data submatrices are obtained by partitioning the second data matrix according to a second partitioning method; the first partitioning method and the second partitioning method are valid partitioning methods that support matrix multiplication of the partitioned data submatrices; and N is a positive integer greater than or equal to 2. The first processing unit is used to initiate the matrix multiplication calculation process, and while processing the target first data submatrix and the target second data submatrix, it copies the first candidate data submatrix in the other N-1 computing devices. The second processing unit is configured to process the copied first candidate data submatrix and its corresponding target data submatrix in response to obtaining a first processing result between the target first data submatrix and the target second data submatrix. The third processing unit is configured to, in response to obtaining a second processing result between the first candidate data submatrix and its corresponding target data submatrix, obtain a target processing result of the first computing device based on the first processing result and the second processing result; wherein, the target processing result of the first computing device is used to concatenate with the N-1 target processing results of the other N-1 computing devices to obtain a target processing result between the first data matrix and the second data matrix.

9. The apparatus according to claim 8, wherein, When processing the target first data submatrix and the target second data submatrix, the first processing unit specifically performs the following: According to the first preset block size, the target first data submatrix is ​​divided into multiple target first matrix blocks, and the target second data submatrix is ​​divided into multiple target second matrix blocks; Based on the plurality of target first matrix blocks and the plurality of target second matrix blocks, the processing result between the target first data submatrix and the target second data submatrix is ​​obtained.

10. The apparatus according to claim 8, wherein, When the first processing unit copies the first candidate data submatrix from the other N-1 computing devices, it specifically performs the following: Based on the first segmentation method, the second segmentation method, and the third segmentation method corresponding to the output data matrix, construct a segmentation method set; The first candidate data submatrix is ​​determined based on the set of segmentation methods; The first candidate data submatrix is ​​copied from the other N-1 computing devices.

11. The apparatus according to claim 10, wherein, When the first processing unit copies the first candidate data submatrix from the other N-1 computing devices, it specifically performs the following: Based on the second preset block size, the first candidate data submatrix in the other N-1 computing devices is copied multiple times.

12. The apparatus according to claim 11, wherein, When the second processing unit processes the copied first candidate data submatrix and its corresponding target data submatrix, it specifically performs the following: Obtain the copy status of the first candidate data sub-matrix; In response to determining that the copy state is incomplete, while processing the already copied first candidate data submatrix and its corresponding target data submatrix, the remaining first candidate data submatrix in the other N-1 computing devices are copied.

13. The apparatus according to claim 8, wherein, When the third processing unit obtains the target processing result of the first computing device based on the first processing result and the second processing result, it specifically performs the following: Based on the first segmentation method, the second segmentation method, and the third segmentation method corresponding to the output data matrix, construct a segmentation method set; Based on the set of segmentation methods, determine the second candidate data submatrix; Copy the second candidate data submatrix from the other N-1 computing devices; Based on the first processing result, the second processing result, and the second candidate data submatrix, the target processing result of the first computing device is obtained.

14. The apparatus according to claim 8, wherein, When the first processing unit copies the first candidate data submatrix from the other N-1 computing devices, it specifically performs the following: Obtain the preset loop order among the N computing devices; According to the preset cyclic order, the first candidate data submatrix in the computing devices located after the first computing device is copied sequentially.

15. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

17. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-7.

Images