Operator operation method and device, electronic equipment and storage medium

Abstract

The application provides an operator operation method and device, electronic equipment and storage medium. The method comprises the following steps: splicing a plurality of first tensors to be operated to obtain a second tensor, and determining index information of each first tensor in the second tensor; based on the index information of each first tensor, performing slice processing on the second tensor to obtain a slice tensor corresponding to each first tensor, and specifying distribution information and a calculation flow of each slice tensor, the distribution information being used to indicate the storage distribution of the slice tensor on a plurality of dies, and the calculation flow being used to indicate a target die where data participating in the operator operation in the slice tensor; based on the distribution information of each slice tensor, reading each slice tensor from the plurality of dies, and based on the calculation flow of each slice tensor, performing operator operation on the data on the target die in each read slice tensor. The method, device, equipment and medium provided by the application reduce the redundant occupation of the display memory and improve the operator operation efficiency.

Description

Technical Field

[0001] This invention relates to the field of high-performance computing technology, and in particular to an operator operation method, apparatus, electronic device, and storage medium. Background Technology

[0002] Currently, when performing GroupGemm (Group General Matrix Multiplication) operations of different shapes based on a multi-die chip architecture, since the operation units of multiple dies need to maintain a unified computational behavior and adapt to the same operation interface, it is necessary to first pad the group tensors of different shapes with zeros to fill them to the same length.

[0003] Although zero-padding can achieve unified calculation, it introduces a large amount of invalid data, resulting in additional memory usage and computational redundancy, which directly affects the computational efficiency of GroupGemm. Summary of the Invention

[0004] This invention provides an operator operation method, apparatus, electronic device, and storage medium to address the shortcomings of GroupGemm operation efficiency caused by the introduction of a large amount of invalid data in related technologies.

[0005] This invention provides an operator operation method, comprising: Multiple first tensors from the reception operation are concatenated to obtain a second tensor, and the index information of each first tensor in the second tensor is determined; Based on the index information of each first tensor, the second tensor is sliced ​​to obtain the slice tensor corresponding to each first tensor. The distribution information and computation flow of each slice tensor are specified. The distribution information is used to indicate the storage distribution of the slice tensor on multiple bare dies. The computation flow is used to indicate the target bare die where the data participating in the operator operation in the slice tensor is located. The target bare die belongs to the multiple bare dies. Based on the distribution information of each slice tensor, each slice tensor is read from the multiple bare wafers, and based on the computational flow of each slice tensor, operator operations are performed on the data on the target bare wafer in each read slice tensor.

[0006] According to an operator operation method provided by the present invention, the plurality of first tensors in the concatenation operation include: Based on the arrangement strategy, multiple first tensors for reception calculation are assembled; The arrangement strategy is used to indicate the distribution method and arrangement order of the plurality of first tensors on the plurality of bare wafers.

[0007] According to an operator operation method provided by the present invention, determining the index information of each first tensor in the second tensor includes: Determine the shape, offset address, and offset length of each first tensor in the second tensor, as the index information of each first tensor in the second tensor.

[0008] According to an operator operation method provided by the present invention, the step of slicing the second tensor based on the index information of each first tensor includes: Based on the shape, offset address, and offset length of each first tensor in the second tensor, data that matches the shape, offset address, and offset length is extracted from the second tensor and used as a slice tensor corresponding to each first tensor.

[0009] According to an operator operation method provided by the present invention, the distribution information includes at least one of the memory distribution format and memory management mode of the slice tensor.

[0010] According to the present invention, an operator operation method is provided, wherein the operator is a grouped general matrix multiplication.

[0011] The present invention also provides an operator operation device, comprising: The splicing unit is used to splice multiple first tensors for receiving operations to obtain a second tensor, and to determine the index information of each first tensor in the second tensor; A slicing unit is used to slice the second tensor based on the index information of each first tensor to obtain a slice tensor corresponding to each first tensor, and to specify the distribution information and computation flow of each slice tensor. The distribution information is used to indicate the storage distribution of the slice tensor on multiple bare dies, and the computation flow is used to indicate the target bare die where the data participating in the operator operation in the slice tensor is located. The target bare die belongs to the multiple bare dies. The computation unit reads each slice tensor from the multiple bare wafers based on the distribution information of each slice tensor, and performs operator operations on the data on the target bare wafer in each read slice tensor based on the computation flow of each slice tensor.

[0012] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the operator operation method described above.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the operator operation method as described above.

[0014] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the operator operation method as described above.

[0015] The operator operation method, apparatus, electronic device, and storage medium provided by this invention, by concatenating multiple first tensors to be operated on into a second tensor, and after slicing, independently specifying distribution information indicating the storage distribution and computation flow indicating the wafer to which the operation belongs for each slice tensor, enables flexible reading and operation of data on different wafers in a multi-wafer architecture based on the specified distribution information and computation flow. This eliminates the need for extensive zero-padding while still meeting the limitations of unified computation in a multi-wafer architecture, effectively reducing redundant memory usage caused by repeated data storage on multiple wafers and excessive zero-padding. Furthermore, allocating different slice tensors to the computation flows of different target wafers for independent operation avoids forced waiting between computation tasks of different shapes, greatly improving the parallel operation efficiency of operators and the utilization rate of memory resources in a multi-wafer architecture. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of tensor distribution in the dual-die amortized computation mode of related technologies.

[0018] Figure 2 This is a schematic diagram of tensor distribution in the dual-die tensor storage mode of related technologies.

[0019] Figure 3 This is a flowchart illustrating the operator operation method provided by the present invention.

[0020] Figure 4 This is a schematic diagram of tensor distribution in a dual-die scenario provided by the present invention.

[0021] Figure 5 This is a schematic diagram of the operator output distribution in a dual-die scenario provided by the present invention.

[0022] Figure 6 This is a schematic diagram of the slicing process of the second tensor provided by the present invention.

[0023] Figure 7 This is a schematic diagram of the operator operation device provided by the present invention.

[0024] Figure 8This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0026] All actions involving the acquisition of signal information or data in this invention are carried out in accordance with the relevant data protection laws and policies of the country where the device is located, and with the authorization granted by the owner of the device.

[0027] In the context of artificial intelligence computing and high-performance computing technologies, operators such as GroupGEMM are widely used in deep learning model training, inference, and scientific computing. Here, GroupGEMM is a computing model that packages multiple independent GEMM subtasks for parallel execution.

[0028] To meet the ever-increasing demand for computing power, multi-die chips, with their advanced chiplet packaging technology offering enhanced computing power, have gradually become one of the mainstream hardware architectures supporting GroupGEMM-type operators. Here, a multi-die chip can be understood as integrating multiple independent dies within a single AI chip, with each die possessing its own dedicated computing unit, memory controller, and on-chip cache. A typical multi-die chip can be a dual-die chip.

[0029] It's important to note that when performing GroupGEMM operations in a multi-die chip architecture, there's a crucial pre-processing step: zero-padding. Taking a dual-die chip architecture as an example, since the computational units on both dies need to maintain consistent computational behavior and adapt to the same computational interface, zero-padding must first be applied to the grouping matrices of different shapes, filling them to the same length. Only after this can the computational tasks be distributed across the two dies for parallel computation.

[0030] While the zero-padding process described above can satisfy the unified computation behavior of multi-die architecture, it will increase the memory usage space. In addition, the zero-padding and subsequent de-padding operations will also increase a lot of computational redundancy.

[0031] In related technologies, there exists an operator operation mode that features amortized computation across two dies and unified memory management. Specifically, in a dual-die architecture, read access operations on tensors and memory must ensure that the size read from both dies remains consistent. This mode employs a shared memory design, distributing computational tasks of different shapes evenly across the two dies in a fixed proportion, and aligning the tensor shapes on both dies by padding with zeros. For example, Figure 1 This is a schematic diagram of tensor distribution in the dual-die amortized computation mode of related technologies, such as... Figure 1 As shown, tensor A and tensor C are assigned to bare disk 0, and tensor B and tensor D are assigned to bare disk 1. To align tensor A and tensor B, zero-padding is applied to tensor B, thus aligning tensor B and padding0 as a whole with tensor A; similarly, to align tensor C and tensor D, zero-padding is applied to tensor C, thus aligning tensor C and padding1 as a whole with tensor D.

[0032] While the computational power allocation logic is clear in this mode, in practical applications, the memory requirements for tensor operations of different shapes vary greatly. If very small tensors are grouped with very large tensors, zero-padding will introduce a large amount of invalid data. Furthermore, since unified memory management cannot dynamically allocate memory resources based on tensor shape, this leads to severe memory redundancy and is prone to memory overflow or low memory utilization. In addition, in the case of dual-die parallel processing, if there is a very small tensor and a very large tensor in the same group, subsequent calculations must wait until the calculation of the large tensor is completed. This waiting mechanism causes significant computational redundancy, increases computational latency, and leads to a significant decrease in overall computational efficiency.

[0033] Furthermore, some related technologies employ operator operation modes that directly store the same tensor on two separate dies. In this mode, computation can be performed using only the tensor on one die, while tensors stored on the other die occupy video memory and do not participate in the computation. For example, Figure 2 This is a schematic diagram of tensor distribution in the dual-die tensor storage mode of related technologies, such as... Figure 2 As shown, each tensor is stored on bare die 0 and bare die 1.

[0034] Although this mode does not require zero-padding, one of the dies not only does not participate in the calculation, but also continuously occupies video memory resources, completely deviating from the original intention of dual-die parallel computing and further exacerbating the waste of computing resources.

[0035] To address the aforementioned problems, this invention provides an operator operation method. Here, the operator can be an operator used to process multiple dynamically shaped matrices, or it can be understood as a group of operators that process multiple dynamically shaped matrices respectively. For example, it can be GroupGEMM, or it can be other types of operators. This invention does not specifically limit the specific type of operator.

[0036] Figure 3 This is a flowchart illustrating the operator operation method provided by the present invention, as shown below. Figure 3 As shown, the method includes: Step 310: Concatenate the multiple first tensors of the reception operation to obtain the second tensor, and determine the index information of each first tensor in the second tensor.

[0037] Specifically, the first tensor is the input data participating in the operator operation. Since the operator in this embodiment of the invention can perform multiple independent operation tasks, multiple first tensors can be obtained here, and the shapes of the multiple first tensors may be different.

[0038] After obtaining multiple first tensors, all first tensors can be concatenated to form a single tensor, which is denoted as the second tensor in this embodiment of the invention. The second tensor can be considered as a global tensor set containing all the first tensors to be computed.

[0039] For example, suppose the first tensor to be operated on includes tensor A, tensor B, tensor C, and tensor D. Tensors A and C can be concatenated along the row dimension, tensors B and D can be concatenated along the row dimension, and the two concatenated tensors along the row dimension can be concatenated along the column dimension and padded with zeros to align them, thus obtaining the second tensor. It is understandable that concatenating multiple first tensors to be operated on can be done along both the row and column dimensions. For example, different first tensors assigned to the same raw chip can be concatenated along the row dimension, and different first tensors assigned to different raw chips can be concatenated along the column dimension, thus obtaining the second tensor.

[0040] After concatenating the first tensors to obtain the second tensor, it is also necessary to determine the index information of each first tensor within the second tensor. Here, for each first tensor, the index information is used to characterize the specific position of the first tensor within the concatenated second tensor, enabling the positioning of the first tensor within the second tensor. In some embodiments, the index information is used to characterize the specific position of the first tensor along the row dimension within the concatenated second tensor.

[0041] Step 320: Based on the index information of each first tensor, slice the second tensor to obtain slice tensors corresponding to each first tensor, and specify the distribution information and computation flow of each slice tensor. The distribution information is used to indicate the storage distribution of the slice tensor on multiple bare chips, and the computation flow is used to indicate the target bare chip where the data participating in the operator operation in the slice tensor is located. The target bare chip belongs to the multiple bare chips.

[0042] Specifically, after obtaining the second tensor and the index information of each first tensor within the second tensor, slicing can be performed on the second tensor based on the index information corresponding to each first tensor. Here, slicing is the process of re-dividing large blocks of contiguous memory data into logically independent tensor objects based on the index information. Through slicing, independent tensors for subsequent batch computations can be extracted from the second tensor. Each extracted tensor is considered a slice tensor, and each slice tensor corresponds one-to-one with a first tensor. Each slice tensor corresponds in data content to a first tensor before concatenation. Furthermore, for any first tensor, the slice tensor corresponding to it necessarily contains the first tensor itself, and the first tensor contained within the slice tensor is the data in the slice tensor that actually participates in the subsequent operator operations.

[0043] For example, based on the index information of any first tensor, slicing the second tensor can extract slice tensors that match the index information from the second tensor. Since the index information reflects the specific position of the first tensor in the row dimension within the second tensor, and multiple first tensors are concatenated in the row and column dimensions to obtain the second tensor, the extracted slice tensors not only contain the first tensor itself, but also contain part or all of the other first tensors concatenated with the first tensor in the column dimension. The first tensors contained in the slice tensors, as well as part or all of the other first tensors concatenated with the first tensor in the column dimension, constitute all the tensors that need to be read synchronously in order to read the first tensor in a multi-die architecture.

[0044] In order to make full use of the storage and computing resources of the multi-die architecture, in this embodiment of the invention, distribution information and computing stream are specified for each slice tensor.

[0045] The distribution information indicates the storage distribution of the slice tensor across multiple wafers. Specifically, in the memory management mechanism of a multi-wafer architecture, slice tensors can choose different storage distribution methods. For example, a slice tensor can be divided into multiple parts and stored on different wafers, or the complete slice tensor can be stored as a mirror image on each wafer. The distribution information thus clarifies how the slice tensor is arranged and occupied in the memory of multiple wafers.

[0046] A computational flow is an identifier that indicates the hardware execution queue and computational affiliation. In this embodiment of the invention, the computational flow is specifically used to indicate which of the multiple raw chips the valid data in the slice tensor that actually participates in the operator operation is located on. Here, the raw chip where the data in the slice tensor that participates in the operator operation is located is denoted as the target raw chip.

[0047] Understandably, in a multi-die architecture, assuming the data of a slice tensor is distributed across multiple dies, specifying the computation flow can force only the computation units on a specific target die to perform operator operations on the data of that slice tensor, while the computation units on other dies do not need to perform redundant parallel computations. For example, if the slice tensor contains not only the first tensor itself, but also some or all of the other first tensors concatenated with it in the column dimension, the computation units on the target die storing the first tensor can be specified to perform operator operations on the first tensor in the slice tensor, while the computation units on other dies do not need to perform redundant parallel computations on some or all of the other first tensors concatenated with the first tensor in the slice tensor in the column dimension.

[0048] Different slice tensors can be assigned to different target dies, thus enabling completely independent parallel computation in different computational flows.

[0049] Step 330: Based on the distribution information of each slice tensor, read each slice tensor from the multiple bare wafers, and perform operator operations on the data on the target bare wafer in each read slice tensor based on the computation flow of each slice tensor.

[0050] Specifically, during the actual execution phase of the operator, the corresponding slice tensor can be read from the video memory of multiple bare chips through addressing based on the distribution information of each slice tensor. Here, the application of distribution information ensures that the read operations of the computing units on each bare chip in the multi-battery architecture remain consistent.

[0051] Furthermore, based on the computation flow of each slice tensor, the target wafer to which the computation task of each slice tensor belongs can be determined. Subsequently, the operation logic can be activated only for the data of the slice tensor on the target wafer to perform operator operations, and the output results can be stored in the corresponding storage location of the target wafer.

[0052] For example, Figure 4 This is a schematic diagram of tensor distribution in a dual-die scenario provided by the present invention, as shown below. Figure 4 As shown, there are four first tensors: tensor A, tensor B, tensor C, and tensor D. Tensor A is... Figure 4 The area represented in pink is tensor B in... Figure 4 The purple area is represented in the middle, and the tensor C is in... Figure 4The blue area is represented in the middle, and the tensor D is in... Figure 4 The orange area represents the data. Additionally, the gray area represents data padded with zeros.

[0053] After concatenating the four first tensors to obtain the second tensor, the slice tensors of tensor B obtained by slicing the second tensor are distributed on bare slice 0 and bare slice 1, which is reflected in... Figure 4 The area marked in green represents the slice tensor of tensor B, which specifically includes a portion of tensor A distributed on die 0 and the entirety of tensor B distributed on die 1. Furthermore, the computation flow of the slice tensor of tensor B is specified to die 1. Therefore, when performing operator operations on tensor B, tensors of the same size can be read simultaneously from both die 0 and die 1: a portion of tensor A corresponding to the green area is read from die 0, and the entirety of tensor B is read from die 1. Based on this, operator operations can be performed only on tensor B read from die 1, excluding the portion of tensor A read from die 0, thus enabling independent operator operations on tensor B.

[0054] In the method provided in this embodiment of the invention, by concatenating multiple first tensors to be computed into a second tensor, and after slicing, independently specifying the distribution information indicating the storage distribution and the computation flow indicating the computing disk to which the operation belongs for each slice tensor, flexible reading and operation of data on different disks can be achieved under a multi-disk architecture based on the specified distribution information and computation flow. Thus, the limitation of unified computation under a multi-disk architecture can be met without extensive zero padding, effectively reducing the redundant memory occupation caused by repeated storage of data on multiple disks and extensive zero padding. Furthermore, allocating different slice tensors to the computation flow of different target disks for independent computation can avoid forced waiting between computation tasks of different shapes, greatly improving the parallel computation efficiency of operators and the utilization rate of memory resources under a multi-disk architecture.

[0055] Based on the above embodiments, in step 310, the plurality of first tensors in the concatenation operation include: Based on the arrangement strategy, multiple first tensors for reception calculation are assembled; The arrangement strategy is used to indicate the distribution method and arrangement order of the plurality of first tensors on the plurality of bare wafers.

[0056] Specifically, for multiple first tensors to be computed, they can be arranged and concatenated based on an arrangement strategy. Here, the arrangement strategy refers to a pre-defined strategy that guides how the first tensors used as operator inputs are grouped, sorted, and their storage space allocated. In particular, the arrangement strategy can indicate the distribution method and arrangement order of multiple first tensors on multiple bare chips.

[0057] The allocation method refers to the way hardware resources are divided among multiple dies for multiple first tensors. The arrangement strategy can plan which first tensor computation tasks and data will be assigned to which die, thus providing conditions for the parallelization of subsequent computation flows. For example, in the case of four first tensors (Tensor A, Tensor B, Tensor C, and Tensor D), the arrangement strategy can be pre-set to assign Tensor A and Tensor C to die 0, and Tensor B and Tensor D to die 1.

[0058] The arrangement order refers to the order in which multiple first tensors are combined into a second tensor. It can also be understood as the order in which each computational stream performs operator operations on each first tensor when multiple computational streams implement operator operations in parallel. For example, given four first tensors—A, B, C, and D—the arrangement strategy can specify that they are arranged sequentially in contiguous memory space in the order A, B, C, and D, or arranged in parallel order of A and C versus B and D, ultimately combining to form a complete second tensor.

[0059] In some embodiments, multiple first tensors to be run can be grouped according to their respective bare dies based on an arrangement strategy, and the first tensors in each group can be sorted. In this way, multiple first tensors to be run can be seamlessly spliced ​​in memory according to the arrangement order to obtain a whole second tensor.

[0060] In this embodiment of the invention, multiple first tensors are concatenated based on an arrangement strategy, effectively avoiding data addressing chaos and uneven distribution of resources across the die caused by disordered concatenation, thus further improving the scheduling efficiency of operator operations. Furthermore, concatenating multiple first tensors based on an arrangement strategy avoids the need for zero-padding for each group of first tensors with different shapes, effectively reducing the scale of irrelevant data introduced by zero-padding and the resulting redundant memory usage, thereby improving the efficiency of operator operations.

[0061] Based on any of the above embodiments, in step 310, determining the index information of each first tensor in the second tensor includes: Determine the shape, offset address, and offset length of each first tensor in the second tensor, as the index information of each first tensor in the second tensor.

[0062] Specifically, in order to accurately locate the first tensor in the second tensor obtained by splicing, the embodiments of the present invention construct the index information of the first tensor by recording the shape, offset address, and offset length of the first tensor in the second tensor.

[0063] For any first tensor, the shape is used to reflect the dimensions and size of each tensor before splicing, such as the number of rows and columns of a matrix; the offset address is used to indicate the starting storage location of the data of the first tensor in the contiguous video memory space where the second tensor is located; the offset length is used to indicate the continuous length of the storage space occupied by the first tensor starting from the offset address, which is usually determined by the shape of the tensor and the bit width of the data type of a single element.

[0064] It is understandable that when splicing multiple first tensors, the shape and size of each first tensor and the storage space information required by it in the chip storage can be obtained and recorded simultaneously, that is, the index information of each first tensor in the second tensor can be obtained.

[0065] For example, suppose there are four first tensors: tensor A, tensor B, tensor C, and tensor D. Concatenating these four first tensors generates a second tensor, total_input. During this process, the shape set of each first tensor can be obtained, for example, [SizeA, SizeB, SizeC, SizeD], which is collectively denoted as tensor shape information region_tensor_infos. Simultaneously, the offset address region_offset of each first tensor in the second tensor's storage space is recorded; and the length of the valid data segment corresponding to each offset address is recorded, denoted as the offset length region_len. These three types of information constitute the index information of each first tensor in the second tensor.

[0066] In this embodiment of the invention, by determining the shape, offset address, and offset length of each first tensor in the second tensor as index information, it is possible to provide a basis for data addressing for subsequent slicing processing of the second tensor after multiple first tensors of different shapes are spliced ​​and stored.

[0067] Based on any of the above embodiments, step 320, which involves slicing the second tensor based on the index information of each first tensor, includes: Based on the shape, offset address, and offset length of each first tensor in the second tensor, data that matches the shape, offset address, and offset length is extracted from the second tensor and used as a slice tensor corresponding to each first tensor.

[0068] Specifically, after concatenating multiple first tensors into a second tensor, the slice tensor used by each first tensor to adapt to the unified computational constraints under a multi-die architecture can be obtained through slicing operations.

[0069] During slicing, the offset address can be used as a memory pointer to precisely lock the starting read point of the first tensor within the contiguous memory space of the second tensor. Then, the offset length is used as the data truncation boundary, and a continuous linear memory data segment is read from the starting point. Furthermore, this linear data segment is reshaped based on its shape. Thus, through the joint constraints of offset address, offset length, and shape, a valid data block matching the first tensor can be precisely extracted from the second tensor and used as the slice tensor corresponding to the first tensor.

[0070] In some embodiments, assuming the concatenated second tensor is `total_input`, the shape offset address and offset length in the index information of each first tensor are stored in `region_tensor_infos`, `region_offset`, and `region_len`, respectively. Furthermore, the list used to store the slice tensors is denoted as `mlp_inputs`. Based on this, the slicing of the second tensor can be implemented using the following code, and the distribution information and computation flow of each slice tensor obtained from the slices can be specified: mlp_inputs = list() for i in len(region_tensor_infos): mlp_inputs.append( torch.flatten_narrow(total_input, region_tensor_infos[i].size,region_offset[i], region_len[i]). view_as_usharp(LayoutInfo, dual_die_ep_expert_shape[i], sbp=total_input.sbp().id, stream=stream0)) In the code above, the torch.flatten_narrow interface is used to implement the slicing operation in this embodiment of the invention. Specifically, torch.flatten_narrow takes the second tensor total_input as the operation object, and based on the shape region_tensor_infos[i].size, offset address region_offset[i], and offset length region_len[i] of the i-th first tensor, it extracts the data content that matches the index information of the i-th first tensor from the second tensor, and uses it as the slice tensor of the i-th first tensor.

[0071] Subsequently, the `view_as_usharp` interface is called in the code above to specify the distribution information and computation flow under the multi-die architecture for the extracted slice tensors. Specifically, in `view_as_usharp`, `LayoutInfo` indicates the physical arrangement rules of the slice tensors in memory; `dual_die_ep_expert_shape[i]` represents the specific computation shape of the i-th slice tensor under the dual-die architecture; `sbp=total_input.sbp().id` is used to specify the SBP (Split, Broadcast, Partial-sum) mode of the i-th slice tensor; `stream=stream0` is used to specify the computation flow of the i-th slice tensor as `stream0`, which can also be `stream1` under the dual-die architecture. Thus, the slice tensors obtained by slicing and with specified distribution information and computation flow are appended to the `mlp_inputs` list for subsequent operator operations.

[0072] In this embodiment of the invention, based on three index dimensions—shape, offset address, and offset length—slice tensors are extracted from each first tensor in the second tensor, providing the data required for computation to adapt to the same computational constraints under a multi-die architecture.

[0073] Based on any of the above embodiments, the distribution information includes at least one of the video memory distribution format and video memory management mode of the slice tensor.

[0074] Specifically, the memory distribution format refers to the specific distribution format of the slice tensor data in the memory. In some embodiments, the memory distribution format can be Usharp information.

[0075] For example, the video memory distribution format may include the logical shape, physical shape, and video memory allocation information of the slice tensor. The logical shape refers to the original data shape of the slice tensor, independent of the underlying hardware device; the physical shape is the physical shape stored in the hardware after the logical shape of the slice tensor has been reshaped according to specific folding rules based on a specific programming model and the dimensional constraints of the underlying hardware; and the video memory allocation information is the information that allocates the corresponding video memory space based on the size of video memory required by the physical shape of the slice tensor. For example, it may include information indicating whether the video memory space belongs to a Uniform Memory Access (UMA) or Non-Uniform Memory Access (NUMA) type.

[0076] The memory management mode refers to the management mode used for memory allocation and retrieval of slice tensors in a multi-die chip architecture, such as the SBP mode.

[0077] For example, the memory management mode can be either SB (Split) or BB (Broadcast / Both). SB mode refers to dividing a complete slice tensor into multiple equal parts, storing them on different dies. For example, in a dual-die architecture, the slice tensor is split in half, with die 0 and die 1 each storing half of the slice tensor. BB mode means storing the same complete slice tensor on all dies.

[0078] In some embodiments, the memory distribution format and memory management mode of the slice tensor can be specified by running the code view_as_usharp(LayoutInfo, shape, sbp=total_input.sbp().id)).

[0079] In this embodiment of the invention, by using at least one of the memory distribution format and memory management mode, the physical storage state of each slice tensor in the multi-die architecture can be flexibly controlled at the underlying hardware level, thereby adapting to the characteristics of the multi-die architecture and providing a data reading foundation for the efficient, independent and parallel execution of operator operations on multiple dies.

[0080] Based on any of the above embodiments, the operator is a grouped general matrix multiplication.

[0081] Specifically, when performing Group-GEMM (Group General Matrix Multiplication), all independent matrices that need to participate in Group-GEMM are first obtained, resulting in multiple first tensors. These first tensors are then concatenated into a second tensor, and the index information of the first tensors within the second tensor is recorded. Next, based on the index information of each first tensor, slices are extracted from the second tensor, resulting in slice tensors that correspond one-to-one with the first tensors for batch computation. The distribution information and computation flow of these slice tensors are specified. Finally, based on the distribution information of the slice tensors, the slice tensors are read from the video memory of multiple bare chips, and according to the specified computation flow, independent matrix multiplication operations are performed on the data participating in the operator operations within the slice tensors on their respective independent target chips. This completes the Group-GEMM operation for multiple first tensors.

[0082] Furthermore, after obtaining the operation results for each slice tensor, the results can be stored in the storage location where the data in each slice tensor that participated in the operator operation resides. For example, Figure 5 This is a schematic diagram of the operator output distribution in a dual-die scenario provided by the present invention, and... Figure 4As can be seen from the comparison, the output result obtained by each first tensor after the operator operation is stored in the storage location of the first tensor.

[0083] In this embodiment of the invention, a general grouped matrix multiplication operation is implemented. In this process, there is no need to force zero-padding for alignment due to the inconsistent shapes of the first tensors. While reducing the redundant use of video memory, it enables matrix multiplication tasks of different shapes to be performed asynchronously in parallel in different computational flows, avoiding forced waiting during the operation process. This achieves parallel optimization that reduces video memory usage and improves the efficiency of grouped matrix multiplication operation.

[0084] Furthermore, the implementation logic of the above operator operation method can also be applied to the data processing of gradient tensors required in the backpropagation stage.

[0085] Based on any of the above embodiments, for the case where the operator is Group-GEMM and the dual-die architecture includes die 0 and die 1, it is assumed that there are four first tensors, namely tensor A, tensor B, tensor C and tensor D. The second tensor can be obtained by arranging and splicing them in parallel according to the arrangement strategy of tensor A, tensor C and tensor B, tensor D. Figure 6 This is a schematic diagram of the slicing process of the second tensor provided by the present invention, as shown below. Figure 6 As shown, the second tensor is obtained by splicing together tensor A, tensor B, tensor C and tensor D, as well as the gray zero-padding area.

[0086] After obtaining the second tensor, the second tensor can be sliced ​​based on the index information of tensors A, B, C, and D within the second tensor. This yields sliced ​​tensors corresponding to tensors A, B, C, and D, respectively. Figure 6 The tensors shown are slice tensor A, slice tensor B, slice tensor C, and slice tensor D. Specifically, the distribution information of slice tensor A indicates that slice tensor A includes tensor A stored on bare die 0, tensor B and a portion of tensor D stored on bare die 1; the distribution information of slice tensor B indicates that slice tensor B includes a portion of tensor A stored on bare die 0 and tensor B stored on bare die 1; the distribution information of slice tensor C indicates that slice tensor C includes tensor C stored on bare die 0 and a portion of tensor D stored on bare die 1; the distribution information of slice tensor D indicates that slice tensor D includes a portion of tensor A, tensor C and zero-padding region stored on bare die 0, and tensor D stored on bare die 1; furthermore, the computational flow of slice tensor A and slice tensor C is computational flow 0, thereby indicating that the data used for operator operations in slice tensor A and slice tensor C are all located on bare die 0; the computational flow of slice tensor B and slice tensor D is computational flow 1, thereby indicating that the data used for operator operations in slice tensor B and slice tensor D are all located on bare die 1.

[0087] After that, for slice tensor A, tensor A can be read from bare die 0 based on the distribution information of slice tensor A, and tensor B and part of tensor D can be read from bare die 1. Furthermore, the computational flow based on slice tensor A is computational flow 0, and GEMM operation can be performed only on tensor A read from bare die 0, and its output result can be stored in the storage location of tensor A in bare die 0. For slice tensor B, a portion of tensor A can be read from bare die 0 and tensor B can be read from bare die 1 based on the distribution information of slice tensor B. Furthermore, the computational flow based on slice tensor B is computational flow 1, so only tensor B read from bare die 1 can be used for GEMM operation, and its output result can be stored in the storage location of tensor B in bare die 1. Moreover, since slice tensors A and B belong to computational flow 0 and computational flow 1 respectively, the GEMM operations on tensors A and B can be performed in parallel without affecting each other.

[0088] For slice tensor C, tensor C can be read from bare die 0 and a portion of tensor D can be read from bare die 1 based on the distribution information of slice tensor C. Furthermore, since the computational flow based on slice tensor C is computational flow 0, GEMM operation can be performed only on tensor C read from bare die 0, and its output can be stored in the storage location of tensor C in bare die 0. Moreover, since slice tensors A and C both belong to computational flow 0, the GEMM operation of tensor C is performed after the GEMM operation on tensor A is completed. For slice tensor D, tensors A, C, and zero-padding regions can be read from bare die 0 based on the distribution information of slice tensor D, and tensor D can be read from bare die 1. Furthermore, the computational flow based on slice tensor D is computational flow 1, so only tensor D read from bare die 1 can be used for GEMM operation, and its output can be stored in the storage location of tensor D in bare die 1. Since slice tensors B and D both belong to computational flow 1, the GEMM operation of tensor D is executed after the GEMM operation of tensor B is completed. And since slice tensors C and D belong to computational flow 0 and computational flow 1 respectively, the GEMM operations of tensors C and D can be performed in parallel without affecting each other.

[0089] In this embodiment of the invention, while adhering to the SBP memory management and access mode under a multi-die architecture, the redundant memory usage caused by data duplication across multiple dies and excessive zero padding is effectively reduced. Furthermore, by specifying distribution information and computational flow for each slice tensor, flexible reading, writing, and computational operations on tensors on different dies are achieved, significantly improving computational efficiency.

[0090] The operator operation device provided by the present invention is described below. The operator operation device described below can be referred to in correspondence with the operator operation method described above.

[0091] Figure 7This is a schematic diagram of the operator operation device provided by the present invention, as shown below. Figure 7 As shown, the device includes: The splicing unit 710 is used to splice multiple first tensors for receiving operations to obtain a second tensor, and to determine the index information of each first tensor in the second tensor; The slicing unit 720 is used to slice the second tensor based on the index information of each first tensor to obtain the slice tensor corresponding to each first tensor, and to specify the distribution information and computation flow of each slice tensor. The distribution information is used to indicate the storage distribution of the slice tensor on multiple bare chips, and the computation flow is used to indicate the target bare chip where the data participating in the operator operation in the slice tensor is located. The target bare chip belongs to the multiple bare chips. The operation unit 730 reads each slice tensor from the multiple bare wafers based on the distribution information of each slice tensor, and performs operator operations on the data on the target bare wafer in each read slice tensor based on the computation flow of each slice tensor.

[0092] In this embodiment of the invention, by concatenating multiple first tensors to be computed into a second tensor, and after slicing, independently assigning distribution information indicating the storage distribution and computational flow indicating the computing die to which the computation belongs to each slice tensor, flexible reading and computation of data on different dies can be achieved under a multi-die architecture based on the specified distribution information and computational flow. Thus, the limitation of unified computation under a multi-die architecture can be met without extensive zero padding, effectively reducing the redundant memory occupation caused by repeated storage of data on multiple dies and extensive zero padding. Furthermore, allocating different slice tensors to the computational flows of different target dies for independent computation can avoid forced waiting between computation tasks of different shapes, greatly improving the parallel computation efficiency of operators and the utilization rate of memory resources under a multi-die architecture.

[0093] Based on the above embodiments, the splicing unit is specifically used for: Based on the arrangement strategy, multiple first tensors for reception calculation are assembled; The arrangement strategy is used to indicate the distribution method and arrangement order of the plurality of first tensors on the plurality of bare wafers.

[0094] Based on any of the above embodiments, the splicing unit is specifically used for: Determine the shape, offset address, and offset length of each first tensor in the second tensor, as the index information of each first tensor in the second tensor.

[0095] Based on any of the above embodiments, the slicing unit is specifically used for: Based on the shape, offset address, and offset length of each first tensor in the second tensor, data that matches the shape, offset address, and offset length is extracted from the second tensor and used as a slice tensor corresponding to each first tensor.

[0096] Based on any of the above embodiments, the distribution information includes at least one of the video memory distribution format and video memory management mode of the slice tensor.

[0097] Based on any of the above embodiments, the operator is a grouped general matrix multiplication.

[0098] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8 As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute operator operation methods, which include: Multiple first tensors from the reception operation are concatenated to obtain a second tensor, and the index information of each first tensor in the second tensor is determined; Based on the index information of each first tensor, the second tensor is sliced ​​to obtain the slice tensor corresponding to each first tensor. The distribution information and computation flow of each slice tensor are specified. The distribution information is used to indicate the storage distribution of the slice tensor on multiple bare dies. The computation flow is used to indicate the target bare die where the data participating in the operator operation in the slice tensor is located. The target bare die belongs to the multiple bare dies. Based on the distribution information of each slice tensor, each slice tensor is read from the multiple bare wafers, and based on the computational flow of each slice tensor, operator operations are performed on the data on the target bare wafer in each read slice tensor.

[0099] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0100] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer being able to execute the operator operation methods provided by the above methods, the method comprising: Multiple first tensors from the reception operation are concatenated to obtain a second tensor, and the index information of each first tensor in the second tensor is determined; Based on the index information of each first tensor, the second tensor is sliced ​​to obtain the slice tensor corresponding to each first tensor. The distribution information and computation flow of each slice tensor are specified. The distribution information is used to indicate the storage distribution of the slice tensor on multiple bare dies. The computation flow is used to indicate the target bare die where the data participating in the operator operation in the slice tensor is located. The target bare die belongs to the multiple bare dies. Based on the distribution information of each slice tensor, each slice tensor is read from the multiple bare wafers, and based on the computational flow of each slice tensor, operator operations are performed on the data on the target bare wafer in each read slice tensor.

[0101] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the operator operation methods provided by the above methods, the method comprising: Multiple first tensors from the reception operation are concatenated to obtain a second tensor, and the index information of each first tensor in the second tensor is determined; Based on the index information of each first tensor, the second tensor is sliced ​​to obtain the slice tensor corresponding to each first tensor. The distribution information and computation flow of each slice tensor are specified. The distribution information is used to indicate the storage distribution of the slice tensor on multiple bare dies. The computation flow is used to indicate the target bare die where the data participating in the operator operation in the slice tensor is located. The target bare die belongs to the multiple bare dies. Based on the distribution information of each slice tensor, each slice tensor is read from the multiple bare wafers, and based on the computational flow of each slice tensor, operator operations are performed on the data on the target bare wafer in each read slice tensor.

[0102] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0103] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An operator operation method, characterized in that, include: Multiple first tensors from the reception operation are concatenated to obtain a second tensor, and the index information of each first tensor in the second tensor is determined; Based on the index information of each first tensor, the second tensor is sliced ​​to obtain the slice tensor corresponding to each first tensor. The distribution information and computation flow of each slice tensor are specified. The distribution information is used to indicate the storage distribution of the slice tensor on multiple bare dies. The computation flow is used to indicate the target bare die where the data participating in the operator operation in the slice tensor is located. The target bare die belongs to the multiple bare dies. Based on the distribution information of each slice tensor, each slice tensor is read from the multiple bare wafers, and based on the computational flow of each slice tensor, operator operations are performed on the data on the target bare wafer in each read slice tensor.

2. The operator operation method according to claim 1, characterized in that, The multiple first tensors in the concatenation operation include: Based on the arrangement strategy, multiple first tensors for reception calculation are assembled; The arrangement strategy is used to indicate the distribution method and arrangement order of the plurality of first tensors on the plurality of bare wafers.

3. The operator operation method according to claim 1, characterized in that, Determining the index information of each first tensor in the second tensor includes: Determine the shape, offset address, and offset length of each first tensor in the second tensor, as the index information of each first tensor in the second tensor.

4. The operator operation method according to claim 3, characterized in that, The step of slicing the second tensor based on the index information of each first tensor includes: Based on the shape, offset address, and offset length of each first tensor in the second tensor, data that matches the shape, offset address, and offset length is extracted from the second tensor and used as a slice tensor corresponding to each first tensor.

5. The operator operation method according to any one of claims 1 to 4, characterized in that, The distribution information includes at least one of the video memory distribution format and video memory management mode of the slice tensor.

6. The operator operation method according to any one of claims 1 to 4, characterized in that, The operator is the grouped general matrix multiplication.

7. An operator arithmetic device, characterized in that, include: The splicing unit is used to splice multiple first tensors for receiving operations to obtain a second tensor, and to determine the index information of each first tensor in the second tensor; A slicing unit is used to slice the second tensor based on the index information of each first tensor to obtain a slice tensor corresponding to each first tensor, and to specify the distribution information and computation flow of each slice tensor. The distribution information is used to indicate the storage distribution of the slice tensor on multiple bare dies, and the computation flow is used to indicate the target bare die where the data participating in the operator operation in the slice tensor is located. The target bare die belongs to the multiple bare dies. The computation unit reads each slice tensor from the multiple bare wafers based on the distribution information of each slice tensor, and performs operator operations on the data on the target bare wafer in each read slice tensor based on the computation flow of each slice tensor.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the operator operation method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the operator operation method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the operator operation method as described in any one of claims 1 to 6.

Images