Operator optimization method and device for hybrid expert model, equipment, medium and program

Abstract

The embodiment of the invention discloses an operator optimization method and device for a hybrid expert model, equipment, a medium and a programmer.The method comprises the steps that a sparse operator suitable for an original hybrid expert model is generated by combining a data partitioning mode with a sparse calculation unit of a calculation chip; wherein the number of the sparse operators is multiple; and updating the original operator of the original hybrid expert model according to the sparse operator to obtain a target hybrid expert model. According to the technical scheme provided by the embodiment of the invention, operator optimization of the hybrid expert model can be realized, so that the hardware acceleration performance of the sparse computing unit in the computing chip is fully played in a model computing scene of the hybrid expert model, and the overhead of computing, bandwidth and storage resources in a sparse matrix multiplication process is greatly optimized.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of artificial intelligence technology, and in particular to an operator optimization method, device, electronic device, storage medium and program of a hybrid expert model. Background Art

[0002] The Mixture of Experts (MoE) model improves the generalization ability of the model by combining multiple expert models to process different input data. It has been widely integrated into large language models.

[0003] The sparse activation characteristics of the hybrid expert model pose new challenges to storage and computing resources. On the one hand, the parameters of the large language model have a certain degree of redundancy, and the sparse method can be used to reduce the amount of calculation while ensuring the accuracy of the model; on the other hand, the hybrid expert model introduces the sparsity of the input data due to the existence of the expert selection strategy.

[0004] In the process of implementing the present invention, the inventors found that the current hybrid expert computing technology mainly focuses on using sparse computing technology to solve the parameter redundancy problem, while ignoring the sparse characteristics of input data. In addition, there is a lack of computing acceleration using hardware instructions to provide structured sparseness in hybrid expert computing. Summary of the invention

[0005] The embodiments of the present invention provide an operator optimization method, device, electronic device, storage medium and program for a hybrid expert model, which realizes operator optimization of the hybrid expert model, fully utilizes the hardware acceleration performance of the sparse computing unit in the computing chip in the model calculation scenario of the hybrid expert model, and thus greatly optimizes the computing, bandwidth and storage resource overhead in the sparse matrix multiplication process.

[0006] According to one aspect of the present invention, there is provided an operator optimization method of a hybrid expert model, comprising:

[0007] A sparse operator suitable for the original hybrid expert model is generated by combining data block with a sparse computing unit of a computing chip; wherein the number of the sparse operators is multiple;

[0008] The original operator of the original hybrid expert model is updated according to the sparse operator to obtain a target hybrid expert model.

[0009] According to another aspect of the present invention, there is provided an operator optimization device of a hybrid expert model, comprising:

[0010] A sparse operator generation module, used to generate a sparse operator suitable for the original hybrid expert model by combining the sparse computing unit of the computing chip in a data block manner; wherein the number of the sparse operators is multiple;

[0011] The sparse operator updating module is used to update the original operator of the original hybrid expert model according to the sparse operator to obtain a target hybrid expert model.

[0012] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the operator optimization method of the hybrid expert model described in any embodiment of the present invention.

[0016] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the operator optimization method of the hybrid expert model described in any embodiment of the present invention when executed.

[0017] According to another aspect of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the operator optimization method of the hybrid expert model described in any embodiment of the present invention is implemented.

[0018] The embodiment of the present invention generates multiple sparse operators suitable for the original hybrid expert model by combining the sparse computing unit of the computing chip in a data blocking manner, thereby updating the original operators of the original hybrid expert model according to the generated sparse operators to obtain the target hybrid expert model. The sparse characteristics of the hybrid expert model when performing matrix multiplication calculations based on sparse matrices are fully considered. By adopting data blocking and data handling technologies that are compatible with the sparse characteristics, the operator optimization of the hybrid expert model is achieved. In the model calculation scenario of the hybrid expert model, the hardware acceleration performance of the sparse computing unit in the computing chip is fully utilized, thereby greatly optimizing the calculation, bandwidth and storage resource overhead in the sparse matrix multiplication operation process.

[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 It is a flow chart of an operator optimization method of a hybrid expert model provided by an embodiment of the present invention;

[0022] Figure 2 is a flow chart of another operator optimization method of a hybrid expert model provided by an embodiment of the present invention;

[0023] Figure 3 It is a schematic diagram of a process for optimizing the original operator of the hybrid expert model provided by an embodiment of the present invention;

[0024] Figure 4 A flowchart of a sparse matrix multiplication acceleration method provided by an embodiment of the present invention;

[0025] Figure 5 It is a schematic diagram of a comparison structure of a first sparse matrix and a matching set of compressed matrices applicable to an embodiment of the present invention;

[0026] Figure 6 It is a block diagram of a data segmentation and multi-stage transfer process applicable to an embodiment of the present invention;

[0027] Figure 7 is a schematic diagram of a register mapping relationship applicable to an embodiment of the present invention;

[0028] Figure 8 It is a structural diagram of a sparse computing unit and an intermediate register used in an embodiment of the present invention to cooperate in computing;

[0029] Fig. 9 is a schematic diagram of an operator optimization device of a hybrid expert model provided by an embodiment of the present invention;

[0030] Fig.10 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] Figure 1 This is a flow chart of an operator optimization method of a hybrid expert model provided by an embodiment of the present invention. This embodiment is applicable to the case where the sparse operator optimized by data block optimization is used to optimize and update the original operator of the hybrid expert model. The method can be executed by an operator optimization device of the hybrid expert model. The device can be implemented by software and / or hardware, and can generally be integrated in an electronic device. The electronic device can be a terminal device or a server device. As long as the operator optimization method of the hybrid expert model can be executed, the embodiment of the present invention does not limit the specific device type of the electronic device. Accordingly, Figure 1 As shown, the method includes the following operations:

[0034] S110, using data block method combined with the sparse computing unit of the computing chip to generate a sparse operator suitable for the original hybrid expert model; wherein the number of the sparse operators is multiple.

[0035] The sparse computing unit may be a hardware computing unit in a computing chip. The original hybrid expert model may be a hybrid expert model that requires optimization of internal operators. The embodiment of the present invention does not limit the model type and model structure of the original hybrid expert model.

[0036] Optionally, the computing chip can be understood as an integrated circuit used to implement a set computing task (for example, Internet of Things control, high-performance computing or mobile computing, etc.). The computing chip can be a general computing chip, such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or a heterogeneous chip combination of "CPU + GPU equipped with a sparse acceleration computing unit", or a dedicated computing chip, such as an NPU (Neural Processing Unit), which is not limited in the embodiments of the present invention.

[0037] Among them, sparse operators refer to operators that only operate on some elements, such as sparse matrices in matrix multiplication. In the calculation process, the processing of sparse operators usually involves how to effectively store and calculate sparse matrices, and how to optimize the computing performance of sparse operators. Sparse operators are mainly used to process sparse matrices, in which most elements are zero and only a small number of non-zero elements. Sparse operators can reduce the use of storage space through compressed storage methods, use algorithms such as fast Fourier transform to accelerate operations such as sparse matrix multiplication, and use memory optimization techniques (such as cache optimization and memory alignment) to improve memory usage efficiency. Parallel computing technologies (such as multi-threading or distributed computing) can also be used to accelerate operations on large-scale sparse matrices.

[0038] In an embodiment of the present invention, in order to achieve computational acceleration of an adaptive sparse acceleration hardware computing unit and fully consider the sparse characteristics of the operating data of the sparse operator, a data block method can be used in combination with the sparse computing unit of the computing chip to optimize the computational, bandwidth and storage resource overhead during the execution of the operator, and through compilation optimization methods such as operator fusion, various types of sparse operators suitable for the original hybrid expert model are generated.

[0039] S120: Update the original operator of the original hybrid expert model according to the sparse operator to obtain a target hybrid expert model.

[0040] The original operator may be the original operator type of the original hybrid expert model, and the target hybrid expert model may be a hybrid expert model with an optimized structure obtained by optimizing and updating the original operator of the original hybrid expert model according to the sparse operator.

[0041] It is understandable that the original operator of the original mixed expert model is usually a matrix multiplication operator. The matrix multiplication operator is usually performed as a dense matrix multiplication operation.

[0042] Correspondingly, after generating various types of sparse operators suitable for the original hybrid expert model, the types and functions of the original operators of the original hybrid expert model can be considered, and the original operators can be updated and replaced with adapted sparse operators to achieve operator optimization and structural update of the original hybrid expert model, and obtain a target hybrid expert model that can be accelerated by hardware instructions.

[0043] Correspondingly, the optimized target hybrid expert model can take the structured sparse matrix as input, optimize the data flow of the sparse operator in this format, and at the same time, arrange the calculation process of the calculation kernel, and use the sparse computing units in the hardware-level computing chip to optimize and accelerate the sparse operators, and finally inject them into the hybrid expert calculation process, thereby accelerating the execution of the hybrid expert model.

[0044] It can be seen that the operator optimization method of the hybrid expert model provided by the embodiment of the present invention can realize the structured sparse transformation suitable for hybrid expert calculation, and finally replace the original hybrid expert model with the converted model, and perform model calculation in the same way as the original one, and realize calculation acceleration by calling a specific sparse acceleration unit.

[0045] The embodiment of the present invention generates multiple sparse operators suitable for the original hybrid expert model by combining the sparse computing unit of the computing chip in a data blocking manner, thereby updating the original operators of the original hybrid expert model according to the generated sparse operators to obtain the target hybrid expert model. The sparse characteristics of the hybrid expert model when performing matrix multiplication calculations based on sparse matrices are fully considered. By adopting data blocking and data handling technologies that are compatible with the sparse characteristics, the operator optimization of the hybrid expert model is achieved. In the model calculation scenario of the hybrid expert model, the hardware acceleration performance of the sparse computing unit in the computing chip is fully utilized, thereby greatly optimizing the calculation, bandwidth and storage resource overhead in the sparse matrix multiplication operation process.

[0046] Figure 2 is a flow chart of another operator optimization method of a hybrid expert model provided by an embodiment of the present invention. Figure 3 1 is a schematic diagram of a process for optimizing the original operator of the hybrid expert model provided by an embodiment of the present invention. This embodiment is specific based on the above embodiment. In this embodiment, multiple specific optional generation methods of different types of sparse operators are provided. Figure 2 and Figure 3 As shown, the method of this embodiment may include:

[0047] S210, using a data block method in combination with a sparse computing unit of a computing chip to generate a first sparse operator, a second sparse operator, and a third sparse operator suitable for the original hybrid expert model.

[0048] The first sparse operator may be an operator in the hybrid expert model for outputting the model calculation result. The second sparse operator may be an operator capable of outputting the output value of the activation function. The third sparse operator may be an operator capable of outputting the output value of the dot product function.

[0049] In an optional embodiment of the present invention, if the sparse operator includes a first sparse operator, then the method of using data blocking in combination with the sparse computing unit of the computing chip to generate a sparse operator suitable for the original hybrid expert model may include: according to the first sparse matrix multiplication requirement, locating a set of compression matrices and a second association matrix matching the first sparse matrix in the global memory of the computing chip; wherein the second association matrix includes a second sparse matrix or a dense matrix corresponding to the second sparse matrix; wherein the first sparse matrix includes a plurality of compression units of a set size, each of the compression units includes at least one valid row and at least one sparse row, and each of the valid rows includes at least one structured sparse computing storage unit; each of the structured sparse computing storage units is in the same sparse mode; compression The matrix set includes a compressed data matrix for storing non-zero data in a first sparse matrix, an index matrix for storing the positions of valid rows in the first sparse matrix in the corresponding compression unit, and a metadata matrix for storing the positions of non-zero data in the corresponding structured sparse computing storage unit; the compressed matrix set and the second association matrix are moved step by step from the global memory to the hardware register of the computing chip in the form of data blocks; the multiplication results of the first sparse matrix and the second association matrix are gradually calculated by the sparse computing unit of the computing chip according to the data loaded in batches in the hardware register; the multiplication results of the first sparse matrix and the second association matrix are processed according to the processing logic of the broadcast multiplication function of the original hybrid expert model to obtain the first sparse operator.

[0050] In an optional embodiment of the present invention, if the sparse operator includes a second sparse operator, then the use of data blocking in combination with the sparse computing unit of the computing chip to generate a sparse operator suitable for the original hybrid expert model may include: according to the second sparse matrix multiplication requirements, locating a compression matrix set and a second association matrix matching the first sparse matrix in the global memory of the computing chip; wherein the second association matrix includes the second sparse matrix or a dense matrix corresponding to the second sparse matrix; wherein the first sparse matrix includes a plurality of compression units of a set size, each of the compression units includes at least one valid row and at least one sparse row, each of the valid rows includes at least one structured sparse computing storage unit; each of the structured sparse computing storage units is in the same sparse mode; the compression matrix set includes a storage unit for storing A compressed data matrix for storing non-zero data in a first sparse matrix, an index matrix for storing the positions of valid rows in the first sparse matrix in the corresponding compression unit, and a metadata matrix for storing the positions of non-zero data in the corresponding structured sparse computing storage unit; the compressed matrix set and the second association matrix are transferred step by step from the global memory to the hardware register of the computing chip in the form of data blocks, and during the transfer process, the data in the original global memory is rearranged or deformed; through the sparse computing unit of the computing chip, the multiplication results of the first sparse matrix and the second association matrix are gradually calculated according to the data loaded in batches in the hardware register; the multiplication results of the first sparse matrix and the second association matrix are processed according to the processing logic of the activation function of the original hybrid expert model to obtain the second sparse operator.

[0051] In an optional embodiment of the present invention, if the sparse operator includes a third sparse operator, then the use of data blocking in combination with the sparse computing unit of the computing chip to generate a sparse operator suitable for the original hybrid expert model may include: according to the third sparse matrix multiplication requirement, locating a compression matrix set and a second association matrix matching the first sparse matrix in the global memory of the computing chip; wherein the second association matrix includes a second sparse matrix or a dense matrix corresponding to the second sparse matrix; wherein the first sparse matrix includes a plurality of compression units of a set size, each of the compression units includes at least one valid row and at least one sparse row, each of the valid rows includes at least one structured sparse computing storage unit; each of the structured sparse computing storage units is in the same sparse mode; the compression matrix set includes a storage unit for storing A compressed data matrix of non-zero data in a first sparse matrix, an index matrix for storing the positions of valid rows in the first sparse matrix in the corresponding compression unit, and a metadata matrix for storing the positions of non-zero data in the corresponding structured sparse computing storage unit; the compressed matrix set and the second association matrix are transferred step by step from the global memory to the hardware register of the computing chip in the form of data blocks, and during the transfer process, the data in the original global memory is rearranged or deformed; through the sparse computing unit of the computing chip, the multiplication results of the first sparse matrix and the second association matrix are gradually calculated according to the data loaded in batches in the hardware register; the multiplication results of the first sparse matrix and the second association matrix are processed according to the processing logic of the point multiplication function of the original hybrid expert model to obtain the third sparse operator.

[0052] Among them, the first sparse matrix can be the left operand of the double matrix multiplication, which is a sparse matrix. The second association matrix can be the right operand of the double matrix multiplication, which can be a sparse matrix or a dense matrix composed of valid data in the sparse matrix. The matrix multiplication between the first sparse matrix and the second association matrix can be called sparse matrix multiplication. When the second association matrix also adopts a sparse matrix, the matrix multiplication between the first sparse matrix and the second association matrix can be called double sparse matrix multiplication. In the following, no matter what type of matrix the second association matrix adopts, the matrix multiplication between the first sparse matrix and the second association matrix is ​​referred to as sparse matrix multiplication.

[0053] In an embodiment of the present invention, the above-mentioned sparse matrix multiplication acceleration method can be encapsulated in the form of an operator interface, for example, a plurality of types of sparse operators are specially constructed for realizing matrix multiplication between a first sparse matrix and a second associated matrix, such as a first sparse operator, a second sparse operator, or a third sparse operator, etc. Furthermore, the multiplication acceleration method between the first sparse matrix and the second associated matrix can be triggered by calling the operator interface of the sparse operator, and according to the fusion requirements of the first sparse operator and other functional functions, additional processing logic is configured for the multiplication result between the first sparse matrix and the second associated matrix, thereby generating a plurality of types of sparse operators, such as a first sparse operator, a second sparse operator, or a third sparse operator, etc.

[0054] Specifically, sparse matrix multiplication can be understood as at least one type of operand in the left operand and the right operand of the matrix multiplication is a sparse matrix. Accordingly, when it is detected that the interface call request for the first sparse operator generates the first sparse operator, it is determined that the first sparse matrix multiplication requirement is detected; when it is detected that the interface call request for the second sparse operator generates the second sparse operator, it is determined that the second sparse matrix multiplication requirement is detected; when it is detected that the interface call request for the third sparse operator generates the third sparse operator, it is determined that the third sparse matrix multiplication requirement is detected. Furthermore, the identification information of the left operand (hereinafter referred to as the first sparse matrix) and the right operand (hereinafter referred to as the second association matrix) that need to perform sparse matrix multiplication can be obtained from the corresponding interface call request. Based on the above identification information, the compressed matrix set and the second association matrix that match the first sparse matrix can be located in the global memory of the computing chip.

[0055] It is understandable that the first sparse matrix and the second association matrix are multiplied in the computing chip. Furthermore, the first sparse matrix and the second association matrix to be calculated need to be preloaded into the computing chip. In order to improve the computing speed of the computing chip, the calculation data is initially stored in the global memory of the computing chip, and then the calculation data can be moved in blocks to the hardware registers adapted to the hardware computing unit in a step-by-step manner, and the hardware computing unit calculates the final calculation result based on the data blocks stored in the hardware registers.

[0056] In this embodiment, in order to achieve the final multiplication acceleration of the sparse operator, the sparse format of the first sparse matrix as the left operand is specially limited. At the same time, the storage method of the first sparse matrix in the global memory of the computing chip is also optimized accordingly.

[0057] Among them, the first sparse matrix includes multiple compression units of set sizes, each compression unit includes at least one valid row and at least one sparse row, each valid row includes at least one structured sparse computing storage unit; each structured sparse computing storage unit is in the same sparse mode; the compression matrix set includes a compression data matrix for storing non-zero data in the first sparse matrix, an index matrix for storing the positions of valid rows in the first sparse matrix in the corresponding compression units, and a metadata matrix for storing the positions of non-zero data in the corresponding structured sparse computing storage units.

[0058] Specifically, the first sparse matrix contains an integer number (one or more) of compression units, that is, the size of the first sparse matrix (number of rows * number of columns) can divide the size of the compression unit. The compression unit can be understood as a small matrix of a specific size, which has at least two matrix rows. The at least two matrix rows have at least one valid row and at least one sparse row. A valid row can be understood as a matrix row containing at least one valid data (non-0), and a sparse row refers to a matrix row in which all data is 0.

[0059] Further, each valid row in the compression unit contains an integer number (one or more) of structured sparse computing storage units. Wherein, each structured sparse computing storage unit contained in the first sparse matrix is ​​in the same sparse mode. Generally speaking, in order to facilitate calculation, the size of the structured sparse computing storage unit can be adapted to the calculation scale of the hardware computing unit in the computing chip.

[0060] The sparse pattern can be understood as the ratio of valid data (non-zero value data) to all data. For example, the sparse pattern can be 2:4 or 4:8. That is, for a 2:4 sparse pattern, the structured sparse computing storage unit contains a total of 4 data, 2 of which are non-zero data. The arrangement positions of the above two non-zero data in the structured sparse computing storage unit are not restricted.

[0061] It is understandable that the first sparse matrix contains a large amount of zero-valued data. If the first sparse matrix is ​​directly stored in the global memory of the computing chip according to the original size of the first sparse matrix, a large amount of storage units will be wasted. In addition, the subsequent sparse matrix multiplication is directly implemented based on the first sparse matrix of the original size, which has low computational efficiency and cannot efficiently use the hardware computing units in the computing chip, that is, the sparse computing units.

[0062] In view of this, in an embodiment of the present invention, a new type of efficient data compression method is creatively proposed for the left operand (first sparse matrix) of the sparse multiplication calculation of the above special structure to solve the above technical problems.

[0063] In an embodiment of the present invention, the first sparse matrix can be compactly stored in the form of a compressed matrix set by a specific data compression method. Specifically, three compressed matrices corresponding to the first sparse matrix are stored in the compressed matrix set, namely, a compressed data matrix, an index matrix, and a metadata matrix.

[0064] Among them, the compressed data matrix is ​​used to store the compressed data matrix of non-zero data in the first sparse matrix, the index matrix is ​​used to store the position of the valid rows in the first sparse matrix in the corresponding compression unit, and the metadata matrix is ​​used to store the position of the non-zero data in the corresponding structured sparse computing storage unit.

[0065] Obviously, through the above-mentioned data compression method, the specific position of each non-zero data in the first sparse matrix in the original first sparse matrix can be clearly determined through three small matrices. This data compression method can also effectively reduce the consumption of storage resources in the computing chip, and can also facilitate the implementation of calculations based on the first sparse matrix.

[0066] In an embodiment of the present invention, the second association matrix may not be stored in compression. However, considering the actual application requirements of sparse matrix multiplication, for example, in a hybrid expert model, the second association matrix generally also has a specific data structure. Typically, when the second association matrix is ​​a second sparse matrix, one or more matrix rows in the second sparse matrix are sparse rows (all 0). When the second association matrix is ​​a dense matrix corresponding to the second sparse matrix, the data in the dense matrix corresponding to the second sparse matrix is ​​valid data in the second sparse matrix. Exemplarily, assuming that the second sparse matrix is ​​[1,3,0,5,0,0], the dense matrix corresponding to the second sparse matrix is ​​[1,3,5].

[0067] Based on this, in addition to completely storing the second association matrix in the global memory of the computing chip, a selection array matching the second association matrix is ​​further stored, and the selection array is used to describe the valid row positions in the second association matrix. In a specific example, assuming that the second association matrix is ​​a second sparse matrix, and all rows except the 0th, 3rd, and 5th rows in the second sparse matrix are sparse rows, a selection array of the form {0,3,5} can be constructed and stored in association with the second association matrix. Assuming that the second association matrix is ​​a dense matrix corresponding to the second sparse matrix, and the second sparse matrix includes valid data for rows 1-3 but does not include sparse rows, a selection array of the form {1,2,3} can be constructed and stored in association with the second association matrix.

[0068] Accordingly, while locating the second association matrix in the global memory of the computing chip, the selection array matching the second association matrix can be located synchronously.

[0069] In an optional implementation of the embodiment of the present invention, the computing chip includes a three-level storage architecture of global memory->shared memory->hardware registers. The three-level storage is gradually approached to the hardware computing units used to perform calculations in the computing chip in terms of physical location, especially the hardware registers are arranged close to the hardware computing units.

[0070] Generally speaking, the closer the storage device is to the computing unit, the faster its data read and write speed is, but the smaller its storage capacity is. Therefore, for a matrix multiplication of C(M×N)=A(M×K)×B(K×N), the complete data of A and B cannot be fully stored in the shared memory or hardware registers, and often need to be moved in blocks to the hardware registers through multi-level transfer, so that the hardware computing unit in the computing chip can calculate the final complete multiplication result in batches.

[0071] In an optional implementation of the present embodiment, the special data structure of the compressed matrix set can be combined to perform data block processing on each matrix in the compressed matrix set. At the same time, combined with the selection array matching the second sparse matrix, the second association matrix can be processed in data blocks to improve the efficiency of data handling and subsequent multiplication calculations.

[0072] In this embodiment, the sparse computing unit can be understood as a special hardware circuit in the computing chip for performing matrix multiplication calculations on sparse matrices. Such special hardware circuits can accelerate sparse matrix multiplication. The computing chip generally includes multiple sparse computing units, and within each sparse computing unit, one or more threads can be opened to implement corresponding sparse computing tasks.

[0073] By using the sparse computing units in the computing chip in coordination, partial data in the first sparse matrix and the second association matrix can be loaded from each hardware register in batches for local calculation, and the local calculation results can be combined or accumulated to finally obtain the multiplication result of the first sparse matrix and the second association matrix.

[0074] In the calculation process of the hybrid expert model, there are many one-to-one operators and matrix multiplications adjacent to each other, but the calculation of two consecutive operators will introduce redundant computing kernel startup overhead and I / O (Input / Output) overhead. In order to eliminate the overhead of loading and storing intermediate results and multiple kernel startups, the computing core of each sparse operator can be improved by operator fusion.

[0075] Among various sparse operators, the first sparse operator is usually used as an operator for outputting model calculation results, for example, Figure 3As shown, the sparse operator 3 in the target hybrid expert model can be used as the first sparse operator. Optionally, for the first sparse operator, after the multiplication results of the first sparse matrix and the second association matrix are gradually calculated according to the data loaded in batches in the hardware register by the sparse computing unit of the computing chip, the multiplication results of the first sparse matrix and the second association matrix can be processed according to the processing logic of the broadcast multiplication function of the original hybrid expert model to obtain the first sparse operator with broadcast function.

[0076] like Figure 3 As shown, the sparse operator 1 in the target hybrid expert model can be used as the second sparse operator. Optionally, for the second sparse operator, after gradually calculating the multiplication results of the first sparse matrix and the second association matrix according to the data loaded in batches in the hardware register through the sparse computing unit of the computing chip, the multiplication results of the first sparse matrix and the second association matrix can be processed according to the processing logic of the activation function of the original hybrid expert model to obtain the second sparse operator. Accordingly, the second sparse operator can directly output the calculation result of the activation function.

[0077] like Figure 3 As shown, the sparse operator 2 in the target hybrid expert model can be used as the third sparse operator. Optionally, for the third sparse operator, after gradually calculating the multiplication results of the first sparse matrix and the second association matrix according to the data loaded in batches in the hardware register through the sparse computing unit of the computing chip, the multiplication results of the first sparse matrix and the second association matrix can be processed according to the processing logic of the point multiplication function of the original hybrid expert model to obtain the third sparse operator. Accordingly, the third sparse operator can directly output the calculation result of the point multiplication function.

[0078] For the first sparse operator, the second sparse operator and the third sparse operator, the overall sparse operator generation process is generally the same, the only difference is that in the process of moving data from the global memory to the hardware register of the computing chip in the form of blocks, the second sparse operator and the third sparse operator need to rearrange or deform the data in the original global memory during the moving process. At the same time, because the functions of the operators are different, the types of specific functional functions fused when the first sparse operator, the second sparse operator and the third sparse operator are different.

[0079] S220 . Construct a sparse operator library according to the first sparse operator, the second sparse operator, and the third sparse operator.

[0080] Among them, the sparse operator library can be used to store various generated sparse operators.

[0081] In the embodiment of the present invention, Figure 3As shown, after various sparse operators suitable for the original hybrid expert model are generated by combining the sparse computing unit of the computing chip in a data block manner, the generated various sparse operators can be stored in the sparse operator library. It can be understood that the generation process of the above sparse operators only schematically illustrates the specific generation process of the first sparse operator, the second sparse operator and the third sparse operator. For other types of sparse operators in the original hybrid expert model, similar methods as described above can also be used to optimize the generation. That is, sparse operators are not limited to the first sparse operator, the second sparse operator and the third sparse operator. Accordingly, in addition to storing the first sparse operator, the second sparse operator and the third sparse operator, the sparse operator library can also store other types of sparse operators, and the embodiment of the present invention does not limit the type of sparse operators.

[0082] S230 , selecting a target sparse operator from a sparse operator library according to the type of the original hybrid expert model, and updating the original operator of the original hybrid expert model according to the target sparse operator to obtain a target hybrid expert model.

[0083] The target sparse operator may be a sparse operator used to update and replace the original operator in the original hybrid expert model. Optionally, there may be multiple target sparse operators.

[0084] For example, Figure 3 As shown, the original operators of the original hybrid expert model may include a first matrix multiplication operator, a second matrix multiplication operator, and a third matrix multiplication operator. Among them, the first matrix multiplication operator may be a matrix multiplication 2 operator in the original hybrid expert model, the second matrix multiplication operator may be a matrix multiplication 1 operator in the original hybrid expert model, and the third matrix multiplication operator may be a matrix multiplication 3 operator in the original hybrid expert model. For this type of original hybrid expert model, the first sparse operator, the second sparse operator, and the third sparse operator may be selected as target sparse operators. Accordingly, when the original operators of the original hybrid expert model are updated according to the target sparse operator, the first matrix multiplication operator of the original hybrid expert model may be replaced and updated according to the first sparse operator, the second matrix multiplication operator of the original hybrid expert model may be replaced and updated according to the second sparse operator, and the third matrix multiplication operator of the original hybrid expert model may be replaced and updated according to the third sparse operator to obtain the target hybrid expert model.

[0085] For example, Figure 3As shown, the original operators of some types of original hybrid expert models only include the first matrix multiplication operator and the second matrix multiplication operator. Among them, the first matrix multiplication operator can be the matrix multiplication 2 operator in the original hybrid expert model, and the second matrix multiplication operator can be the matrix multiplication 1 operator in the original hybrid expert model. For this type of original hybrid expert model, the first sparse operator and the second sparse operator can be screened as the target sparse operator. Correspondingly, when the original operator of the original hybrid expert model is updated according to the target sparse operator, the first matrix multiplication operator of the original hybrid expert model can be replaced and updated according to the first sparse operator, and the second matrix multiplication operator of the original hybrid expert model can be replaced and updated according to the second sparse operator to obtain the target hybrid expert model.

[0086] Similarly, for other types of original hybrid expert models, it is also possible to adapt to generate sparse operator types with other functions, and replace and update the original operators inside them to obtain the corresponding optimized target hybrid expert model.

[0087] The technical solution of the embodiment of the present invention is to compress and store the sparse matrix that needs to be multiplied in a specific compression format, and then move the sparse matrix and the associated matrix in the above-mentioned specific compression format from the global memory to the hardware register of the computing chip in the form of data blocks step by step, and use the sparse computing unit of the computing chip to gradually calculate the multiplication results of the sparse matrix and the associated matrix according to the data loaded in the hardware register in batches. The implementation method fully considers the sparse characteristics when performing matrix multiplication calculations based on sparse matrices. By adopting data blocking and data moving technology that are compatible with the sparse characteristics, the hardware acceleration performance of the sparse computing unit in the computing chip can be fully utilized, thereby greatly optimizing the calculation, bandwidth and storage resource overhead in the sparse matrix multiplication operation process, and is particularly suitable for model calculation scenarios of hybrid expert models.

[0088] Figure 4 A flowchart of a sparse matrix multiplication acceleration method provided by an embodiment of the present invention is provided. This embodiment is optimized based on the above embodiments. In this embodiment, the operation of "moving the compressed matrix set and the second association matrix from the global memory to the hardware register of the computing chip in the form of data blocks" is specifically concretized.

[0089] Correspondingly, such as Figure 4 As shown, the method may include:

[0090] S410. According to the sparse matrix multiplication requirement, locate a set of compressed matrices and a second association matrix matching the first sparse matrix in the global memory of the computing chip.

[0091] The sparse matrix multiplication requirement may include but is not limited to a first sparse matrix multiplication requirement, a second sparse matrix multiplication requirement, and a third sparse matrix multiplication requirement.

[0092] The inventors have found through research that when a hybrid expert model is used for computing tasks (typically, model reasoning tasks), the matrix multiplication calculation between the model weight matrix of each model layer and the sparse matrix of intermediate activation values ​​input to the hybrid model layer in the hybrid expert model belongs to double sparse matrix multiplication. The matrix multiplication calculation between the model weight matrix of each model layer and the dense matrix of intermediate activation values ​​input to the hybrid model layer in the hybrid expert model belongs to sparse matrix multiplication. Furthermore, the methods of the embodiments of the present invention can be applied to model calculation and model optimization scenarios of hybrid expert models.

[0093] In the computing scenario of the hybrid expert model, by executing the sparse matrix multiplication process based on the specific model weight matrix and the matching sparse matrix or dense matrix of the intermediate activation value, combined with the operator fusion strategy, various types of sparse operators can be generated to achieve operator optimization of the hybrid expert model, and then complete the optimization of the internal model structure of the hybrid expert model.

[0094] Accordingly, in an optional implementation of an embodiment of the present invention, the first sparse matrix is ​​a model weight matrix in the target hybrid expert model, and the second association matrix is ​​a sparse matrix or a dense matrix of intermediate activation values ​​of the hybrid model layer input into the target hybrid expert model.

[0095] Furthermore, the first sparse matrix may specifically include a plurality of compression units of size M*V;

[0096] Specifically, each compression unit includes d valid rows, where 1≤d<M; and

[0097] Each structured sparse computing storage unit is in a sparse mode of N:L, where L is the total amount of data contained in the structured sparse computing storage unit, and N is the amount of non-zero data contained in the structured sparse computing storage unit.

[0098] Correspondingly, for the first sparse matrix of size m*k, a compressed data matrix of size (m / M*d)*(k / (L / N)), an index matrix of size (m / M*d)*(k / V), and a metadata matrix of size (m / M*d)*(k / (L / N)) can be obtained.

[0099] For ease of explanation, Figure 5 A schematic diagram of the comparison structure of a first sparse matrix and a matching set of compressed matrices applicable to various embodiments of the present invention is shown in FIG.

[0100] Specifically, Figure 5The diagram shown is a schematic diagram of a set of compressed matrices matching a first sparse matrix with m=4, k=16. In the first sparse matrix, there are 4 compression units with M=2, V=8, each of which contains d=1 valid rows, each of which contains 2 structured sparse computing storage units, and each structured sparse computing storage unit is in a sparse mode with N:L of 2:4. That is, A, B, ..., L filled in each matrix position of the first sparse matrix represent non-zero data, and the blank position represents the data filled with zero value.

[0101] Correspondingly, the size of the compressed data matrix corresponding to the first sparse matrix is ​​(m / M*d)*(k / (L / N))=2*8. The compressed data matrix stores the non-zero data A, B, ..., L in the first sparse matrix in sequence.

[0102] Furthermore, the size of the index matrix corresponding to the first sparse matrix is ​​(m / M*d)*(k / V)=2*2, and the positions of each valid row in the first sparse matrix in the corresponding compression unit are stored in different positions of the index matrix. Figure 5 As shown, the first sparse matrix includes a 2*8 compression unit 1 where A, B, C, and D are located. Since d=1, and the row position where the valid row in the compression unit 1 is located, that is, the row where A, B, C, and D are located, is the 0th row, and further, the above positional relationship can be identified by the 0 stored at the matrix position in the upper left corner of the index matrix.

[0103] Furthermore, the size of the metadata matrix corresponding to the first sparse matrix is ​​(m / M*d)*(k / (L / N))=2*8, and the positions of each non-zero data in the corresponding structured sparse computing storage unit are stored in different positions of the metadata matrix. Figure 5 As shown, the structured sparse computing storage unit 1 containing non-zero elements A and B is located in the upper left corner of the first sparse matrix. Furthermore, the 0 and 2 at the first two column positions of the first row of the metadata matrix represent the column positions of A and B in the structured sparse computing storage unit 1, respectively, with A being located in the 0th column and B being located in the 2nd column.

[0104] It is understandable that by constructing the above-mentioned compressed data matrix, index matrix and metadata data, the first sparse matrix of the above-mentioned specific structure can be uniquely determined. Furthermore, the sparse model weight matrices contained in the target hybrid expert model can be efficiently compressed and stored, thereby effectively reducing the storage overhead of the target hybrid expert model on the configured computing chip.

[0105] S420, dividing the compressed data matrix in the global memory into data blocks, and dividing the index matrix and the metadata matrix into data blocks that match each other according to the data correspondence between the compressed data matrix, the index matrix and the metadata matrix.

[0106] In the embodiment of the present invention, since the compressed data matrix stores all non-zero data in the first sparse matrix, data can be partitioned according to the data size of the compressed data matrix. In the process of partitioning the compressed data matrix, in order to further consider the efficiency of subsequent data handling and multiplication calculations, it can be set that in the process of partitioning the compressed data matrix in the global memory, V (the number of columns contained in each compression unit) is an integer multiple of the partition size in the column partition direction.

[0107] For example, suppose that in global memory, the compressed data matrix is ​​split into multiple m b *k b In the process of data segmentation, V needs to be k. b An integer multiple of, for example, k b It is V / 2 or V / 4, etc.

[0108] As shown above, since there is a data or position correspondence between the compressed data matrix, the index matrix and the metadata matrix, after determining the compressed data matrix blocks cut out from the compressed data matrix, the index matrix blocks and the metadata matrix blocks that match the compressed data matrix blocks can be uniquely determined. That is, according to the data correspondence between the compressed data matrix, the index matrix and the metadata matrix, the data blocks that match the index matrix and the metadata matrix are matched.

[0109] S430, locating a selection array matching the second association matrix in the computing chip, and dividing the second association matrix in the global memory into data blocks according to the selection array.

[0110] The selection array of the second sparse matrix is ​​the index of the non-zero data position therein, and the selection array of the dense matrix corresponding to the second sparse matrix is ​​the position mapping of the data in the dense matrix in the second sparse matrix.

[0111] As shown above, if the second association matrix is ​​a second sparse matrix, the second sparse matrix may contain a large number of sparse rows. If these sparse rows are also divided into data blocks and sent to the hardware computing unit for matrix multiplication calculation, this will result in the loading and calculation of redundant data, thereby reducing the computing efficiency.

[0112] Based on this, in the embodiment of the present invention, in the global memory, only the valid data rows in the second association matrix can be partitioned based on the valid row positions defined by the selection array. Optionally, one or more valid data rows can be taken as a data partition each time, or a set amount of data in a valid data row can be taken as a data partition each time.

[0113] S440, loading each compressed data matrix block, each index matrix block and each metadata matrix block from the global memory into the shared memory one by one.

[0114] After completing the data partitioning for the compressed matrix set in the global memory, each compressed data matrix block, each index matrix block and each metadata matrix block may be loaded from the global memory to the shared memory one by one.

[0115] On the basis of the above embodiments, in order to prevent the data in the shared memory from being restricted by a bank conflict (bank conflict) in which the same storage body (also referred to as a bank) in the computing chip cannot be accessed by multiple threads at the same time during the access (read / write) process, in the embodiments of the present invention, it is further considered to rearrange the storage of data in the shared memory based on the data offset using a specific data rearrangement function.

[0116] Accordingly, in an optional implementation of the embodiment of the present invention, after each compressed data matrix block, each index matrix block, and each metadata matrix block are successively moved from the global memory to the shared memory, the following may also be included:

[0117] Using a preset data rearrangement function, each compressed data matrix block, each index matrix block, and each metadata matrix block are rearranged in a shared memory to avoid storage conflicts.

[0118] Specifically, according to the specific type and model of the computing chip, a matching data rearrangement function can be selected in the adapted function library to rearrange the data in the shared memory, and the embodiment of the present invention is not limited to this.

[0119] S450, dividing each second association matrix into blocks, and moving them from the global memory to the shared memory one by one.

[0120] Similarly, after the data of the second association matrix is ​​divided into blocks in the global memory, the blocks of the second association matrix can be correspondingly moved from the global memory to the shared memory one by one.

[0121] Meanwhile, in order to avoid the problem of memory bank conflict, in an optional implementation of the embodiment of the present invention, after each second association matrix is ​​divided into blocks and moved from the global memory to the shared memory one by one, the following steps may also be included:

[0122] Using a preset data rearrangement function, each second correlation matrix block is rearranged in a shared memory to avoid storage bank conflicts.

[0123] Of course, it is understandable that in addition to using the data rearrangement function to rearrange the compressed data matrix blocks, the index matrix blocks, the metadata matrix blocks and the second association matrix blocks in the shared memory, it is also possible to add data padding to the compressed data matrix blocks, the index matrix blocks, the metadata matrix blocks and the second association matrix blocks in the shared memory to avoid storage conflicts.

[0124] S460, performing secondary block division on each compressed data matrix block, each index matrix block and each metadata matrix block in the shared memory.

[0125] In the embodiment of the present invention, in order to adapt to the storage limitation of the hardware register, each compressed data matrix block, each index matrix block and each metadata matrix block in the shared memory may be divided into two blocks.

[0126] Furthermore, when performing secondary partitioning on each compressed data matrix block, each index matrix block, and each metadata matrix block in the shared memory, the compressed data matrix block can also be firstly secondary partitioned. In the process of secondary partitioning of the compressed data matrix, in order to further consider the efficiency of subsequent data handling and multiplication calculation, it can be set that in the process of secondary partitioning of the compressed data matrix block in the shared memory, V is also an integer multiple of the partition size in the column partitioning direction. For example, secondary partitioning can be selected in the column partitioning direction or no secondary partitioning can be selected (partitioning is performed only in the row direction).

[0127] For example, suppose that in global memory, the compressed data matrix is ​​divided into multiple m blocks b1 *k b1 In the process of new data block size, V needs to be k b1 An integer multiple of, for example, k b1 It is V / 4 or V / 8 ​​etc.

[0128] Similarly, since there is a data or position correspondence between the compressed data matrix, the index matrix and the metadata matrix, after completing the secondary blocking of the compressed data matrix, the secondary blocking of the index matrix and the metadata matrix can be performed accordingly.

[0129] S470, performing secondary block division on each second correlation matrix block in the shared memory.

[0130] S480, according to the register mapping relationship matching the sparse computing unit, at least one item of each compressed data matrix secondary block, each index matrix secondary block and each metadata matrix secondary block is moved from the shared memory to the hardware register one by one.

[0131] For a more intuitive understanding, Figure 6 A block diagram of a data partitioning and multi-stage transfer process applicable to an embodiment of the present invention is shown in FIG. In a specific example, the second sparse matrix is ​​used as an example. Figure 6 As shown, it is described that the compressed data matrix A in the first sparse matrix and the second sparse matrix B are divided into data blocks in the global memory and the shared memory respectively, and are moved step by step in the order of global memory->shared memory->register. After the calculation is completed by the sparse computing unit and written back step by step, the corresponding multiplication result matrix C is obtained in the global memory.

[0132] In this example, an example of data partitioning and data movement for matrix multiplication of any size C(M×N)=A(M×K)×B(K×N) is given. In the embodiment of the present invention, in order to implement the matrix multiplication calculation, the rows and columns of the second sparse matrix B are interchanged, that is, an array is selected to describe the positions of the valid data columns in the second sparse matrix B.

[0133] Specifically, in global memory, since the selection array only contains the locations of valid data columns (rows), the total amount of data in the selection array is len d Then the number of valid values ​​of n dimension in the second sparse matrix B is len d The compressed data matrix A is in m dimensions with m b The second sparse matrix B is divided into several blocks of uniform size. The number of valid column vectors in each block of data is the same as the number of n in the selection array. b Considering the storage size limit of the computing device, the compressed data matrix A and the second sparse matrix B will be further divided into k blocks along the k direction. b To evenly divide the data into different sizes, multiple sub-blocks in the k direction are multiplied and accumulated to obtain the final multiplication result matrix C block result. Based on this data division strategy, the size of the multiplication result matrix C calculated for each block data is m b ×n b , and the block calculation is a part of the final result of the multiplication result matrix C, which corresponds to the same row offset of the data block of the compressed data matrix A in the compressed data matrix A, and the same column offset of the data block of the second sparse matrix B in the selection array. The parallelism of each sparse operator is improved by each thread group being responsible for a block.

[0134] Similarly, in the shared memory, it also involves a process of performing a secondary block division on each data block transferred from the global memory, and after the block is moved to the register for storage, the adapted sparse computing unit performs a matrix multiplication calculation, which will not be described in detail here. In this embodiment, the matrix specification of the compressed data matrix A processed by the sparse computing unit once is mi*ki, and the matrix specification of the second sparse matrix B is ki*ni. In order to facilitate subsequent calculations, the aforementioned V is an integer multiple of ki.

[0135] In an optional implementation of the embodiment of the present invention, each compressed data matrix secondary block, each index matrix secondary block and each metadata matrix secondary block can be moved to the register to implement calculations. Alternatively, considering that register resources are very valuable, only each compressed data matrix secondary block and each metadata matrix secondary block can be moved to the register to implement calculations, while the index matrix can be retained in the shared memory and retrieved from the shared memory only when needed.

[0136] It should be emphasized that, unlike the computing logic of general hardware computing units, when using sparse computing units to accelerate multiplication calculations, the data to be calculated needs to be placed in an adapted instruction register in accordance with the requirements of the sparse instruction set. Therefore, each embodiment of the present invention needs to move the data in the shared memory to the matching hardware register based on the register mapping relationship that matches the sparse computing unit.

[0137] Optionally, the register mapping relationship can be read from a specification file of the factory configuration of the computing chip, and the register hardware file describes the mapping relationship between the computing data at different locations and the hardware registers.

[0138] Specifically, in Figure 7 A schematic diagram of a register mapping relationship applicable to an embodiment of the present invention is shown in FIG. 1 , which register mapping relationship is adapted to the secondary block division of the compressed data matrix. In a specific example, Figure 7 As shown, the T0{a0, a1} in the upper left corner of the register mapping relationship represents: the data in the 0th column and the 1st column of the 0th row in the secondary block of the compressed data matrix are allocated to registers a0 and a1 matching thread T0. Similarly, the T0 in the last position of row 0 in the register mapping relationship 0…3 {a4, a5} represents: the data of the 8th to 15th columns of the 0th row in the secondary block of the compressed data matrix are respectively allocated to register a4 and register a5 matching thread T0, thread T1, thread T2 and thread T3. Different threads T are allocated to different sparse computing units in advance.

[0139] Similarly, the sparse computing unit also has a corresponding register mapping relationship for the metadata matrix secondary blocking and the index matrix secondary blocking. Based on the above register mapping relationship, it is possible to move each calculation data from the shared memory to the hardware register one by one.

[0140] In an optional implementation of the embodiment of the present invention, a preset hardware instruction can be called according to a register mapping relationship matching the sparse computing unit to secondary block each compressed data matrix and secondary block each metadata matrix, or to move each compressed data matrix secondary block, each index matrix secondary block and each metadata matrix secondary block from a shared memory to a hardware register one by one, so as to realize hardware acceleration of data loading.

[0141] The hardware instruction is associated with the specification of the computing chip and can be queried and obtained from the instruction library of the computing chip. For example, it can be a 1dmatrix instruction.

[0142] S490 . Divide each second association matrix into blocks twice according to the register mapping relationship matched with the sparse computing unit, and load them from the shared memory into the hardware register one by one.

[0143] As shown above, by querying and obtaining the register mapping relationship set by the sparse computing unit for the second sparse matrix secondary block, each second sparse matrix secondary block can be loaded from the shared memory to the hardware register one by one.

[0144] Similarly, in an optional implementation of this embodiment, the preset hardware instructions can be called according to the register mapping relationship matching the sparse computing unit to divide each second sparse matrix into secondary blocks and load them one by one from the shared memory to the hardware register to achieve hardware acceleration of data loading.

[0145] It should be noted that, except for the first sparse operator, when the second and third sparse operators are transferred step by step from the global memory to the hardware registers of the computing chip in the form of data blocks, the data in the original global memory needs to be rearranged or deformed during the transfer process.

[0146] S4100. By using the sparse computing unit of the computing chip, the multiplication results of the first sparse matrix and the second association matrix are gradually calculated according to the data loaded in batches in the hardware register.

[0147] S4110. Process the multiplication result of the first sparse matrix and the second correlation matrix according to the processing logic of the corresponding functional function of the original hybrid expert model to obtain a sparse operator.

[0148] S4120. Update the original operator of the original hybrid expert model according to the sparse operator to obtain a target hybrid expert model.

[0149] Optionally, the multiplication result of the first sparse matrix and the second associated matrix can be processed according to the processing logic of the broadcast multiplication function of the original hybrid expert model to obtain a first sparse operator; the multiplication result of the first sparse matrix and the second associated matrix can be processed according to the processing logic of the activation function of the original hybrid expert model to obtain the second sparse operator; the multiplication result of the first sparse matrix and the second associated matrix can be processed according to the processing logic of the point product function of the original hybrid expert model to obtain a third sparse operator.

[0150] In an optional implementation of the embodiment of the present invention, by using a sparse computing unit of a computing chip, gradually calculating the multiplication result of the first sparse matrix and the second sparse matrix according to the data loaded in batches in the hardware register may include:

[0151] Obtaining, through the sparse computing unit, the current compressed data matrix secondary block, the current metadata matrix secondary block, and the current second association matrix secondary block currently loaded in the hardware register;

[0152] Generate a primary sparse block matrix according to the secondary block of the current metadata matrix and the secondary block of the current compressed data matrix through a sparse computing unit;

[0153] The multiplication calculation is performed by the sparse computing unit according to the primary sparse block matrix and the secondary block of the current second correlation matrix.

[0154] As mentioned above, when compressing and storing the first sparse matrix, the compressed data matrix only stores the non-zero data in each valid row, while in the actual first sparse matrix, each valid row contains at least one structured sparse computing storage unit in a set sparse mode. In order to ensure the accuracy of the multiplication calculation, it is necessary to first restore the current compressed data matrix secondary block to the form of the aforementioned structured sparse computing storage unit. Since the metadata matrix stores the position of the non-zero data in the structured sparse computing storage unit, a sparse block matrix can be generated based on the current metadata matrix secondary block and the current compressed data matrix secondary block.

[0155] That is, the primary sparse block matrix can be understood as a matrix after restoring the structured sparse computing storage unit in the secondary block of the current compressed data matrix. In a specific example, if the secondary block of the current compressed data matrix is ​​{A, B}, the corresponding secondary block of the current metadata matrix is ​​{0, 2}, and then, the primary sparse block matrix of the form {A, 0, B, 0} can be restored.

[0156] Among them, through the sparse computing unit, multiplication calculation is performed according to the first sparse block matrix and the current second association matrix secondary blocking, which can specifically include: through the sparse computing unit, the intermediate calculation amount generated in the multiplication calculation process is stored in the intermediate register of the computing chip; and through the sparse computing unit, by obtaining the intermediate calculation amount from the intermediate register, performing matching multiplication calculation.

[0157] When the prior art uses a general computing unit to perform multiplication calculations, the results of the matrix multiplication calculations can be directly accumulated, but this essentially adds a lot of redundant data calculations. In contrast, when the sparse computing units of the various embodiments of the present invention perform sparse multiplication calculations, when the multiplication calculations are iterated along the K direction, since the sparse data comes from different rows, in order to ensure correctness, when different sub-blocks move along the K direction, the output results need to be mapped to different rows. Traditionally, passing the output results to a specific register based on the index may cause the output C matrix to be reloaded into the local memory, which has a significant impact on the performance of the operator.

[0158] In order to avoid the above problems, each embodiment of the present invention introduces an additional intermediate register C IR The frequent memory transfer between global memory and registers is reduced by Figure 8 FIG. 2 shows a schematic diagram of a structure in which a sparse computing unit and an intermediate register cooperate to implement a calculation in accordance with an embodiment of the present invention. Figure 8 As shown, in the sparse computing unit, the current compressed data matrix is ​​divided into two blocks ( Figure 8 A in), the current metadata matrix is ​​divided into two blocks ( Figure 8 metadata in ) and the first second incidence matrix secondary block ( Figure 8 B) When performing sparse matrix multiplication calculations, for multiple result data, the required intermediate results are loaded into C during the calculation process. IR and use the intermediate results for calculation, and store the intermediate results back to the corresponding result data location C after the calculation is completed.

[0159] By adding a new intermediate register to the computing chip, which is a simple hardware improvement, the computing performance and efficiency can be greatly improved during the sparse matrix multiplication calculation of the sparse computing unit.

[0160] On the basis of the above embodiments, if the second association matrix includes a second sparse matrix, then after performing multiplication calculations through a sparse computing unit according to a primary sparse block matrix and a current second sparse matrix secondary block, it may also include: after determining that a complete multiplication calculation result is stored in an intermediate register, converting the complete multiplication calculation result into a secondary sparse block matrix according to a current index matrix secondary block that matches a current compressed data matrix secondary block; and writing the secondary sparse block matrix back to the global memory of the computing chip step by step.

[0161] If the second association matrix includes a dense matrix corresponding to the second sparse matrix, then after performing multiplication calculations through the sparse computing unit according to the first sparse block matrix and the second blocking of the current second association matrix, it may also include: after determining that the complete multiplication calculation result is stored in the intermediate register, writing the multiplication calculation result back to the global memory of the computing chip step by step in the original sparse format.

[0162] Further, since the sparse rows in the first sparse matrix are not considered when the complete multiplication result is calculated, if the right operand adopts the second sparse matrix, it is necessary to obtain the complete multiplication result again, and combine the current index matrix secondary block matching the current compressed data matrix secondary block, and add the matching sparse rows to the complete multiplication result to obtain the secondary sparse block matrix as the real block calculation result. As mentioned above, the current index matrix secondary block can also be moved to the hardware register, and can also be stored only in the shared memory, and this embodiment does not limit this. If the right operand adopts the dense matrix corresponding to the second sparse matrix, then after obtaining the complete multiplication result, the multiplication result can be directly written back to the global memory of the computing chip step by step according to the original sparse format, and there is no need to perform sparse processing on the multiplication result. As mentioned above, the multiplication result can be moved to the hardware register according to the original sparse format, and can also be stored only in the shared memory, and this embodiment does not limit this.

[0163] In summary, combined with the model architecture, matrix multiplication operations with sparse properties are selected from the hybrid expert model calculation flow. The sparsity comes from the sparsity of weights on the one hand, and the redundant properties in its parameters allow the calculation process to skip the calculation of parameters with lower importance; on the other hand, sparsity can also come from the hybrid expert model. The input value selects the expert parameter calculation that matches it through the gate circuit. From the expert's point of view, the input value has sparse characteristics, and only part of the input participates in the current expert's calculation. The selection of such sparse operators effectively improves the execution speed of the hybrid expert model without affecting the accuracy of the model. Based on the strategy of sparse data structure and operator fusion in data flow execution, referring to the compilation optimization scheme of sparse operators, specific sparse operators in the sparse operator library are selected and injected into the artificial intelligence framework and compiler to replace the original dense matrix multiplication operation of the hybrid expert model. For example, the two matrix multiplication operations that accept the original input in the expert architecture of the hybrid expert model will be injected with the operator of the double sparse mode, while the subsequent matrix multiplication that accepts the intermediate activation value will be injected with the unidirectional sparse (sparse × dense) operator.

[0164] The technical solution of the embodiment of the present invention fully considers the sparse characteristics when performing matrix multiplication calculations based on sparse matrices. By adopting data segmentation and data handling technology that are compatible with the sparse characteristics, the operator optimization of the hybrid expert model is achieved, and the hardware acceleration performance of the sparse computing unit in the computing chip is fully utilized in the model calculation scenario of the hybrid expert model, thereby greatly optimizing the computing, bandwidth and storage resource overhead in the sparse matrix multiplication process.

[0165] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data comply with relevant laws, regulations and standards in relevant regions.

[0166] It should be noted that any arrangement and combination of the technical features in the above embodiments also falls within the protection scope of the present invention.

[0167] Fig. 9 is a schematic diagram of an operator optimization device of a hybrid expert model provided by an embodiment of the present invention, such as Fig. 9 As shown, the device includes: a sparse operator generation module 910 and a sparse operator update module 920, wherein:

[0168] A sparse operator generation module 910 is used to generate a sparse operator suitable for the original hybrid expert model by combining the sparse computing unit of the computing chip in a data block manner; wherein the number of the sparse operators is multiple;

[0169] The sparse operator updating module 920 is used to update the original operator of the original hybrid expert model according to the sparse operator to obtain a target hybrid expert model.

[0170] The embodiment of the present invention generates multiple sparse operators suitable for the original hybrid expert model by combining the sparse computing unit of the computing chip in a data blocking manner, thereby updating the original operators of the original hybrid expert model according to the generated sparse operators to obtain the target hybrid expert model. The sparse characteristics of the hybrid expert model when performing matrix multiplication calculations based on sparse matrices are fully considered. By adopting data blocking and data handling technologies that are compatible with the sparse characteristics, the operator optimization of the hybrid expert model is achieved. In the model calculation scenario of the hybrid expert model, the hardware acceleration performance of the sparse computing unit in the computing chip is fully utilized, thereby greatly optimizing the calculation, bandwidth and storage resource overhead in the sparse matrix multiplication operation process.

[0171] Optionally, the sparse operator includes a first sparse operator, and the sparse operator generation module 910 is further used to: locate a set of compressed matrices and a second associated matrix matching the first sparse matrix in the global memory of the computing chip according to the first sparse matrix multiplication requirement; wherein the second associated matrix includes a second sparse matrix or a dense matrix corresponding to the second sparse matrix; wherein the first sparse matrix includes a plurality of compression units of a set size, each of the compression units includes at least one valid row and at least one sparse row, and each of the valid rows includes at least one structured sparse computing storage unit; each of the structured sparse computing storage units is in the same sparse mode; the compressed matrix set includes a compression unit for storing non-zero data in the first sparse matrix A compressed data matrix, an index matrix for storing the positions of valid rows in the first sparse matrix in the corresponding compression unit, and a metadata matrix for storing the positions of non-zero data in the corresponding structured sparse computing storage unit; the compressed matrix set and the second association matrix are moved step by step from the global memory to the hardware register of the computing chip in the form of data blocks; the multiplication results of the first sparse matrix and the second association matrix are gradually calculated by the sparse computing unit of the computing chip according to the data loaded in batches in the hardware register; the multiplication results of the first sparse matrix and the second association matrix are processed according to the processing logic of the broadcast multiplication function of the original hybrid expert model to obtain the first sparse operator.

[0172] Optionally, the sparse operator includes a second sparse operator, and the sparse operator generation module 910 is also used to: locate a compression matrix set and a second association matrix matching the first sparse matrix in the global memory of the computing chip according to the second sparse matrix multiplication requirement; wherein the second association matrix includes the second sparse matrix or a dense matrix corresponding to the second sparse matrix; wherein the first sparse matrix includes a plurality of compression units of a set size, each of the compression units includes at least one valid row and at least one sparse row, each of the valid rows includes at least one structured sparse computing storage unit; each of the structured sparse computing storage units is in the same sparse mode; the compression matrix set includes a compressed data matrix for storing non-zero data in the first sparse matrix, a compressed data matrix for storing An index matrix for storing the positions of valid rows in the first sparse matrix in the corresponding compression unit, and a metadata matrix for storing the positions of non-zero data in the corresponding structured sparse computing storage unit; the compression matrix set and the second association matrix are transferred step by step from the global memory to the hardware register of the computing chip in the form of data blocks, and during the transfer process, the data in the original global memory is rearranged or deformed; through the sparse computing unit of the computing chip, the multiplication results of the first sparse matrix and the second association matrix are gradually calculated according to the data loaded in batches in the hardware register; the multiplication results of the first sparse matrix and the second association matrix are processed according to the processing logic of the activation function of the original hybrid expert model to obtain the second sparse operator.

[0173] Optionally, the sparse operator includes a third sparse operator, and the sparse operator generation module 910 is also used to: locate a compression matrix set and a second association matrix matching the first sparse matrix in the global memory of the computing chip according to the third sparse matrix multiplication requirement; wherein the second association matrix includes a second sparse matrix or a dense matrix corresponding to the second sparse matrix; wherein the first sparse matrix includes a plurality of compression units of a set size, each of the compression units includes at least one valid row and at least one sparse row, and each of the valid rows includes at least one structured sparse computing storage unit; each of the structured sparse computing storage units is in the same sparse mode; the compression matrix set includes a compressed data matrix for storing non-zero data in the first sparse matrix, a compressed data matrix for storing An index matrix of the positions of valid rows in the first sparse matrix in the corresponding compression unit, and a metadata matrix for storing the positions of non-zero data in the corresponding structured sparse computing storage unit; the compression matrix set and the second association matrix are moved step by step from the global memory to the hardware register of the computing chip in the form of data blocks, and during the movement, the data in the original global memory is rearranged or deformed; through the sparse computing unit of the computing chip, the multiplication results of the first sparse matrix and the second association matrix are gradually calculated according to the data loaded in batches in the hardware register; the multiplication results of the first sparse matrix and the second association matrix are processed according to the processing logic of the point multiplication function of the original hybrid expert model to obtain the third sparse operator.

[0174] Optionally, the sparse operator generation module 910 is also used to: partition the compressed data matrix in the global memory into data blocks, and partition the index matrix and the metadata matrix into matching data blocks according to the data correspondence between the compressed data matrix, the index matrix and the metadata matrix; load each compressed data matrix block, each index matrix block and each metadata matrix block from the global memory to the shared memory one by one; perform secondary partitioning on each compressed data matrix block, each index matrix block and each metadata matrix block in the shared memory; and according to the register mapping relationship matching the sparse computing unit, move at least one of each compressed data matrix secondary block, each index matrix secondary block and each metadata matrix secondary block from the shared memory to the hardware register one by one.

[0175] Optionally, the sparse operator generation module 910 is also used to: locate a selection array matching the second association matrix in the computing chip, wherein the selection array is used to describe the valid row positions in the second association matrix; the selection array of the second sparse matrix is ​​the index of the non-zero data positions therein, and the selection array of the dense matrix corresponding to the second sparse matrix is ​​the position mapping of the data in the dense matrix in the second sparse matrix; according to the selection array, block the data of the second association matrix in the global memory; move each second association matrix block from the global memory to the shared memory one by one; perform secondary blocking on each second association matrix block in the shared memory; and according to the register mapping relationship matching the sparse computing unit, load each second association matrix secondary block from the shared memory to the hardware register one by one.

[0176] Optionally, the sparse operator generation module 910 is also used to: use a preset data rearrangement function to rearrange each of the compressed data matrix blocks, each of the index matrix blocks, and each of the metadata matrix blocks in the shared memory to avoid storage conflicts; use the preset data rearrangement function to rearrange each of the second association matrix blocks in the shared memory to avoid storage conflicts.

[0177] Optionally, the sparse operator generation module 910 is also used to: call preset hardware instructions according to the register mapping relationship matching the sparse computing unit, and move each of the compressed data matrix secondary blocks and each of the metadata matrix secondary blocks, or, move each of the compressed data matrix secondary blocks, each of the index matrix secondary blocks and each of the metadata matrix secondary blocks from the shared memory to the hardware register one by one to achieve hardware acceleration of data loading; call preset hardware instructions according to the register mapping relationship matching the sparse computing unit, and load each of the second association matrix secondary blocks from the shared memory to the hardware register one by one to achieve hardware acceleration of data loading.

[0178] Optionally, the sparse operator generation module 910 is also used to: obtain, through the sparse computing unit, the current compressed data matrix secondary block, the current metadata matrix secondary block, and the current second association matrix secondary block currently loaded in the hardware register; generate, through the sparse computing unit, a first sparse block matrix according to the current metadata matrix secondary block and the current compressed data matrix secondary block; and perform multiplication calculations according to the first sparse block matrix and the current second association matrix secondary block.

[0179] Optionally, the sparse operator generation module 910 is also used to: store the intermediate calculation amount generated during the multiplication calculation process in the intermediate register of the computing chip through the sparse computing unit; and, through the sparse computing unit, perform matching multiplication calculations by obtaining the intermediate calculation amount from the intermediate register.

[0180] Optionally, the sparse operator generation module 910 is also used to: if the second associative matrix includes a second sparse matrix, then after determining that the complete multiplication calculation result is stored in the intermediate register, convert the complete multiplication calculation result into a secondary sparse block matrix according to the secondary blocking of the current index matrix that matches the secondary blocking of the current compressed data matrix; write the secondary sparse block matrix back to the global memory of the computing chip step by step; if the second associative matrix includes a dense matrix corresponding to the second sparse matrix, then after determining that the complete multiplication calculation result is stored in the intermediate register, write the multiplication calculation result back to the global memory of the computing chip step by step in the original sparse format.

[0181] Optionally, the first sparse matrix specifically includes multiple compression units of size M*V; each of the compression units specifically includes d valid rows, where 1≤d<M; and each of the structured sparse computing storage units is in a sparse mode of N:L, L is the total number of data contained in the structured sparse computing storage unit, and N is the number of non-zero data contained in the structured sparse computing storage unit.

[0182] Optionally, in each data segmentation process, V is an integer multiple of the segmentation size in the column segmentation direction; and, in each of the sparse operators, the matrix specification of the first sparse matrix processed once is mi*ki, where V is an integer multiple of ki.

[0183] Optionally, the first sparse matrix is ​​the sparse model weight matrix in the target hybrid expert model, the second sparse matrix is ​​the sparse matrix of intermediate activation values ​​input to the hybrid model layer in the target hybrid expert model, and the dense matrix corresponding to the second sparse matrix is ​​the dense matrix of intermediate activation values ​​input to the hybrid model layer in the target hybrid expert model.

[0184] The above-mentioned hybrid expert model operator optimization device can execute the hybrid expert model operator optimization method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. For technical details not described in detail in this embodiment, please refer to the hybrid expert model operator optimization method provided by any embodiment of the present invention.

[0185] Since the operator optimization device of the hybrid expert model introduced above is a device that can execute the operator optimization method of the hybrid expert model in the embodiment of the present invention, based on the operator optimization method of the hybrid expert model introduced in the embodiment of the present invention, the technical personnel of the field can understand the specific implementation of the operator optimization device of the hybrid expert model of this embodiment and its various variations, so how the operator optimization device of the hybrid expert model implements the operator optimization method of the hybrid expert model in the embodiment of the present invention is not introduced in detail here. As long as the technical personnel of the field implement the device used by the operator optimization method of the hybrid expert model in the embodiment of the present invention, it belongs to the scope of protection of this application.

[0186] Fig.10 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0187] like Fig.10 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0188] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0189] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the operator optimization method of the hybrid expert model.

[0190] Optionally, the operator optimization method of the hybrid expert model may include: using data blocking in combination with a sparse computing unit of a computing chip to generate a sparse operator suitable for the original hybrid expert model; wherein the number of the sparse operators is multiple; and updating the original operator of the original hybrid expert model according to the sparse operator to obtain a target hybrid expert model.

[0191] In some embodiments, the operator optimization method of the hybrid expert model may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the operator optimization method of the hybrid expert model described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the operator optimization method of the hybrid expert model in any other appropriate manner (e.g., by means of firmware).

[0192] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0193] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0194] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0195] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0196] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0197] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0198] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0199] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An operator optimization method of a hybrid expert model, characterized in that: include: A sparse operator suitable for the original hybrid expert model is generated by combining data block with a sparse computing unit of a computing chip; wherein the number of the sparse operators is multiple; The original operator of the original hybrid expert model is updated according to the sparse operator to obtain a target hybrid expert model.

2. The method according to claim 1, characterized in that The sparse operator includes a first sparse operator, and the sparse computing unit of the computing chip is combined with the data block method to generate a sparse operator suitable for the original hybrid expert model, including: According to the first sparse matrix multiplication requirement, locate a set of compressed matrices matching the first sparse matrix and a second association matrix in the global memory of the computing chip; wherein the second association matrix includes the second sparse matrix or a dense matrix corresponding to the second sparse matrix; Wherein, the first sparse matrix comprises a plurality of compression units of set size, each of the compression units comprises at least one valid row and at least one sparse row, each of the valid row comprises at least one structured sparse computing storage unit; each of the structured sparse computing storage units is in the same sparse mode; the compression matrix set comprises a compression data matrix for storing non-zero data in the first sparse matrix, an index matrix for storing the position of the valid row in the first sparse matrix in the corresponding compression unit, and a metadata matrix for storing the position of the non-zero data in the corresponding structured sparse computing storage unit; The compressed matrix set and the second association matrix are transferred from the global memory to the hardware register of the computing chip in the form of data blocks step by step; The sparse computing unit of the computing chip gradually calculates the multiplication result of the first sparse matrix and the second association matrix according to the data loaded in batches in the hardware register; The multiplication result of the first sparse matrix and the second association matrix is ​​processed according to the processing logic of the broadcast multiplication function of the original hybrid expert model to obtain the first sparse operator.

3. The method according to claim 1, characterized in that: The sparse operator includes a second sparse operator, and the sparse computing unit of the computing chip is combined with the data block method to generate a sparse operator suitable for the original hybrid expert model, including: According to the second sparse matrix multiplication requirement, locate a set of compressed matrices and a second association matrix matching the first sparse matrix in the global memory of the computing chip; wherein the second association matrix includes the second sparse matrix or a dense matrix corresponding to the second sparse matrix; Wherein, the first sparse matrix comprises a plurality of compression units of set size, each of the compression units comprises at least one valid row and at least one sparse row, each of the valid row comprises at least one structured sparse computing storage unit; each of the structured sparse computing storage units is in the same sparse mode; the compression matrix set comprises a compression data matrix for storing non-zero data in the first sparse matrix, an index matrix for storing the position of the valid row in the first sparse matrix in the corresponding compression unit, and a metadata matrix for storing the position of the non-zero data in the corresponding structured sparse computing storage unit; The compressed matrix set and the second association matrix are transferred from the global memory to the hardware register of the computing chip in the form of data blocks, and during the transfer process, the data in the original global memory is rearranged or deformed; The sparse computing unit of the computing chip gradually calculates the multiplication result of the first sparse matrix and the second association matrix according to the data loaded in batches in the hardware register; The multiplication result of the first sparse matrix and the second association matrix is ​​processed according to the processing logic of the activation function of the original hybrid expert model to obtain the second sparse operator.

4. The method according to claim 1, characterized in that: The sparse operator includes a third sparse operator, and the sparse computing unit of the computing chip is combined with the data block method to generate a sparse operator suitable for the original hybrid expert model, including: According to the third sparse matrix multiplication requirement, locate a set of compressed matrices matching the first sparse matrix and a second association matrix in the global memory of the computing chip; wherein the second association matrix includes the second sparse matrix or a dense matrix corresponding to the second sparse matrix; Wherein, the first sparse matrix comprises a plurality of compression units of set size, each of the compression units comprises at least one valid row and at least one sparse row, each of the valid row comprises at least one structured sparse computing storage unit; each of the structured sparse computing storage units is in the same sparse mode; the compression matrix set comprises a compression data matrix for storing non-zero data in the first sparse matrix, an index matrix for storing the position of the valid row in the first sparse matrix in the corresponding compression unit, and a metadata matrix for storing the position of the non-zero data in the corresponding structured sparse computing storage unit; The compressed matrix set and the second association matrix are transferred from the global memory to the hardware register of the computing chip in the form of data blocks, and during the transfer process, the data in the original global memory is rearranged or deformed; The sparse computing unit of the computing chip gradually calculates the multiplication result of the first sparse matrix and the second association matrix according to the data loaded in batches in the hardware register; The multiplication result of the first sparse matrix and the second correlation matrix is ​​processed according to the processing logic of the point multiplication function of the original hybrid expert model to obtain the third sparse operator.

5. The method according to any one of claims 2 to 4, characterized in that: The compressed matrix set is transferred from the global memory to the hardware register of the computing chip in the form of data blocks, including: The compressed data matrix in the global memory is divided into data blocks, and the index matrix and the metadata matrix are matched into data blocks according to the data correspondence between the compressed data matrix, the index matrix and the metadata matrix; Loading each compressed data matrix block, each index matrix block, and each metadata matrix block from the global memory to the shared memory one by one; Secondarily partitioning each of the compressed data matrix blocks, each of the index matrix blocks, and each of the metadata matrix blocks in the shared memory; According to the register mapping relationship matching the sparse computing unit, at least one item of each of the compressed data matrix secondary blocks, each of the index matrix secondary blocks and each of the metadata matrix secondary blocks is moved from the shared memory to the hardware register one by one.

6. The method according to claim 5, characterized in that While locating the set of compressed matrices matching the first sparse matrix and the second association matrix in the global memory of the computing chip, the method further includes: Locating a selection array matching the second incidence matrix in the computing chip, wherein the selection array is used to describe valid row positions in the second incidence matrix; the selection array of the second sparse matrix is ​​the index of the non-zero data position therein, and the selection array of the dense matrix corresponding to the second sparse matrix is ​​the position mapping of the data in the dense matrix in the second sparse matrix; Accordingly, the second association matrix is ​​moved from the global memory to the hardware register of the computing chip in the form of data blocks step by step, including: According to the selection array, dividing the second association matrix in the global memory into data blocks; Divide each second association matrix into blocks, and transfer them from the global memory to the shared memory one by one; Perform secondary block division on each of the second association matrix blocks in the shared memory; According to the register mapping relationship matching the sparse computing unit, each second association matrix is ​​divided into two blocks and loaded from the shared memory to the hardware register one by one.

7. The method according to claim 6, characterized in that After loading each compressed data matrix block, each index matrix block, and each metadata matrix block from the global memory to the shared memory one by one, the method further includes: Using a preset data rearrangement function, rearrange the compressed data matrix blocks, the index matrix blocks, and the metadata matrix blocks in the shared memory to avoid storage bank conflicts; and After dividing each second correlation matrix into blocks and transferring them from the global memory to the shared memory one by one, the method further includes: The preset data rearrangement function is used to rearrange each of the second association matrix blocks in the shared memory to avoid storage bank conflicts.

8. The method according to claim 6, characterized in that According to the register mapping relationship matched with the sparse computing unit, at least one item of each of the compressed data matrix secondary blocks, each of the index matrix secondary blocks, and each of the metadata matrix secondary blocks is moved from the shared memory to the hardware register one by one, including: According to the register mapping relationship matched with the sparse computing unit, calling the preset hardware instruction, each of the compressed data matrix secondary blocks and each of the metadata matrix secondary blocks, or each of the compressed data matrix secondary blocks, each of the index matrix secondary blocks and each of the metadata matrix secondary blocks, are moved from the shared memory to the hardware register one by one, so as to realize the hardware acceleration of data loading; and According to the register mapping relationship matching the sparse computing unit, each second association matrix is ​​divided into blocks twice, and loaded from the shared memory to the hardware register one by one, including: According to the register mapping relationship matching the sparse computing unit, the preset hardware instructions are called to divide each of the second association matrices into blocks twice, and load them from the shared memory to the hardware registers one by one to achieve hardware acceleration of data loading.

9. The method according to claim 6, characterized in that The multiplication result of the first sparse matrix and the second association matrix is ​​calculated step by step according to the data loaded in batches in the hardware register by the sparse computing unit of the computing chip, including: Obtaining, by means of the sparse computing unit, the current compressed data matrix secondary block, the current metadata matrix secondary block, and the current second association matrix secondary block currently loaded in the hardware register; Generate a primary sparse block matrix according to the secondary block of the current metadata matrix and the secondary block of the current compressed data matrix by the sparse computing unit; The sparse computing unit performs multiplication calculation according to the primary sparse block matrix and the secondary block of the current second correlation matrix.

10. The method according to claim 9, characterized in that The sparse computing unit performs a multiplication calculation according to the primary sparse block matrix and the secondary block of the current second correlation matrix, specifically including: The intermediate calculation amount generated in the multiplication calculation process is stored in the intermediate register of the computing chip through the sparse computing unit; and The sparse computing unit performs a matching multiplication calculation by acquiring the intermediate calculation amount from the intermediate register.

11. The method according to claim 10, characterized in that If the second incidence matrix includes a second sparse matrix, after performing multiplication calculation according to the first sparse block matrix and the second current incidence matrix second block by the sparse computing unit, the method further includes: After determining that the intermediate register stores a complete multiplication calculation result, converting the complete multiplication calculation result into a secondary sparse block matrix according to a current index matrix secondary block matching the current compressed data matrix secondary block; Writing the secondary sparse block matrix back to the global memory of the computing chip step by step; If the second incidence matrix includes a dense matrix corresponding to the second sparse matrix, after performing multiplication calculation according to the first sparse block matrix and the second second incidence matrix second block by the sparse computing unit, the method further includes: After determining that the complete multiplication calculation result is stored in the intermediate register, the multiplication calculation result is written back to the global memory of the computing chip step by step according to the original sparse format.

12. The method according to any one of claims 2 to 4, characterized in that: The first sparse matrix specifically includes a plurality of compression units of size M*V; Specifically, each compression unit includes d valid rows, wherein 1≤d<M; and Each of the structured sparse computing storage units is in a sparse mode of N:L, where L is the total amount of data contained in the structured sparse computing storage unit, and N is the amount of non-zero data contained in the structured sparse computing storage unit.

13. The method according to claim 12, characterized in that In each data segmentation process, V is an integer multiple of the segmentation size in the column segmentation direction; and The matrix specification of the first sparse matrix processed once in each of the sparse operators is mi*ki, where V is an integer multiple of ki.

14. The method according to any one of claims 2 to 4, characterized in that: The first sparse matrix is ​​the sparse model weight matrix in the target hybrid expert model, the second sparse matrix is ​​the sparse matrix of intermediate activation values ​​input to the hybrid model layer in the target hybrid expert model, and the dense matrix corresponding to the second sparse matrix is ​​the dense matrix of intermediate activation values ​​input to the hybrid model layer in the target hybrid expert model.

15. An operator optimization device for a hybrid expert model, characterized in that: include: A sparse operator generation module, used to generate a sparse operator suitable for the original hybrid expert model by combining the sparse computing unit of the computing chip in a data block manner; wherein the number of the sparse operators is multiple; The sparse operator updating module is used to update the original operator of the original hybrid expert model according to the sparse operator to obtain a target hybrid expert model.

16. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the operator optimization method of the hybrid expert model described in any one of claims 1-14.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the operator optimization method of the hybrid expert model described in any one of claims 1 to 14 when executed.

18. A computer program product comprising a computer program / instructions, wherein: When the computer program / instructions are executed by a processor, the operator optimization method of the hybrid expert model described in any one of claims 1 to 14 is implemented.

Images