Optimization method and device of hybrid expert model, equipment, medium and program

Abstract

The embodiment of the invention discloses a hybrid expert model optimization method, device, equipment, medium and program, and the method comprises the steps: carrying out the sparse conversion of each sparse data structure of a target hybrid expert model, and obtaining a structured sparse data structure; performing data arrangement optimization on the structured sparse data structure; optimizing an operator of the target hybrid expert model in combination with a sparse calculation unit of a calculation chip to obtain a sparse operator; and updating an original operator of the target hybrid expert model according to the sparse operator to obtain a structured sparse hybrid expert model. According to the technical scheme provided by the embodiment of the invention, the structured sparse optimization of the hybrid expert model can be realized, so that the acceleration performance of sparse calculation hardware of the hybrid expert model during the calculation of sparse matrix multiplication can be fully exerted, and the overhead of calculation, bandwidth and storage resources in the sparse matrix multiplication operation process can be greatly optimized.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of artificial intelligence technology, and in particular to an optimization method, device, electronic device, storage medium and program for a hybrid expert model. Background Art

[0002] As LLMs (Large Language Models, or LLMs for short) have continued to evolve, the size of model parameters and computational requirements have grown significantly. Newly derived LLMs now have up to 400 billion model parameters, a significant increase from the 110 million parameters of earlier models. This rapid development poses significant challenges to deploying these LLMs within existing AI infrastructure, especially given the limitations imposed by the rate of hardware advancement.

[0003] The new LLM model built on the Mixture-of-Experts (MoE) layer has been widely integrated into emerging large language models due to its enhanced generalization ability and ability to effectively manage multimodal tasks. This architectural innovation places unique demands on storage, bandwidth, and computing resources. Solving the problems of large language model size growth and new architectures is critical to their effective deployment on contemporary AI accelerators.

[0004] In the process of implementing the present invention, the inventors found that in response to the challenges of the new computing load mentioned above, the current cutting-edge work is mainly based on unstructured sparse computing technology for performance optimization and acceleration. However, unstructured sparse computing technology lacks hardware-level instruction support on artificial intelligence acceleration devices such as GPUs (Graphics Processing Units), such as the lack of dedicated sparse digital arithmetic units (sparse ALU) support, and therefore cannot bring much performance improvement. The current hardware-level sparse digital arithmetic unit function mainly supports uniformly proportioned structured sparse technology, and the matrix multiplication operator for double sparse mode data (that is, the left and right operands of the operator are both sparse data) is not implemented. Therefore, directly using the existing solution cannot perform efficient matrix multiplication calculations on double sparse mode data. Summary of the invention

[0005] The embodiments of the present invention provide a hybrid expert model optimization method, device, electronic device, storage medium and program, which can realize structured sparse optimization of the hybrid expert model, so as to give full play to the acceleration performance of the sparse computing hardware of the hybrid expert model when calculating sparse matrix multiplication, thereby greatly optimizing the calculation, bandwidth and storage resource overhead in the sparse matrix multiplication operation process.

[0006] According to one aspect of the present invention, there is provided a method for optimizing a hybrid expert model, comprising:

[0007] Perform sparse transformation on each sparse data structure of the target hybrid expert model to obtain a structured sparse data structure;

[0008] Optimizing data arrangement of the structured sparse data structure;

[0009] Optimizing the operator of the target hybrid expert model in combination with a sparse computing unit of a computing chip to obtain a sparse operator;

[0010] The original operator of the target hybrid expert model is updated according to the sparse operator to obtain a structured sparse hybrid expert model.

[0011] According to another aspect of the present invention, there is provided an optimization device for a hybrid expert model, comprising:

[0012] The data structure sparse conversion module is used to perform sparse conversion on the sparse data structures of the target hybrid expert model to obtain a structured sparse data structure;

[0013] A structured sparse data structure optimization module, used for optimizing the data arrangement of the structured sparse data structure;

[0014] A sparse operator optimization module, used to optimize the operator of the target hybrid expert model in combination with the sparse computing unit of the computing chip to obtain a sparse operator;

[0015] The sparse operator updating module is used to update the original operator of the target hybrid expert model according to the sparse operator to obtain a structured sparse hybrid expert model.

[0016] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the optimization method of the hybrid expert model described in any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the optimization method of the hybrid expert model described in any embodiment of the present invention when executed.

[0021] According to another aspect of the present invention, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the method for optimizing the hybrid expert model described in any embodiment of the present invention is implemented.

[0022] The embodiment of the present invention performs sparse conversion on the sparse data structures of the target hybrid expert model to obtain a structured sparse data structure, and optimizes the data arrangement of the structured sparse data structure. Furthermore, the operator of the target hybrid expert model is optimized in combination with the sparse computing unit of the computing chip to obtain a sparse operator, and then the original operator of the target hybrid expert model is updated according to the sparse operator to obtain a structured sparse hybrid expert model. The above technical scheme can solve the problem that the performance of the existing hybrid expert model is difficult to effectively improve when the performance is optimized and accelerated based on the unstructured sparse computing technology, and can realize the structured sparse optimization of the hybrid expert model to give full play to the acceleration performance of the sparse computing hardware of the hybrid expert model when calculating sparse matrix multiplication, thereby greatly optimizing the calculation, bandwidth and storage resource overhead in the sparse matrix multiplication process.

[0023] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 is a flow chart of an optimization method of a hybrid expert model provided by an embodiment of the present invention;

[0026] Figure 2 It is a schematic diagram of a flow chart of structural optimization processing of a hybrid expert model provided by an embodiment of the present invention;

[0027] Figure 3 is a flow chart of another hybrid expert model optimization method provided by an embodiment of the present invention;

[0028] Figure 4 It is a schematic diagram of a comparison structure of a model weight matrix and a matching compression matrix set applicable to an embodiment of the present invention;

[0029] Figure 5It is a comparison diagram of a sparse computing unit requirement and a matching data re-arrangement result applicable to an embodiment of the present invention;

[0030] Figure 6 It is a schematic diagram of a comparison structure of an intermediate activation value sparse matrix and a matching selection array applicable to an embodiment of the present invention;

[0031] Figure 7 is a flow chart of another hybrid expert model optimization method provided by an embodiment of the present invention;

[0032] Figure 8 It is a schematic diagram of a process for optimizing the original operator of the hybrid expert model provided by an embodiment of the present invention;

[0033] Fig. 9 A flowchart of another hybrid expert model optimization method provided by an embodiment of the present invention;

[0034] Fig.10 It is a block diagram of a data segmentation and multi-stage transfer process applicable to an embodiment of the present invention;

[0035] Fig.11 is a schematic diagram of a register mapping relationship applicable to an embodiment of the present invention;

[0036] Fig.12 It is a structural diagram of a sparse computing unit and an intermediate register used in an embodiment of the present invention to cooperate in computing;

[0037] Fig.13 is a schematic diagram of an optimization device for a hybrid expert model provided by an embodiment of the present invention;

[0038] Fig.14 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0040] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0041] Figure 1 is a flow chart of an optimization method of a hybrid expert model provided by an embodiment of the present invention, Figure 2 The present invention provides a flow chart of a hybrid expert model structure optimization process. The present invention is applicable to the case of hybrid expert model structure optimization. The method can be executed by a hybrid expert model optimization device. The device can be implemented by software and / or hardware and can generally be integrated in an electronic device. The electronic device can be a terminal device or a server device. As long as the hybrid expert model optimization method can be executed, the present invention does not limit the specific device type of the electronic device. Figure 1 As shown, the method includes the following operations:

[0042] S110, performing sparse conversion on each sparse data structure of the target hybrid expert model to obtain a structured sparse data structure.

[0043] Among them, the target hybrid expert model refers to a specific hybrid expert model used to load into the computing chip to implement model reasoning tasks. The hybrid expert model is an efficient neural network architecture that improves overall performance by combining multiple specialized sub-models. The core idea of ​​this model is to integrate different "expert" networks together, with each expert playing a role in his or her area of ​​expertise. These experts can be small multi-layer perceptrons (MLP) or more complex large language models (LLM). The model usually contains a gating network that is responsible for deciding which expert should be activated and involved in the calculation of the output when processing a specific input.

[0044] The structured sparse data structure may be a data structure in a structured sparse data format suitable for a target hybrid expert model.

[0045] The representation and arrangement optimization technology of sparse data is a technical solution that can effectively optimize the computing, bandwidth and storage resource overhead during model training, reasoning and fine-tuning. In the prior art, when performing model calculation, the hybrid expert model mainly involves sparsely compressing the sparse weight data to be calculated and the input sparse intermediate activation data through a specified sparse representation format, and arranging them in a specified storage format in the memory for subsequent calculation. In the process of implementing the present invention, the inventor found that the calculation between the sparse intermediate activation data and the sparse weight data belongs to double sparse calculation, and the current mainstream sparse compression and arrangement optimization technology lacks efficient representation of the double sparse features in the calculation process of the hybrid expert model. Therefore, directly using the existing sparse compression and arrangement optimization scheme cannot fully optimize the calculation process of the hybrid expert model. Based on this, the embodiment of the present invention creatively proposes an efficient data compression method for the sparse model weight matrix in the hybrid expert model to effectively optimize the storage resource overhead of the hybrid expert model.

[0046] like Figure 2 As shown, when the sparse data structures of the target hybrid expert model are sparsely converted, the model weights of the target hybrid expert model can be sparsely converted to obtain a structured sparse data structure of the model weights, and the sparse intermediate activation data input to the target hybrid expert model can also be sparsely converted to obtain a structured sparse data structure of the activation value.

[0047] By converting the sparse model weight matrices of each specific format in the target hybrid expert model into structured sparse data structures of model weights, and optimizing the storage in the computing chip used to implement the hybrid expert model calculation, the storage resource overhead of the computing chip can be effectively reduced. In addition, when using the model weights in the form of structured sparse data structures for model calculations, the sparse computing hardware in the computing chip can be fully utilized, which effectively improves the computing efficiency while reducing the bandwidth overhead.

[0048] S120: Optimize data arrangement of the structured sparse data structure.

[0049] After completing the sparse conversion of various sparse data structures of the target hybrid expert model, the data arrangement of the structured sparse data structure can be further optimized to further improve the computational efficiency of the hybrid expert model.

[0050] In an optional embodiment of the present invention, the data arrangement optimization of the structured sparse data structure may include: before the target hybrid expert model is run, storing the sparse converted model weight data of the target hybrid expert model in a transposed manner; when the sparse operator of the target hybrid expert model performs data calculations in a shared memory, performing a data transposition operation on the intermediate activation value.

[0051] In traditional hybrid expert model calculations, the matrix multiplication is calculated as: xW, where x is the input and W is the model weight. To meet the hardware requirements of the acceleration unit, the model calculation process will be restructured into . From the perspective of continuous memory access and transposition overhead, the embodiment of the present invention optimizes and updates the data arrangement. Specifically, before the target hybrid expert model is run, the sparse converted model weight data of the target hybrid expert model is stored in a transposed manner, that is, the transposition operation of the model weight W is performed during the offline model pruning process, and no additional overhead is introduced during the operation stage. In order to make full use of the bandwidth from global memory to shared memory, when the sparse operator of the target hybrid expert model performs data calculations in the shared memory, a data transposition operation is performed on the intermediate activation value, that is, the transposition of the input value x is completed at the shared memory level. Specifically, the data transposition of the last layer is completed using the fast instructions provided by the GPU hardware.

[0052] S130, optimizing the operator of the target hybrid expert model in combination with the sparse computing unit of the computing chip to obtain a sparse operator.

[0053] The sparse computing unit may be a hardware computing unit in a computing chip. The original hybrid expert model may be a hybrid expert model that requires optimization of internal operators. The embodiment of the present invention does not limit the model type and model structure of the original hybrid expert model.

[0054] Optionally, the computing chip can be understood as an integrated circuit used to implement a set computing task (for example, Internet of Things control, high-performance computing or mobile computing, etc.). The computing chip can be a general computing chip, such as a CPU (Central Processing Unit) or a GPU, or a heterogeneous chip combination of "CPU + GPU equipped with a sparse acceleration computing unit", or a dedicated computing chip, such as an NPU (Neural Processing Unit), which is not limited in the embodiments of the present invention.

[0055] Among them, sparse operators refer to operators that only operate on some elements, such as sparse matrices in matrix multiplication. In the calculation process, the processing of sparse operators usually involves how to effectively store and calculate sparse matrices, and how to optimize the computing performance of sparse operators. Sparse operators are mainly used to process sparse matrices, in which most elements are zero and only a small number of non-zero elements. Sparse operators can reduce the use of storage space through compressed storage methods, use algorithms such as fast Fourier transform to accelerate operations such as sparse matrix multiplication, and use memory optimization techniques (such as cache optimization and memory alignment) to improve memory usage efficiency. Parallel computing technologies (such as multi-threading or distributed computing) can also be used to accelerate operations on large-scale sparse matrices.

[0056] The current hybrid expert computing technology mainly focuses on using sparse computing technology to solve the parameter redundancy problem, while ignoring the sparse characteristics of input data. In addition, there is a lack of computing acceleration using hardware instructions to provide structured sparseness in hybrid expert computing.

[0057] In an embodiment of the present invention, in order to achieve computational acceleration of an adaptive sparse acceleration hardware computing unit and fully consider the sparse characteristics of the operating data of the sparse operator, a data block method can be used in combination with the sparse computing unit of the computing chip to optimize the computational, bandwidth and storage resource overhead during the execution of the operator, and through compilation optimization methods such as operator fusion, various types of sparse operators suitable for the target hybrid expert model are generated.

[0058] S140: Update the original operator of the target hybrid expert model according to the sparse operator to obtain a structured sparse hybrid expert model.

[0059] The original operator may be the original operator type of the target hybrid expert model before structural optimization.

[0060] It is understandable that the original operator of the target hybrid expert model is usually a matrix multiplication operator. The matrix multiplication operator is usually performed as a dense matrix multiplication operation. Accordingly, after generating various types of sparse operators suitable for the target hybrid expert model, the type and function of the original operator of the target hybrid expert model can be considered, and the original operator can be updated and replaced with an adapted sparse operator to achieve operator optimization and structural update of the target hybrid expert model, and obtain a target hybrid expert model that can be accelerated by hardware instructions. The optimized and updated target hybrid expert model is a structured sparse hybrid expert model.

[0061] Correspondingly, the optimized target hybrid expert model can take the sparse matrix of the structured sparse data structure as input, optimize the data flow of the sparse operator in this format, and at the same time arrange the calculation process of the calculation kernel, and use the sparse computing units in the hardware-level computing chip to optimize and accelerate the sparse operators, and finally inject them into the hybrid expert calculation process, thereby accelerating the execution of the hybrid expert model.

[0062] It can be seen that the operator optimization method of the above hybrid expert model can realize structured sparse transformation suitable for hybrid expert calculation, and finally replace the original hybrid expert model with the converted model, use the same method as the original model calculation, and realize calculation acceleration by calling specific sparse acceleration units.

[0063] The embodiment of the present invention performs sparse conversion on the sparse data structures of the target hybrid expert model to obtain a structured sparse data structure, and optimizes the data arrangement of the structured sparse data structure. Furthermore, the operator of the target hybrid expert model is optimized in combination with the sparse computing unit of the computing chip to obtain a sparse operator, and then the original operator of the target hybrid expert model is updated according to the sparse operator to obtain a structured sparse hybrid expert model. The above technical scheme can solve the problem that the performance of the existing hybrid expert model is difficult to effectively improve when the performance is optimized and accelerated based on the unstructured sparse computing technology, and can realize the structured sparse optimization of the hybrid expert model to give full play to the acceleration performance of the sparse computing hardware of the hybrid expert model when calculating sparse matrix multiplication, thereby greatly optimizing the calculation, bandwidth and storage resource overhead in the sparse matrix multiplication process.

[0064] Figure 3 is a flow chart of another hybrid expert model optimization method provided by an embodiment of the present invention. This embodiment is specific based on the above embodiment. In this embodiment, multiple specific optional implementation methods for sparse conversion of various sparse data structures of the target hybrid expert model are provided. Figure 3 As shown, the method of this embodiment may include:

[0065] S210, loading the target hybrid expert model into a computing chip, and obtaining a sparse model weight matrix of each item in the target hybrid expert model.

[0066] In order to facilitate the implementation of subsequent model calculations, in an embodiment of the present invention, the model weight matrix of each model layer in the target hybrid expert model is processed into a sparse model weight matrix of a specific structure.

[0067] Among them, the sparse model weight matrix includes multiple compression units of set sizes, each of the compression units includes at least one valid row and at least one sparse row, each of the valid rows includes at least one structured sparse computing storage unit; each of the structured sparse computing storage units is in the same sparse mode.

[0068] Specifically, each model weight matrix in the target hybrid expert model contains an integer number (one or more) of compression units, that is, the size (number of rows * number of columns) of each model weight matrix can divide the size of the compression unit. The compression unit can be understood as a small matrix of a specific size, which has at least two matrix rows. The at least two matrix rows have at least one valid row and at least one sparse row. A valid row can be understood as a matrix row containing at least one valid data (non-0), and a sparse row refers to a matrix row where all data is 0.

[0069] Furthermore, each valid row in the compression unit contains an integer number (one or more) of structured sparse computing storage units. Among them, each structured sparse computing storage unit contained in the model weight matrix is ​​in the same sparse mode. Generally speaking, in order to facilitate calculation, the size of the structured sparse computing storage unit can be adapted to the calculation scale of the hardware computing unit in the computing chip.

[0070] The sparse pattern can be understood as the ratio of valid data (non-zero value data) to all data. For example, the sparse pattern can be 2:4 or 4:8. That is, for a 2:4 sparse pattern, the structured sparse computing storage unit contains a total of 4 data, 2 of which are non-zero data. The arrangement positions of the above two non-zero data in the structured sparse computing storage unit are not restricted.

[0071] In an embodiment of the present invention, two methods can be used to make each model weight matrix in the target hybrid expert model be in the above-mentioned clearly defined special structure: one is to perform strong constraints on the model weight matrix in each model layer during the training stage of the target hybrid expert model so that each model weight matrix meets the requirements of the above-mentioned special structure; the other is to perform strong constraints on the model weight matrix in each model layer during the model fine-tuning stage after model pre-training so that each model weight matrix meets the requirements of the above-mentioned special structure.

[0072] In this embodiment, the reason for limiting each model weight matrix in the target hybrid expert model to the above-mentioned special structure is to improve the computing efficiency and better adapt to the specific hardware computing unit in the computing chip when performing subsequent calculations based on the target hybrid expert model. The method for determining the above-mentioned special structure is not described in detail here.

[0073] S220. Generate a set of compression matrices corresponding to each of the model weight matrices.

[0074] Among them, the compression matrix set includes a compression data matrix for storing non-zero data in the model weight matrix, an index matrix for storing the positions of valid rows in the model weight matrix in the corresponding compression unit, and a metadata matrix for storing the positions of non-zero data in the corresponding structured sparse computing storage unit.

[0075] As mentioned above, each model weight matrix contains a large amount of zero-valued data. If the above model weight matrices are directly stored in the computing chip according to the original size of the model weight matrix, a large amount of storage units will be wasted. In addition, the subsequent model calculations are directly based on the model weight matrix of the original size, which has low computational efficiency and cannot efficiently use the hardware computing units in the computing chip.

[0076] In view of this, in an embodiment of the present invention, a new type of efficient data compression method is creatively proposed for the model weight matrix with the above-mentioned special structure to solve the above-mentioned technical problems.

[0077] In an embodiment of the present invention, each model weight matrix can be compactly stored in the form of a compressed matrix set through a specific data compression method. Specifically, three compressed matrices corresponding to the model weight matrix are stored in the compressed matrix set, namely, a compressed data matrix, an index matrix, and a metadata matrix.

[0078] Among them, the compressed data matrix is ​​used to store the compressed data matrix of non-zero data in the model weight matrix, the index matrix is ​​used to store the position of valid rows in the model weight matrix in the corresponding compression unit, and the metadata matrix is ​​used to store the position of non-zero data in the corresponding structured sparse computing storage unit.

[0079] Obviously, through the above-mentioned data compression method, the specific position of each non-zero data in the model weight matrix in the original model weight matrix can be clearly identified through three small matrices. This data compression method can also effectively reduce the consumption of storage resources in the computing chip, and can also facilitate the implementation of calculations based on the model weight matrix.

[0080] In an optional embodiment of the present invention, the sparse model weight matrix specifically includes multiple compression units of size M*V; each of the compression units specifically includes d valid rows, where 1≤d<M; and each of the structured sparse computing storage units is in a sparse mode of N:L, L is the total number of data contained in the structured sparse computing storage unit, and N is the number of non-zero data contained in the structured sparse computing storage unit.

[0081] In an optional embodiment of the present invention, generating a set of compressed matrices corresponding to each of the model weight matrices may include: obtaining a current model weight matrix of size m*k currently being processed, where m is an integer multiple of M and k is an integer multiple of V; constructing a first matrix of size (m / M*d)*(k / (L / N)), and filling each non-zero element of the current model weight matrix into the first matrix in a row-by-row traversal manner to obtain a compressed data matrix corresponding to the current model weight matrix; constructing a second matrix of size (m / M*d)*(k / V). An array is constructed, and according to the position of each compression unit in the current model weight matrix and the position of each valid row in the corresponding compression unit, the second matrix is ​​filled to obtain an index matrix corresponding to the current model weight matrix; a third matrix of size (m / M*d)*(k / (L / N)) is constructed, and according to the position of each structured sparse computing storage unit in the current model weight matrix and the position of each non-zero data in the corresponding structured sparse computing storage unit, the third matrix is ​​filled to obtain a metadata matrix corresponding to the current model weight matrix.

[0082] For ease of explanation, Figure 4 A schematic diagram of the comparison structure of a model weight matrix and a matching compression matrix set applicable to various embodiments of the present invention is shown in FIG.

[0083] Specifically, Figure 4 The figure shows a schematic diagram of generating a matching set of compression matrices for a model weight matrix of m=4, k=16. In the model weight matrix, there are 4 compression units of M=2, V=8, each of which contains d=1 valid rows, each of which contains 2 structured sparse computing storage units, and each structured sparse computing storage unit is in a sparse mode of N:L=2:4. That is, A, B, ..., L filled in each matrix position of the model weight matrix represent non-zero data, and the blank position represents the data filled with zero value.

[0084] When constructing the compressed data matrix of the model weight matrix, first construct a first matrix of (m / M*d)*(k / (L / N))=2*8, and then fill the non-zero data A, B,..., L in the model weight matrix into the first matrix in a row-by-row traversal manner to obtain a matching compressed data matrix.

[0085] Similarly, when constructing the index matrix for the model weight matrix, first construct a second matrix of (m / M*d)*(k / V)=2*2. Then, according to the position of each compression unit in the current model weight matrix and the position of each valid row in the corresponding compression unit, the second matrix is ​​filled to obtain the index matrix corresponding to the current model weight matrix.

[0086] In an optional embodiment of the present invention, the filling process of the second matrix according to the position of each compression unit in the current model weight matrix and the position of each valid row in the corresponding compression unit to obtain an index matrix corresponding to the current model weight matrix may include: traversing one current compression unit in turn in the current model weight matrix, and locating d vertical matrix positions matching the current compression unit in the second matrix according to the position of the current compression unit in the current model weight matrix; identifying the row position of each valid row in the current compression unit, and filling the identified row positions into the d vertical matrix positions accordingly; returning to execute the operation of traversing one current compression unit in turn in the current model weight matrix until the processing of all compression units in the current model weight matrix is ​​completed to obtain the index matrix corresponding to the current model weight matrix.

[0087] like Figure 4 As shown, in the model weight matrix, a 2*8 compression unit 1 including A, B, C, and D is first traversed. Since d=1, the corresponding upper left vertical matrix position can be located in the second matrix based on the upper left corner position of the compression unit 1 in the model weight matrix. Afterwards, the row position of the valid row is identified in the compression unit 1, that is, the row where A, B, C, and D are located, that is, the 0th row. Afterwards, 0 is filled into the upper left matrix position in the second matrix. And so on, until the row position of each valid row in the four compression units in the model weight matrix is ​​filled into the second matrix accordingly, and a matching index matrix is ​​obtained.

[0088] Similarly, when constructing the metadata matrix for the model weight matrix, first construct a third matrix of (m / M*d)*(k / (L / N))=2*8. Then, according to the position of each structured sparse computing storage unit in the current model weight matrix and the position of each non-zero data in the corresponding structured sparse computing storage unit, the third matrix is ​​filled to obtain the metadata matrix corresponding to the current model weight matrix.

[0089] In an optional embodiment of the present invention, the third matrix is ​​filled according to the position of each structured sparse computing storage unit in the current model weight matrix and the position of each non-zero data in the corresponding structured sparse computing storage unit to obtain a metadata matrix corresponding to the current model weight matrix, which may include: traversing a current structured sparse computing storage unit in the current model weight matrix in turn, and locating N horizontal matrix positions matching the current structured sparse computing storage unit in the third matrix according to the position of the current structured sparse computing storage unit in the current model weight matrix; filling the N horizontal matrix positions corresponding to the column position of each non-zero data in the current structured sparse computing storage unit; returning to execute the operation of traversing a current structured sparse computing storage unit in the current model weight matrix in turn until the processing of all the structured sparse computing storage units in the current model weight matrix is ​​completed to obtain the metadata matrix corresponding to the current model weight matrix.

[0090] Continue as Figure 4 As shown, each structured sparse computing storage unit contained in the model weight matrix can be traversed in turn by traversing row by row. For example, first traverse to the structured sparse computing storage unit 1 containing non-zero elements A and B. Since the structured sparse computing storage unit 1 is located in the upper left corner of the model weight matrix, it is possible to start from the upper left corner of the third matrix and select two horizontal matrix positions of N=2. After that, the column positions of A and B in the structured sparse computing storage unit 1, A is located in the 0th column, and B is located in the 2nd column, are filled to the first two column positions of the first row of the third matrix. Similarly, after the non-zero data positions in the 2*4 structured sparse computing storage units in the above-mentioned model weight matrix are filled to the third matrix, a matching metadata matrix is ​​obtained.

[0091] It can be understood that by constructing the above-mentioned compressed data matrix, index matrix and metadata data, the sparse model weight matrix of the above-mentioned specific structure can be uniquely determined. Furthermore, the sparse model weight matrices contained in the hybrid expert model can be efficiently compressed and stored, thereby effectively reducing the storage overhead of the hybrid expert model on the configured computing chip. In addition, when the hybrid expert model deployed in the computing chip implements calculations based on the above-mentioned compressed model weight matrix, it can give full play to the hardware computing advantages of each sparse computing unit in the computing chip, while effectively improving the computing efficiency and reducing the bandwidth overhead.

[0092] S230, storing the sparse model weight matrices of each item in the target hybrid expert model as a matching compression matrix set, to obtain a structured sparse data structure of the model weight data.

[0093] Specifically, when it is necessary to store the sparse model weight matrices in the target mixed expert model in the storage space of the computing chip (typically, global memory), the compressed matrix sets obtained after compression are used to replace the sparse model weight matrices for storage to achieve storage optimization.

[0094] The technical solution of the embodiment of the present invention converts the sparse model weight matrices of various specific formats in the target hybrid expert model into a compressed data matrix for storing non-zero data in the model weight matrix, an index matrix for storing the positions of valid rows in the model weight matrix in the corresponding compression unit, and a metadata matrix for storing the positions of non-zero data in the corresponding structured sparse computing storage unit, and then optimizes the storage in a computing chip used to implement hybrid expert model calculations. This can effectively reduce the storage resource overhead of the computing chip. In addition, when using the model weight matrix in the above-mentioned data compression form for model calculations, the sparse computing hardware in the computing chip can be fully utilized, which effectively improves the computing efficiency while reducing the bandwidth overhead.

[0095] In an optional embodiment of the present invention, after storing the sparse model weight matrices of each item in the target hybrid expert model as matching compressed matrix sets, it can also include: obtaining a sparse computing unit requirement file that matches the computing chip; wherein the sparse computing unit requirement file defines the data access location of each model weight matrix in the target hybrid expert model by each thread executed in the sparse operator when performing sparse calculations; according to the sparse computing unit requirement file, re-arranging the data arrangement of the compressed data matrix, index matrix and metadata matrix in each compressed matrix set; and storing each re-arranged compressed matrix set in the set memory of the computing chip to realize continuous storage of data with continuous access characteristics.

[0096] Generally speaking, computing chips (for example, GPUs) use memory access coalescing when loading data. The core idea is to merge memory accesses from multiple threads into a larger memory access operation, which can reduce access latency and improve bandwidth utilization. At the same time, its data loading also has the following characteristics:

[0097] 1. Aligned access: GPU memory access is usually aligned according to a certain step size, usually 32 bytes, 64 bytes or larger. If multiple threads operate on consecutive memory addresses, these accesses can usually be merged into a single memory access operation, thereby reducing the overhead of memory access; 2. Merged memory access: For example, if multiple threads access adjacent addresses in memory, the GPU can merge these accesses into a single memory request to avoid multiple memory accesses, thereby reducing latency and bandwidth waste.

[0098] Based on the above data loading concept, each embodiment of the present invention re-arranges the data arrangement of the compressed data matrix, index matrix and metadata matrix in each compression matrix set based on the data reading requirements of the sparse computing unit in the computing chip used to implement the calculation, so as to further improve the data reading efficiency during subsequent calculations.

[0099] Specifically, by parsing the sparse computing unit requirement file that comes with the computing chip, the sparse computing unit's data reading requirements can be obtained. Specifically, the sparse computing unit requirement file records the data access locations of each thread in the sparse computing unit to each model weight matrix in the target hybrid expert model.

[0100] Among them, the sparse computing unit can be understood as a special hardware circuit in the computing chip used to perform matrix multiplication calculations on sparse matrices. This type of special hardware circuit can accelerate sparse matrix multiplication. The computing chip generally contains multiple sparse computing units, and inside each sparse computing unit, one or more threads can be opened to implement the corresponding sparse computing tasks.

[0101] By way of example and not limitation, Figure 5 A schematic diagram showing a comparison between a sparse computing unit requirement and a matching data re-arrangement result applicable to an embodiment of the present invention is shown in FIG.

[0102] like Figure 5 As shown, in Figure 5 The table on the left records the access positions of matrix elements in a specific matrix (e.g., a compressed data matrix) by each thread in each sparse computing unit. 29 They represent threads respectively, and different threads belong to different sparse computing units. T0{3...0} in the 0th row, 0th column, 0th...3rd column in the left table represents that the data in the 0th row, 0th column, 0th...3rd column of the compressed data matrix is ​​loaded into the data position {3...0} in the target vector used by the T0 thread to perform calculations. Then, the T0 thread needs to read the data in the 0th row, 0th...3rd column of the compressed data matrix from the storage space where the compressed data matrix is ​​located and store it in the matching position in the target vector.

[0103] Similarly, T0{19...16} in the 0...3 column of the 8th row in the table on the left means that the data in the 0...3 column of the 8th row in the compressed data matrix is ​​loaded into the data position {19...16} in the target vector used by the T0 thread to perform calculations. Then, the T0 thread needs to read the data in the 0...3 column of the 8th row in the compressed data matrix from the storage space where the compressed data matrix is ​​located and store it in the matching position in the target vector.

[0104] It is understandable that, considering the computational characteristics of an actual sparse matrix, each thread does not read data from a continuous position in a matrix. This data reading method is inefficient and has poor bandwidth utilization, which reduces the computational efficiency of the dual sparse matrix to a certain extent. Based on this, in various embodiments of the present invention, the data arrangement of the compressed data matrix, index matrix, and metadata matrix in each compressed matrix set is rearranged according to the data reading characteristics of each thread in the sparse computing unit.

[0105] Specific, combined Figure 5 As shown in the table on the left and right sides, the total number of computational data required by the T0 thread is 32, which are located in the 0th row, columns 0...15, and the 8th row, columns 0...15 in the compressed data matrix. However, according to the conventional data storage method, these 32 data are not continuous. In other words, a thread needs to transfer the memory twice to obtain the data required by the sparse computing unit, which wastes the memory bandwidth of the computing chip.

[0106] To improve memory utilization, such as Figure 5 As can be seen from the table on the right, the 32 data that are not stored continuously can be arranged continuously in a new matrix. Through such optimization, each thread needs to load continuous 32 bits of data, and the data loading operation can be completed through one memory transfer. At this time, the T0 thread can continuously read the data that needs to be accessed continuously, greatly reducing the memory access overhead and data reading delay, and effectively improving bandwidth utilization.

[0107] S240 , optimizing the operator of the target hybrid expert model in combination with the sparse computing unit of the computing chip to obtain a sparse operator.

[0108] S250 . Update the original operator of the target hybrid expert model according to the sparse operator to obtain a structured sparse hybrid expert model.

[0109] S260: Update the original operator of the target hybrid expert model according to the sparse operator to obtain a structured sparse hybrid expert model.

[0110] In an optional embodiment of the present invention, after the sparse model weight matrices of each item in the target hybrid expert model are stored as matching compressed matrix sets, it can also include: calling each sparse operator on the computing chip to implement matching sparse calculations based on the stored optimized target hybrid expert model.

[0111] After completing the sparse conversion, data arrangement optimization, operator fusion and operator injection processing of the target hybrid expert model, a structured sparse hybrid expert model is obtained. Further, data can be input to the optimized target hybrid expert model. The optimized target hybrid expert model can call each sparse operator on the computing chip to implement a matching sparse computing process on the input data.

[0112] In an optional embodiment of the present invention, before calling each of the sparse operators on the computing chip and implementing the matching sparse calculation based on the target hybrid expert model after storage optimization, it may also include: identifying each hybrid model layer contained in the target hybrid expert model, and obtaining the expert allocation mode corresponding to each of the hybrid model layers; wherein the expert allocation mode is used to indicate the row position of the valid row of the intermediate activation value assigned to its own hybrid model layer when implementing the calculation; according to the expert allocation mode, generating a selection array corresponding to each of the hybrid model layers; associating and storing the selection array of each of the hybrid model layers with the compressed matrix set of each of the model weight matrices, so that when implementing the calculation, the selection array is combined with the sparse matrix or dense matrix of the intermediate activation value to be processed, and sparse calculation is performed with the matching compressed matrix set.

[0113] As mentioned above, the characteristic of the hybrid expert model is that it is necessary to determine which expert should be activated and participate in the calculation of the output when processing a specific input. That is, when implementing calculations based on the target hybrid expert model, for any hybrid model layer, its intermediate activation values ​​are assigned to different experts for processing by the intermediate routing layer. Optionally, the intermediate activation values ​​can include two forms: a sparse matrix of intermediate activation values ​​and a dense matrix of intermediate activation values.

[0114] The inventors have found through research that when a hybrid expert model is used for computing tasks (typically, model reasoning tasks), the matrix multiplication calculation between the model weight matrix of each model layer and the sparse matrix of intermediate activation values ​​input to the hybrid model layer in the hybrid expert model belongs to double sparse matrix multiplication. The matrix multiplication calculation between the model weight matrix of each model layer and the dense matrix of intermediate activation values ​​input to the hybrid model layer in the hybrid expert model belongs to sparse matrix multiplication. Furthermore, the methods of the embodiments of the present invention can be applied to model calculation and model optimization scenarios of hybrid expert models.

[0115] Optionally, for any expert, the intermediate activation values ​​that are not assigned to its processing can be transformed into structured sparse values, that is, the input to each hybrid model layer can be a sparse matrix of intermediate activation values ​​in the form of a sparse matrix, and one or more matrix rows in the intermediate activation value sparse matrix are sparse rows (all 0). Optionally, the input to each hybrid model layer can also be a dense matrix of intermediate activation values ​​in the form of a dense matrix, and all matrix elements in the dense matrix of intermediate activation values ​​are valid data (all non-0).

[0116] The traditional processing method is that each hybrid model layer copies the data (data in non-sparse rows) of the original intermediate activation value sparse matrix or dense matrix for each expert as new data, which will bring additional computing, memory and bandwidth requirements. Based on this, the embodiments of the present invention further propose a structured sparse representation method based on column vectors to avoid additional data copying processes, thereby greatly reducing runtime overhead and improving model operation efficiency.

[0117] That is, a sparse matrix or a dense matrix of intermediate activation values ​​is directly stored for each hybrid model layer, avoiding the extra cost of copying new data.

[0118] In an embodiment of the present invention, since the intermediate activation value sparse matrix or dense matrix is ​​directly stored for each hybrid model layer respectively, in order to improve the computing requirements of the sparse computing unit for the intermediate activation value sparse matrix or dense matrix, it is necessary to synchronously store the selection array for the intermediate activation value sparse matrix or dense matrix.

[0119] The selection array is used to describe the valid row positions in the sparse matrix or dense matrix of intermediate activation values.

[0120] By way of example and not limitation, Figure 6 A schematic diagram of a comparison structure of an intermediate activation value sparse matrix and a matching selection array applicable to an embodiment of the present invention is shown in FIG. Figure 6 The left matrix of shows the specific form of an intermediate activation value sparse matrix adapted by a hybrid model layer. In the intermediate activation value sparse matrix, the 0th, 3rd and 5th rows are set as valid rows, and the remaining rows are all-0 sparse rows. At this time, the expert allocation mode for the hybrid model layer is the intermediate activation values ​​output by the 0th, 3rd and 5th experts. Furthermore, a selection array of the form {0,3,5} can be constructed to describe the location of the valid rows in the intermediate activation value sparse matrix. Assuming that the matrix corresponding to the intermediate activation value is a dense matrix, which includes valid data in rows 1-3 and does not include sparse rows, a selection array of the form {1,2,3} can be constructed and stored in association with the second association matrix.

[0121] The technical solution of the embodiment of the present invention converts the sparse model weight matrices of various specific formats in the target hybrid expert model into a compressed data matrix for storing non-zero data in the model weight matrix, an index matrix for storing the positions of valid rows in the model weight matrix in the corresponding compression unit, and a metadata matrix for storing the positions of non-zero data in the corresponding structured sparse computing storage unit, and then optimizes the storage in a computing chip used to implement hybrid expert model calculations. This can effectively reduce the storage resource overhead of the computing chip. In addition, when using the model weight matrix in the above-mentioned data compression form for model calculations, the sparse computing hardware in the computing chip can be fully utilized, which effectively improves the computing efficiency while reducing the bandwidth overhead.

[0122] In the embodiment of the present invention, since the target hybrid expert model needs to perform matrix multiplication between the model weight matrix of each model layer and the input activation value matrix input by the model layer when performing model inference, sparse matrix multiplication is involved in this calculation process. In each embodiment of the present invention, by compressing and storing the model weight matrix as a compressed matrix set and synchronously storing it with the selection array of each hybrid model layer, the sparse computing unit in the computing chip can be fully utilized to improve computing efficiency.

[0123] Figure 7 is a flowchart of another hybrid expert model optimization method provided by an embodiment of the present invention, Figure 8 1 is a schematic diagram of a process for optimizing the original operator of a hybrid expert model provided by an embodiment of the present invention. This embodiment is specific based on the above embodiment. In this embodiment, multiple specific optional implementation methods for sparse conversion of various sparse data structures of the target hybrid expert model are provided. Figure 7 and Figure 8 As shown, the method of this embodiment may include:

[0124] S310, performing sparse conversion on each sparse data structure of the target hybrid expert model to obtain a structured sparse data structure, and performing data arrangement optimization on the structured sparse data structure.

[0125] S320, using a data block method in combination with a sparse computing unit of a computing chip to generate a first sparse operator, a second sparse operator, and a third sparse operator suitable for a target hybrid expert model.

[0126] The first sparse operator may be an operator in the hybrid expert model for outputting the model calculation result. The second sparse operator may be an operator capable of outputting the output value of the activation function. The third sparse operator may be an operator capable of outputting the output value of the dot product function.

[0127] In an optional embodiment of the present invention, if the sparse operator includes a first sparse operator, the sparse computing unit combined with the computing chip optimizes the operator of the target hybrid expert model to obtain a sparse operator, which may include: according to the first sparse matrix multiplication requirement, locating a compressed matrix set and a second associated matrix matching the first sparse matrix in the global memory of the computing chip; wherein the second associated matrix includes a second sparse matrix or a dense matrix corresponding to the second sparse matrix; the compressed matrix set and the second associated matrix are moved step by step from the global memory to the hardware register of the computing chip in the form of data blocks; through the sparse computing unit of the computing chip, the multiplication results of the first sparse matrix and the second associated matrix are gradually calculated according to the data loaded in the hardware register in batches; the multiplication results of the first sparse matrix and the second associated matrix are processed according to the processing logic of the broadcast multiplication function of the target hybrid expert model to obtain the first sparse operator.

[0128] In an optional embodiment of the present invention, if the sparse operator includes a second sparse operator, the sparse computing unit combined with the computing chip optimizes the operator of the target hybrid expert model to obtain the sparse operator, which may include: according to the second sparse matrix multiplication requirement, locating a compressed matrix set and a second associated matrix matching the first sparse matrix in the global memory of the computing chip; wherein the second associated matrix includes the second sparse matrix or a dense matrix corresponding to the second sparse matrix; the compressed matrix set and the second associated matrix are moved step by step from the global memory to the hardware register of the computing chip in the form of data blocks, and during the moving process, the data in the original global memory is rearranged or deformed; through the sparse computing unit of the computing chip, the multiplication results of the first sparse matrix and the second associated matrix are gradually calculated according to the data loaded in batches in the hardware register; the multiplication results of the first sparse matrix and the second associated matrix are processed according to the processing logic of the activation function of the target hybrid expert model to obtain the second sparse operator.

[0129] In an optional embodiment of the present invention, if the sparse operator includes a third sparse operator, the sparse computing unit combined with the computing chip optimizes the operator of the target hybrid expert model to obtain the sparse operator, which may include: according to the third sparse matrix multiplication requirement, locating a compressed matrix set and a second associated matrix matching the first sparse matrix in the global memory of the computing chip; wherein the second associated matrix includes a second sparse matrix or a dense matrix corresponding to the second sparse matrix; the compressed matrix set and the second associated matrix are moved step by step from the global memory to the hardware register of the computing chip in the form of data blocks, and during the moving process, the data in the original global memory is rearranged or deformed; through the sparse computing unit of the computing chip, the multiplication results of the first sparse matrix and the second associated matrix are gradually calculated according to the data loaded in batches in the hardware register; the multiplication results of the first sparse matrix and the second associated matrix are processed according to the processing logic of the point multiplication function of the target hybrid expert model to obtain the third sparse operator.

[0130] Among them, the first sparse matrix can be the left operand of the dual matrix multiplication, which is a sparse matrix. In an embodiment of the present invention, the first sparse matrix can be a sparse model weight matrix obtained by sparse conversion processing in the above embodiment, that is, a structured sparse data structure of model weight data. The second association matrix can be the right operand of the dual matrix multiplication, which can be a sparse matrix, or a dense matrix composed of valid data in the sparse matrix. Optionally, the second association matrix can be a sparse matrix of intermediate activation values ​​input to the mixed model layer in the target mixed expert model, and the dense matrix corresponding to the second sparse matrix can be a dense matrix of intermediate activation values ​​input to the mixed model layer in the target mixed expert model. The matrix multiplication between the first sparse matrix and the second association matrix can be called sparse matrix multiplication. When the second association matrix also uses a sparse matrix, the matrix multiplication between the first sparse matrix and the second association matrix can be called a dual sparse matrix multiplication. In the following, no matter what type of matrix the second association matrix uses, the matrix multiplication between the first sparse matrix and the second association matrix is ​​referred to as sparse matrix multiplication.

[0131] In an embodiment of the present invention, the above-mentioned sparse matrix multiplication acceleration method can be encapsulated in the form of an operator interface, for example, a plurality of types of sparse operators are specially constructed for realizing matrix multiplication between a first sparse matrix and a second associated matrix, such as a first sparse operator, a second sparse operator, or a third sparse operator, etc. Furthermore, the multiplication acceleration method between the first sparse matrix and the second associated matrix can be triggered by calling the operator interface of the sparse operator, and according to the fusion requirements of the first sparse operator and other functional functions, additional processing logic is configured for the multiplication result between the first sparse matrix and the second associated matrix, thereby generating a plurality of types of sparse operators, such as a first sparse operator, a second sparse operator, or a third sparse operator, etc.

[0132] Specifically, sparse matrix multiplication can be understood as at least one type of operand in the left operand and the right operand of the matrix multiplication is a sparse matrix. Accordingly, when it is detected that the interface call request for the first sparse operator generates the first sparse operator, it is determined that the first sparse matrix multiplication requirement is detected; when it is detected that the interface call request for the second sparse operator generates the second sparse operator, it is determined that the second sparse matrix multiplication requirement is detected; when it is detected that the interface call request for the third sparse operator generates the third sparse operator, it is determined that the third sparse matrix multiplication requirement is detected. Furthermore, the identification information of the left operand (hereinafter referred to as the first sparse matrix) and the right operand (hereinafter referred to as the second association matrix) that need to perform sparse matrix multiplication can be obtained from the corresponding interface call request. Based on the above identification information, the compressed matrix set and the second association matrix that match the first sparse matrix can be located in the global memory of the computing chip.

[0133] It is understandable that the first sparse matrix and the second association matrix are multiplied in the computing chip. Furthermore, the first sparse matrix and the second association matrix to be calculated need to be preloaded into the computing chip. In order to improve the computing speed of the computing chip, the calculation data is initially stored in the global memory of the computing chip, and then the calculation data can be moved in blocks to the hardware registers adapted to the hardware computing unit in a step-by-step manner, and the hardware computing unit calculates the final calculation result based on the data blocks stored in the hardware registers.

[0134] In the embodiment of the present invention, in order to achieve the final multiplication acceleration of the sparse operator, the sparse format of the first sparse matrix as the left operand is specially limited. At the same time, the storage method of the first sparse matrix in the global memory of the computing chip is also optimized accordingly.

[0135] Among them, the first sparse matrix includes multiple compression units of set sizes, each compression unit includes at least one valid row and at least one sparse row, each valid row includes at least one structured sparse computing storage unit; each structured sparse computing storage unit is in the same sparse mode; the compression matrix set includes a compression data matrix for storing non-zero data in the first sparse matrix, an index matrix for storing the positions of valid rows in the first sparse matrix in the corresponding compression units, and a metadata matrix for storing the positions of non-zero data in the corresponding structured sparse computing storage units.

[0136] Specifically, the first sparse matrix contains an integer number (one or more) of compression units, that is, the size of the first sparse matrix (number of rows * number of columns) can divide the size of the compression unit. The compression unit can be understood as a small matrix of a specific size, which has at least two matrix rows. The at least two matrix rows have at least one valid row and at least one sparse row. A valid row can be understood as a matrix row containing at least one valid data (non-0), and a sparse row refers to a matrix row in which all data is 0.

[0137] Further, each valid row in the compression unit contains an integer number (one or more) of structured sparse computing storage units. Wherein, each structured sparse computing storage unit contained in the first sparse matrix is ​​in the same sparse mode. Generally speaking, in order to facilitate calculation, the size of the structured sparse computing storage unit can be adapted to the calculation scale of the hardware computing unit in the computing chip.

[0138] It is understandable that the first sparse matrix contains a large amount of zero-valued data. If the first sparse matrix is ​​directly stored in the global memory of the computing chip according to the original size of the first sparse matrix, a large amount of storage units will be wasted. In addition, the subsequent sparse matrix multiplication is directly implemented based on the first sparse matrix of the original size, which has low computational efficiency and cannot efficiently use the hardware computing units in the computing chip, that is, the sparse computing units.

[0139] In view of this, in an embodiment of the present invention, a new type of efficient data compression method is creatively proposed for the left operand (first sparse matrix) of the sparse multiplication calculation of the above special structure to solve the above technical problems. In an embodiment of the present invention, the first sparse matrix can be compactly stored in the form of a compressed matrix set through a specific data compression method. Specifically, three compressed matrices corresponding to the first sparse matrix are stored in the compressed matrix set, namely, a compressed data matrix, an index matrix, and a metadata matrix. Among them, the compressed data matrix is ​​used to store the compressed data matrix of non-zero data in the first sparse matrix, the index matrix is ​​used to store the position of the valid rows in the first sparse matrix in the corresponding compression unit, and the metadata matrix is ​​used to store the position of the non-zero data in the structured sparse computing storage unit.

[0140] Obviously, through the above-mentioned data compression method, the specific position of each non-zero data in the first sparse matrix in the original first sparse matrix can be clearly determined through three small matrices. This data compression method can also effectively reduce the consumption of storage resources in the computing chip, and can also facilitate the implementation of calculations based on the first sparse matrix.

[0141] In an embodiment of the present invention, the second association matrix may not be stored in compression. However, considering the actual application requirements of sparse matrix multiplication, for example, in a hybrid expert model, the second association matrix generally also has a specific data structure. Typically, when the second association matrix is ​​a second sparse matrix, one or more matrix rows in the second sparse matrix are sparse rows (all 0). When the second association matrix is ​​a dense matrix corresponding to the second sparse matrix, the data in the dense matrix corresponding to the second sparse matrix is ​​valid data in the second sparse matrix. Exemplarily, assuming that the second sparse matrix is ​​[1,3,0,5,0,0], the dense matrix corresponding to the second sparse matrix is ​​[1,3,5].

[0142] Based on this, in addition to storing the second association matrix completely in the global memory of the computing chip, a selection array matching the second association matrix is ​​further stored to describe the valid row positions in the second association matrix. Accordingly, while locating the second association matrix in the global memory of the computing chip, the selection array matching the second association matrix can be located synchronously.

[0143] In an optional implementation of the embodiment of the present invention, the computing chip includes a three-level storage architecture of global memory->shared memory->hardware registers. The three-level storage is gradually approached to the hardware computing units used to perform calculations in the computing chip in terms of physical location, especially the hardware registers are arranged close to the hardware computing units.

[0144] Generally speaking, the closer the storage device is to the computing unit, the faster its data read and write speed is, but the smaller its storage capacity is. Therefore, for a matrix multiplication of C(M×N)=A(M×K)×B(K×N), the complete data of A and B cannot be fully stored in the shared memory or hardware registers, and often need to be moved in blocks to the hardware registers through multi-level transfer, so that the hardware computing unit in the computing chip can calculate the final complete multiplication result in batches.

[0145] In an optional implementation of the present embodiment, the special data structure of the compressed matrix set can be combined to perform data block processing on each matrix in the compressed matrix set. At the same time, combined with the selection array matching the second sparse matrix, the second association matrix can be processed in data blocks to improve the efficiency of data handling and subsequent multiplication calculations.

[0146] The computing chip generally includes multiple sparse computing units, and inside each sparse computing unit, one or more threads can be started to implement the corresponding sparse computing tasks. By using the sparse computing units in the computing chip in coordination, partial data in the first sparse matrix and the second association matrix can be loaded from each hardware register in batches for local calculation, and the local calculation results can be combined or accumulated to finally obtain the multiplication result of the first sparse matrix and the second association matrix.

[0147] In the calculation process of the hybrid expert model, there are many one-to-one operators and matrix multiplications adjacent to each other, but the calculation of two consecutive operators will introduce redundant computing kernel startup overhead and I / O (Input / Output) overhead. In order to eliminate the overhead of loading and storing intermediate results and multiple kernel startups, the computing core of each sparse operator can be improved by operator fusion.

[0148] Among various sparse operators, the first sparse operator is usually used as an operator for outputting model calculation results, for example, Figure 8 As shown, the sparse operator 3 in the target hybrid expert model can be used as the first sparse operator. Optionally, for the first sparse operator, after the multiplication results of the first sparse matrix and the second association matrix are gradually calculated according to the data loaded in batches in the hardware register by the sparse computing unit of the computing chip, the multiplication results of the first sparse matrix and the second association matrix can be processed according to the processing logic of the broadcast multiplication function of the original hybrid expert model to obtain the first sparse operator with broadcast function.

[0149] like Figure 8As shown, the sparse operator 1 in the target hybrid expert model can be used as the second sparse operator. Optionally, for the second sparse operator, after gradually calculating the multiplication results of the first sparse matrix and the second association matrix according to the data loaded in batches in the hardware register through the sparse computing unit of the computing chip, the multiplication results of the first sparse matrix and the second association matrix can be processed according to the processing logic of the activation function of the original hybrid expert model to obtain the second sparse operator. Accordingly, the second sparse operator can directly output the calculation result of the activation function.

[0150] like Figure 8 As shown, the sparse operator 2 in the target hybrid expert model can be used as the third sparse operator. Optionally, for the third sparse operator, after gradually calculating the multiplication results of the first sparse matrix and the second association matrix according to the data loaded in batches in the hardware register through the sparse computing unit of the computing chip, the multiplication results of the first sparse matrix and the second association matrix can be processed according to the processing logic of the point multiplication function of the original hybrid expert model to obtain the third sparse operator. Accordingly, the third sparse operator can directly output the calculation result of the point multiplication function.

[0151] For the first sparse operator, the second sparse operator and the third sparse operator, the overall sparse operator generation process is generally the same, the only difference is that in the process of moving data from the global memory to the hardware register of the computing chip in the form of blocks, the second sparse operator and the third sparse operator need to rearrange or deform the data in the original global memory during the moving process. At the same time, because the functions of the operators are different, the types of specific functional functions fused when the first sparse operator, the second sparse operator and the third sparse operator are different.

[0152] S330: construct a sparse operator library according to the first sparse operator, the second sparse operator, and the third sparse operator.

[0153] Among them, the sparse operator library can be used to store various generated sparse operators.

[0154] In the embodiment of the present invention, Figure 8As shown, after various sparse operators suitable for the original hybrid expert model are generated by combining the sparse computing unit of the computing chip in a data block manner, the generated various sparse operators can be stored in the sparse operator library. It can be understood that the generation process of the above sparse operators only schematically illustrates the specific generation process of the first sparse operator, the second sparse operator and the third sparse operator. For other types of sparse operators in the original hybrid expert model, similar methods as described above can also be used to optimize the generation. That is, sparse operators are not limited to the first sparse operator, the second sparse operator and the third sparse operator. Accordingly, in addition to storing the first sparse operator, the second sparse operator and the third sparse operator, the sparse operator library can also store other types of sparse operators, and the embodiment of the present invention does not limit the type of sparse operators.

[0155] S340 , selecting a target sparse operator from a sparse operator library according to the type of the target hybrid expert model, and updating the original operator of the target hybrid expert model according to the target sparse operator to obtain a structured sparse hybrid expert model.

[0156] The target sparse operator may be a sparse operator used to update and replace the original operator in the original hybrid expert model. Optionally, there may be multiple target sparse operators.

[0157] For example, Figure 8 As shown, the original operator of the target hybrid expert model may include a first matrix multiplication operator, a second matrix multiplication operator, and a third matrix multiplication operator. Among them, the first matrix multiplication operator may be a matrix multiplication 2 operator in the target hybrid expert model before optimization, the second matrix multiplication operator may be a matrix multiplication 1 operator in the target hybrid expert model before optimization, and the third matrix multiplication operator may be a matrix multiplication 3 operator in the target hybrid expert model before optimization. For this type of target hybrid expert model, the first sparse operator, the second sparse operator, and the third sparse operator may be selected as the target sparse operator. Accordingly, when the original operator of the target hybrid expert model is updated according to the target sparse operator, the first matrix multiplication operator of the target hybrid expert model may be replaced and updated according to the first sparse operator, the second matrix multiplication operator of the target hybrid expert model may be replaced and updated according to the second sparse operator, and the third matrix multiplication operator of the target hybrid expert model may be replaced and updated according to the third sparse operator to obtain a structured sparse hybrid expert model.

[0158] For example, Figure 8As shown, the original operators of some types of target hybrid expert models only include the first matrix multiplication operator and the second matrix multiplication operator. Among them, the first matrix multiplication operator can be the matrix multiplication 2 operator in the target hybrid expert model before optimization, and the second matrix multiplication operator can be the matrix multiplication 1 operator in the target hybrid expert model before optimization. For this type of target hybrid expert model, the first sparse operator and the second sparse operator can be screened as the target sparse operator. Correspondingly, when the original operator of the target hybrid expert model is updated according to the target sparse operator, the first matrix multiplication operator of the target hybrid expert model can be replaced and updated according to the first sparse operator, and the second matrix multiplication operator of the target hybrid expert model can be replaced and updated according to the second sparse operator to obtain a structured sparse hybrid expert model.

[0159] Similarly, for other types of target hybrid expert models, it is also possible to adapt to generate sparse operator types with other functions, and replace and update the original operators inside them to obtain the corresponding structured sparse hybrid expert models.

[0160] The technical solution of the embodiment of the present invention is to compress and store the sparse matrix that needs to be multiplied in a specific compression format, and then move the sparse matrix and the associated matrix in the above-mentioned specific compression format from the global memory to the hardware register of the computing chip in the form of data blocks step by step, and use the sparse computing unit of the computing chip to gradually calculate the multiplication results of the sparse matrix and the associated matrix according to the data loaded in the hardware register in batches. The implementation method fully considers the sparse characteristics when performing matrix multiplication calculations based on sparse matrices. By adopting data blocking and data moving technology that are compatible with the sparse characteristics, the hardware acceleration performance of the sparse computing unit in the computing chip can be fully utilized, thereby greatly optimizing the calculation, bandwidth and storage resource overhead in the sparse matrix multiplication operation process, and is particularly suitable for model calculation scenarios of hybrid expert models.

[0161] Fig. 9 A flowchart of another hybrid expert model optimization method provided by an embodiment of the present invention, which is optimized based on the above embodiments. In this embodiment, the operation of "moving the compressed matrix set and the second association matrix from the global memory to the hardware register of the computing chip in the form of data blocks" is specifically concretized.

[0162] Correspondingly, such as Fig. 9 As shown, the method may include:

[0163] S410, performing sparse conversion on each sparse data structure of the target hybrid expert model to obtain a structured sparse data structure, and performing data arrangement optimization on the structured sparse data structure.

[0164] S420. According to the sparse matrix multiplication requirement, locate a set of compressed matrices and a second association matrix matching the first sparse matrix in the global memory of the computing chip.

[0165] The sparse matrix multiplication requirement may include but is not limited to a first sparse matrix multiplication requirement, a second sparse matrix multiplication requirement, and a third sparse matrix multiplication requirement.

[0166] In an optional implementation of an embodiment of the present invention, the first sparse matrix may be a sparse model weight matrix in the target hybrid expert model, the second sparse matrix may be a sparse matrix of intermediate activation values ​​input to the hybrid model layer in the target hybrid expert model, and the dense matrix corresponding to the second sparse matrix may be a dense matrix of intermediate activation values ​​input to the hybrid model layer in the target hybrid expert model.

[0167] In the computing scenario of the hybrid expert model, by executing the sparse matrix multiplication process based on the specific model weight matrix and the matching sparse matrix or dense matrix of the intermediate activation value, combined with the operator fusion strategy, various types of sparse operators can be generated to achieve operator optimization of the hybrid expert model, and then complete the optimization of the internal model structure of the hybrid expert model.

[0168] S430, dividing the compressed data matrix in the global memory into data blocks, and dividing the index matrix and the metadata matrix into data blocks that match each other according to the data correspondence between the compressed data matrix, the index matrix and the metadata matrix.

[0169] In the embodiment of the present invention, since the compressed data matrix stores all non-zero data in the first sparse matrix, data can be partitioned according to the data size of the compressed data matrix. In the process of partitioning the compressed data matrix, in order to further consider the efficiency of subsequent data handling and multiplication calculations, it can be set that in the process of partitioning the compressed data matrix in the global memory, V (the number of columns contained in each compression unit) is an integer multiple of the partition size in the column partition direction.

[0170] For example, suppose that in global memory, the compressed data matrix is ​​split into multiple m b *k b In the process of data segmentation, V needs to be k. b An integer multiple of, for example, k b It is V / 2 or V / 4, etc.

[0171] As shown above, since there is a data or position correspondence between the compressed data matrix, the index matrix and the metadata matrix, after determining the compressed data matrix blocks cut out from the compressed data matrix, the index matrix blocks and the metadata matrix blocks that match the compressed data matrix blocks can be uniquely determined. That is, according to the data correspondence between the compressed data matrix, the index matrix and the metadata matrix, the data blocks that match the index matrix and the metadata matrix are matched.

[0172] S440, locating a selection array matching the second association matrix in the computing chip, and dividing the second association matrix in the global memory into data blocks according to the selection array.

[0173] Among them, the selection array is used to describe the valid row positions in the second sparse matrix, the selection array of the second sparse matrix is ​​the index of the non-zero data positions therein, and the selection array of the dense matrix corresponding to the second sparse matrix is ​​the position mapping of the data in the dense matrix in the second sparse matrix.

[0174] As shown above, if the second association matrix is ​​a second sparse matrix, the second sparse matrix may contain a large number of sparse rows. If these sparse rows are also divided into data blocks and sent to the hardware computing unit for matrix multiplication calculation, this will result in the loading and calculation of redundant data, thereby reducing the computing efficiency.

[0175] Based on this, in the embodiment of the present invention, in the global memory, only the valid data rows in the second association matrix can be partitioned based on the valid row positions defined by the selection array. Optionally, one or more valid data rows can be taken as a data partition each time, or a set amount of data in a valid data row can be taken as a data partition each time.

[0176] S450, loading each compressed data matrix block, each index matrix block and each metadata matrix block from the global memory into the shared memory one by one.

[0177] After completing the data partitioning for the compressed matrix set in the global memory, each compressed data matrix block, each index matrix block and each metadata matrix block may be loaded from the global memory to the shared memory one by one.

[0178] On the basis of the above embodiments, in order to prevent the data in the shared memory from being restricted by a bank conflict (bank conflict) in which the same storage body (also referred to as a bank) in the computing chip cannot be accessed by multiple threads at the same time during the access (read / write) process, in the embodiments of the present invention, it is further considered to rearrange the storage of data in the shared memory based on the data offset using a specific data rearrangement function.

[0179] Accordingly, in an optional implementation of the embodiment of the present invention, after each compressed data matrix block, each index matrix block, and each metadata matrix block are successively moved from the global memory to the shared memory, the following may also be included:

[0180] Using a preset data rearrangement function, each compressed data matrix block, each index matrix block, and each metadata matrix block are rearranged in a shared memory to avoid storage conflicts.

[0181] Specifically, according to the specific type and model of the computing chip, a matching data rearrangement function can be selected in the adapted function library to rearrange the data in the shared memory, and the embodiment of the present invention is not limited to this.

[0182] S460, dividing each second association matrix into blocks, and moving them from the global memory to the shared memory one by one.

[0183] Similarly, after the data of the second association matrix is ​​divided into blocks in the global memory, the blocks of the second association matrix can be correspondingly moved from the global memory to the shared memory one by one.

[0184] Meanwhile, in order to avoid the problem of memory bank conflict, in an optional implementation of the embodiment of the present invention, after each second association matrix is ​​divided into blocks and moved from the global memory to the shared memory one by one, the following steps may also be included:

[0185] Using a preset data rearrangement function, each second correlation matrix block is rearranged in a shared memory to avoid storage bank conflicts.

[0186] Of course, it is understandable that in addition to using the data rearrangement function to rearrange the compressed data matrix blocks, the index matrix blocks, the metadata matrix blocks and the second association matrix blocks in the shared memory, it is also possible to add data padding to the compressed data matrix blocks, the index matrix blocks, the metadata matrix blocks and the second association matrix blocks in the shared memory to avoid storage conflicts.

[0187] S470, performing secondary block division on each compressed data matrix block, each index matrix block and each metadata matrix block in the shared memory, and performing secondary block division on each second association matrix block in the shared memory.

[0188] In the embodiment of the present invention, in order to adapt to the storage limitation of the hardware register, each compressed data matrix block, each index matrix block and each metadata matrix block in the shared memory may be divided into two blocks.

[0189] Furthermore, when performing secondary partitioning on each compressed data matrix block, each index matrix block, and each metadata matrix block in the shared memory, the compressed data matrix block can also be firstly secondary partitioned. In the process of secondary partitioning of the compressed data matrix, in order to further consider the efficiency of subsequent data handling and multiplication calculation, it can be set that in the process of secondary partitioning of the compressed data matrix block in the shared memory, V is also an integer multiple of the partition size in the column partitioning direction. For example, secondary partitioning can be selected in the column partitioning direction or no secondary partitioning can be selected (partitioning is performed only in the row direction).

[0190] For example, suppose that in global memory, the compressed data matrix is ​​divided into multiple m blocks b1 *k b1 In the process of new data block size, V needs to be k b1 An integer multiple of, for example, k b1 It is V / 4 or V / 8 ​​etc.

[0191] Similarly, since there is a data or position correspondence between the compressed data matrix, the index matrix and the metadata matrix, after completing the secondary blocking of the compressed data matrix, the secondary blocking of the index matrix and the metadata matrix can be performed accordingly.

[0192] S480, according to the register mapping relationship matching the sparse computing unit, at least one item of each compressed data matrix secondary block, each index matrix secondary block and each metadata matrix secondary block is moved from the shared memory to the hardware register one by one.

[0193] For a more intuitive understanding, Fig.10 A block diagram of a data partitioning and multi-stage transfer process applicable to an embodiment of the present invention is shown in FIG. In a specific example, the second sparse matrix is ​​used as an example. Fig.10 As shown, it is described that the compressed data matrix A in the first sparse matrix and the second sparse matrix B are divided into data blocks in the global memory and the shared memory respectively, and are moved step by step in the order of global memory->shared memory->register. After the calculation is completed by the sparse computing unit and written back step by step, the corresponding multiplication result matrix C is obtained in the global memory.

[0194] In this example, an example of data partitioning and data movement for matrix multiplication of any size C(M×N)=A(M×K)×B(K×N) is given. In the embodiment of the present invention, in order to implement the matrix multiplication calculation, the rows and columns of the second sparse matrix B are interchanged, that is, an array is selected to describe the positions of the valid data columns in the second sparse matrix B.

[0195] Specifically, in global memory, since the selection array only contains the locations of valid data columns (rows), the total amount of data in the selection array is len d Then the number of valid values ​​of n dimension in the second sparse matrix B is len d The compressed data matrix A is in m dimensions with m b The second sparse matrix B is divided into several blocks of uniform size. The number of valid column vectors in each block of data is the same as the number of n in the selection array. b Considering the storage size limit of the computing device, the compressed data matrix A and the second sparse matrix B will be further divided into k blocks along the k direction. b To evenly divide the data into different sizes, multiple sub-blocks in the k direction are multiplied and accumulated to obtain the final multiplication result matrix C block result. Based on this data division strategy, the size of the multiplication result matrix C calculated for each block data is m b ×n b , and the block calculation is a part of the final result of the multiplication result matrix C, which corresponds to the same row offset of the data block of the compressed data matrix A in the compressed data matrix A, and the same column offset of the data block of the second sparse matrix B in the selection array. The parallelism of each sparse operator is improved by each thread group being responsible for a block.

[0196] Similarly, in the shared memory, it also involves a process of performing a secondary block division on each data block transferred from the global memory, and after the block is moved to the register for storage, the adapted sparse computing unit performs a matrix multiplication calculation, which will not be described in detail here. In this embodiment, the matrix specification of the compressed data matrix A processed by the sparse computing unit once is mi*ki, and the matrix specification of the second sparse matrix B is ki*ni. In order to facilitate subsequent calculations, the aforementioned V is an integer multiple of ki.

[0197] In an optional implementation of the embodiment of the present invention, each compressed data matrix secondary block, each index matrix secondary block and each metadata matrix secondary block can be moved to the register to implement calculations. Alternatively, considering that register resources are very valuable, only each compressed data matrix secondary block and each metadata matrix secondary block can be moved to the register to implement calculations, while the index matrix can be retained in the shared memory and retrieved from the shared memory only when needed.

[0198] It should be emphasized that, unlike the computing logic of general hardware computing units, when using sparse computing units to accelerate multiplication calculations, the data to be calculated needs to be placed in an adapted instruction register in accordance with the requirements of the sparse instruction set. Therefore, each embodiment of the present invention needs to move the data in the shared memory to the matching hardware register based on the register mapping relationship that matches the sparse computing unit.

[0199] Optionally, the register mapping relationship can be read from a specification file of the factory configuration of the computing chip, and the register hardware file describes the mapping relationship between the computing data at different locations and the hardware registers.

[0200] Specifically, in Fig.11 A schematic diagram of a register mapping relationship applicable to an embodiment of the present invention is shown in FIG. 1 , which register mapping relationship is adapted to the secondary block division of the compressed data matrix. In a specific example, Fig.11 As shown, the T0{a0, a1} in the upper left corner of the register mapping relationship represents: the data in the 0th column and the 1st column of the 0th row in the secondary block of the compressed data matrix are allocated to registers a0 and a1 matching thread T0. Similarly, the T0 in the last position of row 0 in the register mapping relationship 0…3 {a4, a5} represents: the data of the 8th to 15th columns of the 0th row in the secondary block of the compressed data matrix are respectively allocated to register a4 and register a5 matching thread T0, thread T1, thread T2 and thread T3. Different threads T are allocated to different sparse computing units in advance.

[0201] Similarly, the sparse computing unit also has a corresponding register mapping relationship for the metadata matrix secondary blocking and the index matrix secondary blocking. Based on the above register mapping relationship, it is possible to move each calculation data from the shared memory to the hardware register one by one.

[0202] In an optional implementation of the embodiment of the present invention, a preset hardware instruction can be called according to a register mapping relationship matching the sparse computing unit to secondary block each compressed data matrix and secondary block each metadata matrix, or to move each compressed data matrix secondary block, each index matrix secondary block and each metadata matrix secondary block from a shared memory to a hardware register one by one, so as to realize hardware acceleration of data loading.

[0203] The hardware instruction is associated with the specification of the computing chip and can be queried and obtained from the instruction library of the computing chip. For example, it can be a 1dmatrix instruction.

[0204] S490 . Divide each second association matrix into blocks twice according to the register mapping relationship matched with the sparse computing unit, and load them from the shared memory into the hardware register one by one.

[0205] As shown above, by querying and obtaining the register mapping relationship set by the sparse computing unit for the second sparse matrix secondary block, each second sparse matrix secondary block can be loaded from the shared memory to the hardware register one by one.

[0206] Similarly, in an optional implementation of an embodiment of the present invention, the preset hardware instructions can be called according to the register mapping relationship matching the sparse computing unit to divide each second sparse matrix into secondary blocks and load them one by one from the shared memory into the hardware register to achieve hardware acceleration of data loading.

[0207] It should be noted that, except for the first sparse operator, when the second and third sparse operators are transferred step by step from the global memory to the hardware registers of the computing chip in the form of data blocks, the data in the original global memory needs to be rearranged or deformed during the transfer process.

[0208] S4100. By using the sparse computing unit of the computing chip, the multiplication results of the first sparse matrix and the second association matrix are gradually calculated according to the data loaded in batches in the hardware register.

[0209] S4110. Process the multiplication result of the first sparse matrix and the second correlation matrix according to the processing logic of the corresponding functional function of the original hybrid expert model to obtain a sparse operator.

[0210] S4120. Update the original operator of the target hybrid expert model according to the sparse operator to obtain a structured sparse hybrid expert model.

[0211] Optionally, the multiplication result of the first sparse matrix and the second associated matrix can be processed according to the processing logic of the broadcast multiplication function of the original hybrid expert model to obtain a first sparse operator; the multiplication result of the first sparse matrix and the second associated matrix can be processed according to the processing logic of the activation function of the original hybrid expert model to obtain the second sparse operator; the multiplication result of the first sparse matrix and the second associated matrix can be processed according to the processing logic of the point product function of the original hybrid expert model to obtain a third sparse operator.

[0212] In an optional implementation of an embodiment of the present invention, the multiplication results of the first sparse matrix and the second sparse matrix are gradually calculated according to the data loaded in batches in the hardware register through the sparse computing unit of the computing chip, which may include: obtaining the current compressed data matrix secondary block, the current metadata matrix secondary block and the current second association matrix secondary block currently loaded in the hardware register through the sparse computing unit; generating a sparse block matrix according to the current metadata matrix secondary block and the current compressed data matrix secondary block through the sparse computing unit; performing multiplication calculation according to the first sparse block matrix and the current second association matrix secondary block through the sparse computing unit.

[0213] As mentioned above, when compressing and storing the first sparse matrix, the compressed data matrix only stores the non-zero data in each valid row, while in the actual first sparse matrix, each valid row contains at least one structured sparse computing storage unit in a set sparse mode. In order to ensure the accuracy of the multiplication calculation, it is necessary to first restore the current compressed data matrix secondary block to the form of the aforementioned structured sparse computing storage unit. Since the metadata matrix stores the position of the non-zero data in the structured sparse computing storage unit, a sparse block matrix can be generated based on the current metadata matrix secondary block and the current compressed data matrix secondary block.

[0214] That is, the primary sparse block matrix can be understood as a matrix after restoring the structured sparse computing storage unit in the secondary block of the current compressed data matrix. In a specific example, if the secondary block of the current compressed data matrix is ​​{A, B}, the corresponding secondary block of the current metadata matrix is ​​{0, 2}, and then, the primary sparse block matrix of the form {A, 0, B, 0} can be restored.

[0215] Among them, through the sparse computing unit, multiplication calculation is performed according to the first sparse block matrix and the current second association matrix secondary blocking, which can specifically include: through the sparse computing unit, the intermediate calculation amount generated in the multiplication calculation process is stored in the intermediate register of the computing chip; and, through the sparse computing unit, by obtaining the intermediate calculation amount from the intermediate register, performing matching multiplication calculation.

[0216] When the prior art uses a general computing unit to perform multiplication calculations, the results of the matrix multiplication calculations can be directly accumulated, but this essentially adds a lot of redundant data calculations. In contrast, when the sparse computing units of the various embodiments of the present invention perform sparse multiplication calculations, when the multiplication calculations are iterated along the K direction, since the sparse data comes from different rows, in order to ensure correctness, when different sub-blocks move along the K direction, the output results need to be mapped to different rows. Traditionally, passing the output results to a specific register based on the index may cause the output C matrix to be reloaded into the local memory, which has a significant impact on the performance of the operator.

[0217] In order to avoid the above problems, each embodiment of the present invention introduces an additional intermediate register C IR The frequent memory transfer between global memory and registers is reduced by Fig.12 FIG. 2 shows a schematic diagram of a structure in which a sparse computing unit and an intermediate register cooperate to implement a calculation in accordance with an embodiment of the present invention. Fig.12 As shown, in the sparse computing unit, the current compressed data matrix is ​​divided into two blocks ( Fig.12 A in), the current metadata matrix is ​​divided into two blocks ( Fig.12metadata in ) and the first second incidence matrix secondary block ( Fig.12 B) When performing sparse matrix multiplication calculations, for multiple result data, the required intermediate results are loaded into C during the calculation process. IR and use the intermediate results for calculation, and store the intermediate results back to the corresponding result data location C after the calculation is completed.

[0218] By adding a new intermediate register to the computing chip, which is a simple hardware improvement, the computing performance and efficiency can be greatly improved during the sparse matrix multiplication calculation of the sparse computing unit.

[0219] On the basis of the above embodiments, if the second association matrix includes a second sparse matrix, then after performing multiplication calculations through a sparse computing unit according to a primary sparse block matrix and a current second sparse matrix secondary block, it may also include: after determining that a complete multiplication calculation result is stored in an intermediate register, converting the complete multiplication calculation result into a secondary sparse block matrix according to a current index matrix secondary block that matches a current compressed data matrix secondary block; and writing the secondary sparse block matrix back to the global memory of the computing chip step by step.

[0220] If the second association matrix includes a dense matrix corresponding to the second sparse matrix, then after performing multiplication calculations through the sparse computing unit according to the first sparse block matrix and the second blocking of the current second association matrix, it may also include: after determining that the complete multiplication calculation result is stored in the intermediate register, writing the multiplication calculation result back to the global memory of the computing chip step by step in the original sparse format.

[0221] Further, since the sparse rows in the first sparse matrix are not considered when the complete multiplication result is calculated, if the right operand adopts the second sparse matrix, it is necessary to obtain the complete multiplication result again, and combine the current index matrix secondary block matching the current compressed data matrix secondary block, and add the matching sparse rows to the complete multiplication result to obtain the secondary sparse block matrix as the real block calculation result. As mentioned above, the current index matrix secondary block can also be moved to the hardware register, and can also be stored only in the shared memory, and this embodiment does not limit this. If the right operand adopts the dense matrix corresponding to the second sparse matrix, then after obtaining the complete multiplication result, the multiplication result can be directly written back to the global memory of the computing chip step by step according to the original sparse format, and there is no need to perform sparse processing on the multiplication result. As mentioned above, the multiplication result can be moved to the hardware register according to the original sparse format, and can also be stored only in the shared memory, and this embodiment does not limit this.

[0222] In summary, combined with the model architecture, matrix multiplication operations with sparse properties are selected from the hybrid expert model calculation flow. The sparsity comes from the sparsity of weights on the one hand, and the redundant properties in its parameters allow the calculation process to skip the calculation of parameters with lower importance; on the other hand, sparsity can also come from the hybrid expert model. The input value selects the expert parameter calculation that matches it through the gate circuit. From the expert's point of view, the input value has sparse characteristics, and only part of the input participates in the current expert's calculation. The selection of such sparse operators effectively improves the execution speed of the hybrid expert model without affecting the accuracy of the model. Based on the strategy of sparse data structure and operator fusion in data flow execution, referring to the compilation optimization scheme of sparse operators, specific sparse operators in the sparse operator library are selected and injected into the artificial intelligence framework and compiler to replace the original dense matrix multiplication operation of the hybrid expert model. For example, the two matrix multiplication operations that accept the original input in the expert architecture of the hybrid expert model will be injected with the operator of the double sparse mode, while the subsequent matrix multiplication that accepts the intermediate activation value will be injected with the unidirectional sparse (sparse × dense) operator.

[0223] The technical solution of the embodiment of the present invention fully considers the sparse characteristics when performing matrix multiplication calculations based on sparse matrices. By adopting data segmentation and data handling technology that are compatible with the sparse characteristics, the operator optimization of the hybrid expert model is achieved, and the hardware acceleration performance of the sparse computing unit in the computing chip is fully utilized in the model calculation scenario of the hybrid expert model, thereby greatly optimizing the computing, bandwidth and storage resource overhead in the sparse matrix multiplication process.

[0224] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data comply with relevant laws, regulations and standards in relevant regions.

[0225] It should be noted that any arrangement and combination of the technical features in the above embodiments also falls within the protection scope of the present invention.

[0226] Fig.13 is a schematic diagram of an optimization device for a hybrid expert model provided by an embodiment of the present invention, such as Fig.13 As shown, the device includes: a data structure sparse conversion module 510, a structured sparse data structure optimization module 520, a sparse operator optimization module 530 and a sparse operator update module 540, wherein:

[0227] The data structure sparse conversion module 510 is used to perform sparse conversion on each sparse data structure of the target hybrid expert model to obtain a structured sparse data structure;

[0228] A structured sparse data structure optimization module 520, used for optimizing the data arrangement of the structured sparse data structure;

[0229] A sparse operator optimization module 530, used to optimize the operator of the target hybrid expert model in combination with the sparse computing unit of the computing chip to obtain a sparse operator;

[0230] The sparse operator updating module 540 is used to update the original operator of the target hybrid expert model according to the sparse operator to obtain a structured sparse hybrid expert model.

[0231] The embodiment of the present invention performs sparse conversion on the sparse data structures of the target hybrid expert model to obtain a structured sparse data structure, and optimizes the data arrangement of the structured sparse data structure. Furthermore, the operator of the target hybrid expert model is optimized in combination with the sparse computing unit of the computing chip to obtain a sparse operator, and then the original operator of the target hybrid expert model is updated according to the sparse operator to obtain a structured sparse hybrid expert model. The above technical scheme can solve the problem that the performance of the existing hybrid expert model is difficult to effectively improve when the performance is optimized and accelerated based on the unstructured sparse computing technology, and can realize the structured sparse optimization of the hybrid expert model to give full play to the acceleration performance of the sparse computing hardware of the hybrid expert model when calculating sparse matrix multiplication, thereby greatly optimizing the calculation, bandwidth and storage resource overhead in the sparse matrix multiplication process.

[0232] Optionally, the data structure sparse conversion module 510 is also used to: load the target hybrid expert model into a computing chip, and obtain the sparse model weight matrices of each item in the target hybrid expert model; wherein the sparse model weight matrix includes multiple compression units of a set size, each of the compression units includes at least one valid row and at least one sparse row, and each of the valid rows includes at least one structured sparse computing storage unit; each of the structured sparse computing storage units is in the same sparsity mode; generate a compression matrix set corresponding to each of the model weight matrices; wherein the compression matrix set includes a compression data matrix for storing non-zero data in the model weight matrix, an index matrix for storing the position of the valid rows in the model weight matrix in the corresponding compression unit, and a metadata matrix for storing the position of the non-zero data in the corresponding structured sparse computing storage unit; the sparse model weight matrices of each item in the target hybrid expert model are stored as matching compression matrix sets to obtain a structured sparse data structure of the model weight data.

[0233] Optionally, the sparse model weight matrix specifically includes multiple compression units of size M*V; each of the compression units specifically includes d valid rows, where 1≤d<M; and each of the structured sparse computing storage units is in a sparse mode of N:L, L is the total number of data contained in the structured sparse computing storage unit, and N is the number of non-zero data contained in the structured sparse computing storage unit.

[0234] Optionally, the data structure sparse conversion module 510 is also used to: obtain the current model weight matrix of the size m*k being processed, where m is an integer multiple of M and k is an integer multiple of V; construct a first matrix of the size (m / M*d)*(k / (L / N)), and fill the non-zero elements of the current model weight matrix into the first matrix in a row-by-row traversal manner to obtain a compressed data matrix corresponding to the current model weight matrix; construct a second matrix of the size (m / M*d)*(k / V), and fill the second matrix according to the position of each compression unit in the current model weight matrix and the position of each valid row in the corresponding compression unit to obtain an index matrix corresponding to the current model weight matrix; construct a third matrix of the size (m / M*d)*(k / (L / N)), and fill the third matrix according to the position of each structured sparse computing storage unit in the current model weight matrix and the position of each non-zero data in the corresponding structured sparse computing storage unit to obtain a metadata matrix corresponding to the current model weight matrix.

[0235] Optionally, the data structure sparse conversion module 510 is also used to: traverse a current compression unit in the current model weight matrix in turn, and locate d vertical matrix positions matching the current compression unit in the second matrix according to the position of the current compression unit in the current model weight matrix; identify the row position of each valid row in the current compression unit, and fill the identified row positions in the d vertical matrix positions accordingly; return to execute the operation of traversing a current compression unit in the current model weight matrix in turn until the processing of all compression units in the current model weight matrix is ​​completed to obtain the index matrix corresponding to the current model weight matrix.

[0236] Optionally, the data structure sparse conversion module 510 is also used to: traverse a current structured sparse computing storage unit in the current model weight matrix in turn, and locate N horizontal matrix positions matching the current structured sparse computing storage unit in the third matrix according to the position of the current structured sparse computing storage unit in the current model weight matrix; fill the N horizontal matrix positions with the corresponding column positions of each non-zero data in the current structured sparse computing storage unit; return to execute the operation of traversing a current structured sparse computing storage unit in the current model weight matrix in turn until the processing of all the structured sparse computing storage units in the current model weight matrix is ​​completed to obtain the metadata matrix corresponding to the current model weight matrix.

[0237] Optionally, the above-mentioned device also includes a sparse computing module, which is used to: call each of the sparse operators on the computing chip, and implement matching sparse computing based on the target hybrid expert model after storage optimization.

[0238] Optionally, the above-mentioned device also includes a matrix set re-arrangement storage module, which is used to: obtain a sparse computing unit requirement file matching the computing chip; wherein the sparse computing unit requirement file defines the data access location of each model weight matrix in the target hybrid expert model by each thread executed in the sparse operator when performing sparse calculations; re-arrange the data arrangement of the compressed data matrix, index matrix and metadata matrix in each compressed matrix set according to the sparse computing unit requirement file; and store the re-arranged compressed matrix sets in the set memory of the computing chip to realize continuous storage of data with continuous access characteristics.

[0239] Optionally, the sparse computing module is also used to: identify each hybrid model layer included in the target hybrid expert model, and obtain the expert allocation pattern corresponding to each of the hybrid model layers; wherein the expert allocation pattern is used to indicate the row position of the valid row of intermediate activation values ​​allocated to its own hybrid model layer when implementing the calculation; generate a selection array corresponding to each of the hybrid model layers according to the expert allocation pattern; associate and store the selection array of each of the hybrid model layers with the compressed matrix set of each of the model weight matrices, so that when implementing the calculation, the selection array is combined with the sparse matrix or dense matrix of the intermediate activation value to be processed, and sparse calculation is performed with the matching compressed matrix set.

[0240] Optionally, the structured sparse data structure optimization module 520 is also used to: store the sparse converted model weight data of the target hybrid expert model in a transposed manner before the target hybrid expert model is run; and perform data transposition operations on intermediate activation values ​​when the sparse operator of the target hybrid expert model performs data operations in the shared memory.

[0241] Optionally, the sparse operator includes a first sparse operator, and the sparse operator optimization module 530 is also used to: locate a set of compressed matrices and a second associated matrix matching the first sparse matrix in the global memory of the computing chip according to the first sparse matrix multiplication requirement; wherein the second associated matrix includes a second sparse matrix or a dense matrix corresponding to the second sparse matrix; move the set of compressed matrices and the second associated matrix step by step from the global memory to the hardware registers of the computing chip in the form of data blocks; through the sparse computing unit of the computing chip, gradually calculate the multiplication results of the first sparse matrix and the second associated matrix according to the data loaded in batches in the hardware registers; process the multiplication results of the first sparse matrix and the second associated matrix according to the processing logic of the broadcast multiplication function of the target hybrid expert model to obtain the first sparse operator.

[0242] Optionally, the sparse operator includes a second sparse operator, and the sparse operator optimization module 530 is also used to: locate a set of compressed matrices and a second associated matrix matching the first sparse matrix in the global memory of the computing chip according to the second sparse matrix multiplication requirements; wherein the second associated matrix includes a second sparse matrix or a dense matrix corresponding to the second sparse matrix; the set of compressed matrices and the second associated matrix are moved step by step from the global memory to the hardware register of the computing chip in the form of data blocks, and during the movement, the data in the original global memory is rearranged or deformed; through the sparse computing unit of the computing chip, the multiplication results of the first sparse matrix and the second associated matrix are gradually calculated according to the data loaded in batches in the hardware register; the multiplication results of the first sparse matrix and the second associated matrix are processed according to the processing logic of the activation function of the target hybrid expert model to obtain the second sparse operator.

[0243] Optionally, the sparse operator includes a third sparse operator, and the sparse operator optimization module 530 is also used to: locate a set of compressed matrices and a second associated matrix matching the first sparse matrix in the global memory of the computing chip according to the third sparse matrix multiplication requirement; wherein the second associated matrix includes a second sparse matrix or a dense matrix corresponding to the second sparse matrix; the set of compressed matrices and the second associated matrix are moved step by step from the global memory to the hardware register of the computing chip in the form of data blocks, and during the movement, the data in the original global memory is rearranged or deformed; through the sparse computing unit of the computing chip, the multiplication results of the first sparse matrix and the second associated matrix are gradually calculated according to the data loaded in batches in the hardware register; the multiplication results of the first sparse matrix and the second associated matrix are processed according to the processing logic of the point multiplication function of the target hybrid expert model to obtain the third sparse operator.

[0244] Optionally, the first sparse matrix is ​​the sparse model weight matrix in the target hybrid expert model, the second sparse matrix is ​​the sparse matrix of intermediate activation values ​​input to the hybrid model layer in the target hybrid expert model, and the dense matrix corresponding to the second sparse matrix is ​​the dense matrix of intermediate activation values ​​input to the hybrid model layer in the target hybrid expert model.

[0245] Optionally, the sparse operator optimization module 530 is also used to: partition the compressed data matrix in the global memory into data blocks, and partition the index matrix and the metadata matrix into matching data blocks according to the data correspondence between the compressed data matrix, the index matrix and the metadata matrix; load each compressed data matrix block, each index matrix block and each metadata matrix block from the global memory to the shared memory one by one; perform secondary partitioning on each compressed data matrix block, each index matrix block and each metadata matrix block in the shared memory; and according to the register mapping relationship matching the sparse computing unit, move at least one of each compressed data matrix secondary block, each index matrix secondary block and each metadata matrix secondary block from the shared memory to the hardware register one by one.

[0246] Optionally, the sparse operator optimization module 530 is also used to: locate a selection array matching the second association matrix in the computing chip, wherein the selection array is used to describe the valid row positions in the second sparse matrix; the selection array of the second sparse matrix is ​​the index of the non-zero data positions therein, and the selection array of the dense matrix corresponding to the second sparse matrix is ​​the position mapping of the data in the dense matrix in the second sparse matrix; according to the selection array, block the data of the second association matrix in the global memory; move each second association matrix block from the global memory to the shared memory one by one; perform secondary blocking on each second association matrix block in the shared memory; and load each second association matrix secondary block from the shared memory to the hardware register one by one according to the register mapping relationship matching the sparse computing unit.

[0247] Optionally, the sparse operator optimization module 530 is also used to: use a preset data rearrangement function to rearrange each of the compressed data matrix blocks, each of the index matrix blocks, and each of the metadata matrix blocks in the shared memory to avoid storage conflicts; use the preset data rearrangement function to rearrange each of the second association matrix blocks in the shared memory to avoid storage conflicts.

[0248] Optionally, the sparse operator optimization module 530 is also used to: call preset hardware instructions according to the register mapping relationship matching the sparse computing unit, and move each of the compressed data matrix secondary blocks and each of the metadata matrix secondary blocks, or each of the compressed data matrix secondary blocks, each of the index matrix secondary blocks and each of the metadata matrix secondary blocks from the shared memory to the hardware register one by one to achieve hardware acceleration of data loading; call preset hardware instructions according to the register mapping relationship matching the sparse computing unit, and load each of the second association matrix secondary blocks from the shared memory to the hardware register one by one to achieve hardware acceleration of data loading.

[0249] Optionally, the sparse operator optimization module 530 is also used to: obtain, through the sparse computing unit, the current compressed data matrix secondary block, the current metadata matrix secondary block and the current second association matrix secondary block currently loaded in the hardware register; generate, through the sparse computing unit, a primary sparse block matrix according to the current metadata matrix secondary block and the current compressed data matrix secondary block; and perform multiplication calculations according to the primary sparse block matrix and the current second association matrix secondary block.

[0250] Optionally, the sparse operator optimization module 530 is also used to: store the intermediate calculation amount generated during the multiplication calculation process in the intermediate register of the computing chip through the sparse computing unit; and, through the sparse computing unit, perform matching multiplication calculations by obtaining the intermediate calculation amount from the intermediate register.

[0251] Optionally, if the second associative matrix includes a second sparse matrix, the sparse operator optimization module 530 is also used to: after determining that the complete multiplication calculation result is stored in the intermediate register, convert the complete multiplication calculation result into a secondary sparse block matrix according to the secondary blocking of the current index matrix that matches the secondary blocking of the current compressed data matrix; write the secondary sparse block matrix back to the global memory of the computing chip step by step; if the second associative matrix includes a dense matrix corresponding to the second sparse matrix, the sparse operator optimization module 530 is also used to: after determining that the complete multiplication calculation result is stored in the intermediate register, write the multiplication calculation result back to the global memory of the computing chip step by step in the original sparse format.

[0252] Optionally, in each data segmentation process, V is an integer multiple of the segmentation size in the column segmentation direction; and, in each of the sparse operators, the matrix specification of the first sparse matrix processed once is mi*ki, where V is an integer multiple of ki.

[0253] The above-mentioned hybrid expert model optimization device can execute the hybrid expert model optimization method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. For technical details not fully described in this embodiment, please refer to the hybrid expert model optimization method provided by any embodiment of the present invention.

[0254] Since the above-mentioned optimization device of the hybrid expert model is a device that can execute the optimization method of the hybrid expert model in the embodiment of the present invention, based on the optimization method of the hybrid expert model introduced in the embodiment of the present invention, the technical personnel of the field can understand the specific implementation method and various variations of the optimization device of the hybrid expert model in this embodiment, so how the optimization device of the hybrid expert model implements the optimization method of the hybrid expert model in the embodiment of the present invention is not described in detail here. As long as the technical personnel of the field implement the device used by the optimization method of the hybrid expert model in the embodiment of the present invention, it belongs to the scope of protection of this application.

[0255] Fig.14A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0256] like Fig.14 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0257] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0258] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the optimization method of the hybrid expert model.

[0259] Optionally, the optimization method of the hybrid expert model may include: performing sparse conversion on the sparse data structures of the target hybrid expert model to obtain a structured sparse data structure; performing data arrangement optimization on the structured sparse data structure; optimizing the operator of the target hybrid expert model in combination with the sparse computing unit of the computing chip to obtain a sparse operator; and updating the original operator of the target hybrid expert model according to the sparse operator to obtain a structured sparse hybrid expert model.

[0260] In some embodiments, the optimization method of the hybrid expert model may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the optimization method of the hybrid expert model described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the optimization method of the hybrid expert model in any other appropriate manner (e.g., by means of firmware).

[0261] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0262] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0263] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0264] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0265] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0266] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0267] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0268] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A hybrid expert model optimization method, characterized in that: include: Perform sparse transformation on each sparse data structure of the target hybrid expert model to obtain a structured sparse data structure; Optimizing data arrangement of the structured sparse data structure; Optimizing the operator of the target hybrid expert model in combination with a sparse computing unit of a computing chip to obtain a sparse operator; The original operator of the target hybrid expert model is updated according to the sparse operator to obtain a structured sparse hybrid expert model.

2. The method according to claim 1, characterized in that The sparse conversion of each sparse data structure of the target hybrid expert model is performed to obtain a structured sparse data structure, including: Loading the target hybrid expert model into a computing chip, and obtaining a sparse model weight matrix of each item in the target hybrid expert model; The sparse model weight matrix includes a plurality of compression units of set sizes, each of the compression units includes at least one valid row and at least one sparse row, each of the valid rows includes at least one structured sparse computing storage unit; each of the structured sparse computing storage units is in the same sparse mode; Generate a set of compression matrices corresponding to each of the model weight matrices; The compression matrix set includes a compression data matrix for storing non-zero data in the model weight matrix, an index matrix for storing the position of valid rows in the model weight matrix in the corresponding compression unit, and a metadata matrix for storing the position of non-zero data in the corresponding structured sparse computing storage unit; The sparse model weight matrices of each item in the target hybrid expert model are stored as matching compressed matrix sets respectively, so as to obtain a structured sparse data structure of the model weight data.

3. The method according to claim 2, characterized in that The sparse model weight matrix specifically includes a plurality of compression units of size M*V; Specifically, each compression unit includes d valid rows, wherein 1≤d<M; and Each of the structured sparse computing storage units is in a sparse mode of N:L, where L is the total amount of data contained in the structured sparse computing storage unit, and N is the amount of non-zero data contained in the structured sparse computing storage unit.

4. The method according to claim 3, characterized in that The generating a set of compressed matrices corresponding to each of the model weight matrices comprises: Get the current model weight matrix of the current processing size m*k, where m is an integer multiple of M and k is an integer multiple of V; Construct a first matrix of size (m / M*d)*(k / (L / N)), and fill each non-zero element of the current model weight matrix into the first matrix in a row-by-row traversal manner to obtain a compressed data matrix corresponding to the current model weight matrix; Constructing a second matrix of size (m / M*d)*(k / V), and filling the second matrix according to the position of each compression unit in the current model weight matrix and the position of each valid row in the corresponding compression unit, to obtain an index matrix corresponding to the current model weight matrix; Construct a third matrix of size (m / M*d)*(k / (L / N)), and fill the third matrix according to the position of each structured sparse computing storage unit in the current model weight matrix and the position of each non-zero data in the corresponding structured sparse computing storage unit to obtain a metadata matrix corresponding to the current model weight matrix.

5. The method according to claim 4, characterized in that The filling process is performed on the second matrix according to the position of each compression unit in the current model weight matrix and the position of each valid row in the corresponding compression unit to obtain an index matrix corresponding to the current model weight matrix, including: Traversing a current compression unit in the current model weight matrix in turn, and locating d longitudinal matrix positions matching the current compression unit in the second matrix according to the position of the current compression unit in the current model weight matrix; Identifying the row positions of the valid rows in the current compression unit, and filling the identified row positions into d longitudinal matrix positions accordingly; Return to execute the operation of traversing one current compression unit in the current model weight matrix in turn until all compression units in the current model weight matrix are processed to obtain the index matrix corresponding to the current model weight matrix.

6. The method according to claim 4, characterized in that The method of filling the third matrix according to the position of each structured sparse computing storage unit in the current model weight matrix and the position of each non-zero data in the corresponding structured sparse computing storage unit to obtain a metadata matrix corresponding to the current model weight matrix includes: Traversing a current structured sparse computing storage unit in the current model weight matrix in turn, and locating N horizontal matrix positions matching the current structured sparse computing storage unit in the third matrix according to the position of the current structured sparse computing storage unit in the current model weight matrix; Fill the column positions of each non-zero data in the current structured sparse computing storage unit into the N horizontal matrix positions; Return to execute the operation of traversing one current structured sparse computing storage unit in the current model weight matrix in turn until the processing of all the structured sparse computing storage units in the current model weight matrix is ​​completed to obtain the metadata matrix corresponding to the current model weight matrix.

7. The method according to any one of claims 2 to 6, characterized in that: After the model weight matrices of the sparse items in the target hybrid expert model are stored as matching compressed matrix sets, the method further includes: The sparse operators on the computing chip are called to implement matching sparse calculations based on the target hybrid expert model after storage optimization.

8. The method according to claim 7, characterized in that After the model weight matrices of the sparse items in the target hybrid expert model are stored as matching compressed matrix sets, the method further includes: Obtaining a sparse computing unit requirement file matching the computing chip; The sparse computing unit requirement file defines the data access location of each model weight matrix in the target hybrid expert model by each thread executed in the sparse operator when performing sparse computing; According to the sparse computing unit requirement file, re-arrange the data arrangement of the compressed data matrix, the index matrix and the metadata matrix in each of the compressed matrix sets; The reorganized compression matrix sets are stored in a setting memory of a computing chip to achieve continuous storage of data with continuous access characteristics.

9. The method according to claim 8, characterized in that Before calling each of the sparse operators on the computing chip to implement the matched sparse calculation based on the target hybrid expert model after storage optimization, the method further includes: Identify each hybrid model layer included in the target hybrid expert model, and obtain the expert allocation mode corresponding to each of the hybrid model layers; The expert allocation mode is used to indicate the row position of the valid row of the intermediate activation value allocated to the self-mixed model layer when performing the calculation; According to the expert allocation mode, generating selection arrays corresponding to each of the hybrid model layers; The selection array of each hybrid model layer is associated with the compressed matrix set of each model weight matrix and stored so that when performing calculations, the selection array is combined with the sparse matrix or dense matrix of the intermediate activation values ​​to be processed and sparse calculations are performed with the matching compressed matrix set.

10. The method according to claim 1, characterized in that The step of optimizing the data arrangement of the structured sparse data structure includes: Before the target hybrid expert model is run, storing the sparsely converted model weight data of the target hybrid expert model in a transposed manner; When the sparse operator of the target hybrid expert model performs data operations in the shared memory, a data transposition operation is performed on the intermediate activation value.

11. The method according to claim 3, characterized in that The sparse operator includes a first sparse operator, and the sparse computing unit combined with the computing chip optimizes the operator of the target hybrid expert model to obtain a sparse operator, including: According to the first sparse matrix multiplication requirement, locate a set of compressed matrices matching the first sparse matrix and a second association matrix in the global memory of the computing chip; wherein the second association matrix includes the second sparse matrix or a dense matrix corresponding to the second sparse matrix; The compressed matrix set and the second association matrix are transferred from the global memory to the hardware register of the computing chip in the form of data blocks step by step; The sparse computing unit of the computing chip gradually calculates the multiplication result of the first sparse matrix and the second association matrix according to the data loaded in batches in the hardware register; The multiplication result of the first sparse matrix and the second association matrix is ​​processed according to the processing logic of the broadcast multiplication function of the target hybrid expert model to obtain the first sparse operator.

12. The method according to claim 3, characterized in that The sparse operator includes a second sparse operator, and the sparse computing unit combined with the computing chip optimizes the operator of the target hybrid expert model to obtain a sparse operator, including: According to the second sparse matrix multiplication requirement, locate a set of compressed matrices and a second association matrix matching the first sparse matrix in the global memory of the computing chip; wherein the second association matrix includes the second sparse matrix or a dense matrix corresponding to the second sparse matrix; The compressed matrix set and the second association matrix are transferred from the global memory to the hardware register of the computing chip in the form of data blocks, and during the transfer process, the data in the original global memory is rearranged or deformed; The sparse computing unit of the computing chip gradually calculates the multiplication result of the first sparse matrix and the second association matrix according to the data loaded in batches in the hardware register; The multiplication result of the first sparse matrix and the second association matrix is ​​processed according to the processing logic of the activation function of the target hybrid expert model to obtain the second sparse operator.

13. The method according to claim 3, characterized in that The sparse operator includes a third sparse operator, and the sparse computing unit combined with the computing chip optimizes the operator of the target hybrid expert model to obtain a sparse operator, including: According to the third sparse matrix multiplication requirement, locate a set of compressed matrices matching the first sparse matrix and a second association matrix in the global memory of the computing chip; wherein the second association matrix includes the second sparse matrix or a dense matrix corresponding to the second sparse matrix; The compressed matrix set and the second association matrix are transferred from the global memory to the hardware register of the computing chip in the form of data blocks, and during the transfer process, the data in the original global memory is rearranged or deformed; The sparse computing unit of the computing chip gradually calculates the multiplication result of the first sparse matrix and the second association matrix according to the data loaded in batches in the hardware register; The multiplication result of the first sparse matrix and the second association matrix is ​​processed according to the processing logic of the point multiplication function of the target hybrid expert model to obtain the third sparse operator.

14. The method according to any one of claims 11 to 13, characterized in that: The first sparse matrix is ​​the sparse model weight matrix in the target hybrid expert model, the second sparse matrix is ​​the sparse matrix of intermediate activation values ​​input to the hybrid model layer in the target hybrid expert model, and the dense matrix corresponding to the second sparse matrix is ​​the dense matrix of intermediate activation values ​​input to the hybrid model layer in the target hybrid expert model.

15. The method according to claim 14, characterized in that The compressed matrix set is transferred from the global memory to the hardware register of the computing chip in the form of data blocks, including: The compressed data matrix in the global memory is divided into data blocks, and the index matrix and the metadata matrix are matched into data blocks according to the data correspondence between the compressed data matrix, the index matrix and the metadata matrix; Loading each compressed data matrix block, each index matrix block, and each metadata matrix block from the global memory to the shared memory one by one; Secondarily partitioning each of the compressed data matrix blocks, each of the index matrix blocks, and each of the metadata matrix blocks in the shared memory; According to the register mapping relationship matched with the sparse computing unit, at least one item of each compressed data matrix secondary block, each index matrix secondary block and each metadata matrix secondary block is moved from the shared memory to the hardware register one by one.

16. The method according to claim 15, characterized in that While locating the set of compressed matrices matching the first sparse matrix and the second association matrix in the global memory of the computing chip, the method further includes: Locating a selection array matching the second association matrix in the computing chip, wherein the selection array is used to describe valid row positions in the second sparse matrix; the selection array of the second sparse matrix is ​​the index of the non-zero data position therein, and the selection array of the dense matrix corresponding to the second sparse matrix is ​​the position mapping of the data in the dense matrix in the second sparse matrix; Accordingly, the second association matrix is ​​moved from the global memory to the hardware register of the computing chip in the form of data blocks step by step, including: According to the selection array, dividing the second association matrix in the global memory into data blocks; Divide each second association matrix into blocks, and transfer them from the global memory to the shared memory one by one; Perform secondary block division on each of the second association matrix blocks in the shared memory; According to the register mapping relationship matching the sparse computing unit, each second association matrix is ​​divided into two blocks and loaded from the shared memory to the hardware register one by one.

17. The method according to claim 16, characterized in that After loading each compressed data matrix block, each index matrix block, and each metadata matrix block from the global memory to the shared memory one by one, the method further includes: Using a preset data rearrangement function, rearrange the compressed data matrix blocks, the index matrix blocks, and the metadata matrix blocks in the shared memory to avoid storage bank conflicts; and After dividing each second correlation matrix into blocks and transferring them from the global memory to the shared memory one by one, the method further includes: The preset data rearrangement function is used to rearrange each of the second association matrix blocks in the shared memory to avoid storage bank conflicts.

18. The method according to claim 16, characterized in that According to the register mapping relationship matched with the sparse computing unit, at least one item of each compressed data matrix secondary block, each index matrix secondary block and each metadata matrix secondary block is moved from the shared memory to the hardware register one by one, including: According to the register mapping relationship matched with the sparse computing unit, calling the preset hardware instruction, each of the compressed data matrix secondary blocks and each of the metadata matrix secondary blocks, or each of the compressed data matrix secondary blocks, each of the index matrix secondary blocks and each of the metadata matrix secondary blocks, are moved from the shared memory to the hardware register one by one, so as to realize the hardware acceleration of data loading; and According to the register mapping relationship matching the sparse computing unit, each second association matrix is ​​divided into blocks twice, and loaded from the shared memory to the hardware register one by one, including: According to the register mapping relationship matching the sparse computing unit, the preset hardware instructions are called to divide each of the second association matrices into blocks twice, and load them from the shared memory to the hardware registers one by one to achieve hardware acceleration of data loading.

19. The method according to claim 16, characterized in that The multiplication result of the first sparse matrix and the second association matrix is ​​calculated step by step according to the data loaded in batches in the hardware register by the sparse computing unit of the computing chip, including: Obtaining, by means of the sparse computing unit, the current compressed data matrix secondary block, the current metadata matrix secondary block, and the current second association matrix secondary block currently loaded in the hardware register; Generate a primary sparse block matrix according to the secondary block of the current metadata matrix and the secondary block of the current compressed data matrix by the sparse computing unit; The sparse computing unit performs multiplication calculation according to the primary sparse block matrix and the secondary block of the current second correlation matrix.

20. The method according to claim 19, characterized in that The sparse computing unit performs a multiplication calculation according to the primary sparse block matrix and the secondary block of the current second correlation matrix, specifically including: The intermediate calculation amount generated in the multiplication calculation process is stored in the intermediate register of the computing chip through the sparse computing unit; and The sparse computing unit performs a matching multiplication calculation by acquiring the intermediate calculation amount from the intermediate register.

21. The method according to claim 20, characterized in that If the second incidence matrix includes a second sparse matrix, after performing multiplication calculation according to the first sparse block matrix and the second current incidence matrix second block by the sparse computing unit, the method further includes: After determining that the intermediate register stores a complete multiplication calculation result, converting the complete multiplication calculation result into a secondary sparse block matrix according to a current index matrix secondary block matching the current compressed data matrix secondary block; Writing the secondary sparse block matrix back to the global memory of the computing chip step by step; If the second incidence matrix includes a dense matrix corresponding to the second sparse matrix, after performing multiplication calculation according to the first sparse block matrix and the second second incidence matrix second block by the sparse computing unit, the method further includes: After determining that the complete multiplication calculation result is stored in the intermediate register, the multiplication calculation result is written back to the global memory of the computing chip step by step according to the original sparse format.

22. The method according to claim 14, characterized in that In each data segmentation process, V is an integer multiple of the segmentation size in the column segmentation direction; and The matrix specification of the first sparse matrix processed once in each of the sparse operators is mi*ki, where V is an integer multiple of ki.

23. An optimization device for a hybrid expert model, characterized in that: include: The data structure sparse conversion module is used to perform sparse conversion on the sparse data structures of the target hybrid expert model to obtain a structured sparse data structure; A structured sparse data structure optimization module, used for optimizing the data arrangement of the structured sparse data structure; A sparse operator optimization module, used to optimize the operator of the target hybrid expert model in combination with the sparse computing unit of the computing chip to obtain a sparse operator; The sparse operator updating module is used to update the original operator of the target hybrid expert model according to the sparse operator to obtain a structured sparse hybrid expert model.

24. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the optimization method of the hybrid expert model described in any one of claims 1-22.

25. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the optimization method of the hybrid expert model described in any one of claims 1-22 when executed.

26. A computer program product comprising a computer program / instructions, wherein: When the computer program / instructions are executed by a processor, the optimization method of the hybrid expert model described in any one of claims 1-22 is implemented.

Images