Storage optimization method and device for hybrid expert model, equipment, medium and product

Abstract

The invention discloses a storage optimization method and device for a hybrid expert model, equipment, a medium and a product. The method comprises the following steps: loading a target hybrid expert model into a computing chip, and obtaining various sparse model weight matrixes in the target hybrid expert model; the sparse model weight matrix comprises a plurality of compression units with set sizes, each compression unit comprises at least one effective row and at least one sparse row, and each effective row comprises at least one structured sparse calculation storage unit; generating a compression matrix set corresponding to each model weight matrix; the compression matrix set comprises a compression data matrix, an index matrix and a metadata matrix; and storing each sparse model weight matrix in the target hybrid expert model as a matched compression matrix set so as to realize storage optimization of the target hybrid expert model. According to the technical scheme provided by the embodiment of the invention, the storage resource overhead of the hybrid expert model can be effectively optimized.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a storage optimization method, device, equipment, medium and product of a hybrid expert model. Background Art

[0002] Sparse data representation and arrangement optimization technology is a technical solution that can effectively optimize computing, bandwidth and storage resource overhead during model training, reasoning and fine-tuning.

[0003] In the prior art, when performing model calculation, the hybrid expert model mainly involves sparsely compressing the sparse weight data to be calculated and the input sparse intermediate activation data through a specified sparse representation format, and arranging them in a specified storage format in the memory for subsequent calculations.

[0004] In the process of implementing the present invention, the inventors found that the calculation between sparse intermediate activation data and sparse weight data belongs to double sparse calculation. However, the current mainstream sparse compression and orchestration optimization technology lacks efficient representation of double sparse features in the calculation process of the hybrid expert model. Therefore, directly using the existing sparse compression and orchestration optimization scheme cannot fully optimize the calculation process of the hybrid expert model. Summary of the invention

[0005] The embodiments of the present invention provide a storage optimization method, device, equipment, medium and product for a hybrid expert model, and creatively propose an efficient data compression method for a sparse model weight matrix in a hybrid expert model to effectively optimize the storage resource overhead of the hybrid expert model.

[0006] According to one aspect of an embodiment of the present invention, a storage optimization method of a hybrid expert model is provided, comprising:

[0007] Loading the target hybrid expert model into the computing chip, and obtaining the sparse model weight matrix of each item in the target hybrid expert model;

[0008] The sparse model weight matrix includes a plurality of compression units of set sizes, each compression unit includes at least one valid row and at least one sparse row, each valid row includes at least one structured sparse computing storage unit; each structured sparse computing storage unit is in the same sparse mode;

[0009] Generate a set of compressed matrices corresponding to each model weight matrix;

[0010] The compressed matrix set includes a compressed data matrix for storing non-zero data in the model weight matrix, an index matrix for storing the position of valid rows in the model weight matrix in the corresponding compression unit, and a metadata matrix for storing the position of non-zero data in the corresponding structured sparse computing storage unit;

[0011] The sparse model weight matrices of each item in the target hybrid expert model are stored as matching compressed matrix sets to achieve storage optimization of the target hybrid expert model.

[0012] According to another aspect of an embodiment of the present invention, a storage optimization device of a hybrid expert model is also provided, comprising:

[0013] A model weight matrix acquisition module is used to load the target hybrid expert model into the computing chip and obtain the sparse model weight matrix of each item in the target hybrid expert model;

[0014] The sparse model weight matrix includes a plurality of compression units of set sizes, each compression unit includes at least one valid row and at least one sparse row, each valid row includes at least one structured sparse computing storage unit; each structured sparse computing storage unit is in the same sparse mode;

[0015] A compression matrix set generation module, used to generate a compression matrix set corresponding to each model weight matrix;

[0016] The compressed matrix set includes a compressed data matrix for storing non-zero data in the model weight matrix, an index matrix for storing the position of valid rows in the model weight matrix in the corresponding compression unit, and a metadata matrix for storing the position of non-zero data in the corresponding structured sparse computing storage unit;

[0017] The storage optimization module is used to store the sparse model weight matrices of each item in the target hybrid expert model as a matching compressed matrix set to achieve storage optimization of the target hybrid expert model.

[0018] According to another aspect of an embodiment of the present invention, an electronic device is provided, the electronic device comprising:

[0019] at least one computing chip; and

[0020] A memory communicatively connected to the at least one computing chip; wherein,

[0021] The memory stores a computer program that can be executed by the at least one computing chip, and the computer program is executed by the at least one computing chip so that the at least one computing chip can execute the storage optimization method of the hybrid expert model described in any embodiment of the present invention.

[0022] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to implement the storage optimization method of the hybrid expert model described in any embodiment of the present invention when a computing chip executes the computer instructions.

[0023] According to another aspect of an embodiment of the present invention, a computer program product is also provided, comprising computer instructions, which, when executed by a computing chip, implement the steps of the storage optimization method of the hybrid expert model as described in any embodiment of the present invention.

[0024] The technical solution of the embodiment of the present invention converts the sparse model weight matrices of various specific formats in the target hybrid expert model into a compressed data matrix for storing non-zero data in the model weight matrix, an index matrix for storing the positions of valid rows in the model weight matrix in the corresponding compression unit, and a metadata matrix for storing the positions of non-zero data in the corresponding structured sparse computing storage unit, and then optimizes the storage in a computing chip used to implement hybrid expert model calculations. This can effectively reduce the storage resource overhead of the computing chip. In addition, when using the model weight matrix in the above-mentioned data compression form for model calculations, the sparse computing hardware in the computing chip can be fully utilized, which effectively improves the computing efficiency while reducing the bandwidth overhead.

[0025] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 is a flow chart of a storage optimization method of a hybrid expert model provided according to an embodiment of the present invention;

[0028] Figure 2 It is a schematic diagram of a comparison structure of a model weight matrix and a matching compression matrix set applicable to an embodiment of the present invention;

[0029] Figure 3 is a flow chart of another storage optimization method of a hybrid expert model provided by an embodiment of the present invention;

[0030] Figure 4 It is a comparison diagram of a sparse computing unit requirement and a matching data re-arrangement result applicable to an embodiment of the present invention;

[0031] Figure 5 It is a schematic diagram of a comparison structure of an intermediate activation value sparse matrix and a matching selection array applicable to an embodiment of the present invention;

[0032] Figure 6 is a structural schematic diagram of a storage optimization device of a hybrid expert model provided according to an embodiment of the present invention;

[0033] Figure 7 It is a structural schematic diagram of an electronic device for implementing the storage optimization method of the hybrid expert model of an embodiment of the present invention;

[0034] Figure 8 It is a structural diagram of a computing chip applicable to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] Figure 1The flowchart of a storage optimization method of a hybrid expert model provided by an embodiment of the present invention is applicable to the case where each sparse model weight matrix in the hybrid expert model is converted into a matching compressed matrix set and then optimized for storage in a computing chip used to implement hybrid expert model calculation. The method can be executed by a storage optimization device for a hybrid expert model, which can be implemented in the form of hardware and / or software and can generally be configured in an electronic device where the computing chip is located. Figure 1 As shown, the method includes:

[0038] S110, loading the target hybrid expert model into the computing chip, and obtaining the sparse model weight matrix of each item in the target hybrid expert model.

[0039] Optionally, the computing chip can be understood as an integrated circuit used to implement a set computing task (for example, Internet of Things control, high-performance computing or mobile computing, etc.). The computing chip can be a general computing chip, such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or a dedicated computing chip, such as an NPU (Neural Processing Unit), which is not limited in this embodiment.

[0040] The target hybrid expert model refers to a specific hybrid expert model that is loaded into a computing chip to implement model reasoning tasks. The hybrid expert model is an efficient neural network architecture that improves overall performance by combining multiple specialized sub-models. The core idea of ​​this model is to integrate different "expert" networks together, with each expert playing a role in his or her area of ​​expertise. These experts can be small multilayer perceptrons (MLP) or more complex large language models (LLM). The model usually contains a gating network that is responsible for deciding which expert should be activated and involved in the calculation of the output when processing a specific input.

[0041] Specifically, the hybrid expert model can be applied in the fields of computer vision, natural language processing, medical treatment, or autonomous driving;

[0042] The computer vision field includes image classification, target detection or face recognition; the natural language processing field includes text classification, machine translation or speech recognition, etc.

[0043] In order to facilitate the implementation of subsequent model calculations, in this embodiment, the model weight matrix of each model layer in the target hybrid expert model is processed into a sparse model weight matrix of a specific structure.

[0044] Among them, the sparse model weight matrix includes multiple compression units of set sizes, each compression unit includes at least one valid row and at least one sparse row, each valid row includes at least one structured sparse computing storage unit; each structured sparse computing storage unit is in the same sparse mode.

[0045] Specifically, each model weight matrix in the target hybrid expert model contains an integer number (one or more) of compression units, that is, the size (number of rows * number of columns) of each model weight matrix can divide the size of the compression unit. The compression unit can be understood as a small matrix of a specific size, which has at least two matrix rows. The at least two matrix rows have at least one valid row and at least one sparse row. A valid row can be understood as a matrix row containing at least one valid data (non-0), and a sparse row refers to a matrix row where all data is 0.

[0046] Furthermore, each valid row in the compression unit contains an integer number (one or more) of structured sparse computing storage units. Among them, each structured sparse computing storage unit contained in the model weight matrix is ​​in the same sparse mode. Generally speaking, in order to facilitate calculation, the size of the structured sparse computing storage unit can be adapted to the calculation scale of the hardware computing unit in the computing chip.

[0047] The sparse pattern can be understood as the ratio of valid data (non-zero value data) to all data. For example, the sparse pattern can be 2:4 or 4:8. That is, for a 2:4 sparse pattern, the structured sparse computing storage unit contains a total of 4 data, 2 of which are non-zero data. The arrangement positions of the above two non-zero data in the structured sparse computing storage unit are not restricted.

[0048] In this embodiment, two methods can be used to make each model weight matrix in the target hybrid expert model be in the special structure clearly defined above: one is to perform strong constraints on the model weight matrix in each model layer during the training stage of the target hybrid expert model so that each model weight matrix meets the requirements of the special structure; the other is to perform strong constraints on the model weight matrix in each model layer during the model fine-tuning stage after model pre-training so that each model weight matrix meets the requirements of the special structure.

[0049] In this embodiment, the reason why each model weight matrix in the target hybrid expert model is limited to the above-mentioned special structure is to improve the computational efficiency and better adapt to the specific hardware computing unit in the computing chip when the subsequent calculation is performed based on the target hybrid expert model. The determination method of the above-mentioned special structure is not described in detail here. That is, the model weight matrix is ​​adapted to the computing performance of the sparse computing unit in the computing chip (the computing scale of the included computing components, the number of included registers, and the number of cache buffers, etc.).

[0050] S120. Generate a set of compression matrices corresponding to each model weight matrix.

[0051] As mentioned above, each model weight matrix contains a large amount of zero-valued data. If the above model weight matrices are directly stored in the computing chip according to the original size of the model weight matrix, a large amount of storage units will be wasted. In addition, the subsequent model calculations are directly based on the model weight matrix of the original size, which has low computational efficiency and cannot efficiently use the hardware computing units in the computing chip.

[0052] In view of this, in this embodiment, a new type of efficient data compression method is creatively proposed for the model weight matrix with the above special structure to solve the above technical problems.

[0053] In this embodiment, each model weight matrix can be compactly stored in the form of a compressed matrix set through a specific data compression method. Specifically, three compressed matrices corresponding to the model weight matrix are stored in the compressed matrix set, namely, a compressed data matrix, an index matrix, and a metadata matrix.

[0054] Among them, the compressed data matrix is ​​used to store the compressed data matrix of non-zero data in the model weight matrix, the index matrix is ​​used to store the position of valid rows in the model weight matrix in the corresponding compression unit, and the metadata matrix is ​​used to store the position of non-zero data in the corresponding structured sparse computing storage unit.

[0055] Obviously, through the above-mentioned data compression method, the specific position of each non-zero data in the model weight matrix in the original model weight matrix can be clearly identified through three small matrices. This data compression method can also effectively reduce the consumption of storage resources in the computing chip, and can also facilitate the implementation of calculations based on the model weight matrix.

[0056] S130, storing the sparse model weight matrices of each item in the target hybrid expert model as matching compressed matrix sets, so as to achieve storage optimization of the target hybrid expert model.

[0057] Specifically, when it is necessary to store the sparse model weight matrices in the target mixed expert model in the storage space of the computing chip (typically, global memory), the compressed matrix sets obtained after compression are used to replace the sparse model weight matrices for storage to achieve storage optimization.

[0058] The technical solution of the embodiment of the present invention converts the sparse model weight matrices of various specific formats in the target hybrid expert model into a compressed data matrix for storing non-zero data in the model weight matrix, an index matrix for storing the positions of valid rows in the model weight matrix in the corresponding compression unit, and a metadata matrix for storing the positions of non-zero data in the corresponding structured sparse computing storage unit, and then optimizes the storage in a computing chip used to implement hybrid expert model calculations. This can effectively reduce the storage resource overhead of the computing chip. In addition, when using the model weight matrix in the above-mentioned data compression form for model calculations, the sparse computing hardware in the computing chip can be fully utilized, which effectively improves the computing efficiency while reducing the bandwidth overhead.

[0059] In an optional implementation of the present embodiment, the sparse model weight matrix specifically includes multiple compression units of size M*V; each compression unit specifically includes d valid rows, where 1≤d<M; and each structured sparse computing storage unit is in a sparse mode of N:L, L is the total number of data contained in the structured sparse computing storage unit, and N is the number of non-zero data contained in the structured sparse computing storage unit.

[0060] Accordingly, generating a set of compressed matrices corresponding to each model weight matrix may include:

[0061] Get the current model weight matrix of the current processing size m*k, where m is an integer multiple of M and k is an integer multiple of V;

[0062] Construct a first matrix of size (m / M*d)*(k / (L / N)), and fill each non-zero element of the current model weight matrix into the first matrix in a row-by-row traversal manner to obtain a compressed data matrix corresponding to the current model weight matrix;

[0063] Construct a second matrix of size (m / M*d)*(k / V), and fill the second matrix according to the position of each compression unit in the current model weight matrix and the position of each valid row in the corresponding compression unit to obtain an index matrix corresponding to the current model weight matrix;

[0064] Construct a third matrix of size (m / M*d)*(k / (L / N)), and fill the third matrix according to the position of each structured sparse computing storage unit in the current model weight matrix and the position of each non-zero data in the corresponding structured sparse computing storage unit to obtain a metadata matrix corresponding to the current model weight matrix.

[0065] For ease of explanation, Figure 2 A schematic diagram of the comparison structure of a model weight matrix and a matching compression matrix set applicable to various embodiments of the present invention is shown in FIG.

[0066] Specifically, Figure 2 The figure shows a schematic diagram of generating a matching set of compression matrices for a model weight matrix of m=4, k=16. In the model weight matrix, there are 4 compression units of M=2, V=8, each of which contains d=1 valid rows, each of which contains 2 structured sparse computing storage units, and each structured sparse computing storage unit is in a sparse mode of N:L=2:4. That is, A, B, ..., L filled in each matrix position of the model weight matrix represent non-zero data, and the blank position represents the data filled with zero value.

[0067] When constructing the compressed data matrix of the model weight matrix, first construct a first matrix of (m / M*d)*(k / (L / N))=2*8, and then fill the non-zero data A, B,..., L in the model weight matrix into the first matrix in a row-by-row traversal manner to obtain a matching compressed data matrix.

[0068] Similarly, when constructing the index matrix for the model weight matrix, first construct a second matrix of (m / M*d)*(k / V)=2*2. Then, according to the position of each compression unit in the current model weight matrix and the position of each valid row in the corresponding compression unit, the second matrix is ​​filled to obtain the index matrix corresponding to the current model weight matrix.

[0069] Specifically, in an optional implementation of this embodiment, according to the position of each compression unit in the current model weight matrix and the position of each valid row in the corresponding compression unit, the second matrix is ​​filled to obtain an index matrix corresponding to the current model weight matrix, which may specifically include:

[0070] Traversing a current compression unit in the current model weight matrix in turn, and locating d longitudinal matrix positions matching the current compression unit in the second matrix according to the position of the current compression unit in the current model weight matrix;

[0071] Identify the row positions of each valid row in the current compression unit, and fill the identified row positions into d longitudinal matrix positions accordingly;

[0072] Returns the operation of traversing one current compression unit in the current model weight matrix in turn until all compression units in the current model weight matrix are processed to obtain an index matrix corresponding to the current model weight matrix.

[0073] like Figure 2 As shown, in the model weight matrix, a 2*8 compression unit 1 including A, B, C, and D is first traversed. Since d=1, the corresponding upper left vertical matrix position can be located in the second matrix based on the upper left corner position of the compression unit 1 in the model weight matrix. Afterwards, the row position of the valid row is identified in the compression unit 1, that is, the row where A, B, C, and D are located, that is, the 0th row. Afterwards, 0 is filled into the upper left matrix position in the second matrix. And so on, until the row position of each valid row in the four compression units in the model weight matrix is ​​filled into the second matrix accordingly, and a matching index matrix is ​​obtained.

[0074] Similarly, when constructing the metadata matrix for the model weight matrix, first construct a third matrix of (m / M*d)*(k / (L / N))=2*8. Then, according to the position of each structured sparse computing storage unit in the current model weight matrix and the position of each non-zero data in the corresponding structured sparse computing storage unit, the third matrix is ​​filled to obtain the metadata matrix corresponding to the current model weight matrix.

[0075] Accordingly, in an optional implementation of this embodiment, according to the position of each structured sparse computing storage unit in the current model weight matrix, and the position of each non-zero data in the corresponding structured sparse computing storage unit, the third matrix is ​​filled to obtain a metadata matrix corresponding to the current model weight matrix, which may include:

[0076] Traversing a current structured sparse computing storage unit in the current model weight matrix in turn, and locating N horizontal matrix positions matching the current structured sparse computing storage unit in the third matrix according to the position of the current structured sparse computing storage unit in the current model weight matrix;

[0077] Fill the column positions of each non-zero data in the current structured sparse computing storage unit into the N horizontal matrix positions;

[0078] Returns the operation of traversing one current structured sparse computing storage unit in turn in the current model weight matrix until all structured sparse computing storage units in the current model weight matrix are processed to obtain a metadata matrix corresponding to the current model weight matrix.

[0079] Continue as Figure 2 As shown, each structured sparse computing storage unit contained in the model weight matrix can be traversed in turn by traversing row by row. For example, first traverse to the structured sparse computing storage unit 1 containing non-zero elements A and B. Since the structured sparse computing storage unit 1 is located in the upper left corner of the model weight matrix, it is possible to start from the upper left corner of the third matrix and select two horizontal matrix positions of N=2. After that, the column positions of A and B in the structured sparse computing storage unit 1, A is located in the 0th column, and B is located in the 2nd column, are filled to the first two column positions of the first row of the third matrix. Similarly, after the non-zero data positions in the 2*4 structured sparse computing storage units in the above-mentioned model weight matrix are filled to the third matrix, a matching metadata matrix is ​​obtained.

[0080] It can be understood that by constructing the above-mentioned compressed data matrix, index matrix and metadata data, the sparse model weight matrix of the above-mentioned specific structure can be uniquely determined. Furthermore, the sparse model weight matrices contained in the hybrid expert model can be efficiently compressed and stored, thereby effectively reducing the storage overhead of the hybrid expert model on the configured computing chip. In addition, when the hybrid expert model deployed in the computing chip implements calculations based on the above-mentioned compressed model weight matrix, it can give full play to the hardware computing advantages of each sparse computing unit in the computing chip, while effectively improving the computing efficiency and reducing the bandwidth overhead.

[0081] Figure 3 A flowchart of another storage optimization method for a hybrid expert model provided by an embodiment of the present invention is provided. This embodiment is refined based on the above embodiments. In this embodiment, specifically: after the operation of storing the sparse model weight matrices of each item in the target hybrid expert model as a matching compressed matrix set, an operation of calling each sparse computing unit on the computing chip is further added to implement the matching sparse computing based on the target hybrid expert model after storage optimization. And the re-arrangement operation of the compressed matrix set and the associated storage operation of the selection array are added simultaneously.

[0082] Correspondingly, such as Figure 3 As shown, the method includes:

[0083] S310, loading the target hybrid expert model into the computing chip, and obtaining the sparse model weight matrix of each item in the target hybrid expert model.

[0084] Among them, the sparse model weight matrix includes multiple compression units of set sizes, each compression unit includes at least one valid row and at least one sparse row, each valid row includes at least one structured sparse computing storage unit; each structured sparse computing storage unit is in the same sparse mode.

[0085] S320. Generate a set of compression matrices corresponding to each model weight matrix.

[0086] Among them, the compression matrix set includes a compressed data matrix for storing non-zero data in the model weight matrix, an index matrix for storing the positions of valid rows in the model weight matrix in the corresponding compression unit, and a metadata matrix for storing the positions of non-zero data in the corresponding structured sparse computing storage unit.

[0087] S330, storing the sparse model weight matrices of each item in the target hybrid expert model as matching compressed matrix sets, so as to achieve storage optimization of the target hybrid expert model.

[0088] S340, obtaining a sparse computing unit requirement file that matches the computing chip.

[0089] The sparse computing unit requirement file defines the data access location of each thread executed in the sparse computing unit to each model weight matrix in the target hybrid expert model when performing sparse computing.

[0090] S350, re-arranging the data arrangement of the compressed data matrix, the index matrix and the metadata matrix in each compressed matrix set according to the sparse computing unit requirement file.

[0091] S360, storing the reorganized compression matrix sets in the setting memory of the computing chip to achieve continuous storage of data with continuous access characteristics.

[0092] Generally speaking, computing chips (for example, GPUs) use memory access coalescing when loading data. The core idea is to merge memory accesses from multiple threads into a larger memory access operation, which can reduce access latency and improve bandwidth utilization. At the same time, its data loading also has the following characteristics:

[0093] 1. Aligned access: GPU memory access is usually aligned according to a certain step size, usually 32 bytes, 64 bytes or larger. If multiple threads operate on consecutive memory addresses, these accesses can usually be merged into a single memory access operation, thereby reducing the overhead of memory access; 2. Merged memory access: For example, if multiple threads access adjacent addresses in memory, the GPU can merge these accesses into a single memory request to avoid multiple memory accesses, thereby reducing latency and bandwidth waste.

[0094] Based on the above data loading concept, each embodiment of the present invention re-arranges the data arrangement of the compressed data matrix, index matrix and metadata matrix in each compression matrix set based on the data reading requirements of the sparse computing unit in the computing chip used to implement the calculation, so as to further improve the data reading efficiency during subsequent calculations.

[0095] Specifically, by parsing the sparse computing unit requirement file that comes with the computing chip, the sparse computing unit's data reading requirements can be obtained. Specifically, the sparse computing unit requirement file records the data access locations of each thread in the sparse computing unit to each model weight matrix in the target hybrid expert model.

[0096] The sparse computing unit can be understood as a special hardware circuit in the computing chip for performing matrix multiplication calculations on double sparse matrices, where double sparse matrices refer to the left operand matrix and the right operand matrix used for matrix multiplication calculations being both sparse matrices. A computing chip generally includes multiple sparse computing units, and within each sparse computing unit, one or more threads can be opened to implement corresponding computing tasks.

[0097] By way of example and not limitation, Figure 4 A schematic diagram showing a comparison between a sparse computing unit requirement and a matching data re-arrangement result applicable to an embodiment of the present invention is shown in FIG.

[0098] like Figure 4 As shown, in Figure 4 The table on the left records the access positions of matrix elements in a specific matrix (e.g., a compressed data matrix) by each thread in each sparse computing unit. 29They represent threads respectively, and different threads belong to different sparse computing units. T0{3...0} in the 0th row, 0th column, 0th...3rd column in the left table represents that the data in the 0th row, 0th column, 0th...3rd column of the compressed data matrix is ​​loaded into the data position {3...0} in the target vector used by the T0 thread to perform calculations. Then, the T0 thread needs to read the data in the 0th row, 0th...3rd column of the compressed data matrix from the storage space where the compressed data matrix is ​​located and store it in the matching position in the target vector.

[0099] Similarly, T0{19...16} in the 0...3 column of the 8th row in the table on the left means that the data in the 0...3 column of the 8th row in the compressed data matrix is ​​loaded into the data position {19...16} in the target vector used by the T0 thread to perform calculations. Then, the T0 thread needs to read the data in the 0...3 column of the 8th row in the compressed data matrix from the storage space where the compressed data matrix is ​​located and store it in the matching position in the target vector.

[0100] It is understandable that, considering the actual calculation characteristics of the dual sparse matrix, each thread does not read data from a continuous position in a matrix. This data reading method is inefficient and has poor bandwidth utilization, which will reduce the calculation efficiency of the dual sparse matrix to a certain extent. Based on this, in various embodiments of the present invention, according to the data reading characteristics of each thread in the sparse computing unit, the data arrangement of the compressed data matrix, the index matrix and the metadata matrix in each compressed matrix set is rearranged.

[0101] Specific, combined Figure 4 As shown in the table on the left and right sides, the total number of computational data required by the T0 thread is 32, which are located in the 0th row, columns 0...15, and the 8th row, columns 0...15 in the compressed data matrix. However, according to the conventional data storage method, these 32 data are not continuous. In other words, a thread needs to transfer the memory twice to obtain the data required by the sparse computing unit, which wastes the memory bandwidth of the computing chip.

[0102] To improve memory utilization, such as Figure 4 As can be seen from the table on the right, the 32 data that are not stored continuously can be arranged continuously in a new matrix. Through such optimization, each thread needs to load continuous 32 bits of data, and the data loading operation can be completed through one memory transfer. At this time, the T0 thread can continuously read the data that needs to be accessed continuously, greatly reducing the memory access overhead and data reading delay, and effectively improving bandwidth utilization.

[0103] S370, identifying each hybrid model layer included in the target hybrid expert model, and obtaining the expert allocation mode corresponding to each hybrid model layer.

[0104] The expert allocation mode is used to indicate the row positions of valid rows of intermediate activation values ​​allocated to the self-mixed model layer when performing calculations.

[0105] As mentioned above, the characteristic of the hybrid expert model is that it is necessary to determine which expert should be activated and participate in the calculation of the output when processing a specific input. That is, when implementing calculations based on the target hybrid expert model, for any hybrid model layer, its intermediate activation value is assigned to different experts for processing by the intermediate routing layer.

[0106] For any expert, the intermediate activation values ​​that are not assigned to its processing therefore become structured sparse values, that is, the input to each hybrid model layer is a sparse matrix of intermediate activation values ​​in the form of a sparse matrix, in which one or more matrix rows are sparse rows (all 0).

[0107] The traditional processing method is that each hybrid model layer copies the data (data in non-sparse rows) assigned to the original intermediate activation value sparse matrix for each expert as new data, which will bring additional computing, memory and bandwidth requirements. Based on this, the embodiments of the present invention further propose a structured sparse representation method based on column vectors to avoid additional data copying processes, thereby greatly reducing runtime overhead and improving model operation efficiency.

[0108] That is, a sparse matrix of intermediate activation values ​​with sparse characteristics is directly stored for each hybrid model layer, thereby avoiding the additional cost of copying new data.

[0109] S380. Generate selection arrays corresponding to each hybrid model layer according to the expert allocation mode.

[0110] In this embodiment, since the intermediate activation value sparse matrix is ​​directly stored for each hybrid model layer, in order to improve the computing requirements of the sparse computing unit for the intermediate activation value sparse matrix, it is necessary to synchronously store the selection array for the intermediate activation value sparse matrix.

[0111] Among them, the selection array is used to describe which matrix rows in the sparse matrix of intermediate activation values ​​corresponding to each hybrid model layer are non-sparse rows containing valid inputs.

[0112] By way of example and not limitation, Figure 5 A schematic diagram of a comparison structure of an intermediate activation value sparse matrix and a matching selection array applicable to an embodiment of the present invention is shown in FIG. Figure 5The left matrix of shows the specific form of an intermediate activation value sparse matrix adapted by a hybrid model layer. In the intermediate activation value sparse matrix, the 0th, 3rd and 5th rows are set as valid rows, and the remaining rows are sparse rows of all 0s. At this time, the expert allocation mode for the hybrid model layer is the intermediate activation values ​​output by the 0th, 3rd and 5th experts. Furthermore, a selection array of the form {0,3,5} can be constructed to describe the location of the valid rows in the intermediate activation value sparse matrix.

[0113] S390. The selection array of each hybrid model layer is associated with the compression matrix set of each model weight matrix and stored so that when performing calculations, the selection array is combined with the sparse matrix of the intermediate activation value to be processed and sparse calculations are performed with the matching compression matrix set.

[0114] S3100, calling each sparse computing unit on the computing chip, and performing matching sparse computing based on the target hybrid expert model after storage optimization.

[0115] In this embodiment, since the target hybrid expert model needs to perform matrix multiplication on the model weight matrix of each model layer and the input activation value matrix input by the model layer when performing model inference, double sparse matrix multiplication is involved in this calculation process. In each embodiment of the present invention, by compressing and storing the model weight matrix as a compressed matrix set and synchronously storing it with the selection array of each hybrid model layer, the sparse computing unit in the computing chip can be fully utilized to improve computing efficiency.

[0116] Figure 6 A schematic diagram of the structure of a storage optimization device of a hybrid expert model provided by an embodiment of the present invention. Figure 6 As shown, the device includes a model weight matrix acquisition module 610, a compression matrix set generation module 620 and a storage optimization module 630, wherein:

[0117] A model weight matrix acquisition module 610 is used to load the target hybrid expert model into the computing chip and obtain the sparse model weight matrix of each item in the target hybrid expert model;

[0118] The sparse model weight matrix includes a plurality of compression units of set sizes, each compression unit includes at least one valid row and at least one sparse row, each valid row includes at least one structured sparse computing storage unit; each structured sparse computing storage unit is in the same sparse mode;

[0119] A compression matrix set generation module 620 is used to generate a compression matrix set corresponding to each model weight matrix;

[0120] The compressed matrix set includes a compressed data matrix for storing non-zero data in the model weight matrix, an index matrix for storing the position of valid rows in the model weight matrix in the corresponding compression unit, and a metadata matrix for storing the position of non-zero data in the corresponding structured sparse computing storage unit;

[0121] The storage optimization module 630 is used to store the sparse model weight matrices of each item in the target hybrid expert model as a matching compressed matrix set, so as to achieve storage optimization of the target hybrid expert model.

[0122] The technical solution of the embodiment of the present invention converts the sparse model weight matrices of various specific formats in the target hybrid expert model into a compressed data matrix for storing non-zero data in the model weight matrix, an index matrix for storing the positions of valid rows in the model weight matrix in the corresponding compression unit, and a metadata matrix for storing the positions of non-zero data in the corresponding structured sparse computing storage unit, and then optimizes the storage in a computing chip used to implement hybrid expert model calculations. This can effectively reduce the storage resource overhead of the computing chip. In addition, when using the model weight matrix in the above-mentioned data compression form for model calculations, the sparse computing hardware in the computing chip can be fully utilized, which effectively improves the computing efficiency while reducing the bandwidth overhead.

[0123] Based on the above embodiments, the sparse model weight matrix may specifically include a plurality of compression units of size M*V;

[0124] Specifically, each compression unit includes d valid rows, where 1≤d<M; and

[0125] Each structured sparse computing storage unit is in a sparse mode of N:L, where L is the total amount of data contained in the structured sparse computing storage unit, and N is the amount of non-zero data contained in the structured sparse computing storage unit.

[0126] Based on the above embodiments, the compression matrix set generation module 620 may specifically include:

[0127] A current model weight matrix acquisition unit is used to acquire a current model weight matrix of a size of m*k that is currently being processed, where m is an integer multiple of M and k is an integer multiple of V;

[0128] The compressed data matrix construction subunit is used to construct a first matrix of size (m / M*d)*(k / (L / N)), and fill each non-zero element of the current model weight matrix into the first matrix in a row-by-row traversal manner to obtain a compressed data matrix corresponding to the current model weight matrix;

[0129] The index matrix construction subunit is used to construct a second matrix of (m / M*d)*(k / V) size, and fill the second matrix according to the position of each compression unit in the current model weight matrix and the position of each valid row in the corresponding compression unit to obtain an index matrix corresponding to the current model weight matrix;

[0130] The metadata matrix construction subunit is used to construct a third matrix of (m / M*d)*(k / (L / N)) size, and fill the third matrix according to the position of each structured sparse computing storage unit in the current model weight matrix and the position of each non-zero data in the corresponding structured sparse computing storage unit to obtain a metadata matrix corresponding to the current model weight matrix.

[0131] Based on the above embodiments, the index matrix constructs a sub-unit, which is specifically used for:

[0132] Traversing a current compression unit in the current model weight matrix in turn, and locating d longitudinal matrix positions matching the current compression unit in the second matrix according to the position of the current compression unit in the current model weight matrix;

[0133] Identify the row positions of each valid row in the current compression unit, and fill the identified row positions into d longitudinal matrix positions accordingly;

[0134] Returns the operation of traversing one current compression unit in the current model weight matrix in turn until all compression units in the current model weight matrix are processed to obtain an index matrix corresponding to the current model weight matrix.

[0135] Based on the above embodiments, the metadata matrix constructs a sub-unit, which is specifically used for:

[0136] Traversing a current structured sparse computing storage unit in the current model weight matrix in turn, and locating N horizontal matrix positions matching the current structured sparse computing storage unit in the third matrix according to the position of the current structured sparse computing storage unit in the current model weight matrix;

[0137] Fill the column positions of each non-zero data in the current structured sparse computing storage unit into the N horizontal matrix positions;

[0138] Returns the operation of traversing one current structured sparse computing storage unit in turn in the current model weight matrix until all structured sparse computing storage units in the current model weight matrix are processed to obtain a metadata matrix corresponding to the current model weight matrix.

[0139] On the basis of the above embodiments, a sparse computing module may also be included, which is used to:

[0140] After each sparse model weight matrix in the target hybrid expert model is stored as a matching compressed matrix set, each sparse computing unit on the computing chip is called to implement the matching sparse computing based on the target hybrid expert model after storage optimization.

[0141] Based on the above embodiments, a re-arrangement module may be further included, which is used to:

[0142] After storing the sparse model weight matrices of each item in the target hybrid expert model as matching compressed matrix sets, obtaining a sparse computing unit requirement file matching the computing chip;

[0143] The sparse computing unit requirement file defines the data access location of each thread executed in the sparse computing unit to each model weight matrix in the target hybrid expert model when performing sparse computing;

[0144] According to the sparse computing unit requirement file, the data arrangement of the compressed data matrix, the index matrix and the metadata matrix in each compressed matrix set is rearranged;

[0145] The reorganized compressed matrix sets are stored in the setting memory of the computing chip to achieve continuous storage of data with continuous access characteristics.

[0146] On the basis of the above embodiments, the method may further include selecting an array building module for:

[0147] Before calling each sparse computing unit on the computing chip to implement the matching sparse computing based on the target hybrid expert model after storage optimization, identify each hybrid model layer included in the target hybrid expert model, and obtain the expert allocation mode corresponding to each hybrid model layer;

[0148] Among them, the expert allocation mode is used to indicate the row position of the valid row of the intermediate activation value assigned to the own hybrid model layer when implementing the calculation;

[0149] According to the expert allocation mode, generating selection arrays corresponding to each hybrid model layer respectively;

[0150] The selection array of each hybrid model layer is associated with the compressed matrix set of each model weight matrix and stored so that when the calculation is implemented, the selection array is combined with the sparse matrix of the intermediate activation value to be processed and sparse calculation is performed with the matching compressed matrix set.

[0151] The storage optimization device of the hybrid expert model provided in the embodiment of the present invention can execute the storage optimization method of the hybrid expert model provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0152] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0153] Figure 7 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0154] like Figure 7 As shown, the electronic device 10 includes at least one computing chip 11, and a memory connected to the at least one computing chip 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one computing chip, and the computing chip 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The computing chip 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0155] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0156] The computing chip 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing chip 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various computing chips running machine learning model algorithms, digital signal computing chips (DSPs), and any appropriate computing chips, controllers, microcontrollers, etc. The computing chip 11 executes the various methods and processes described above, for example, executing the storage optimization method of the hybrid expert model as described in any embodiment of the present invention.

[0157] That is, the target hybrid expert model is loaded into the computing chip, and the sparse model weight matrix of each item in the target hybrid expert model is obtained;

[0158] The sparse model weight matrix includes a plurality of compression units of set sizes, each compression unit includes at least one valid row and at least one sparse row, each valid row includes at least one structured sparse computing storage unit; each structured sparse computing storage unit is in the same sparse mode;

[0159] Generate a set of compressed matrices corresponding to each model weight matrix;

[0160] The compressed matrix set includes a compressed data matrix for storing non-zero data in the model weight matrix, an index matrix for storing the position of valid rows in the model weight matrix in the corresponding compression unit, and a metadata matrix for storing the position of non-zero data in the corresponding structured sparse computing storage unit;

[0161] The sparse model weight matrices of each item in the target hybrid expert model are stored as matching compressed matrix sets to achieve storage optimization of the target hybrid expert model.

[0162] Further, in Figure 8 FIG. 2 shows a structural diagram of a computing chip applicable to an embodiment of the present invention. Figure 8 As shown, the computing chip further includes: a plurality of sparse computing units 810; the sparse computing units 810 are used to implement sparse computing.

[0163] In some embodiments, the storage optimization method of the hybrid expert model as described in any embodiment of the present invention may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the computing chip 11, one or more steps of the storage optimization method of the hybrid expert model as described in any embodiment of the present invention described above may be executed. Alternatively, in other embodiments, the computing chip 11 may be configured to execute the storage optimization method of the hybrid expert model as described in any embodiment of the present invention by any other appropriate means (for example, by means of firmware).

[0164] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable computing chip, which can be a dedicated or general-purpose programmable computing chip that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0165] The computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a computing chip of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that when the computer program is executed by the computing chip, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0166] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0167] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0168] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0169] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0170] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0171] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A storage optimization method for a hybrid expert model, characterized in that: include: Loading the target hybrid expert model into the computing chip, and obtaining the sparse model weight matrix of each item in the target hybrid expert model; The sparse model weight matrix includes a plurality of compression units of set sizes, each compression unit includes at least one valid row and at least one sparse row, each valid row includes at least one structured sparse computing storage unit; each structured sparse computing storage unit is in the same sparse mode; Generate a set of compressed matrices corresponding to each model weight matrix; The compressed matrix set includes a compressed data matrix for storing non-zero data in the model weight matrix, an index matrix for storing the position of valid rows in the model weight matrix in the corresponding compression unit, and a metadata matrix for storing the position of non-zero data in the corresponding structured sparse computing storage unit; The sparse model weight matrices of each item in the target hybrid expert model are stored as matching compressed matrix sets respectively, so as to achieve storage optimization of the target hybrid expert model.

2. The method according to claim 1, characterized in that The sparse model weight matrix specifically includes multiple compression units of size M*V; Specifically, each compression unit includes d valid rows, where 1≤d<M; and Each structured sparse computing storage unit is in a sparse mode of N:L, where L is the total amount of data contained in the structured sparse computing storage unit, and N is the amount of non-zero data contained in the structured sparse computing storage unit.

3. The method according to claim 2, characterized in that Generate a set of compressed matrices corresponding to each model weight matrix, including: Get the current model weight matrix of the current processing size m*k, where m is an integer multiple of M and k is an integer multiple of V; Construct a first matrix of size (m / M*d)*(k / (L / N)), and fill each non-zero element of the current model weight matrix into the first matrix in a row-by-row traversal manner to obtain a compressed data matrix corresponding to the current model weight matrix; Construct a second matrix of size (m / M*d)*(k / V), and fill the second matrix according to the position of each compression unit in the current model weight matrix and the position of each valid row in the corresponding compression unit to obtain an index matrix corresponding to the current model weight matrix; Construct a third matrix of size (m / M*d)*(k / (L / N)), and fill the third matrix according to the position of each structured sparse computing storage unit in the current model weight matrix and the position of each non-zero data in the corresponding structured sparse computing storage unit to obtain a metadata matrix corresponding to the current model weight matrix.

4. The method according to claim 3, characterized in that According to the position of each compression unit in the current model weight matrix and the position of each valid row in the corresponding compression unit, the second matrix is ​​filled to obtain an index matrix corresponding to the current model weight matrix, including: Traversing a current compression unit in the current model weight matrix in turn, and locating d longitudinal matrix positions matching the current compression unit in the second matrix according to the position of the current compression unit in the current model weight matrix; Identify the row positions of each valid row in the current compression unit, and fill the identified row positions into d longitudinal matrix positions accordingly; Returns the operation of traversing one current compression unit in the current model weight matrix in turn until all compression units in the current model weight matrix are processed to obtain an index matrix corresponding to the current model weight matrix.

5. The method according to claim 3, characterized in that: According to the position of each structured sparse computing storage unit in the current model weight matrix and the position of each non-zero data in the corresponding structured sparse computing storage unit, the third matrix is ​​filled to obtain a metadata matrix corresponding to the current model weight matrix, including: Traversing a current structured sparse computing storage unit in the current model weight matrix in turn, and locating N horizontal matrix positions matching the current structured sparse computing storage unit in the third matrix according to the position of the current structured sparse computing storage unit in the current model weight matrix; Fill the column positions of each non-zero data in the current structured sparse computing storage unit into the N horizontal matrix positions; Returns the operation of traversing one current structured sparse computing storage unit in turn in the current model weight matrix until all structured sparse computing storage units in the current model weight matrix are processed to obtain a metadata matrix corresponding to the current model weight matrix.

6. The method according to any one of claims 1 to 5, characterized in that: After the sparse model weight matrices of each item in the target hybrid expert model are stored as a matching compressed matrix set, it also includes: The sparse computing units on the computing chip are called to implement matching sparse computing based on the target hybrid expert model after storage optimization.

7. The method according to claim 6, characterized in that After the sparse model weight matrices of each item in the target hybrid expert model are stored as a matching compressed matrix set, it also includes: Obtain the sparse computing unit requirement file that matches the computing chip; The sparse computing unit requirement file defines the data access location of each thread executed in the sparse computing unit to each model weight matrix in the target hybrid expert model when performing sparse computing; According to the sparse computing unit requirement file, the data arrangement of the compressed data matrix, the index matrix and the metadata matrix in each compressed matrix set is rearranged; The reorganized compressed matrix sets are stored in the setting memory of the computing chip to achieve continuous storage of data with continuous access characteristics.

8. The method according to claim 7, characterized in that Before calling each sparse computing unit on the computing chip to implement the matching sparse computing based on the target hybrid expert model after storage optimization, it also includes: Identify each mixed model layer included in the target mixed expert model, and obtain the expert allocation mode corresponding to each mixed model layer; Among them, the expert allocation mode is used to indicate the row position of the valid row of the intermediate activation value assigned to the own hybrid model layer when implementing the calculation; According to the expert allocation mode, generating selection arrays corresponding to each hybrid model layer respectively; The selection array of each hybrid model layer is associated with the compressed matrix set of each model weight matrix and stored so that when the calculation is implemented, the selection array is combined with the sparse matrix of the intermediate activation value to be processed and sparse calculation is performed with the matching compressed matrix set.

9. A storage optimization device for a hybrid expert model, characterized in that: include: A model weight matrix acquisition module is used to load the target hybrid expert model into the computing chip and obtain the sparse model weight matrix of each item in the target hybrid expert model; The sparse model weight matrix includes a plurality of compression units of set sizes, each compression unit includes at least one valid row and at least one sparse row, each valid row includes at least one structured sparse computing storage unit; each structured sparse computing storage unit is in the same sparse mode; A compression matrix set generation module, used to generate a compression matrix set corresponding to each model weight matrix; The compressed matrix set includes a compressed data matrix for storing non-zero data in the model weight matrix, an index matrix for storing the position of valid rows in the model weight matrix in the corresponding compression unit, and a metadata matrix for storing the position of non-zero data in the corresponding structured sparse computing storage unit; The storage optimization module is used to store the sparse model weight matrices of each item in the target hybrid expert model as a matching compressed matrix set to achieve storage optimization of the target hybrid expert model.

10. An electronic device, characterized in that: The electronic device comprises: at least one computing chip; and A memory communicatively connected to the at least one computing chip; wherein, The memory stores a computer program executed by the at least one computing chip, and the computer program is executed by the at least one computing chip so that the at least one computing chip can execute the storage optimization method of the hybrid expert model according to any one of claims 1 to 8.

11. The electronic device according to claim 10, characterized in that: The computing chip further includes: a plurality of sparse computing units; The sparse computing unit is used to implement sparse computing.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to implement the storage optimization method of the hybrid expert model according to any one of claims 1 to 8 when executed by a computing chip.

13. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a computing chip, implements the storage optimization method of the hybrid expert model according to any one of claims 1 to 8.

Images