A reram-based weight matrix processing method and device

Abstract

The application discloses a weight matrix processing method and device based on ReRAM, and the method comprises the following steps: acquiring the activated row number and the activated column number of a memistor ReRAM; performing non-structure pruning and compression on a weight matrix to obtain a compressed matrix block, wherein the row number of the compressed matrix block is less than or equal to the activated row number, the column number of the compressed matrix block is less than or equal to the activated column number, the row positions of each weight value in the same row in the compressed matrix block correspond to the same row position of the weight matrix, and the column positions of each weight value in the same column in the compressed matrix block correspond to the same column position of the weight matrix. By implementing the application, the matching application of the compressed matrix block and the ReRAM can be realized, the utilization efficiency of the ReRAM is improved, and the operation of the neural network is accelerated.

Description

Technical Field

[0001] This application relates to the field of neural networks, and in particular to a method and apparatus for processing weight matrices based on ReRAM. Background Technology

[0002] Neural networks are inherently large in size, and the size of a neural network model is often influenced by the size of its weight matrix. Pruning algorithms are commonly used to prune the weight matrix of a neural network to remove redundant elements (e.g., weights smaller than a preset threshold). This can reduce the size of the network model without compromising its accuracy, thereby accelerating the inference process.

[0003] Resistive Random-Access Memory (ReRAM) is often used to store the weight matrix of neural networks. However, when pruning the weight matrix in related technologies, the hardware limitation of the number of activation rows and columns in ReRAM is not taken into account, which makes the pruned weight matrix unsuitable for ReRAM-based accelerator applications. Summary of the Invention

[0004] This application discloses a weight matrix processing method and apparatus based on ReRAM, which can realize the matching application of compressed matrix and variable memory ReRAM, improve the utilization efficiency of ReRAM, and help accelerate the operation of neural network.

[0005] In a first aspect, this application provides a weight matrix processing method based on ReRAM. The method includes: obtaining the number of active rows and active columns of the memristor ReRAM; performing unstructured pruning and compression on the weight matrix to obtain a compressed matrix block, wherein the number of rows in the compressed matrix block is less than or equal to the number of active rows, the number of columns in the compressed matrix block is less than or equal to the number of active columns, the row positions of the weight matrices corresponding to each weight in the same row of the compressed matrix block are the same, and the column positions of the weight matrices corresponding to each weight in the same column of the compressed matrix block are the same.

[0006] In the above method, when performing unstructured pruning and compression on the weight matrix, the number of active rows and columns of the memristor ReRAM are taken into account, so that the number of rows of the obtained compressed matrix block is less than or equal to the number of active rows of the ReRAM and the number of columns of the compressed matrix block is less than or equal to the number of active columns of the ReRAM. The compressed matrix block can be matched with ReRAM-based accelerators, improving the utilization efficiency of ReRAM, effectively reducing the size of the weight matrix, and helping to accelerate the operation of neural networks.

[0007] In one possible implementation of the first aspect, when there are multiple compressed matrix blocks, unstructured pruning and compression are performed on the weight matrix to obtain compressed matrix blocks, including: performing unstructured pruning on the weight matrix to obtain a sparse matrix, wherein the number of non-zero weights in each column of the sparse matrix is ​​the same as the number of rows in the compressed matrix block; processing each column of the sparse matrix to obtain a mask matrix, wherein the mask matrix consists of 0s and 1s, and each column in the mask matrix belongs to multiple classes, each class includes the same number of columns, and the row positions of non-zero elements in columns belonging to the same class are the same; training the weight matrix based on the mask matrix to obtain a target matrix; the target matrix has non-zero weights at the positions corresponding to the non-zero elements in the mask matrix; and extracting the non-zero weights of each column belonging to each class in the target matrix to obtain multiple compressed matrix blocks.

[0008] By implementing the above method, unstructured pruning is performed on the weight matrix to obtain a sparse matrix. The columns of the sparse matrix are then classified based on the similarity of the row position distribution of non-zero weights along the column direction to obtain a mask matrix. Each column in the mask matrix belongs to multiple classes, with each class containing the same number of columns, and non-zero elements in columns belonging to the same class have the same row position distribution. The weight matrix is ​​trained based on the mask matrix to obtain the target matrix. The mask matrix indicates the position distribution of weights in the target matrix; therefore, the columns of the target matrix and the columns of the mask matrix have the same characteristics. Column rearrangement and row compression are performed on the target matrix along both the row and column directions to obtain multiple compressed matrix blocks. This improves the compression ratio of the weight matrix, reduces its size, and allows for the application of multiple compressed matrix blocks on ReRAM-based accelerators using time-division multiplexing techniques. Therefore, the compressed matrix blocks obtained by this method can better adapt to the hardware limitations of ReRAM.

[0009] In one possible implementation of the first aspect, after processing each column of the sparse matrix, row index information and column index information are also obtained. The column index information includes the column position of each column in each class in the mask matrix; the row index information includes the row position of the non-zero elements of each column in each class in the mask matrix; the non-zero weights of each column belonging to each class in the target matrix are extracted to obtain multiple compressed matrix blocks, including: rearranging each column of the target matrix based on the column index information so that columns belonging to the same class in the rearranged target matrix are arranged together; and extracting the non-zero rows of each class in the rearranged target matrix according to the row index information to obtain multiple compressed matrix blocks, the number of compressed matrix blocks being equal to the number of classes.

[0010] By implementing the above method, the target matrix is ​​rearranged in columns and compressed in rows in both the row and column directions. Column rearrangement means arranging columns of the same class in the target matrix together, with the row positions of the weights in the columns of the same class being the same. Row compression means removing all zero rows in the matrix formed by the columns of each class, thereby obtaining compressed matrix blocks and improving the compression rate of the weight matrix.

[0011] In one possible implementation of the first aspect, the more times the sparse matrix is ​​processed, the higher the similarity of the row position distribution of the non-zero weights of each column included in each class.

[0012] Some columns of a sparse matrix have similar row position distributions of weights. By performing clustering and other processing on the sparse matrix, columns with similar row position distributions of weights in the sparse matrix can be grouped into one class. The more times the processing is performed, the more similar the row position distributions of the non-zero weights of each column in each class become, which can effectively improve the compression ratio of the weight matrix and reduce the size of the weight matrix.

[0013] In one possible implementation of the first aspect, the proportion of zero weights in the sparse matrix is ​​greater than or equal to the sparsity, which indicates the degree of unstructured pruning of the weight matrix.

[0014] When performing unstructured pruning on the weight matrix, sparsity can also be considered so that the resulting compressed matrix block can both adapt to the hardware limitations of ReRAM and achieve a good compression ratio.

[0015] Secondly, this application provides a weight matrix processing method based on ReRAM. The method includes: storing a compressed matrix block into a memristor ReRAM, wherein the number of rows in the compressed matrix block is less than or equal to the number of active rows in the ReRAM, and the number of columns in the compressed matrix block is less than or equal to the number of active columns in the ReRAM; the compressed matrix block is obtained based on unstructured pruning and compression of the weight matrix; obtaining index information, which includes the position mapping information of the weights in the compressed matrix block; extracting data at the row positions indicated by the index information from image data to obtain input data corresponding to each column of the compressed matrix block; the image data being the input data corresponding to each column of the weight matrix; inputting the input data into a ReRAM-based accelerator to obtain output data corresponding to each column of the compressed matrix block; and restoring the output data corresponding to each column to the correct column position in the weight matrix indicated by the index information.

[0016] In the above method, the compressed matrix block is obtained by unstructured pruning and compression of the weight matrix. In addition, index information is also obtained, which indicates the position mapping information of the weights in the compressed matrix block. Based on the index information, not only can the input data corresponding to the compressed matrix block be determined, but the output data corresponding to each column of the compressed matrix block output by the ReRAM-based accelerator can also be restored to the correct column position. This allows the compressed matrix block to better adapt to the hardware limitations of ReRAM, realizes the application of the compressed matrix block on the ReRAM-based accelerator, and improves the computational efficiency of the neural network.

[0017] In one possible implementation of the second aspect, the row positions of the weight matrices corresponding to each weight in the same row of the compressed matrix block are the same, and the column positions of the weight matrices corresponding to each weight in the same column of the compressed matrix block are the same.

[0018] It can be seen that the row positions of the weights in each column of the same compressed matrix block are the same, and the original column positions corresponding to the weights in the same column of the compressed matrix block are also the same. That is, the compressed matrix is ​​obtained by compressing in both row and column dimensions, which effectively improves the compression ratio of the weight matrix, reduces the size of the weight matrix, and improves the computational efficiency of the neural network.

[0019] In one possible implementation of the second aspect, the index information includes row index information and column index information. The row index information includes the row position of each weight in each column of the compressed matrix block corresponding to the weight matrix, and the column index information includes the column position of each column of the compressed matrix corresponding to the weight matrix. Extracting data at the row positions indicated by the index information from the image data to obtain the input data corresponding to each column of the compressed matrix block includes: extracting data at the row positions indicated by the row index information from the image data to obtain the input data corresponding to each column of the compressed matrix block. Restoring the output data corresponding to each column to the correct column position in the weight matrix indicated by the index information includes: restoring the output data corresponding to each column to the correct column position in the weight matrix indicated by the column index information.

[0020] By using row index information and image data to determine the correct input data corresponding to each column of the compression matrix block, and by using column index information to ensure that the output result corresponding to each column of the compression matrix block corresponds to the correct column position, the compression matrix block can be adapted to the hardware limitations of ReRAM, thus realizing the application of the compression matrix block on ReRAM-based accelerators.

[0021] Thirdly, this application provides an apparatus comprising: an acquisition unit for acquiring the number of active rows and active columns of a memristor ReRAM; and a processing unit for performing unstructured pruning and compression on a weight matrix to obtain a compressed matrix block, wherein the number of rows in the compressed matrix block is less than or equal to the number of active rows, the number of columns in the compressed matrix block is less than or equal to the number of active columns, the row positions of the weight matrices corresponding to each weight in the same row of the compressed matrix block are the same, and the column positions of the weight matrices corresponding to each weight in the same column of the compressed matrix block are the same.

[0022] In one possible implementation of the third aspect, the number of compressed matrix blocks is multiple, and the processing unit is specifically used for: performing unstructured pruning on the weight matrix to obtain a sparse matrix, wherein the number of non-zero weights in each column of the sparse matrix is ​​the same as the number of rows in the compressed matrix block; processing each column of the sparse matrix to obtain a mask matrix, which consists of 0s and 1s, wherein each column of the mask matrix belongs to multiple classes, each class includes the same number of columns, and the non-zero elements in each column belonging to the same class have the same row position; training the weight matrix based on the mask matrix to obtain the target matrix; the target matrix has non-zero weights at the positions corresponding to the non-zero elements in the mask matrix; and extracting the non-zero weights of each column belonging to each class in the target matrix to obtain multiple compressed matrix blocks.

[0023] In one possible implementation of the third aspect, the processing unit is further configured to obtain row index information and column index information. The column index information includes the column position of each column in each class in the mask matrix; the row index information includes the row position of the non-zero elements of each column in each class in the mask matrix; the processing unit is further configured to: rearrange the columns of the target matrix based on the column index information so that the columns belonging to the same class in the rearranged target matrix are arranged together; and extract the non-zero rows of each class in the rearranged target matrix according to the row index information to obtain multiple compressed matrix blocks, the number of compressed matrix blocks being equal to the number of classes.

[0024] In one possible implementation of the third aspect, the more times the sparse matrix is ​​processed, the higher the similarity of the row position distribution of the non-zero weights of each column included in each class.

[0025] In one possible implementation of the third aspect, the proportion of zero weights in the sparse matrix is ​​greater than or equal to the sparsity, which indicates the degree of unstructured pruning of the weight matrix.

[0026] Fourthly, this application provides an apparatus comprising: a storage unit for storing compressed matrix blocks to a memristor ReRAM, wherein the number of rows in the compressed matrix block is less than or equal to the number of active rows in the ReRAM, and the number of columns in the compressed matrix block is less than or equal to the number of active columns in the ReRAM; the compressed matrix block is obtained based on unstructured pruning and compression of a weight matrix; an acquisition unit for acquiring index information, the index information including position mapping information of the weights in the compressed matrix block; a row conversion unit for extracting data at the row positions indicated by the index information from image data to obtain input data corresponding to each column of the compressed matrix block; the image data being the input data corresponding to each column of the weight matrix; an acceleration unit for inputting the input data into a ReRAM-based accelerator to obtain output data corresponding to each column of the compressed matrix block; and a column conversion unit for restoring the output data corresponding to each column to the correct column position in the weight matrix indicated by the index information.

[0027] In one possible implementation of the fourth aspect, the row positions of the weight matrices corresponding to each weight in the same row of the compressed matrix block are the same, and the column positions of the weight matrices corresponding to each weight in the same column of the compressed matrix block are the same.

[0028] In one possible implementation of the fourth aspect, the index information includes row index information and column index information. The row index information includes the row position of each weight in each column of the compressed matrix block corresponding to the weight matrix, and the column index information includes the column position of each column of the compressed matrix corresponding to the weight matrix. The row conversion unit is specifically used to: extract the data at the row position indicated by the row index information from the image data to obtain the input data corresponding to each column of the compressed matrix block; the column conversion unit is specifically used to: restore the output data corresponding to each column to the correct column position in the weight matrix indicated by the column index information.

[0029] Fifthly, this application provides a computer-readable storage medium storing program code for execution by a device, the program code including instructions for performing the method in the first aspect or any possible implementation of the first aspect.

[0030] In a sixth aspect, this application provides a computer-readable storage medium storing program code for execution by a device, the program code including instructions for performing the method in the second aspect or any possible implementation thereof.

[0031] In a seventh aspect, this application provides a computer software product comprising program instructions that, when executed by a device, perform the methods described in the first aspect or any possible embodiment of the first aspect. The computer software product may be a software installation package. When the methods provided by any possible design of the first aspect are required, the computer software product may be downloaded and executed on a device to implement the methods described in the first aspect or any possible embodiment of the first aspect.

[0032] Eighthly, this application provides a computer software product comprising program instructions that, when executed by a device, perform the methods described in the second aspect or any possible embodiment of the second aspect. The computer software product may be a software installation package. When the methods provided by any possible design of the second aspect are required, the computer software product may be downloaded and executed on a device to implement the methods described in the second aspect or any possible embodiment of the second aspect. Attached Figure Description

[0033] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1A This is a structural diagram of an accelerator based on ReRAM;

[0035] Figure 1B This is a schematic diagram of a ReRAM-based accelerator performing convolution calculations;

[0036] Figure 2 These are some pruning diagrams based on columns in the Crossbar;

[0037] Figure 3 This is a system architecture for an application provided in the embodiments of this application;

[0038] Figure 4 This is a flowchart of a weight matrix processing method based on ReRAM provided in an embodiment of this application;

[0039] Figure 5 This is a flowchart of a weight matrix compression method provided in an embodiment of this application;

[0040] Figure 6These are schematic diagrams of some weight matrices provided in the embodiments of this application before and after unstructured pruning;

[0041] Figure 7A This is a schematic diagram illustrating the effect of sparse matrix clustering provided in an embodiment of this application;

[0042] Figure 7B This is a schematic diagram illustrating the effect of processing a clustered sparse matrix according to an embodiment of this application;

[0043] Figure 8A This is a schematic diagram of a target matrix after column rearrangement provided in this embodiment of the application;

[0044] Figure 8B This is a schematic diagram of a target matrix after column rearrangement and row compression, provided in an embodiment of this application;

[0045] Figure 9 This is a schematic diagram illustrating how to obtain the input data corresponding to each matrix block of a compression matrix, as provided in this embodiment of the application.

[0046] Figure 10 This is a schematic diagram illustrating how the output result of each matrix block corresponds to the actual output position, as provided in this embodiment of the application.

[0047] Figure 11 This is a schematic diagram of the structure of a device provided in this application and this embodiment;

[0048] Figure 12 This is a functional structure diagram of a device provided in this embodiment of the application. Detailed Implementation

[0049] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. Terms such as "first," "second," etc., in the specification and claims of this application are used to distinguish different objects, not to describe a particular order.

[0050] To facilitate understanding, the relevant terms that may be involved in the embodiments of this application will be introduced below.

[0051] (1) Resistive Random Access Memory (ReRAM)

[0052] Resistive Random Access Memory (ReRAM), also known as memristor, is a non-volatile memory with high integration density. It boasts advantages such as zero static power consumption, fast read / write speeds, and small cell size, enabling the integration of storage and computation to reduce memory access overhead. Furthermore, ReRAM cells can represent multiple values, unlike traditional cells which can only represent 0 and 1. ReRAM arrays, constructed using a cross-point array structure, maximize area utilization efficiency.

[0053] ReRAM-based accelerators primarily consist of crossbar arrays. See also Figure 1A , Figure 1A This is a schematic diagram of an accelerator based on ReRAM. As can be seen, ReRAM is a cross array composed of word lines and bit lines, where the word lines are... Figure 1A The horizontal lines in the code are used to receive input signals; the resistors connecting the word lines and bit lines are called cells and are used to store weights. For example, each weight can be stored in one cell, and in some possible embodiments, each weight can also be stored in two or other numbers of cells; the bit lines are... Figure 1A In the diagram, the vertical lines represent the input signal multiplied by the corresponding weights of the bit lines. In addition, the ReRAM-based accelerator includes a Digital-to-Analog Converter (DAC), a Sample and Hold (S&H) unit, an Analog-to-Digital Converter (ADC), and a Shift and Add (S&A) unit. The DAC is connected to each word line of the crossbar array to convert the input data into an analog signal. The accumulated current on the bit lines (the dot product of the analog signal and the weights within the unit) is read from the S&H unit and fed to the shared ADC. The ADC converts the accumulated current on the bit lines into a digital signal, and the S&A unit sums the dot products of each column of bit lines. Thus, each unit in the Crossbar can perform a multiplication operation simultaneously, and the multiplication results in the same column can be accumulated. This architecture, when used in neural networks, can accelerate the computation of convolutional or fully connected layers, thereby speeding up the inference process of the neural network.

[0054] The following is a brief explanation of the working principle of ReRAM-based accelerators. (See also...) Figure 1B , Figure 1B This is a schematic diagram illustrating how a ReRAM-based accelerator performs convolution calculations, as shown below. Figure 1B As shown, assuming this convolutional layer has two 2x2 filters, after the `img2col` transformation, a 4x2 weight matrix is ​​obtained, where each column of the weight matrix represents a filter. Additionally, assuming there is a 4x4 crossbar array, each cell can represent 2 bits. Therefore, each weight in the weight matrix can be represented by 4 bits (i.e., two cells). Thus, this weight matrix can be processed according to... Figure 1B The method described in the diagram is mapped onto a crossbar. Each weight is stored in two cells, and filters belonging to the same category are stored in the same column of the crossbar according to their correspondence. The inputs to these filters are identical; for example, if the input to a filter in its first cycle is the content within a 2x2 bounding box in the feature map, its corresponding input is converted into a 4x1 vector. This vector is input into the crossbar, where it is multiplied by the corresponding cell in each column. The results from the same column are accumulated, and the accumulated result for each column is output to its corresponding position. Therefore, for ReRAM's crossbar, data is input row-wise, and the accumulated result is output column-wise. This crossbar architecture allows neural network accelerators to leverage the parallelism of convolutional layer computations to accelerate convolutional calculations.

[0055] (2) Hardware limitations on the number of active rows and columns in ReRAM

[0056] In actual activation, ReRAM chips do not ideally activate all rows and columns simultaneously. Instead, only some rows and columns can be activated at the same time. This is due to the hardware limitations inherent in ReRAM in the row and column directions.

[0057] In the row direction, the Crossbar itself exhibits leakage current when activated. When too many rows are activated, it leads to excessive accumulated error, thus reducing the accuracy of the output. In the column direction, due to the large area of ​​the Analog-to-Digital Converter (ADC), to avoid wasting chip area resources by matching each column with an ADC, in actual ReRAM, multiple columns share a low-frequency ADC. However, the limitation on the number of ADCs restricts the chip's throughput during operation, preventing multiple columns from being activated simultaneously.

[0058] (3) Network pruning

[0059] Network pruning is an effective method for miniaturizing and accelerating neural networks. It removes redundant parameters from a network while maintaining its accuracy, allowing the pruned network to achieve a speedup within existing neural network inference frameworks. Based on the structural characteristics of the pruned weight matrix, network pruning methods can be divided into structured pruning and unstructured pruning. Structured pruning involves deleting redundant rows or columns from the weight matrix. For example, if the weight matrix is ​​10x10, pruning the first and fourth columns, as well as the third and sixth rows, results in a regular 8x8 matrix. Unstructured pruning involves deleting any redundant elements from the weight matrix. For example, if the weight matrix is ​​10x10, setting weights smaller than a preset threshold (redundant elements) to 0 results in a 10x10 weight matrix. Since the positions of the weights set to 0 are random, the unstructured pruning weight matrix is ​​a sparse matrix.

[0060] In related technologies, some have proposed pruning the weight matrix column-wise based on the sparsity of the weight matrix in the column direction of the neural network to reduce the size of the network model. See also Figure 2 , Figure 2 This is a pruning diagram based on columns in a Crossbar. The original weight matrix can be mapped to 9 Crossbars in ReRAM. Pruning is performed using the weight of each column in the Crossbar as the basic pruning granularity, such as... Figure 2 In (1), the column that was pruned is... Figure 2 The white part in (1) can be obtained by shifting the remaining columns to the left to fill in the blanks. Figure 2 The pruning effect shown in (2) is due to Figure 2 As can be seen from (2), the weight matrix after pruning only occupies 4 crossbars in ReRAM, which saves 5 crossbars compared to the original matrix.

[0061] On the one hand, this method only utilizes the sparsity along the column direction of the weight matrix, but not the sparsity along the row direction, thus limiting the compression ratio of the weight matrix, resulting in a low compression ratio. On the other hand, this method does not consider the number of activated rows and columns of the Crossbar in ReRAM, due to... Figure 2As can be seen from (2) in the figure, the compressed weight matrix consumes 5 crossbars in the ReRAM, and the number of rows in each crossbar is occupied, but the number of columns occupied varies. Since the number of activated rows and columns in the crossbar in the ReRAM is not taken into account, the number of activated rows and columns in the crossbar is less than the number of rows and columns in the crossbar. This results in some rows or columns of the weight matrix in the crossbar being unable to participate in the calculation of the convolutional layer or the fully connected layer. In other words, it is difficult to directly combine the weight matrix obtained by compression using this method with the hardware ReRAM for application.

[0062] The following describes a system architecture used in an embodiment of this application. See also Figure 3 , Figure 3 A schematic block diagram of the weight matrix compression system used in the embodiments of this application is provided as an example. Figure 3 As shown, the system 10 includes a computing module, a memristor-based ReRAM accelerator, a storage module, a row conversion module, and a column conversion module. The storage module, the memristor-based ReRAM accelerator, and the computing module are connected or coupled via a bus, and the row conversion module, the column conversion module, and the memristor-based ReRAM accelerator are connected or coupled via a bus.

[0063] The computation module, which can be a processor, performs unstructured pruning of the neural network's weight matrix based on the ReRAM hardware conditions. It then trains the pruned weight matrix to obtain the target matrix, and finally compresses the target matrix to obtain a compressed matrix. The compressed matrix consists of multiple compressed matrix blocks, each of the same size. The number of rows and columns in each compressed matrix block are less than or equal to the number of activation rows and columns in the ReRAM. The compressed matrix is ​​then stored in a memristor-based ReRAM accelerator. It can be understood that compared to the weight matrix, the compressed matrix not only changes in size but also in the weight distribution along the row and column directions.

[0064] The storage module is used to store data to be processed, such as images, feature maps of images, etc. This data to be processed is used to perform multiplication operations with a compressed matrix stored in a memristor-based ReRAM accelerator.

[0065] The row conversion module is used to store the row position of the weight matrix corresponding to each row of each compressed matrix block. When the data to be processed in the storage module is multiplied and other operations are performed with the compressed matrix, the input data corresponding to each compressed matrix block is determined according to the row position information of each compressed matrix block and the data to be processed.

[0066] An accelerator based on memristor ReRAM is used to receive input data corresponding to each compressed matrix block of the compressed matrix. The input data is multiplied with each weight in each column of each compressed matrix block, and the multiplication results corresponding to each weight in each column are accumulated to obtain the accumulated result corresponding to each column of each compressed matrix block in the compressed matrix.

[0067] The column transformation module is used to store the column position of the weight matrix corresponding to each column of each compressed matrix block of the compressed matrix. When the accelerator based on memristor ReRAM outputs the cumulative result corresponding to each column of each compressed matrix block, it uses the column position information to take the cumulative result corresponding to each column as the result of that column in the weight matrix. In other words, it restores the cumulative result corresponding to each column in the compressed matrix to its original column position.

[0068] See Figure 4 Based on the system architecture described above, this application first describes a ReRAM-based weight matrix processing method, which includes, but is not limited to, the following steps:

[0069] S101. Obtain the number of active rows and columns of ReRAM.

[0070] In this embodiment, the number of activated rows and columns of the ReRAM are obtained. The ReRAM is a compressed matrix used to store the weight matrix of a neural network after unstructured pruning and compression. The number of activated rows in the ReRAM refers to the number of word lines that can be activated per cycle in the Crossbar (or XB) array of the ReRAM. The number of activated columns in the ReRAM refers to the number of bit lines that can be activated per cycle in the Crossbar (or XB) array of the ReRAM.

[0071] It should be noted that there is a correspondence between the number of activated rows / columns in the ReRAM and the number of activated rows / columns in the compressed matrix subsequently stored in the ReRAM. The compressed matrix corresponding to the weight matrix is ​​stored in the ReRAM. For an introduction to ReRAM, please refer to the relevant descriptions above; they will not be repeated here. The maximum number of activated rows in the compressed matrix is ​​the number of activated rows in the ReRAM. The number of activated columns in the compressed matrix depends on the mapping relationship between the ReRAM cells and the weights, where each cell represents 2 bits. For example, if the mapping relationship between the ReRAM cells and weights is 2:1, meaning one weight requires 2 cells (4 bits), and the number of activated columns in the ReRAM is L, then the maximum number of activated columns in the compressed matrix is ​​L / 2. As another example, if the mapping relationship between the ReRAM cells and weights is 1:1, and the number of activated columns in the ReRAM is L, then the maximum number of activated columns in the compressed matrix is ​​L.

[0072] It should be noted that the number of active rows and columns of ReRAM can be obtained by the user through the keyboard or touch screen, or it can be preset by the program. This application does not make any specific restrictions.

[0073] In some possible embodiments, it is also necessary to obtain sparsity, which is used to indicate the sparsity of the target matrix before compression, and can also be used to indicate the degree of pruning of the weight matrix. Sparsity can be user-inputted or preset, and this application does not specifically limit it.

[0074] S102. Perform unstructured pruning and compression on the weight matrix based on the number of activated rows and columns to obtain the compressed matrix and index information.

[0075] In this embodiment, the weight matrix of the neural network is unstructured pruning and compressed based on the number of activated rows and columns to obtain a compressed matrix and index information. The compressed matrix includes at least one compressed matrix block, and each compressed matrix block corresponds to a matrix of equal size. The number of rows in each compressed matrix block is less than or equal to the number of activated rows in the ReRAM, and the number of columns in each compressed matrix block is less than or equal to the number of activated columns in the ReRAM. The index information includes the position mapping information of the weights in the compressed matrix.

[0076] Since the number of rows and columns activated per cycle on ReRAM is fixed, a certain number of rows and columns on ReRAM can be activated using time-division multiplexing to activate each compressed matrix block in the compressed matrix.

[0077] In some possible embodiments, the index information includes row index information and column index information. The row index information includes the row position of the weight matrix corresponding to each row of each compressed matrix block; the column index information includes the column position of the weight matrix corresponding to each column of each compressed matrix block.

[0078] To improve the utilization of ReRAM, the number of rows in each compressed matrix block is equal to the number of active rows in ReRAM, and there is the same mapping relationship between the number of columns in each compressed matrix block and the number of active columns in ReRAM. Specifically, the number of columns in each compressed matrix block is equal to the number of active weight columns in ReRAM. The conversion relationship between the number of active weight columns and the number of active columns in ReRAM can be found in the relevant description in S101, and will not be repeated here.

[0079] To clearly explain the process of obtaining the compression matrix and index information, see [link to documentation]. Figure 5 , Figure 5 This application provides a method for compressing weight matrices. Figure 5 The example is Figure 4S102 further includes, but is not limited to, the following steps:

[0080] S1021. Perform unstructured pruning on the weight matrix based on the number of activated rows to obtain a sparse matrix.

[0081] Specifically, the weight matrix of the neural network is unstructured and pruned according to the number of activation rows to obtain a sparse matrix. The size of the sparse matrix is ​​the same as the weight matrix, and the number of non-zero weights in each column of the sparse matrix is ​​equal to the number of activation rows. It can be understood that, due to the unstructured pruning, the resulting sparse matrix is ​​a matrix with sparse weight distribution, consisting of 0s and weights.

[0082] In some possible embodiments, unstructured pruning of each column of the weight matrix based on the number of activated rows can be performed as follows: for each column of the weight matrix, retain the top R largest weights in the column and delete the weights in the remaining positions of the column (i.e., set the weights in the remaining positions of the column to 0), thereby obtaining a sparse matrix. The number of weights in each column of the sparse matrix is ​​the number of activated rows, and the size of the sparse matrix is ​​the same as the size of the weight matrix.

[0083] See Figure 6 , Figure 6 This is a schematic diagram of a non-structural pruning method provided in this application. For example... Figure 6 As shown, dark squares represent weighted values, and light squares represent unweighted values. Figure 6 (1) in the matrix is ​​the weight matrix. Figure 6 (2) is the sparse matrix obtained after unstructured pruning of the weight matrix. The number of activation rows is 5 and the number of rows of the weight matrix is ​​15, which satisfies the sparsity of 66%. It can be seen that the weight distribution of the sparse matrix is ​​sparser than that of the weight matrix, and each column of the sparse matrix has only 5 weights, that is, the number of weights in each column is equal to the number of activation rows.

[0084] It should be noted that the weights of each column in the sparse matrix can be derived from the top R largest weights of that column in the weight matrix, or from the top R smallest weights of that column in the weight matrix, or can be obtained from the columns of the weight matrix through other methods. This application does not impose any specific limitations on these values.

[0085] In some possible embodiments, sparsity must also be considered when performing unstructured pruning on the weight matrix. The number of rows in the resulting sparse matrix is ​​determined by the sparsity and the number of activation rows. The relationship between the number of rows, sparsity, and number of activation rows of the sparse matrix can be expressed as formula (1):

[0086]

[0087] Among them, X RLet R represent the number of rows in the sparse matrix, p represent the sparsity, and (1-p) represent the compression ratio. Therefore, the number of rows in the sparse matrix is ​​the floor function of the quotient of the number of activation rows and the compression ratio, and the number of columns in the sparse matrix is ​​the number of columns in the weight matrix.

[0088] In some possible embodiments, unstructured pruning is performed on the weight matrix based on sparsity and the number of activated rows. For example, if the weight matrix is ​​25*25 and the number of rows in the sparse matrix, determined by sparsity (e.g., 66%) and the number of activated rows (e.g., 5), is 15, meaning the number of rows in the sparse matrix is ​​less than the number of rows in the weight matrix, then first, each column of the matrix (15*25) consisting of the first 15 rows of the weight matrix is ​​pruned to retain the top 5 largest weights in each column, resulting in the first pruning result. To ensure consistent sparsity, each column of the matrix (10*25) consisting of the last 10 rows of the weight matrix is ​​pruned to retain the top 3 largest weights in each column, resulting in the second pruning result. The sparse matrix is ​​then obtained by concatenating the matrix corresponding to the first pruning result and the matrix corresponding to the second pruning result.

[0089] S1022. Process each column of the sparse matrix to obtain the mask matrix.

[0090] Specifically, the columns of the sparse matrix are clustered based on the similarity of the number of activated columns and the row position distribution to obtain multiple classes. The row positions of the non-zero weights of each column in each class are similar, and the more times the clustering is performed, the more similar the row positions of the non-zero weights of each column in each class are. The row position distribution of each column in each class is processed to obtain a mask matrix. The row position distribution of each column belonging to the same class in the mask matrix is ​​the same.

[0091] The mask matrix is ​​a matrix composed of 0 and 1, where "1" represents an active position and "0" represents an inactive position. Each column in the mask matrix belongs to multiple classes, and each class includes the same number of columns. Furthermore, the non-zero elements in the columns belonging to the same class are in the same row position.

[0092] On the one hand, the so-called clustering of columns of a sparse matrix based on the similarity of the number of activated columns and the distribution of row positions to obtain multiple clusters means that, on a column-by-column basis, the columns of the sparse matrix are clustered according to the similarity of the number of activated columns and the distribution of weight positions in the column direction. This achieves the goal of grouping the columns with the most similar row position distributions in the sparse matrix into the same cluster. Each cluster corresponds to a distribution pattern, meaning that the clustering results include the aforementioned multiple distribution patterns. Each distribution pattern includes an equal number of columns, and the number of columns in each distribution pattern has the same quantitative relationship with the number of activated columns.

[0093] For example, if the number of activated columns in the ReRAM is L, and storing one weight in the ReRAM requires 2 units, each unit representing 2 bits, i.e., occupying 4 bits, then to improve the utilization efficiency of the ReRAM, the maximum number of activated weight columns in the ReRAM is L / 2. Therefore, the number of columns included in each class is the maximum number of activated weight columns in the ReRAM (e.g., L / 2). Since the size of the sparse matrix is ​​equal to the size of the weight matrix, when clustering the columns of the sparse matrix, the number of classes in the clustering result is the ratio of the number of columns in the sparse matrix to the maximum number of activated weight columns in the ReRAM.

[0094] In some possible implementations, the sparse matrix has multiple columns with the same distribution pattern, but due to the limitation on the number of activated columns, at least two classes with the same distribution pattern exist in the clustering results.

[0095] For example, suppose the sparse matrix has N columns, the ReRAM has L active columns, and the maximum number of active weight columns in the ReRAM is L'. Then, the sparse matrix is ​​divided into K classes based on the similarity of the weight positions along the column directions, where K is the ratio of N to L'. In other words, the N columns of the sparse matrix are divided into K classes, with each class containing L' columns. It should be noted that each column in the sparse matrix can be called a weight column.

[0096] For example, the K-means clustering method can be used to cluster sparse matrices, and the process is as follows: Steps A1-A3:

[0097] Step A1: Randomly select K weight columns from the sparse matrix as K cluster centers.

[0098] Step A2: Compare the first distance between each remaining weight column in the sparse matrix and the K cluster centers, and divide each weight column into the cluster center that is closest to the row position distribution of the weights in that column according to the first distance.

[0099] It should be noted that the first distance measures the difference between the row position distribution of the weights in the weight column to be compared in the sparse matrix and the row position distribution represented by a certain cluster center. The smaller the first distance, the more similar the row position distribution of the weights in the weight column to be compared is to the row position distribution represented by the cluster center.

[0100] For example, each weight column of a sparse matrix can be represented by 0-1 to show the row position distribution of each weight column. Here, 1 indicates that the corresponding position has a weight, and 0 indicates that the corresponding position has no weight. See, for example, [link to documentation]. Figure 6In (2), the row position distribution of the first column of the sparse matrix can be represented as "001010100100100", which means that only the 3rd, 5th, 7th, 10th and 13th rows of this column have weights, if the first column is one of the K cluster centers. Taking the classification of the second column (non-cluster centers) of a sparse matrix as an example, the row position distribution of the second column of the sparse matrix can be represented as "100100010001001". Calculate the first distance between the second column and the first column (cluster centers). In this case, perform a bitwise AND operation between "100100010001001" in the second column and "001010100100100" in the first column to obtain "101110110101001". Then sum "101110110101001" to obtain the first distance of 9. Based on this method, calculate the first distance between the second column and the remaining (K-1) cluster centers. Compare the K first distances corresponding to the second column and assign the second column to the cluster center with the smallest first distance among the K first distances.

[0101] For example, the row coordinates of the rows containing the weights in each weight column can also be recorded. See, for example, [link to relevant documentation]. Figure 6 In (2), the row coordinates of the weights in the first column of the sparse matrix are labeled as (3,5,7,10,13). If the first column is one of the K cluster centers, taking the classification of the second column (non-cluster center) of the sparse matrix as an example, the row coordinates of the weights in the second column of the sparse matrix are labeled as (1,4,8,12,15). The first distance between the second column and the first column (cluster center) is calculated. In this case, the sum of the absolute error values ​​of the row coordinates between (1,4,8,12,15) of the second column and (3,5,7,10,13) of the first column is calculated to obtain the first distance as 8. According to this method, the first distance between the second column and the remaining (K-1) cluster centers is calculated. The K first distances corresponding to the second column are compared, and the second column is assigned to the cluster center corresponding to the smallest first distance among the K first distances.

[0102] Step A3: Update the cluster centers of each class and determine whether the cluster centers of each class have changed.

[0103] Specifically, each weight column in the sparse matrix is ​​assigned to K cluster centers. The cluster centers of each class are recalculated to update the cluster centers. It is then determined whether the cluster centers of each class have changed. If they have not changed, the classification ends. If the cluster centers of any class have changed, the current cluster centers are updated, and steps A2-A3 are continued until the cluster centers of each class no longer change, so that the number of weight columns in each class in the final clustering result is equal to the maximum number of weight columns that can be activated in ReRAM. For ease of description, the maximum number of weight columns that can be activated in ReRAM can be simply referred to as the number of activated weight columns.

[0104] For example, such as Figure 7A As shown, Figure 7A This is a schematic diagram of a clustering result provided in an embodiment of this application. To clearly display the clustering result of the sparse matrix, columns belonging to the same class (i.e., the same distribution pattern) are arranged together. It can be seen that the sparse matrix is ​​divided into 6 classes, each containing 4 weight columns, and the columns in the same class have similar row position distributions. Taking the fourth class as an example from left to right, the weight position distribution pattern corresponding to the fourth class is "010001010010100", that is, for the fourth class, the weights of each weight column are concentrated in rows 2, 6, 8, 11, and 13.

[0105] In some possible implementations, multiple clustering operations are required for each column of the sparse matrix to ensure that each cluster contains the same number of columns. That is, after one clustering operation on the sparse matrix, the number of weight columns in each cluster is inconsistent; some clusters have more weight columns than others. In this case, it is necessary to unify the number of weight columns in each cluster. This can be achieved by extracting and / or re-clustering clusters with weight columns greater than or less than a preset threshold, ensuring that each cluster contains the same number of activated weight columns. It should be noted that the preset threshold is the number of activated weight columns, denoted as L'.

[0106] Specifically, the process of determining the number of weight columns included in each cluster of the clustering results can be illustrated by taking the i-th cluster as an example, where i = 1, 2, ..., K. The determination process is as follows:

[0107] B1. The number of weight columns included in the i-th cluster in the statistical clustering results.

[0108] B2. Determine whether the number of weight columns currently included in the i-th category is greater than the number of activated weight columns L'.

[0109] Specifically, determine whether the number of weight columns currently included in the i-th cluster in the clustering result is greater than the number of activated weight columns L'. If the number of weight columns currently included in the i-th cluster is greater than L', execute B3; if the number of weight columns included in the i-th cluster is less than or equal to L', execute B4.

[0110] B3. Extract the top L' weight columns with the smallest first distance from the cluster center of the i-th class as a class and store them in the class set.

[0111] Specifically, when the number of weight columns currently included in the i-th class is greater than L', L' weight columns are extracted from the i-th class and stored as a class in the class set. The row position distribution of the extracted L' weight columns is most similar to the row position distribution corresponding to the cluster center of the i-th class. Therefore, the extracted L' weight columns are the first L' weight columns with the smallest distance to the cluster center of the i-th class.

[0112] It should be noted that after the weight columns in the i-th class are extracted, B2 still needs to be executed until the number of weight columns currently included in the i-th class is less than or equal to the number of activated weight columns L'.

[0113] B4. Determine whether the number of weight columns currently included in the i-th category is equal to the number of activated weight columns L'.

[0114] Specifically, determine whether the number of weight columns currently included in the i-th category is equal to the number of activated weight columns L'. If the number of weight columns currently included in the i-th category is equal to L', execute B5; if the number of weight columns included in the i-th category is less than L', execute B6.

[0115] B5. Treat the weight columns currently included in the i-th class as a class and store them in the class set.

[0116] B6. Add the weight column currently included in the i-th class to the set of remaining columns.

[0117] Specifically, when the number of weight columns currently included in the i-th class is less than the number of activated weight columns L', the weight columns currently included in the i-th class are added to the set of remaining columns.

[0118] After performing the above B1-B6 processing on each cluster in the clustering results, the remaining column set is obtained. Then, according to the relevant descriptions in steps A1-A3 above, the weight columns in the remaining column set are clustered. The re-clustered clustering results are then processed again using the above B1-B6 processing. This process is repeated to obtain... Figure 7A The clustering effect shown indicates that the set of clusters can be used as the final clustering result of the sparse matrix. The number of weight columns included in each cluster of this set is equal to the number of activated weight columns L', thus ensuring that the number of weight columns included in each cluster of the clustering result is consistent. It can be understood that when clustering the remaining column set, in order to ensure that the total number of clusters in the sparse matrix is ​​consistent, the number of clusters corresponding to the remaining column set is less than or equal to the total number of clusters.

[0119] For example, when clustering a sparse matrix (matrix size 15*24), assuming the number of activated weight columns is 4, the desired outcome is that each cluster includes 4 weight columns. However, after one clustering operation, the columns in the sparse matrix are divided into 6 clusters. The number of weight columns in each cluster from the first to the sixth is 7, 3, 4, 4, 4, and 2 respectively. It can be seen that the number of weight columns in each cluster is inconsistent. Referring to the processing flow of B1-B6 above, we know that 4 columns need to be extracted from the first cluster and stored in the category set. The remaining 3 columns in the first cluster are added to the remaining column set. Since the number of columns in the second and sixth clusters is less than 4, the 3 columns in the second cluster and the 2 columns in the sixth cluster are added to the remaining column set. The third, fourth, and fifth clusters each include 4 columns, which is equal to the number of activated weight columns, so the third, fourth, and fifth clusters are stored in the category set. In summary, the category set contains 4 categories, each corresponding to a distribution pattern. The remaining column set contains 8 columns. Clustering is performed on the columns in the remaining column set, that is, the 8 columns in the remaining column set are divided into 2 classes based on the similarity of the row position distribution of the column weights, and each class contains 4 columns. These two classes obtained from the second clustering are then added to the category set, which can be used as the final clustering result. This category set contains 6 classes, each containing 4 columns.

[0120] It should be noted that regardless of how many times the columns of the sparse matrix are clustered, the ultimate goal is to make the number of weight columns included in each cluster equal, and the row positions of the non-zero weights of the columns belonging to the same cluster most similar.

[0121] On the other hand, the so-called processing of the row position distribution of each column in each class to obtain the mask matrix means that, due to the randomness of the distribution of non-zero weights in the sparse matrix, the row position distribution of non-zero weights in some columns of each class is not exactly the same as the distribution pattern corresponding to that class. In this case, the columns in each class that are different from the distribution pattern corresponding to that class are processed to unify the row positions of the non-zero weights in each column of each class to make the row position distribution of each column in each class the same as the position distribution corresponding to that class, so as to obtain the mask matrix.

[0122] For example, see Figure 7A , Figure 7AThis is a visualization of the clustering effect of a sparse matrix. Columns belonging to the same cluster are grouped together for easy display of the clustering results. The sparse matrix is ​​divided into 6 clusters, each containing 4 columns. However, for each cluster, at least one column has a different row position distribution than the corresponding cluster. Let's take the fourth cluster (from left to right) as an example. The weight distribution pattern for the fourth cluster is "010001010010100", meaning each column has non-zero weights in rows 2, 6, 8, 11, and 13. However, the first column in the fourth cluster has non-zero weights in rows 2, 6, 8, 12, and 13, which is different from the fourth cluster's distribution pattern. In this case, the weight in row 12 of the first column in the fourth cluster is deleted, and the weight in row 11 of the first column in the fourth cluster is marked. Figure 7A After uniformly processing the row position distribution of each column in each category, we can obtain Figure 7B , Figure 7B Columns belonging to the same category have the same distribution pattern. It should be noted that, similar to... Figure 7A , Figure 7B The columns in each cluster are arranged together to display the clustering effect, but in reality, the column positions within each cluster are not... Figure 7B As shown, each column in each class corresponds to a column position in the sparse matrix. For example, Figure 7B The second column from left to right in the text actually corresponds to Figure 6 The sixth column of the sparse matrix in (2), i.e. Figure 7B The second column is composed of Figure 6 It is obtained by changing the sixth column of (2) in the middle.

[0123] The mask matrix can be obtained by creating a zero-based marker matrix of the same size as the sparse matrix. Based on the row position distribution of the weights of each column in each class after processing and the column position of each column in the corresponding sparse matrix, the value at the position corresponding to the weight in the corresponding column of the marker matrix is ​​marked from "0" to "1". The marker matrix obtained after marking is the mask matrix.

[0124] For example, see Figure 7A , Figure 7B and Figure 6 (2) Figure 7A It can be seen as a Figure 6 The clustering effect of the sparse matrix shown in Figure (2) is displayed. Figure 7B Yes Figure 7A The image shows the result after unifying the positional distribution of each column within each category. Figure 7B The columns in Figure 7A The columns in the table correspond one-to-one from left to right. Figure 7B The first column corresponds to Figure 7A The first column in the text, Figure 7BThe 24th column corresponds to Figure 7A Column 24. But for Figure 7A and Figure 6 From (2), it can be seen that Figure 7A The 13th column from left to right (i.e., the first column of the fourth category) actually corresponds to Figure 6 The 15th column of the sparse matrix shown in (2) is... Figure 7B The 13th column from left to right also corresponds. Figure 6 The 15th column of the sparse matrix shown in (2) is therefore referred to Figure 7B The 13th column, in the 15th column of the marker matrix, will Figure 7B In the 13th column, the value corresponding to the weight is marked as "1". Based on this method, each column of the marking matrix is ​​marked to obtain the mask matrix.

[0125] It should be noted that the elements in the mask matrix consist of 0 and 1, where "1" represents an active position and "0" represents an inactive position (i.e., a weight of 0). The mask matrix represents the weight position distribution of the weight matrix after processing such as unstructured pruning and column clustering.

[0126] S1023. Obtain index information.

[0127] In this embodiment of the application, during the process of obtaining the mask matrix by performing a series of processes such as clustering on the sparse matrix, it is also necessary to obtain index information, which includes row index information and column index information. Specifically, the column positions of each column belonging to each class in the mask matrix are stored to obtain column index information; the row positions of each non-zero row corresponding to each class in the mask matrix are stored to obtain row index information.

[0128] It can be understood that each column in each class can form a submatrix, and the number of columns in the submatrix is ​​equal to the number of activation weight columns in the ReRAM. The number of rows in the submatrix is ​​equal to the number of rows in the sparse matrix. Therefore, each non-zero row corresponding to each class is equivalent to each non-zero row in the submatrix corresponding to that class. Each non-zero element in the same row of the submatrix corresponds to the same row position in the mask matrix (or weight matrix), and each non-zero element in the same column of the submatrix corresponds to the same column position in the mask matrix (or weight matrix).

[0129] In some possible embodiments, the row index information and column index information can be mapping tables similar to those shown in Table 1 and Table 2. As shown in Table 1, Table 1 is an example of column index information, which records the column positions of each column in the mask matrix corresponding to each category. Taking category 1 in Table 1 as an example, the column position corresponding to category 1 is "100001000000000010000010", that is, the column positions corresponding to the columns included in category 1 are columns 1, 6, 17, and 23 in sequence. As shown in Table 2, Table 2 is an example of row index information, which records the row positions of each non-zero row corresponding to each category in the mask matrix, and the columns of the same category have the same row position distribution of non-zero elements. Taking category 1 as an example, the row position corresponding to category 1 is represented by the row vector "010010100100100", which means that each column in category 1 has weights in rows 2, 5, 7, 10, and 13.

[0130] Table 1

[0131]

[0132]

[0133] Table 2

[0134]

[0135] S1024. Train the weight matrix based on the mask matrix to obtain the target matrix.

[0136] In this embodiment, a weight matrix is ​​trained based on the mask matrix to obtain the target matrix. The target matrix contains weights at the activation positions corresponding to the mask matrix, and no weights at the inactive positions corresponding to the mask matrix. It can be understood that the target matrix, mask matrix, and weight matrix are all of the same size.

[0137] It should be noted that the Alternating Direction Method of Multipliers (ADMM) can be used to progressively train the weight matrix based on the mask matrix. This allows the weights at the non-activated positions of the corresponding mask matrix to be progressively reduced to 0, and the weights at the activated positions of the corresponding mask matrix to be continuously adjusted to compensate for the accuracy loss caused by pruning, ultimately obtaining the target matrix, which is the weight matrix after training.

[0138] It is understandable that the target matrix and the mask matrix are similar in that the distribution of non-zero elements in the target matrix is ​​the same as that in the mask matrix; the difference is that all non-zero elements in the mask matrix are "1", while the non-zero elements in the target matrix include weights with different values.

[0139] It should be noted that in some possible embodiments, the backpropagation algorithm or other methods can be used to train the weight matrix based on the mask matrix to obtain the target matrix, and this application does not make specific limitations.

[0140] S1025. Compress the target matrix according to the index information to obtain the compressed matrix.

[0141] In this embodiment of the application, the index information includes row index information and column index information. Compressing the target matrix according to the index information to obtain a compressed matrix means: rearranging each column of the target matrix according to the column index information so that columns belonging to the same category in the rearranged target matrix are arranged together; then extracting the non-zero rows in each category of the rearranged target matrix according to the row index information to obtain multiple compressed matrix blocks, and the multiple compressed matrix blocks form a compressed matrix.

[0142] Specifically, since the column index information records the column positions of each column belonging to each category in the mask matrix, the columns of the target matrix are rearranged according to the column index information, and the rearrangement result is as follows: Figure 8A As shown, columns belonging to the same category in the target matrix are arranged in one column. Figure 8A Different gray levels are used to distinguish different classes to display the target matrix after column rearrangement; then, based on the row index information, the rearranged target matrix is ​​compressed in the row direction to delete the rows with all zeros corresponding to each class, obtaining a compressed matrix, such as... Figure 8B As shown, in Figure 8B In the compression matrix, there are 6 compression matrix blocks, each of which is 5*4 in size. The number of rows (5) in each compression matrix block is the number of active rows in the ReRAM, and the number of columns (4) in each compression matrix block is the number of active weight columns (which is different from the number of active columns in the ReRAM).

[0143] It should be noted that each category corresponds to a compressed matrix block in the compressed matrix. After obtaining the compressed matrix, it is mapped onto ReRAM, that is, each weight in the compressed matrix is ​​mapped to a cell in the cross array of ReRAM. The number of rows and columns of each compressed matrix block satisfies the hardware limitations of ReRAM. Therefore, a ReRAM-based accelerator can activate one compressed matrix block stored in ReRAM per cycle. In other words, through time-division multiplexing, the ReRAM-based accelerator improves the utilization efficiency of ReRAM by activating a matrix consisting of a specific number of rows and columns (i.e., one compressed matrix block in the compressed matrix).

[0144] The mapping of the compression matrix in ReRAM can be changed according to the number of rows and columns of the cross array in ReRAM. If each weight consumes one cell of ReRAM storage, it means that each row of ReRAM can store a sufficient number of weights. Figure 8BThe six compression matrix blocks in the middle compression matrix can be used as follows: Figure 8B The diagram shown is mapped to the cells of the cross array of ReRAM, meaning that the six compressed matrix blocks can be mapped sequentially along the horizontal direction (word line direction). If each weight consumes four cells of ReRAM storage, it means that the number of weights that can be stored in each row of ReRAM is limited. In this case, Figure 8B The six compressed matrix blocks shown can be mapped sequentially along the vertical direction (bit line direction), and this application does not impose any specific limitations.

[0145] As can be seen, by implementing the embodiments of this application, the target matrix is ​​obtained by performing unstructured pruning, clustering and other processing on the weight matrix of the neural network based on the hardware limitations of ReRAM (i.e., limiting the number of activated rows and columns). Then, the target matrix is ​​compressed by utilizing the sparsity of the target matrix in the column and row directions. This not only improves the compression ratio of the weight matrix, but also makes the compressed matrix match the hardware conditions of ReRAM, thereby improving the row and column utilization of ReRAM.

[0146] S103. Obtain the input data corresponding to each column of each compression matrix block in the compression matrix based on the index information and image data.

[0147] In this embodiment, obtaining the input data corresponding to each column of each compression matrix block in the compression matrix based on row index information and image data means: extracting the data at the row positions indicated by the row index information from the image data to obtain the input data corresponding to each column of each compression matrix block. Here, the image data refers to the input data corresponding to each column of the weight matrix. The image data can be a feature map of the image, a grayscale image, a color image, or an intermediate result of image processing, etc., and this application does not impose specific limitations. The image data can also be referred to as the original input data corresponding to each column of the compression matrix block.

[0148] On the one hand, the weights in each column of each compressed matrix block have the same row position distribution, so the input data corresponding to each compressed matrix block is the same; on the other hand, each column of each compressed matrix block is equivalent to a filter after compression. It can be understood that the input data corresponding to the filter after compression is different from the input data corresponding to the original filter (uncompressed). Therefore, it is necessary to determine the input data corresponding to each column of each compressed matrix block based on the index information and image data.

[0149] For example, see Figure 9 , Figure 9 This is a schematic diagram illustrating how to determine the input data corresponding to a compressed matrix block. Each column of the weight matrix is ​​equivalent to a raw filter. Assuming the size of the raw filter is 5*3, the raw input data corresponding to the raw filter is as follows: Figure 9As shown, represented as a row vector "428967653434253", the row position corresponding to the first matrix block (compressed matrix block) is determined based on the row index information as follows: Figure 9 As shown, if the row vector is represented as "010010100100100", then the column vector formed by taking the corresponding weights from the 2nd, 5th, 7th, 10th and 13th rows of the original input data according to the row position of the first matrix block is the input data corresponding to the first matrix block, which is represented as "26642" in row vector.

[0150] S104. Input the input data into the ReRAM-based accelerator to obtain the output data corresponding to each column in each compressed matrix block.

[0151] In this embodiment of the application, the obtained input data is input into a ReRAM-based accelerator. The ReRAM-based accelerator stores a compression matrix, which includes multiple compression matrix blocks, so that the ReRAM-based accelerator can calculate the output data corresponding to each column of each compression matrix block based on the input data and each compression matrix block.

[0152] For ReRAM-based accelerators, the received input data is multiplied by the weights of each column in each compressed matrix block, and then the products corresponding to the weights of each column are summed to obtain the output data for each column of each compressed matrix block. Here, the input data corresponds one-to-one with the weights of each column in each compressed matrix block. For details of the above acceleration process, please refer to [reference needed]. Figure 1B The relevant descriptions in the document will not be repeated here.

[0153] It should be noted that ReRAM is a cross array with word lines and bit lines. The input data is simply input into the word lines of ReRAM as analog voltages. The analog voltage of each word line in ReRAM is multiplied by the weight in the cell corresponding to each bit line. The accelerator based on ReRAM outputs the cumulative result corresponding to each column of bit lines.

[0154] It should be noted that if each cell in the ReRAM stores one weight, the cumulative result corresponding to each column of bit lines is the output data corresponding to each column in each compressed matrix block in this application; if every two cells in the ReRAM store one weight, the dot product result corresponding to each cell of the two bit lines needs to be processed according to the conversion relationship between the two bit lines corresponding to the column to obtain the cumulative result, which is the output data corresponding to the column.

[0155] By leveraging the parallel computing capabilities of ReRAM-based accelerators, the size of the weight matrix is ​​reduced through methods such as unstructured pruning, clustering, and compression. This not only improves the compression ratio of the weight matrix but also reduces the number of weights involved in each column of the convolution operation, effectively increasing the computation speed.

[0156] S105. Based on the index information, ensure that the output data corresponding to each column in each compressed matrix block corresponds to the correct column in the weight matrix.

[0157] In this embodiment of the application, making the output result corresponding to each column in each compressed matrix block correspond to the correct column in the weight matrix according to the index information means: restoring the output data corresponding to each column to the correct column position in the weight matrix indicated by the column index information.

[0158] It should be noted that the compressed matrix block is obtained through compression, that is, by rearranging the columns and compressing the rows of the target matrix. It can be seen that the column position of each column in the compressed matrix block has changed. The actual column position of each column in the compressed matrix block is the column position of the target matrix corresponding to that column, which is also the position of that column before the column rearrangement operation, and also the correct column position in the weight matrix indicated by the column index information.

[0159] For example, see Figure 10 , Figure 10 This is a schematic diagram for determining the actual output position of the output data in each column of a compressed matrix block. Taking the first matrix block (compressed matrix block) as an example, assuming the size of the first matrix block is 5*4, that is, it has four columns, and assuming the output corresponding to the four columns in the first matrix block is (15, 26, 32, 28), according to the column index information in Table 1 above, the column position corresponding to the first matrix block (i.e., category one) is "100001000000000010000010". Therefore, it can be known that in the first matrix block, output result 15 should be output to column 1, output result 26 should be output to column 6, output result 32 should be output to column 17, and output result 28 should be output to column 23.

[0160] As can be seen, by implementing the embodiments of this application, when performing unstructured pruning and compression on the weight matrix, the hardware limitations of ReRAM (number of activated rows and columns) are considered to obtain the compression matrix and index information, so that the compression matrix matches and adapts to the hardware limitations of ReRAM. This not only reduces the size of the weight matrix and improves the compression ratio, but also improves the utilization efficiency of ReRAM. The compression matrix can maximize the utilization of the rows and columns of ReRAM based on the index information, thereby accelerating the operation speed of the neural network.

[0161] See Figure 11 , Figure 11This is a schematic diagram of a device provided in this application. Device 20 includes at least a processor 110, a memory 111, a receiver 112, a row conversion module 113, a column conversion module 114, and a memristor-based ReRAM accelerator 115. Optionally, the memory 111, receiver 112, row conversion module 113, column conversion module 114, and memristor-based ReRAM accelerator 115 are connected or coupled to the processor 110 via a bus. The row conversion module 113 and column conversion module 114 are also connected or coupled to the memristor-based ReRAM accelerator 115. In some possible embodiments, the memristor-based ReRAM accelerator 115 may also be independent of device 20.

[0162] It should be noted that the row conversion module 113 and the column conversion module 114 can be independent of the processor 110, or they can be integrated into the processor 110 so that the processor 110 has the functions of the row conversion module 113 and the column conversion module 114. This application does not make any specific limitations.

[0163] Receiver 112 is used to obtain the number of active rows and columns of the ReRAM. Receiver 112 can be a wired interface or a wireless interface. The wired interface can be an Ethernet interface, a Local Interconnect Network (LIN), etc., and the wireless interface can be a cellular network interface or a wireless LAN interface, etc.

[0164] The specific implementation of each operation by processor 110 can be found in the above method embodiments, such as unstructured pruning of the weight matrix, clustering of the sparse matrix, and compression of the target matrix. Processor 110 can be composed of one or more general-purpose processors, such as a central processing unit (CPU), or a combination of a CPU and hardware chips. The aforementioned hardware chips can be application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or combinations thereof. The aforementioned PLDs can be complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), generic array logic (GALs), or any combination thereof.

[0165] The row conversion module 113 is used to determine the input data corresponding to each compressed matrix block in the compressed matrix based on the row index information and image data, and send the input data to the memristor-based ReRAM-based accelerator 115. Optionally, the row conversion module 113 includes at least a first multiplexer, a second multiplexer, and a first decoder, wherein the first multiplexer is used to select the row position information corresponding to the target compressed matrix block in the compressed matrix from the row index information, the second multiplexer is used to obtain the original input data corresponding to the target compressed matrix block, and the first decoder is used to determine the input data corresponding to the target compressed matrix block based on the row position information output by the first multiplexer and the original input data output by the second multiplexer, wherein the target compressed matrix block is any one of multiple compressed matrix blocks in the compressed matrix.

[0166] The memristor-based ReRAM accelerator 115 is used to store the compression matrix. In addition, the memristor-based ReRAM accelerator 115 is also used to receive the input data corresponding to the target compression matrix block sent by the row conversion module 113, and output the output data corresponding to each column in the target compression matrix block.

[0167] The column conversion module 114 is used to map the output data corresponding to each column of the target matrix output by the memristor-based ReRAM accelerator 115 to the initial column position in the weight matrix. Optionally, the column conversion module 114 includes at least a third multiplexer, a fourth multiplexer, and a second decoder. The third multiplexer is used to select the column position information corresponding to the target compressed matrix block in the compressed matrix from the column index information. The fourth multiplexer is used to obtain the output data corresponding to each column of the target compressed matrix block. The second decoder is used to restore the output data corresponding to each column to the corresponding initial column position based on the column position information output by the third multiplexer and the output data corresponding to each column of the target matrix output by the fourth multiplexer.

[0168] The memory 111 may include volatile memory, such as random access memory (RAM); the memory 111 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); the memory 111 may also include combinations of the above types. The memory 111 can store programs and data, wherein the stored programs include: unstructured pruning algorithms, clustering algorithms, etc., and the stored data includes: weight matrices, mask matrices, target matrices, etc. The memory 111 may exist independently or be integrated within the processor 110.

[0169] It can be seen that device 20 can both compress the weight matrix to obtain a compressed matrix and match the compressed matrix with a ReRAM-based accelerator. In some possible embodiments, the function of device 20 can also be split into two different devices, such as a first device and a second device. The second device is used to perform unstructured pruning and compression operations on the weight matrix to obtain a compressed matrix, while the first device is used to match the compressed matrix with a ReRAM-based accelerator. Specifically, the first device sends the weight matrix, the number of active rows and columns of the ReRAM to the second device. The second device processes the weight matrix according to the number of active rows and columns of the ReRAM to obtain a compressed matrix and row / column index information. The second device sends the compressed matrix and row / column index information to the first device. The first device stores the received compressed matrix in the ReRAM-based accelerator, the row index information in the row conversion module, and the column index information in the column conversion module. The first device uses the row conversion module and column conversion module to match the compressed matrix with the ReRAM-based accelerator. For details on the functional implementation of each part of the first device and the functional implementation of each part of the second device, please refer to [reference needed]. Figure 11 For the sake of brevity, the relevant descriptions in the manual will not be repeated here.

[0170] In this embodiment of the application, the device 20 is used to implement the above. Figure 4 and Figure 5 The method described in the embodiments.

[0171] See Figure 12 , Figure 12This is a functional structure diagram of a device provided in an embodiment of this application. The device 22 includes an acquisition unit 220, a processing unit 221, a first conversion unit 222, an acceleration unit 223, and a second conversion unit 224. The device 22 can be implemented by hardware, software, or a combination of hardware and software.

[0172] The system includes: an acquisition unit 220 for acquiring the number of active rows and columns of the memristor ReRAM; a processing unit 221 for performing unstructured pruning and compression on the weight matrix to obtain a compressed matrix block, wherein the number of rows in the compressed matrix block is less than or equal to the number of active rows, the number of columns in the compressed matrix block is less than or equal to the number of active columns, the row positions of the weight matrices corresponding to each weight in the same row of the compressed matrix block are the same, and the column positions of the weight matrices corresponding to each weight in the same column of the compressed matrix block are the same; a first conversion unit 222 for extracting data at the row positions indicated by the index information from the image data to obtain the input data corresponding to each column of the compressed matrix block; the image data is the input data corresponding to each column of the weight matrix; an acceleration unit 223 for inputting the input data into the ReRAM-based accelerator to obtain the output data corresponding to each column of the compressed matrix block; and a second conversion unit 224 for restoring the output data corresponding to each column to the correct column position in the weight matrix indicated by the index information.

[0173] The functional modules of the device 22 can be used to achieve Figure 4 The method described in the embodiments. Figure 4 In this embodiment, the acquisition unit 220 can be used to execute S101, the processing unit 221 can be used to execute S102, the first conversion unit 222 can be used to execute S103, the acceleration unit 223 can be used to execute S104, and the second conversion unit 224 can be used to execute S105. In addition, the processing unit 221 is also used to implement... Figure 5 The method described in the embodiment, the processing unit 221 can be used to execute S1021-S1025.

[0174] In the embodiments described above, each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0175] It should be noted that those skilled in the art will recognize that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. This program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0176] The technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to cause a device (which may be a personal computer, server, network device, robot, microcontroller, chip, robot, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

Claims

1. A ReRAM-based weight matrix processing method, characterized in that, The method includes: Obtain the number of active rows and columns of the memristor ReRAM; Unstructured pruning and compression are performed on the weight matrix to obtain a compressed matrix block. The number of rows in the compressed matrix block is less than or equal to the number of active rows, and the number of columns in the compressed matrix block is less than or equal to the number of active columns. The row positions of the weight matrix corresponding to each weight in the same row of the compressed matrix block are the same, and the column positions of the weight matrix corresponding to each weight in the same column of the compressed matrix block are the same.

2. The method of claim 1, wherein, The number of compressed matrix blocks is multiple, and the unstructured pruning and compression of the weight matrix to obtain compressed matrix blocks includes: Unstructured pruning is performed on the weight matrix to obtain a sparse matrix, wherein the number of non-zero weights in each column of the sparse matrix is ​​the same as the number of rows in the compressed matrix block; The columns of the sparse matrix are processed to obtain a mask matrix. The mask matrix consists of 0s and 1s. Each column of the mask matrix belongs to multiple classes. Each class includes the same number of columns. The non-zero elements in the columns of the same class are in the same row position. The weight matrix is ​​trained based on the mask matrix to obtain the target matrix; the target matrix has non-zero weights at the positions corresponding to the non-zero elements of the mask matrix. Extract the non-zero weights of each column belonging to each class in the target matrix to obtain multiple compressed matrix blocks.

3. The method of claim 2, wherein, After processing each column of the sparse matrix, row index information and column index information are also obtained. The column index information includes the column position of each column in each class in the mask matrix; the row index information includes the row position of the non-zero element of each column in each class in the mask matrix. The step of extracting the non-zero weights of each column belonging to each class in the target matrix to obtain multiple compressed matrix blocks includes: Based on the column index information, the columns of the target matrix are rearranged so that columns belonging to the same category in the rearranged target matrix are arranged together. Based on the row index information, extract the non-zero rows from each class in the rearranged target matrix to obtain the plurality of compressed matrix blocks, the number of which is equal to the number of classes.

4. The method according to claim 2 or 3, characterized in that, The more times the sparse matrix is ​​processed, the higher the similarity of the row position distribution of the non-zero weights of each column included in each class.

5. The method according to claim 2 or 3, characterized in that, The proportion of zero weights in the sparse matrix is ​​greater than or equal to the sparsity, where the sparsity indicates the degree of unstructured pruning of the weight matrix.

6. A ReRAM-based weight matrix processing method, characterized in that, The method includes: A compressed matrix block is stored in a memristor ReRAM, wherein the number of rows in the compressed matrix block is less than or equal to the number of active rows in the ReRAM, and the number of columns in the compressed matrix block is less than or equal to the number of active columns in the ReRAM; the compressed matrix block is obtained based on unstructured pruning and compression of the weight matrix; Obtain index information, which includes the position mapping information of the weights in the compressed matrix block; Data at the row positions indicated by the index information is extracted from the image data to obtain the input data corresponding to each column of the compression matrix block; the image data is the input data corresponding to each column of the weight matrix. The input data is input into the ReRAM-based accelerator to obtain the output data corresponding to each column of the compression matrix block; The output data corresponding to each column is restored to the correct column position in the weight matrix indicated by the index information.

7. The method of claim 6, wherein, In the compressed matrix block, the row positions of the weight matrix corresponding to each weight in the same row are the same, and the column positions of the weight matrix corresponding to each weight in the same column are the same.

8. The method according to claim 7, characterized in that, The index information includes row index information and column index information. The row index information includes the row position of each weight in each column of the compressed matrix block corresponding to the weight matrix. The column index information includes the column position of each column of the compressed matrix corresponding to the weight matrix. The step of extracting data at the row position indicated by the index information from the image data to obtain the input data corresponding to each column of the compressed matrix block includes: Extract the data at the row position indicated by the row index information from the image data to obtain the input data corresponding to each column of the compressed matrix block; Restoring the output data corresponding to each column to the correct column position in the weight matrix indicated by the index information includes: The output data corresponding to each column is restored to the correct column position in the weight matrix indicated by the column index information.

9. An apparatus for weight matrix processing of ReRAM, characterized in that, The device includes: The acquisition unit is used to acquire the number of active rows and columns of the memristor ReRAM; The processing unit is used to perform unstructured pruning and compression on the weight matrix to obtain a compressed matrix block. The number of rows in the compressed matrix block is less than or equal to the number of active rows, the number of columns in the compressed matrix block is less than or equal to the number of active columns, the row position of the weight matrix corresponding to each weight in the same row of the compressed matrix block is the same, and the column position of the weight matrix corresponding to each weight in the same column of the compressed matrix block is the same.

10. The apparatus of claim 9, wherein, The number of compressed matrix blocks is multiple, and the processing unit is specifically used for: Unstructured pruning is performed on the weight matrix to obtain a sparse matrix, wherein the number of non-zero weights in each column of the sparse matrix is ​​the same as the number of rows in the compressed matrix block; The columns of the sparse matrix are processed to obtain a mask matrix. The mask matrix consists of 0s and 1s. Each column of the mask matrix belongs to multiple classes. Each class includes the same number of columns. The non-zero elements in the columns of the same class are in the same row position. The weight matrix is ​​trained based on the mask matrix to obtain the target matrix; The target matrix has non-zero weights at the positions corresponding to the non-zero elements of the mask matrix; Extract the non-zero weights of each column belonging to each class in the target matrix to obtain multiple compressed matrix blocks.

11. The apparatus of claim 10, wherein, The processing unit is further configured to obtain row index information and column index information, wherein the column index information includes the column position of each column in each category in the mask matrix; the row index information includes the row position of the non-zero element of each column in each category in the mask matrix; the processing unit is further configured to: Based on the column index information, the columns of the target matrix are rearranged so that columns belonging to the same category in the rearranged target matrix are arranged together. Based on the row index information, extract the non-zero rows from each class in the rearranged target matrix to obtain the plurality of compressed matrix blocks, the number of compressed matrix blocks being equal to the number of classes.

12. The apparatus of claim 10 or 11, wherein, The more times the sparse matrix is ​​processed, the higher the similarity of the row position distribution of the non-zero weights of each column included in each class.

13. The apparatus of claim 10 or 11, wherein, The proportion of zero weights in the sparse matrix is ​​greater than or equal to the sparsity, where the sparsity indicates the degree of unstructured pruning of the weight matrix.

14. An apparatus for weight matrix processing of ReRAM, characterized in that, The device includes: A storage unit is used to store compressed matrix blocks into a memristor ReRAM, wherein the number of rows in the compressed matrix block is less than or equal to the number of active rows in the ReRAM, and the number of columns in the compressed matrix block is less than or equal to the number of active columns in the ReRAM; the compressed matrix block is obtained based on unstructured pruning and compression of the weight matrix; An acquisition unit is used to acquire index information, the index information including the position mapping information of the weights in the compressed matrix block; A row conversion unit is used to extract data at the row position indicated by the index information from the image data to obtain the input data corresponding to each column of the compression matrix block; the image data is the input data corresponding to each column of the weight matrix. An acceleration unit is used to input the input data into an accelerator based on the ReRAM to obtain the output data corresponding to each column of the compression matrix block; The column transformation unit is used to restore the output data corresponding to each column to the correct column position in the weight matrix indicated by the index information.

15. The apparatus according to claim 14, characterized in that, In the compressed matrix block, the row positions of the weight matrix corresponding to each weight in the same row are the same, and the column positions of the weight matrix corresponding to each weight in the same column are the same.

16. The apparatus of claim 15, wherein, The index information includes row index information and column index information. The row index information includes the row position of each weight in each column of the compressed matrix block corresponding to the weight matrix. The column index information includes the column position of each column of the compressed matrix corresponding to the weight matrix. The row conversion unit is specifically used to: extract data at the row position indicated by the row index information from the image data to obtain the input data corresponding to each column of the compression matrix block; The column conversion unit is specifically used to restore the output data corresponding to each column to the correct column position in the weight matrix indicated by the column index information.

Images