Compression neural network

Abstract

The invention relates to a compressed neural network. The method for compressing the neural network comprises the steps of receiving the neural network; adjacent layers of the neural network are selected, the adjacent layers are provided with input channels and output channels, and the output channels of the first layer correspond to the input channels of the second subsequent layer; determining first and second matrices, rows or columns of the first and second matrices representing output channels and other rows or columns representing input channels; forming an array by aligning columns or rows of the first matrix with columns or rows of the second matrix; rearranging rows and / or columns of the array; aggregating elements representing non-zero values included in the first matrix and the second matrix into a first sub-matrix and a second sub-matrix, the first and second sub-matrices have a ratio of the number of elements representing non-zero values to the total number of elements of the first and second sub-matrices greater than the ratio of the number of elements representing non-zero values to the total number of elements of the first and second matrices; and outputting a compression neural network, the compression neural network comprising a first compression layer and a second subsequent compression layer.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to UK patent application 2302839.2 filed on February 27, 2023, which is incorporated herein by reference in its entirety. This application also claims priority to UK patent application 2302840.0 filed on February 27, 2023, which is incorporated herein by reference in its entirety. This application also claims priority to UK patent application 2401650.3 filed on February 7, 2024, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates to methods and processing systems for compressing neural networks. Background Art

[0004] A neural network (NN) is a form of artificial network comprising multiple interconnected layers (eg, "layers") that can be used for machine learning applications. In particular, neural networks can be used to perform signal processing applications, including but not limited to image processing.

[0005] Figure 1 An example neural network 100 is shown that includes a plurality of adjacent layers (e.g., a first layer 102-1, a second layer 102-2, a third layer 102-3). Each layer (e.g., a first layer 102-1, a second layer 102-2, a third layer 102-3) receives a set of input activation values. The set of input activation values ​​received by a layer may be referred to as input activation data for the layer. Each layer 102-1, 102-2, 102-3 processes its respective received input activation data by performing an operation according to the layer to produce output data. The output data is provided to a subsequent layer as the input activation data for the layer (e.g., the output data is the "output activation data"), or is output as final output data of the neural network. For example, in the neural network 100, the first layer 102-1 receives input activation data 104 that is initially input to the neural network 100, and processes the input activation data by performing a first operation according to the first layer 102-1 to produce output activation data. The output activation data of the first layer 102-1 becomes the input activation data of the subsequent second layer 102-2, and the subsequent second layer processes the input activation data by performing a second operation according to the second layer 102-2 to generate output activation data. The output activation data of the second layer 102-2 becomes the input activation data of the subsequent third layer 102-3, and the subsequent third layer processes the input activation data by performing a third operation according to the third layer 102-3 to generate output data. The output data of the third layer 102-3 is output as the output data 106 of the neural network.

[0006] Each layer of a neural network can be one of a number of different types. The type of operation performed on the input activation data of the layer depends on the type of layer. Fully connected layers (sometimes called dense layers or linear layers) and convolutional layers are example types of neural network layers. It will be apparent to those skilled in the art that this is not an exhaustive list of example neural network layer types.

[0007] In a fully connected layer, a fully connected operation is performed by performing a matrix multiplication between a coefficient matrix comprising a set of coefficients of the fully connected layer and an input matrix comprising a set of input activation values ​​received by the fully connected layer. The purpose of a fully connected layer is to cause a dimensionality change between the activation data input to the layer and the data output from the layer. The coefficient matrix comprising the set of coefficients of the fully connected layer may have a dimension C out ×C in That is, the number of rows of the matrix can represent the number of output channels of the fully connected layer (“C out ”), and the number of columns of the matrix can represent the number of input channels of the fully connected layer (“C in ”). In the fully connected layer, a matrix multiplication WX=Y can be performed, where: W is a matrix that includes a set of coefficients and has dimension C out ×C in The coefficient matrix of C; X is the input matrix including the input activation value set and has dimension M×N, where C in = M; and Y is a set of output values ​​and has dimension C out Alternatively, the coefficient matrix comprising the coefficient set of the fully connected layer may have dimension C in ×C out That is, the number of rows of the matrix can represent the number of input channels of the fully connected layer (“C in ”), and the number of columns of the matrix can represent the number of output channels of the fully connected layer (“C out ”). In this alternative, in the fully connected layer, a matrix multiplication XW=Y can be performed, where: X is the input matrix including the set of input activation values ​​and having dimension M×N; W is the matrix including the set of coefficients and having dimension C in ×C out The coefficient matrix of C in = N; and Y is a set of output values ​​and has dimensions M×C out The output matrix of . Matrix multiplication involves performing multiple element-by-element multiplications between the coefficients of the coefficient matrix and the activation values ​​of the input matrix. The results of the element-by-element multiplications can be summed (e.g., accumulated) to form the output data values ​​of the output matrix.

[0008] It will be apparent to those skilled in the art that other types of neural network layers also use coefficient matrices comprising sets of coefficients to perform matrix multiplication.

[0009] In a convolutional layer, a convolution operation is performed using a set of input activation values ​​received by the convolutional layer and a set of coefficients of the convolutional layer. Figure 2a shows the data structures used in the example convolutional layer. Figure 2a In the example, the input activation data of the convolutional layer can have dimension C in ×H a ×W a In other words, the input activation data can be arranged as C in input channels, each of which has spatial dimension H a ×W a —— Among them, H a and W a are the height and width dimensions respectively. Figure 2a In the figure, the input activation data is shown to include three input channels (i.e., C in =3). The coefficient sets of the convolutional layer can be arranged in one or more input channels and one or more output channels. The output channels can alternatively be referred to as filters. Figure 2a In the equation, the coefficient set has dimension C out ×C in ×H w ×W w The number of input channels in the coefficient set of the convolutional layer corresponds to (e.g., is equal to) the number of input channels in the input activation data to be convolved with the coefficient set (e.g., in Figure 2a In the example shown, C in =3). Each input channel of each filter of the coefficient set input to the convolutional layer has a spatial dimension H w ×W w —— Among them, H w and W w are the height and width dimensions, respectively. Each input channel of each filter includes a corresponding subset of the set of coefficients of the convolutional layer. Each coefficient is contained in (e.g., is included by or is part of) an input channel and a filter. C out The dimension (e.g., the number of output channels or the number of filters) is Figure 2a , but represents the number of channels generated in the output data by performing a convolution operation using a coefficient set.

[0010] Figure 2b An example convolutional layer 200 is schematically shown. Figure 2b In FIG, a convolutional layer 200 is arranged to perform a convolution operation using input activation data 202 and a coefficient set 204. Figure 2b , activation data 202 input to layer 200 is arranged in three input channels 1, 2, 3. The number of input channels in coefficient set 204 corresponds to (e.g., is equal to) the number of input channels in activation data 202 to be convolved with coefficient set 204. Therefore, coefficient set 204 is arranged in three input channels 1, 2, 3. Coefficient set 204 is also arranged in four filters (e.g., output channels) A, B, C, D. The number of filters in coefficient set 204 corresponds to (e.g., is equal to) the number of channels in output data 206. Each coefficient is contained in (e.g., is included by or is part of) one input channel and one filter. For example, coefficient 210 is contained in input channel 1 and filter A.

[0011] In convolutional layer 200, input activation data 202 is convolved with coefficient set 204 to produce output data 206 having four data channels A, B, C, D. More specifically, the first input channel of input activation data 202 is convolved with the first input channel of each filter in coefficient set 204. For example, returning to Figure 2a , with spatial dimension H a ×W a The first input channel of the input activation data 202 may be associated with the coefficient set 204 having a spatial dimension H w ×W w The first input channel of each filter is convolved in directions s and t in multiple steps. Return to Figure 2b In a similar manner, the second input channel of input activation data 202 is convolved with the second input channel of each filter in coefficient set 204, and the third input channel of input activation data 202 is convolved with the third input channel of each filter in coefficient set 204. Convolving each input channel of input activation data with each input channel of each filter involves performing a plurality of element-wise multiplications between the activation values ​​of each input channel of input activation data and the coefficients of each input channel of each filter. The results of the element-wise multiplications may be summed (e.g., accumulated) to form output data values ​​for each channel of output data 206.

[0012] The coefficient sets used by the layers of a typical neural network typically include a large number of coefficients. When the neural network is implemented at a neural network accelerator, the coefficient sets are typically stored in "off-chip" memory. The neural network accelerator can implement the layers of the neural network by reading the coefficient sets of the layers at runtime. A large amount of memory bandwidth may be required to read a large number of coefficient sets from the off-chip memory. The memory bandwidth required to read in the coefficient sets can be referred to as "weight bandwidth". It is desirable to reduce the weight bandwidth required to implement the neural network at the neural network accelerator. Summary of the invention

[0013] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0014] According to a first aspect of the present invention, there is provided a computer-implemented method for compressing a neural network, the method comprising: receiving a neural network; selecting two or more adjacent layers of the received neural network, each of the two or more adjacent layers having one or more input channels and one or more output channels, the one or more output channels of a first layer of the two or more adjacent layers corresponding to the one or more input channels of a second subsequent layer of the two or more adjacent layers, the first layer being arranged to perform a first operation and the second layer being arranged to perform a second operation; determining a first matrix representing a set of coefficients of the first layer of the received neural network, the first matrix comprising a plurality of elements representing non-zero values ​​and a plurality of elements representing zero values, one or more rows or columns of the first matrix representing the one or more output channels of the first layer, and one or more other rows or columns of the rows or columns of the first matrix representing the one or more input channels of the first layer; determining a second matrix representing a set of coefficients of the second layer of the received neural network, the second matrix comprising a plurality of elements representing non-zero values ​​and a plurality of elements representing zero values, one or more rows or columns of the second matrix representing the one or more output channels of the second layer, and one or more other rows or columns of the rows or columns of the second matrix representing the one or more input channels; forming an array by one of the following ways: transposing the first matrix and forming the array including the transposed first matrix and the second matrix by aligning the columns or rows of the transposed first matrix representing the one or more output channels of the first layer with the columns or rows of the second matrix representing the one or more input channels of the second layer; or transposing the second matrix and forming the array including the transposed second matrix and the first matrix by aligning the rows or columns of the transposed second matrix representing the one or more input channels of the second layer with the rows or columns of the first matrix representing the one or more output channels of the first layer; or forming the array including the first matrix and the second matrix by aligning the rows or columns of the first matrix representing the one or more output channels of the first layer with the rows or columns of the second matrix representing the one or more input channels of the second layer; rearranging the rows and / or columns of the array so as to: aggregate the multiple elements representing non-zero values ​​included in the first matrix or the transposed first matrix into first one or more sub-matrices, the first one or more sub-matrices having a ratio of the number of elements representing non-zero values ​​to the total number of elements of the first one or more sub-matrices that is greater than the ratio of the number of elements representing non-zero values ​​to the total number of elements of the first matrix;and aggregating the plurality of elements representing non-zero values ​​included in the second matrix or the transposed second matrix into a second one or more sub-matrices, the second one or more sub-matrices having a ratio of the number of elements representing non-zero values ​​to the total number of elements of the second one or more sub-matrices that is greater than the ratio of the number of elements representing non-zero values ​​to the total number of elements of the second matrix; and outputting a compressed neural network, the compressed neural network comprising a first compression layer arranged to perform a first compression operation based on the first one or more sub-matrices, and a second subsequent compression layer arranged to perform a second compression operation based on the second one or more sub-matrices.;

[0015] Each of the first one or more sub-matrices may have a ratio of the number of elements representing non-zero values ​​to the total number of elements of the first sub-matrix that is greater than the ratio of the number of elements representing non-zero values ​​to the total number of elements of the first matrix; and / or each of the second one or more sub-matrices may have a ratio of the number of elements representing non-zero values ​​to the total number of elements of the second sub-matrix that is greater than the ratio of the number of elements representing non-zero values ​​to the total number of elements of the second matrix.

[0016] The first matrix may include the coefficient set of the first layer, the multiple elements representing non-zero values ​​may be multiple non-zero coefficients, the multiple elements representing zero values ​​may be multiple zero coefficients, and the one or more first sub-matrices may include a subset of the coefficient set of the first layer; and / or the second matrix may include the coefficient set of the second layer, the multiple elements representing non-zero values ​​may be multiple non-zero coefficients, the multiple elements representing zero values ​​may be multiple zero coefficients, and the one or more second sub-matrices may include a subset of the coefficient set of the second layer.

[0017] The first layer of the received neural network may be arranged to perform the first operation by performing matrix multiplications using the first matrix including the set of coefficients of the first layer and a first input matrix including the set of input activation values ​​of the first layer, and the compressed neural network may be configured such that the first compression layer is arranged to perform the first compression operation by performing one or more matrix multiplications using the first one or more sub-matrices and one or more subsets of the set of coefficients of the first layer included in the one or more first input sub-matrices, each of the one or more first input sub-matrices including a corresponding subset of the set of input activation values ​​of the first layer; and / or the second layer of the received neural network may be arranged to perform the second operation by performing matrix multiplications using the second matrix including the set of coefficients of the second layer and a second input matrix including the set of input activation values ​​of the second layer, and the compressed neural network may be configured such that the second compression layer is arranged to perform the second compression operation by performing one or more matrix multiplications using the second one or more sub-matrices and one or more subsets of the set of coefficients of the second layer included in the one or more second input sub-matrices, each of the one or more second input sub-matrices including a corresponding subset of the set of input activation values ​​of the second layer.

[0018] The first compression layer may be arranged to perform the first compression operation to form one or more first output sub-matrices, each of the one or more first output sub-matrices comprising a corresponding subset of the output activation values ​​of the first compression layer, the one or more first output sub-matrices of the first compression layer being the one or more second input sub-matrices.

[0019] The first layer of the received neural network may be a first convolutional layer comprising a set of coefficients arranged in one or more filters, each of the one or more filters being arranged in one or more input channels, each input channel of each filter comprising a corresponding subset of the set of coefficients of the first convolutional layer, and determining the first matrix may comprise: for each input channel of each filter: determining whether the input channel of the filter comprises a non-zero coefficient; and in response to determining that the input channel of the filter comprises at least one non-zero coefficient, representing the input channel of the filter with elements in the first matrix representing non-zero values; or in response to determining that the input channel of the filter comprises only zero coefficients, representing the input channel of the filter with elements in the first matrix representing zero values.

[0020] The first convolutional layer of the received neural network may be arranged to perform the first operation by convolving a set of input activation values ​​of the first convolutional layer with the set of coefficients of the first convolutional layer; the first one or more sub-matrices may include multiple elements representing a subset of the input channels of the filters of the coefficient set of the first convolutional layer; and the compression neural network may be configured such that the first compression layer is arranged to perform the first compression operation by convolving one or more subsets of the input activation values ​​of the first convolutional layer with the subset of the set of coefficients of the first convolutional layer represented by elements in the first one or more sub-matrices included in the one or more subsets of the input channels of the filters.

[0021] The second layer of the received neural network may be a second convolutional layer comprising a set of coefficients arranged in one or more filters, each of the one or more filters being arranged in one or more input channels, each input channel of each filter comprising a corresponding subset of the set of coefficients of the second convolutional layer, and determining the second matrix may comprise: for each input channel of each filter: determining whether the input channel of the filter comprises a non-zero coefficient; and in response to determining that the input channel of the filter comprises at least one non-zero coefficient, representing the input channel of the filter with elements in the second matrix representing non-zero values; or in response to determining that the input channel of the filter comprises only zero coefficients, representing the input channel of the filter with elements in the second matrix representing zero values.

[0022] The second convolutional layer of the received neural network may be arranged to perform the second operation by convolving a set of input activation values ​​of the second convolutional layer with the set of coefficients of the second convolutional layer; the second one or more sub-matrices may include multiple elements representing a subset of the input channels of the filters of the coefficient set of the second convolutional layer; and the compression neural network may be configured such that the second compression layer is arranged to perform the second compression operation by convolving one or more subsets of the input activation values ​​of the second convolutional layer with the subset of the set of coefficients of the second convolutional layer represented by elements in the one or more sub-matrices included in the one or more subsets of the input channels of the filters.

[0023] The first compression layer may be arranged to perform the first compression operation to form one or more subsets of output activation values ​​of the first compression layer, the one or more subsets of output activation values ​​of the first compression layer being one or more subsets of input activation values ​​of the second compression layer.

[0024] The method may include rearranging rows and / or columns of the array according to a hypergraph model.

[0025] The method may include: forming a hypergraph model based on the corresponding row and column positions of each of the multiple elements representing non-zero values ​​within the array; partitioning the hypergraph model; and rearranging the rows and / or columns of the array based on the partitioned hypergraph model.

[0026] The method may also include: selecting a third layer of the received neural network, the third layer being arranged to perform a third operation, the third layer being located after the second layer and adjacent to the second layer within the received neural network, the third layer having one or more input channels corresponding to the one or more output channels of the second layer; determining a third matrix representing a set of coefficients of the third layer, the third matrix including a plurality of elements representing non-zero values ​​and a plurality of elements representing zero values, one or more rows or columns of the third matrix representing the one or more output channels of the third layer, and one or more other rows or columns of the rows or columns of the first matrix representing the one or more input channels of the third layer; wherein forming the array further includes any of the following: transposing the third matrix and by merging the columns or rows of the transposed third matrix representing the one or more input channels of the third layer with the second matrix or the transposed third matrix representing the one or more output channels of the second layer or forming the array including the third matrix by aligning the columns or rows of the third matrix representing the one or more input channels of the third layer with the columns or rows of the second matrix or the transposed second matrix representing the one or more output channels of the second layer; wherein rearranging the rows and / or columns of the array further comprises: aggregating the plurality of elements representing non-zero values ​​included in the third matrix or the transposed third matrix into a third one or more sub-matrices, the third one or more sub-matrices having a ratio of the number of elements representing non-zero values ​​to the total number of elements of the third one or more sub-matrices that is greater than the ratio of the number of elements representing non-zero values ​​to the total number of elements of the third matrix; and outputting the compressed neural network, the compressed neural network further comprising a third compression layer, the third compression layer being arranged to perform a third compression operation based on the third one or more sub-matrices.

[0027] The method may further include: selecting a first scatter layer of the received neural network, the first scatter layer being arranged to perform a first scatter operation, the first scatter layer being after and adjacent to the second layer within the received neural network, the first scatter layer having one or more input channels corresponding to the one or more output channels of the second layer; determining a first scatter matrix representing a set of coefficients of the first scatter layer, the one or more columns or rows of the first scatter matrix representing the one or more input channels of the first scatter layer; determining a rearranged second matrix from the rearranged array by either: identifying the rearranged second matrix within the rearranged array; or identifying a rearranged transposed second matrix within the rearranged array and transposing the rearranged transposed second matrix; rearranging the columns or rows of the first scatter matrix so that an order of the one or more columns or rows of the first scatter matrix representing the one or more input channels of the first scatter layer corresponds to an order of the one or more rows or columns of the rearranged second matrix representing the one or more output channels of the second layer; and outputting the compressed neural network including the first scatter layer, the first scatter layer being arranged to perform the first scatter operation according to the rearranged first scatter matrix.

[0028] The first scatter matrix may include a plurality of elements representing non-zero values ​​and a plurality of elements representing zero values; and the method includes selecting the first scatter layer of the received neural network based on the number of elements representing non-zero values ​​exceeding a threshold value according to the total number of elements of the first scatter matrix.

[0029] The method may include: selecting a second scatter layer of the received neural network, the second scatter layer being arranged to perform a second scatter operation, the second scatter layer being before and adjacent to the first layer within the received neural network, the second scatter layer having one or more output channels corresponding to the one or more input channels of the first layer; determining a second scatter matrix representing a set of coefficients of the second scatter layer, the one or more rows or columns of the second scatter matrix representing the one or more output channels of the second scatter layer; determining the rearranged first matrix from the rearranged array by either: identifying the rearranged first matrix within the rearranged array; or identifying a rearranged transposed first matrix within the rearranged array and transposing the rearranged transposed first matrix; rearranging the rows or columns of the second scatter matrix such that an order of one or more rows or columns of the second scatter matrix representing the one or more output channels of the second scatter layer corresponds to an order of one or more columns or rows of the rearranged first matrix representing the one or more input channels of the first layer; and outputting the compressed neural network including the second scatter layer, the second scatter layer being arranged to perform the second scatter operation according to the rearranged second scatter matrix.

[0030] The second scatter matrix may include a plurality of elements representing non-zero values ​​and a plurality of elements representing zero values; and the method may include selecting the second scatter layer of the received neural network based on the number of elements representing non-zero values ​​exceeding a threshold value according to the total number of elements of the second scatter matrix.

[0031] The method may further include storing the compressed neural network for subsequent implementation.

[0032] The method may also include outputting a computer-readable description of the compressed neural network, which, when implemented at a system for implementing a neural network, causes the compressed neural network to be executed.

[0033] The method may also include configuring hardware logic to implement the compressed neural network.

[0034] Hardware logic can include neural network accelerators.

[0035] The method may include using the compressed neural network to perform image processing.

[0036] The one or more rows of the first matrix may represent the one or more output channels of the first layer, the one or more columns of the first matrix may represent the one or more input channels of the first layer, the one or more rows of the second matrix may represent the one or more output channels of the second layer, and the one or more columns of the second matrix may represent the one or more input channels of the second layer; or the one or more columns of the first matrix may represent the one or more output channels of the first layer, the one or more rows of the first matrix may represent the one or more input channels of the first layer, the one or more columns of the second matrix may represent the one or more output channels of the second layer, and the one or more rows of the second matrix may represent the one or more input channels of the second layer; Or the one or more rows of the first matrix may represent the one or more output channels of the first layer, the one or more columns of the first matrix may represent the one or more input channels of the first layer, the one or more columns of the second matrix may represent the one or more output channels of the second layer, and the one or more rows of the second matrix may represent the one or more input channels of the second layer; or the one or more columns of the first matrix may represent the one or more output channels of the first layer, the one or more rows of the first matrix may represent the one or more input channels of the first layer, the one or more rows of the second matrix may represent the one or more output channels of the second layer, and the one or more columns of the second matrix may represent the one or more input channels of the second layer.

[0037] According to a second aspect of the present invention, there is provided a processing system for compressing a neural network, the processing system comprising at least one processor, the at least one processor being configured to: receive a neural network; select two or more adjacent layers of the received neural network, each of the two or more adjacent layers having one or more input channels and one or more output channels, the one or more output channels of a first layer of the two or more adjacent layers corresponding to the one or more input channels of a second subsequent layer of the two or more adjacent layers, the first layer being arranged to perform a first operation and the second layer being arranged to perform a second operation; determine a first matrix representing a set of coefficients of the first layer of the received neural network, the first matrix comprising a plurality of elements representing non-zero values ​​and a plurality of elements representing zero values, one or more rows or columns of the first matrix representing the one or more output channels of the first layer, and one or more other rows or columns of the rows or columns of the first matrix representing the one or more input channels of the first layer; determine a second matrix representing a set of coefficients of the second layer of the received neural network, the second matrix comprising a plurality of elements representing non-zero values ​​and a plurality of elements representing zero values, one or more rows or columns of the second matrix representing the one or more output channels of the second layer, and one or more other rows or columns of the rows or columns of the second matrix representing the one or more input channels of the first layer The one or more input channels of the second layer; forming an array by one of the following ways: transposing the first matrix and forming the array including the transposed first matrix and the second matrix by aligning the columns or rows of the transposed first matrix representing the one or more output channels of the first layer with the columns or rows of the second matrix representing the one or more input channels of the second layer; or transposing the second matrix and forming the array including the transposed second matrix and the first matrix by aligning the rows or columns of the transposed second matrix representing the one or more input channels of the second layer with the rows or columns of the first matrix representing the one or more output channels of the first layer. an array; or forming the array including the first matrix and the second matrix by aligning the rows or columns of the first matrix representing the one or more output channels of the first layer with the rows or columns of the second matrix representing the one or more input channels of the second layer; rearranging the rows and / or columns of the array so as to: aggregate the multiple elements representing non-zero values ​​included in the first matrix or the transposed first matrix into first one or more sub-matrices, the first one or more sub-matrices having a ratio of the number of elements representing non-zero values ​​to the total number of elements of the first one or more sub-matrices that is greater than the ratio of the number of elements representing non-zero values ​​to the total number of elements of the first matrix;and aggregating the plurality of elements representing non-zero values ​​included in the second matrix or the transposed second matrix into a second one or more sub-matrices, the second one or more sub-matrices having a ratio of the number of elements representing non-zero values ​​to the total number of elements of the second one or more sub-matrices that is greater than the ratio of the number of elements representing non-zero values ​​to the total number of elements of the second matrix; and outputting a compressed neural network, the compressed neural network comprising a first compression layer arranged to perform a first compression operation based on the first one or more sub-matrices, and a second subsequent compression layer arranged to perform a second compression operation based on the second one or more sub-matrices.;

[0038] The processing system may further include a memory, and the at least one processor may be further configured to store the compressed neural network in the memory for subsequent implementation.

[0039] The at least one processor may also be configured to configure hardware logic to implement the compressed neural network.

[0040] Hardware logic can include neural network accelerators.

[0041] According to a third aspect of the present invention, there is provided a processing system for compressing a neural network, the processing system comprising at least one processor, the at least one processor being configured to: receive a neural network; determine a matrix representing a set of coefficients of a layer of the received neural network, the layer being arranged to perform an operation, the matrix comprising a plurality of elements representing non-zero values ​​and a plurality of elements representing zero values; rearrange the rows and / or columns of the matrix so as to aggregate the plurality of elements representing non-zero values ​​of the matrix into one or more sub-matrices, the one or more sub-matrices having a ratio of the number of elements representing non-zero values ​​to the total number of elements of the one or more sub-matrices that is greater than the ratio of the number of elements representing non-zero values ​​to the total number of elements of the matrix; and output a compressed neural network comprising a compression layer arranged to perform a compression operation based on the one or more sub-matrices.

[0042] According to a fourth aspect of the present invention, there is provided a computer-implemented method for compressing a neural network, the method comprising: receiving a neural network; determining a matrix representing a set of coefficients of a layer of the received neural network, the layer being arranged to perform an operation, the matrix comprising a plurality of elements representing non-zero values ​​and a plurality of elements representing zero values; rearranging the rows and / or columns of the matrix so as to cluster the plurality of elements representing non-zero values ​​of the matrix into one or more sub-matrices, the one or more sub-matrices having a ratio of the number of elements representing non-zero values ​​to the total number of elements of the one or more sub-matrices that is greater than the ratio of the number of elements representing non-zero values ​​to the total number of elements of the matrix; and outputting a compressed neural network, the compressed neural network comprising a compression layer arranged to perform a compression operation based on the one or more sub-matrices.

[0043] The processing system may be embodied in hardware on an integrated circuit. A method of manufacturing a processing system at an integrated circuit manufacturing system may be provided. An integrated circuit definition data set may be provided that, when processed in the integrated circuit manufacturing system, configures the system to manufacture the processing system. A non-transitory computer-readable storage medium may be provided on which a computer-readable description of the processing system is stored that, when processed in the integrated circuit manufacturing system, causes the integrated circuit manufacturing system to manufacture an integrated circuit embodying the processing system.

[0044] An integrated circuit manufacturing system may be provided, the integrated circuit manufacturing system comprising: a non-transitory computer-readable storage medium having stored thereon a computer-readable description of a processing system; a layout processing system configured to process the computer-readable description to generate a circuit layout description of an integrated circuit embodying the processing system; and an integrated circuit production system configured to manufacture the processing system based on the circuit layout description.

[0045] A computer program code for performing any of the methods described herein may be provided. A non-transitory computer readable storage medium may be provided on which computer readable instructions are stored which, when executed at a computer system, cause the computer system to perform any of the methods described herein.

[0046] As will be apparent to those skilled in the art, the above-described features may be combined as appropriate and may be combined with any of the aspects of the examples described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Examples will now be described in detail with reference to the accompanying drawings, in which:

[0048] Figure 1 An example neural network is shown.

[0049] Figure 2a shows the data structures used in an example convolutional layer.

[0050] Figure 2b An example convolutional layer is shown.

[0051] Figure 3 An example system for implementing a neural network is shown.

[0052] Figure 4 A processing system for compressing a neural network is shown.

[0053] Figure 5 A first method of compressing a neural network is shown.

[0054] Figure 6 An example matrix is ​​shown, the matrix including a plurality of elements representing non-zero values ​​and a plurality of elements representing zero values.

[0055] Figure 7 An example rearranged matrix including a plurality of sub-matrices is shown.

[0056] Figure 8 An example hypergraph model is shown.

[0057] Fig. 9 It is shown how compression operations can be performed using multiple sub-matrices.

[0058] Fig.10 It is shown how compression operations can be performed using multiple sub-matrices.

[0059] Fig.11 A second method of compressing a neural network is shown.

[0060] Fig.12 An example array is shown.

[0061] Fig.13 An example rearranged array is shown.

[0062] Fig.14 In the example shown, output activation values ​​formed by a first layer from a plurality of sub-matrices serving as input activation values ​​for a second subsequent layer are shown.

[0063] Fig.15 In the example a rearranged matrix interspersed between rearranged matrices comprising a plurality of sub-matrices is shown.

[0064] Fig.16a shows a graph with dimension C out ×C in ×H w ×W w The coefficient set C out×C in flat.

[0065] Fig.16b It shows how a compressed convolution operation can be performed using a subset of the coefficient set of a convolutional layer.

[0066] Fig.17 A computer system is shown in which the processing system is implemented; and

[0067] Fig.18 An integrated circuit manufacturing system for producing an integrated circuit embodying a processing system is shown.

[0068] The accompanying drawings show various examples. It will be appreciated by the skilled person that the element boundaries (e.g., boxes, groups of boxes, or other shapes) shown in the accompanying drawings represent an example of boundaries. In some examples, it may be the case that an element can be designed as multiple elements, or multiple elements can be designed as one element. Where appropriate, common reference numerals are used throughout the accompanying drawings to indicate similar features. DETAILED DESCRIPTION

[0069] The following description is presented by way of example to enable one skilled in the art to make and use the invention.The present invention is not limited to the embodiments described herein, and various modifications to the disclosed embodiments will be apparent to those skilled in the art.

[0070] Embodiments will now be described by way of example only.

[0071] Neural networks can be used to perform image processing. Examples of image processing techniques that can be performed by neural networks include: image super-resolution processing, semantic image segmentation processing, and object detection. For example, performing image super-resolution processing involves the neural network processing a lower resolution image input to the neural network in order to output a higher resolution image. It should be understood that the principles described herein are not limited to use in compressing neural networks for performing image processing. For example, the principles described herein can be used to compress neural networks to perform speech recognition / speech to text applications or any other suitable type of application. A skilled person will understand how to configure a neural network to perform any of the processing techniques mentioned in this paragraph, so for the sake of brevity, these techniques will not be discussed in further detail.

[0072] A neural network can be defined by a software model. For example, the software model can define a series of layers of a neural network (e.g., the number of layers, the order of the layers, and the connectivity between the layers), and define each layer in the series according to the operations it is configured to perform and the set of coefficients it will use. In general, a neural network can be implemented in hardware, software, or any combination thereof.

[0073] Figure 3An example system 300 for implementing a neural network in hardware is shown. The system 300 includes hardware components (e.g., hardware logic, such as processing elements implemented in hardware) and software components (e.g., firmware, and processes and tasks for execution at the hardware logic). The system 300 includes hardware logic for implementing a neural network. The hardware logic may include a neural network accelerator (NNA) 302. In other words, the system 300 may include a hardware neural network accelerator (NNA). That is, the system 300 may include a neural network accelerator (NNA) implemented in hardware. The neural network accelerator (NNA) 302 may be configured to perform operations to be performed by a layer of a neural network. The neural network accelerator (NNA) 302 includes a plurality of configurable resources that enable implementation of different types of neural network layers, such as fully connected layers and convolutional layers. In other words, implementing a neural network may include configuring the hardware logic included by the system 300 to process data input to the system according to the layers of the neural network. A neural network accelerator (NNA) is a hardware accelerator designed to accelerate the processing of a neural network. As known to those skilled in the art, a hardware accelerator is hardware designed to perform a specific set of one or more functions more efficiently than a general-purpose processing unit, such as a central processing unit (CPU). Therefore, compared to a general-purpose CPU that can be configured to perform any number of functions, an accelerator can only perform a limited set of one or more functions. An NNA includes one or more hardware accelerators designed to accelerate one or more neural network operations. Therefore, a graphics processing unit (GPU) having one or more hardware accelerators designed to accelerate one or more neural network operations can be understood as an NNA.

[0074] In more detail, system 300 includes an input 301 for receiving input data. The input data received at input 301 includes input activation data. For example, when the implemented neural network is configured to perform image processing, the input activation data may include image data representing one or more images. For example, for RGB images, the image data may be in the format C in ×H a ×W a , where H a and W a is the image across the three input color channels C in The input data received at input 301 also includes the coefficient sets for each layer of the neural network. The coefficient sets may also be referred to as weights. As described herein, the coefficient sets for the fully connected layer may have dimensions C out ×C in or C in ×C out , and the coefficient set of the convolutional layer can have dimension C out ×Cin ×H w ×W w .

[0075] Input data received at input 301 may be written to a memory 304 included in system 300. Memory 304 may be accessible by neural network accelerator (NNA) 302. Memory 304 may be a system memory accessible to neural network accelerator (NNA) 302 via a data bus. Neural network accelerator (NNA) 302 may be implemented on a chip (e.g., a semiconductor die and / or an integrated circuit package), and memory 304 may not be physically located on the same chip (e.g., a semiconductor die and / or an integrated circuit package) as neural network accelerator (NNA) 302. Thus, memory 304 may be referred to as "off-chip memory" and / or "external memory". Memory 304 may be coupled to input buffer 306 at neural network accelerator (NNA) 302 to provide input activation data to neural network accelerator (NNA) 302. Memory 304 may be coupled to coefficient buffer 330 at neural network accelerator (NNA) 302 to provide coefficient sets to neural network accelerator (NNA) 302.

[0076] Input buffer 306 may be arranged to store input activation data required by neural network accelerator (NNA) 302. Coefficient buffer 330 may be arranged to store coefficient sets required by neural network accelerator (NNA) 302. Input buffer 306 may contain some or all of the input activation data associated with one or more operations performed at neural network accelerator (NNA) 302 on a given cycle, as described herein. Coefficient buffer 330 may contain some or all of the coefficient sets associated with one or more operations processed at neural network accelerator (NNA) 302 on a given cycle, as described herein. Figure 3 The various buffers of the neural network accelerator (NNA) 302 shown may be implemented in any suitable manner, for example, as any number of data storage repositories local to the neural network accelerator (NNA) 302 (e.g., on the same semiconductor die and / or disposed within the same integrated circuit package) or accessible to the neural network accelerator (NNA) 302 via a data bus or other interconnect.

[0077] exist Figure 3In the embodiment, the system 300 also includes a memory 328 for storing at least some of the coefficient sets and / or input activation data required for the neural network implemented at the neural network accelerator (NNA) 302. The neural network accelerator (NNA) 302 can be implemented on a chip (e.g., a semiconductor die and / or an integrated circuit package), and the memory 328 can be physically located on the same chip (e.g., a semiconductor die and / or an integrated circuit package) as the neural network accelerator (NNA) 302. As such, the memory 328 can be referred to as "on-chip memory" and / or "local memory". Data stored in the on-chip memory 328 can be accessed by the neural network accelerator (NNA) 302 without consuming memory bandwidth to the off-chip memory 304. That is, the on-chip memory 328 may not have enough storage space to simultaneously store all the input data required to implement the neural network. Therefore, a subset of the input data can be periodically written from the off-chip memory 304 to the on-chip memory 328. The coefficient buffer 330 may be configured to receive coefficient sets from the on-chip memory 328 to reduce the bandwidth between the off-chip memory 304 and the coefficient buffer 330. The input buffer 306 may be configured to receive input activation data from the on-chip memory 328 to reduce the bandwidth between the off-chip memory 304 and the input buffer 306.

[0078] exist Figure 3 In the figure, the neural network accelerator (NNA) 302 includes an input buffer 306, a coefficient buffer 330, a plurality of multiplication engines 308, a plurality of accumulators 310, an accumulation buffer 312, and an output buffer 316. Each multiplication engine 308, together with its corresponding accumulator 310 and its share of the resources of the accumulation buffer 312, represents a processing element 314 of the neural network accelerator (NNA) 302. Figure 3 Three processing elements 314 are shown in FIG. 3 , but in general any number of processing elements may be present.

[0079] Each processing element 314 may receive a set of input activation values ​​from the input buffer 306 and a set of coefficients from the coefficient buffer 330. By operating on the set of input activation values ​​and the set of coefficients, the processing element may be used to perform operations of a layer of a neural network. The processing element 314 of the neural network accelerator (NNA) 302 may be an independent processing subsystem of the neural network accelerator (NNA) 302 that may operate in parallel. Each processing element 314 includes a multiplication engine 308 configured to perform multiplication between a set of coefficients and an input activation value. In an example, the multiplication engine 308 may be configured to perform a fully connected operation (e.g., when a fully connected layer is implemented) or a convolution operation (e.g., when a convolution layer is implemented) between the set of coefficients and the input activation value. The multiplication engine 308 may perform these operations by means of each multiplication engine 308 including a plurality of multipliers, each multiplier being configured to multiply a coefficient and a corresponding input activation value to produce a multiplication output value. The multipliers may be followed, for example, by a tree of adders arranged to compute the sum of the multiplication outputs in a manner dictated by the operations performed by the layer. In some examples, these multiply-accumulate calculations may be pipelined.

[0080] As described herein, neural networks are often described as including many layers. A large number of multiplication-accumulation calculations must often be performed at the neural network accelerator (NNA) 302 in order to perform the operations performed by each layer of the neural network. This is because the input activation data and coefficient sets of each layer are often very large. Since more than one pass of the multiplication engine 308 may be used to produce the complete output of the operation (e.g., because the multiplication engine 308 may only receive and process a portion of the coefficient set and input activation values), the neural network accelerator (NNA) may include multiple accumulators 310. Each accumulator 310 receives the output of the multiplication engine 308 and adds the output to the previous output of the multiplication engine 308 associated with the same operation. Depending on the implementation of the neural network accelerator (NNA) 302, the multiplication engine 308 may not process the same operation in consecutive cycles, so an accumulation buffer 312 may be provided to store the partially accumulated output of a given operation. Appropriate partial results may be provided to the accumulator 310 by the accumulation buffer 312 in each cycle.

[0081] The accumulation buffer 312 may be coupled to the output buffer 316 to allow the output buffer 316 to receive output activation data of intermediate layers of the neural network operated at the neural network accelerator (NNA) 302, and output data of a final layer (e.g., a layer that performs the final operation of the network implemented at the neural network accelerator (NNA) 302). The output buffer 316 may be coupled to the on-chip memory 328 and / or the off-chip memory 304, and the output data stored in the output buffer 316 (e.g., output activation data input as input activation data to a subsequent layer, or final output data output by the neural network) may be written to the on-chip memory and / or the off-chip memory.

[0082] In general, the neural network accelerator (NNA) 302 may also include any other suitable processing logic. For example, in some examples, the neural network accelerator (NNA) 302 may include reduction logic (e.g., for implementing a maximum pooling or average pooling operation), activation logic (e.g., for applying an activation function, such as a sigmoid function or a step function), or any other suitable processing logic. For simplicity, Figure 3 Such a unit is not shown in FIG.

[0083] As described herein, the coefficient sets used by the layers of a typical neural network generally include a large number of coefficients. A neural network accelerator, such as neural network accelerator 302, can implement the layers of a neural network by reading the input activation values ​​and coefficient sets of the layers at runtime, such as directly reading from off-chip memory 304 or reading via on-chip memory 328, as described herein with reference to Figure 3 As described. A large amount of memory bandwidth may be required to read a large number of coefficient sets from memory. The memory bandwidth required to read in the coefficient set may be referred to as "weight bandwidth". The memory bandwidth required to read in the activation value set may be referred to as "activation bandwidth". In addition, the time taken to read a large coefficient set from memory may also affect the time taken for the neural network to perform the operations performed by the layer. This may affect the latency of the neural network. The large coefficient set may also place a large amount of computational demands on the processing elements of the neural network accelerator that implements the neural network, such as the processing element 314 of the neural network accelerator (NNA) 302 described herein. That is, layers with large coefficient sets may require those processing elements to perform a large number of multiplication and accumulation operations.

[0084] Furthermore, the inventors have observed that, typically, a large portion of the coefficient sets of a layer of a typical neural network have coefficients equal to zero (e.g., "zero coefficients" or "0"). This is particularly true in trained neural networks, as the training process can typically drive most of the coefficients to zero. Performing element-wise multiplication between an input activation value and a zero coefficient will inevitably produce a zero output value, regardless of the value of the input activation value.

[0085] Therefore, it is undesirable to incur the weight bandwidth, latency, and computational requirements penalties caused by the use of large coefficient sets by the layers of the neural network for most element-wise multiplications performed using coefficients of the coefficient set to inevitably produce zero output values. It is also undesirable to incur the activation bandwidth "cost" of reading activation values ​​from memory for element-wise multiplications performed using only the activation values ​​and zero coefficients to inevitably produce zero output values.

[0086] Methods and processing systems are described herein for compressing neural networks in order to solve one or more of the problems described in the preceding paragraphs.

[0087] Figure 4 A processing system 400 for compressing a neural network according to the principles described herein is shown. The processing system 400 includes at least one processor 404 and a memory 406.

[0088] At least one processor 404 may be implemented in hardware, software, or any combination thereof. At least one processor 404 may be a microprocessor, a controller, or any other suitable type of processor for processing computer-executable instructions. At least one processor 404 may be configured to perform a method for compressing a neural network according to the principles described herein (e.g., as will be described herein with reference to Figure 5 and Fig.11 one of the methods described).

[0089] The memory 406 is accessible by at least one processor 404. The memory 406 may be a system memory accessible to the at least one processor 404 via a data bus. The at least one processor 404 may be implemented on a chip (e.g., a semiconductor die and / or an integrated circuit package), and the memory 406 may not be physically located on the same chip (e.g., a semiconductor die and / or an integrated circuit package) as the at least one processor 404. Thus, the memory 406 may be referred to as “off-chip memory” and / or “external memory”. Alternatively, the at least one processor 404 may be implemented on a chip (e.g., a semiconductor die and / or an integrated circuit package), and the memory 406 may be physically located on the same chip (e.g., a semiconductor die and / or an integrated circuit package) as the at least one processor 404. Thus, the memory 406 may be referred to as “on-chip memory” and / or “local memory”. Alternatively, Figure 4 The memory 406 shown in may again represent any combination of "on-chip" and "off-chip" memory, such as one or more "on-chip" memories and one or more "off-chip" memories arranged in a memory hierarchy.

[0090] The memory 406 may store computer executable instructions for executing a method of compressing a neural network according to the principles described herein (e.g., as will be referred to herein). Figure 5 and Fig.11 Memory 406 may store a neural network received at processing system 400 for compression (e.g., a software model defining the neural network, as described herein). Memory 406 may store compressed neural network outputs as at least one processor executes computer-executable instructions stored by memory 406. Memory 406 may store the compressed neural network for subsequent implementation (e.g., at a system for implementing a neural network).

[0091] Processing system 400 may be used to configure system 300 for implementing a neural network. Figure 4 The system 300 shown in FIG. 3 may have the same Figure 3 That is, system 300 may include hardware logic for implementing a neural network, the hardware logic including a neural network accelerator (NNA) 302. Because at least one processor executes a method for compressing a neural network according to the principles described herein (e.g., as will be referred to herein Figure 5 and 11 ), so at least one processor 404 can configure hardware logic to implement compressed neural network output.

[0092] Figure 5 A first method of compressing a neural network is shown. Figure 5 The method of compressing a neural network shown in is a computer-implemented method. Figure 4 The processing system 400 for compressing a neural network shown in FIG. 4 may be configured to perform Figure 5 The method for compressing neural networks shown in .

[0093] In step S502, a neural network is received. The received neural network may be defined by a software model. For example, the software model may define a series of layers of the received neural network (e.g., the number of layers, the order of the layers, and the connectivity between the layers), and define each layer in the series according to the operations it is configured to perform and the set of coefficients it will use. The received neural network may be a trained neural network. That is, as the technician will understand, the received neural network may have been previously iteratively trained by the following steps: processing training data in a forward pass; evaluating the accuracy of the output of the forward pass; and updating the coefficient set of the layer in a backward pass. As described herein, the training process may typically drive most of the coefficients of the coefficient set used by the layers of the neural network to zero. The neural network (e.g., the software model defining the neural network) may be trained in a forward pass. Figure 4The data is received at the processing system 400 and stored in the memory 406 .

[0094] A layer of the received neural network may be selected for compression. In step S504, a matrix representing a set of coefficients of the selected layer of the received neural network is determined. The matrix includes a plurality of elements representing non-zero values ​​and a plurality of elements representing zero values. The matrix representing the set of coefficients of the selected layer of the received neural network may not have a subgraph partitioning. Figure 4 The at least one processor 404 shown in FIG. 4 may be configured to perform step S504 .

[0095] Figure 6 An example matrix 600 is shown, the matrix including a plurality of elements representing non-zero values ​​and a plurality of elements representing zero values. Figure 6 , elements of the matrix 600 representing non-zero values ​​are marked with "X", while elements of the matrix 600 representing zero values ​​are not marked. For example, elements located in row 1, column 7 and row 1, column 10 of the matrix 600 represent non-zero values, while other elements located in row 1 of the matrix 600 represent zero values. The example matrix 600 does not have subgraph partitioning.

[0096] In a first example, the selected layer of the received neural network is a fully connected layer arranged to perform a fully connected operation, or any other type of layer arranged to perform a matrix multiplication. In a first example, the determined matrix 600 may include a set of coefficients of the layer. The plurality of elements representing non-zero values ​​may be a plurality of non-zero coefficients. A non-zero coefficient is any coefficient having a positive or negative value that is not equal to zero. The plurality of elements representing zero values ​​may be a plurality of zero coefficients. A zero coefficient is a coefficient having a value equal to zero. Reference Figure 6 , in a first example, the matrix 600 includes a plurality of non-zero coefficients marked with an "X" and a plurality of zero coefficients that are not marked.

[0097] In a first example, the selected layer of the received neural network can be arranged to perform a fully connected operation by performing a matrix multiplication using a matrix 600 including a set of coefficients of the layer and an input matrix including a set of input activation values ​​of the layer. For example, as described herein, in a fully connected layer, a matrix multiplication WX=Y can be performed, where: W is a matrix including a set of coefficients and having a dimension C out ×C in (Right now, Figure 6 14×14) coefficient matrix (e.g., matrix 600); X is an input matrix including input activation value sets and having dimensions M×N, where C in = M; and Y is a set of output values ​​and has dimension C out ×N output matrix. That is, the number of rows in the coefficient matrix can represent the number of output channels of the fully connected layer (“C out”), and the number of columns of the coefficient matrix can represent the number of input channels of the fully connected layer (“C in ”). Alternatively, in a fully connected layer, a matrix multiplication XW=Y can be performed, where: X is an input matrix with dimensions M×N and including a set of input activation values; W is a matrix with dimensions C and including a set of coefficients. in ×C out (Right now, Figure 6 14×14) coefficient matrix (e.g., matrix 600), where C in = N; and Y is a set of output values ​​and has dimensions M×C out That is, the number of rows in the coefficient matrix can represent the number of input channels of the fully connected layer (“C in ”), and the number of columns of the coefficient matrix can represent the number of output channels of the fully connected layer (“C out ”).

[0098] In a second example, the selected layer of the received neural network is a convolutional layer. Figure 2a and Figure 2b As described, a convolutional layer includes one or more filters (e.g., Figure 2b The coefficient sets in the filters A, B, C, D) shown in FIG. 1 , each of the one or more filters is arranged on one or more input channels (e.g., Figure 2b In the input channels 1, 2, 3) shown in , each input channel of each filter includes a corresponding subset of the coefficient set of the convolutional layer. That is, reference Figure 2b , the input channel 1 of filter A includes a subset of the coefficient set 204 of the convolution layer 200, the input channel 2 of filter A includes a subset of the coefficient set 204 of the convolution layer 200, the input channel 3 of filter A includes a subset of the coefficient set 204 of the convolution layer 200, the input channel 1 of filter B includes a subset of the coefficient set 204 of the convolution layer 200, the input channel 2 of filter B includes a subset of the coefficient set 204 of the convolution layer 200, the input channel 3 of filter B includes a subset of the coefficient set 204 of the convolution layer 200, and so on, until the input channel 3 of filter D includes a subset of the coefficient set 204 of the convolution layer 200. Each input channel of each filter has dimension H w ×W w , for example, H, which includes the coefficient sets of the convolutional layers w ×W w coefficients.

[0099] In a second example, the selected convolutional layer of the received neural network can be arranged to perform a convolution operation by convolving a set of input activation values ​​of the convolutional layer with a set of coefficients of the convolutional layer, as described with reference to Figure 2a and Figure 2bThe description of this article understands that. With space dimension H a ×W a The Nth input channel of the input activation data 202 may be associated with the coefficient set 204 having a spatial dimension H w ×W w Convolution of each filter with the Nth input channel.

[0100] In the second example, in step S504, it is determined that the matrix includes for each input channel of each filter (eg, reference Figure 2b , for input channel 1 of filter A, for input channel 2 of filter A, and so on, to input channel 3 of filter D) to determine whether the input channel of the filter includes a non-zero coefficient. In response to determining that the input channel of the filter includes at least one non-zero coefficient, the input channel of the filter can be represented by an element representing a non-zero value in a matrix. In response to determining that the input channel of the filter includes only zero coefficients (i.e., excluding any non-zero coefficients in a subset of the coefficient set of the convolutional layer), the input channel of the filter can be represented by an element representing a zero value in a matrix. For completeness, binary "1" can be used to represent non-zero values ​​in a matrix, and binary "0" can be used to represent zero values ​​in a matrix, but this is not necessary. For example, binary "0" can be used to represent non-zero values ​​in a matrix, and binary "1" can be used to represent zero values ​​in a matrix. It should also be understood that any other arbitrarily selected value can be used to represent non-zero values ​​or zero values ​​in a matrix. As an illustrative example, one or more bits indicating an even number can be used to represent non-zero values ​​in a matrix, and one or more bits indicating an odd number can be used to represent zero values ​​in a matrix.

[0101] In a second example, each row of the matrix can represent a filter in one or more filters of a convolutional layer. In other words, each row of the matrix can represent an output channel in one or more output channels of a convolutional layer. That is, each row of the matrix can represent a corresponding output channel (e.g., a filter) of a convolutional layer. Each column of the matrix can represent an input channel in one or more input channels of a convolutional layer. That is, each column of the matrix can represent a corresponding input channel of a convolutional layer.

[0102] refer to Figure 6 , in the second example, the matrix 600 includes a plurality of elements marked with an "X" representing non-zero values ​​and a plurality of unmarked elements representing zero values. In the second example, it has been determined for a convolutional layer having a set of coefficients arranged in 14 filters (e.g., output channels) (e.g., represented by 14 rows of the matrix 600) Figure 6600 having 14 rows and 14 columns, each of the 14 filters is arranged in 14 input channels (e.g., represented by the 14 columns of the matrix 600). For example, row 1 of the matrix 600 may represent the first filter of the convolutional layer, and each column in row 1 represents one of the 14 input channels of the first filter. Therefore, the elements located in row 1, column 7 and row 1, column 10 of the matrix 600 represent input channels of the first filter including at least one non-zero coefficient, while the other elements located in row 1 of the matrix 600 represent input channels of the first filter including only zero coefficients.

[0103] In other words, in the second example, the matrix 600 may represent a matrix having dimension C. out ×C in ×H w ×W w The coefficient set C of the convolutional layer out ×C in This is in Fig.16a In order to have a dimension C out ×C in ×H w ×W w The four-dimensional coefficient set of the convolutional layer can be visualized, for example, by replacing w ×W w The coefficients of each two-dimensional input channel of each filter of H are represented as corresponding one-dimensional coefficient sequences to make H w ×W w The dimension is "flattened". These one-dimensional coefficient sequences are Fig.16a , which spans a three-dimensional cuboid 1600 representing a set of coefficients of a convolutional layer and is labeled H w W w One dimension. C out ×C in Plane Fig.16a 1600. w W w can be called in C out ×C in In the plane, there is (C in , C out ) coordinates. As described herein, matrix 600 is Fig.16a The three-dimensional cuboid 1600 shown in C out ×C in As described herein, in the second example, each element of matrix 600 represents an input channel of a filter of a convolutional layer. Fig.16a The elements marked with "X" in the matrix 600 representing non-zero values ​​represent the out ×Cin The plane has the (C in , C out ) coordinates of the one-dimensional coefficient sequence H w W w At least one coefficient in is a non-zero coefficient. Fig.16a The unlabeled zero-valued elements in the matrix 600 in C out ×C in The plane has the (C in , C out ) coordinates of the one-dimensional coefficient sequence H w W w There are no non-zero coefficients in .

[0104] In the second example, it is described herein that in step S504, a matrix may be determined such that each row of the matrix represents a corresponding output channel (e.g., a filter) of the convolutional layer, and each column of the matrix represents a corresponding input channel of the convolutional layer. It should be understood that, alternatively, in the second example, it is described herein that in step S504, a matrix may be determined such that each row of the matrix represents a corresponding input channel of the convolutional layer, and each column of the matrix represents a corresponding output channel (e.g., a filter) of the convolutional layer. After defining the matrix to be filled in this manner, the elements of the matrix may be filled accordingly by evaluating whether the input channel of the filter represented by each element includes a non-zero coefficient.

[0105] In step S506, the rows and / or columns of the matrix determined in step S504 are rearranged (e.g., reordered) so that multiple elements of the matrix representing non-zero values ​​are clustered into one or more sub-matrices. The one or more sub-matrices have a ratio of the number of elements representing non-zero values ​​to the total number of elements of the one or more sub-matrices that is greater than the ratio of the number of elements representing non-zero values ​​to the total number of elements of the matrix. In other words, the "non-zero density" of the one or more sub-matrices as a whole is greater than the "non-zero density" of the matrix. Figure 4 The at least one processor 404 shown in FIG. 4 may be configured to perform step S506 .

[0106] In some examples, each of the one or more sub-matrices can have a ratio of the number of elements representing non-zero values ​​to the total number of elements of the sub-matrix that is greater than the ratio of the number of elements representing non-zero values ​​to the total number of elements of the matrix. In other words, in these examples, the "non-zero density" of each of the one or more sub-matrices is greater than the "non-zero density" of the matrix, but this need not be the case.

[0107] In the first example, the one or more sub-matrices include a subset of the coefficient set of the layer selected in step S504. In the second example, the one or more sub-matrices include elements of a subset of the input channels of the filters of the coefficient set of the convolutional layer selected in step S504. Step S506 is performed in the same manner in the first example and the second example.

[0108] Step S506 can refer to Figure 7 It is understood that the figure shows an example rearranged matrix 710 including a plurality of sub-matrices 702-1, 702-2, 702-3, 703-4. The rearranged matrix 710 may be rearranged (e.g., reordered) Figure 6 The rows and columns of the matrix 600 shown in FIG. Figure 7 The rearranged matrix 710 shown in FIG. That is, Figure 7 Shown in different order Figure 6 14 rows and 14 columns of the matrix 600 shown in . For example, Figure 6 Column 2 of the matrix 600 shown in FIG. 6 having elements in rows 2, 7, and 13 representing non-zero values ​​has been shifted "to the left" and is now Figure 7 . In another example, Figure 6 Row 7 of the matrix 600 shown in FIG. 6 having elements in columns 2, 4, and 7 representing non-zero values ​​has been shifted "upwards" and is now Figure 7 The first row of the rearranged matrix 710 is shown in . By rearranging the rows and columns as described herein, multiple elements of the matrix 600 representing non-zero values ​​are grouped into sub-matrices 702-1, 702-2, 702-3, 703-4.

[0109] Matrix 600 includes 45 elements representing non-zero values, and a total of 196 (i.e., 14×14) elements. Therefore, the "non-zero density" of matrix 600 is 0.23 (i.e., 45 / 196). Submatrices 702-1, 702-2, 702-3, 703-4 also include 45 elements representing non-zero values, but include a total of 103 (i.e., (3×4)+(4×5)+(3×5)+(4×14)) elements. Therefore, the "non-zero density" of multiple submatrices 702-1, 702-2, 702-3, 703-4 is 0.44 (i.e., 45 / 103). Therefore, the "non-zero density" of multiple submatrices 702-1, 702-2, 702-3, 703-4 as a whole is greater than the "non-zero density" of matrix 600.

[0110] Submatrix 702-1 includes 8 elements representing non-zero values, and a total of 12 (i.e., 3×4) elements. Therefore, the "non-zero density" of submatrix 702-1 is 0.67 (i.e., 8 / 12). Submatrix 702-2 includes 9 elements representing non-zero values, and a total of 20 (i.e., 4×5) elements. Therefore, the "non-zero density" of submatrix 702-2 is 0.45 (i.e., 9 / 20). Submatrix 702-3 includes 8 elements representing non-zero values, and a total of 15 (i.e., 3×5) elements. Therefore, the "non-zero density" of submatrix 702-3 is 0.53 (i.e., 8 / 15). Submatrix 702-4 includes 20 elements representing non-zero values, and a total of 56 (i.e., 4×14) elements. Therefore, the "non-zero density" of submatrix 702-4 is 0.36 (i.e., 20 / 56). Therefore, the “non-zero density” of each of the plurality of sub-matrices 702 - 1 , 702 - 2 , 702 - 3 , and 703 - 4 is greater than the “non-zero density” of the matrix 600 .

[0111] Figure 7 The rearranged matrix 710 shown in FIG. 7 is in a "single-border block-diagonal matrix form". This means that the rearranged matrix includes one or more block arrays arranged on a diagonal line and a border array arranged along one side of the rearranged matrix. The one or more block arrays and the border array are non-overlapping. Figure 7 In FIG. 1 , the block arrays are labeled 702 - 1 , 702 - 2 , and 702 - 3 , and are arranged on a diagonal line from top left to bottom right. Figure 7, the border array is labeled 702-4 and is arranged along the bottom of the rearranged matrix 710. In other examples, one or more block arrays may be arranged on different diagonals (e.g., from top right to bottom left), or may not be arranged on a diagonal at all. That is, for example, the columns of the rearranged matrix 710 may be arranged differently so that the "horizontal" positions of two or more block arrays are swapped. For example, the columns of the rearranged matrix 710 may be arranged differently so that the "horizontal" positions of block arrays 702-1 and 702-2 are swapped, such as so that block array 702-2 is the "leftmost" block array. For example, more specifically, the columns of the rearranged matrix 710 may be ordered alternatively: [9, 13, 3, 1, 11, 2, 7, 10, 4, 6, 8, 14, 12, 5]. In other examples, the border array can be arranged along different sides of the rearranged matrix (e.g., along the top, left or right side of the matrix), or there can be more than one border matrix, wherein each border matrix is ​​arranged along different sides of the rearranged matrix. The border array can be referred to as a horizontal array or a vertical array. The horizontal array can be arranged horizontally on the rearranged matrix. The horizontal array can span the entire width of the rearranged matrix. The vertical array can be arranged vertically on the rearranged matrix. The vertical array can span the entire height of the rearranged matrix. In these other examples, the rearranged matrix can be referred to as a "border block matrix form". A matrix in the form of a single border block-diagonal matrix can also be referred to as being in the form of a border block matrix. As the technician will understand, the border array does not need to be arranged along one side of the rearranged matrix. That is, for example, the border array can be arranged between two block arrays in the block array. For example, the rows of the rearranged matrix 710 can be arranged differently so that the "vertical" positions of one or more block arrays in the border array and the block array are exchanged. For example, the rows of the rearranged matrix 710 may be arranged differently so that the "vertical" positions of the boundary array 702-4 and the block array 702-3 are swapped. For example, more specifically, the rows of the rearranged matrix 710 may be ordered alternatively: [7, 1, 2, 6, 8, 3, 14, 13, 12, 4, 10, 9, 11, 5]. In summary, the rows and / or columns of the matrix may be rearranged to form a rearranged matrix, the rearranged matrix comprising: one or more block arrays arranged along the diagonal of the rearranged matrix, and / or one or more block arrays not arranged along the diagonal of the rearranged matrix; and one or more horizontal arrays arranged horizontally on the rearranged matrix, and / or one or more vertical arrays arranged vertically on the rearranged matrix. The multiple elements representing non-zero values ​​included in the matrix may be aggregated into one or more block arrays and one or more horizontal and / or vertical arrays of the rearranged matrix.In other words, the rows and / or columns of the matrix can be rearranged to form: a rearranged matrix in the form of a border block matrix; or a rearranged matrix as a block matrix, wherein the block matrix includes an array that can be arranged into (e.g., capable of being rearranged into) a border block matrix form.

[0112] As described herein, matrix 600 does not have a subgraph partitioning. As will be appreciated by the skilled person, this means that it is not possible to rearrange matrix 600 into a block-diagonal matrix form consisting of (e.g., exclusively including) a plurality of block arrays arranged on a diagonal line in which all non-zero values ​​of matrix 600 are clustered.

[0113] In step S506, the rows and / or columns of the matrix may be rearranged according to the hypergraph model. The hypergraph model may be used to convert the matrix into a "single-boundary block-diagonal matrix form". The hypergraph model may be formed according to the corresponding row and column positions of each of the plurality of elements representing non-zero values ​​within the matrix.

[0114] In one example, the hypergraph model is a "rownet" hypergraph model. Figure 6 The matrix 600 shown in FIG. 6 can be rearranged using a rownet hypergraph model to form Figure 7 The rearranged matrix 710 shown in . Forming a row-net hypergraph model includes forming a vertex representing each column of the matrix, the vertex including an element representing a non-zero value; and forming a net representing each row of the matrix, the net including an element representing a non-zero value. For each of a plurality of elements representing non-zero values ​​within the matrix, a vertex representing a column of the matrix including the element representing the non-zero value is connected to a net representing a row of the matrix including the element representing the non-zero value. Reference may be made to Figure 8 To understand this example, the diagram shows Figure 6 The matrix 600 shown in FIG. 8 forms an example row-net hypergraph model 812 .

[0115] exist Figure 8 , the vertices representing the columns of the matrix 600 are shown as v i , where i is the index of the column. For example, column 10 of matrix 600 consists of vertex v 10 The net representing the rows of matrix 600 is shown as n j , where j is the index of the row. For example, row 1 of matrix 600 consists of vertex n 1 As described herein, the elements located in row 1, column 10 of matrix 600 represent non-zero values. Thus, as an example, vertex v of column 10 of matrix 600 including an element representing a non-zero value is represented. 10 The net n connected to row 1 of the matrix 600 including elements representing non-zero values 1 In order to form Figure 8The row network hypergraph model shown in has been Figure 6 The same vertex-net connection process is repeated for each of the non-zero valued elements in the matrix 600 shown in FIG. 6 , as shown by comparing Figure 6 and Figure 8 Will understand.

[0116] In other words, we can construct a row-net hypergraph model for the coefficient matrix W as follows. Let H = (V, N) be a hypergraph H with vertex set V and net set N. Each column W(:, i) consists of vertex v i ∈V, and each row W(j,:) is represented by the network n j ∈N indicates that if there is an element in the coefficient matrix W that represents a non-zero value W(i,j), then the net n j Connect vertices v i . By the network j The connected vertices can be represented as

[0117] It should be understood that when forming a hypergraph model (e.g., a row-net hypergraph model), vertices may not be formed for columns of a matrix that do not include any elements representing non-zero values ​​(not shown in the figure), and a net may not be formed for rows of a matrix that do not include any elements representing non-zero values ​​(not shown in the figure).

[0118] In another example, the hypergraph model is a "columnnet" hypergraph model. Forming the columnnet hypergraph model includes forming a net representing each column of a matrix, the net including elements representing non-zero values; and forming a vertex representing each row of the matrix, the vertex including elements representing non-zero values. For each of a plurality of elements within the matrix representing non-zero values, a net representing a column of the matrix including the element representing the non-zero value is connected to a vertex representing a row of the matrix including the element representing the non-zero value.

[0119] In other words, we can construct a column-net hypergraph model for the coefficient matrix W as follows. Let H = (V, N) be a hypergraph H with vertex set V and net set N. Each row W(j,:) consists of vertex v j ∈V, and each column W(:,i) is represented by the network n i ∈N indicates that if there is an element in the coefficient matrix W that represents a non-zero value W(i,j), then the net n i Connect vertices v j . By the network i The connected vertices can be represented as

[0120] It should be understood that when forming a hypergraph model (e.g., a column-net hypergraph model), a net may not be formed for columns of a matrix that do not include any elements representing non-zero values ​​(not shown in the figure), and / or vertices may not be formed for rows of a matrix that do not include any elements representing non-zero values ​​(not shown in the figure).

[0121] Once formed, the hypergraph model can be partitioned. Figure 8 A row-net hypergraph model that has been divided into three parts 812-1, 812-2, 812-3 is shown. It should be understood that the hypergraph model can be divided into any suitable number of parts. The rows and / or columns of the matrix can be rearranged according to the partitioned hypergraph model so that multiple elements of the matrix representing non-zero values ​​are aggregated into one or more sub-matrices.

[0122] For example, in Figure 8 , section 812-1 corresponds to Figure 7 The elements representing non-zero values ​​located in the rows of the matrix represented by the nets are gathered into the block array 702-1, and the nets are connected only to the vertices of the columns of the matrix within the representation portion 812-1. For example, in Figure 8 China, Network 1 、n 2 and n 7 Only connected to the vertices in part 812-1, that is, connected to the vertex v 7 and v 10 ; Vertex v 2 、v 4 and v 10 ; and vertex v 2 、v 4 and v 7 Therefore, the rows and columns of the matrix 600 may be rearranged so that the elements representing non-zero values ​​located in rows 1, 2, and 7 are gathered into a block array 702-1, as shown in FIG. Figure 7 In a similar manner: Section 812-2 corresponds to Figure 7 The block array 702-2 shown in FIG. 8; and the portion 812-3 corresponds to Figure 7 Block array 702-3 shown in .

[0123] Elements representing non-zero values ​​located in rows of the matrix represented by nets connected to vertices representing columns of the matrix in more than one section are gathered into a boundary array 702-4. Figure 8 China, Network 4 Connected to the vertices in parts 812-1 and 812-2, namely, vertex v 1 、v 7 、v 9 、v 10 and v 11 .like Figure 8As shown in the 10 、n 12 and n 13 Also connected to vertices in more than one part. Therefore, the rows and columns of matrix 600 are rearranged so that the elements representing non-zero values ​​located in rows 4, 10, 12, and 13 are gathered into boundary array 702-4, as shown in FIG. Figure 7 The hypergraph model can be partitioned according to a load balancing constraint that aims to minimize the number of nets connected to vertices in more than one part, for example, to minimize the number of rows of the border array.

[0124] As will be appreciated by those skilled in the art, a hypergraph model formed for a matrix with subgraph partitioning (not shown in the figure) will not include any nets (or vertices) connected to vertices (or nets) in more than one part. That is, there will be no nets (or vertices) that are "connected" to any of the parts. This will enable the matrix to be converted into a block-diagonal matrix form consisting of (e.g., exclusively including) a plurality of block arrays arranged on a diagonal in which all non-zero values ​​of the matrix are clustered.

[0125] It should be understood that any row or column of the matrix that does not contain any elements representing non-zero values ​​(e.g., any row or column that does not form a net or vertex, as appropriate, when forming a hypergraph model) can be rearranged (e.g., arbitrarily) to any row or column position within the rearranged matrix. Alternatively, another "empty" block array (not shown in the figure) can be formed, and the elements of the rows and columns that do not contain any elements representing non-zero values ​​can be gathered into the other "empty" block array. The "empty" block array can be used in a manner equivalent to the "non-empty" block array during future calculations performed in the compression layer (as will be further described herein), or not used for future calculations performed in the compression layer (e.g., discarded therefrom).

[0126] In other words, a K-way vertex partition of a hypergraph model H can be defined as a set of mutually incoherent and exhaustive subsets V of vertices. m ∈V composed of Π(H)={V 1 ,V 2 ,…V K}, where for all V m ∈Π(H), if m≠n and but Make According to the partition Π(H), by the network n j The connected partial set can be defined as the connected set Λ(n j ), and by the network j The number of connected parts can be defined as the connectivity λ(n j )=|Λ(n j )|. If the network jconnected to multiple parts, the net is called a cut (external) (ie, λ(n j )>1), otherwise it is called uncut (inner) (i.e., λ(n j )=1). If network n j Connect only part of V m The vertex in (i.e., Λ(n j )=V m and pins(n j )=V m ), then the network is part V m Section V m The internal network set is denoted as N m , and the external network set of partition Π(H) is denoted as N S Therefore, a K-way vertex partition Π(H) can also induce a (K+1)-way partition {N 1 ,N 2 ,…N K ; N S Here, N S is a mesh separator, removing the mesh separator yields K disconnected vertex parts V 1 ,V 2 ,…V K and K disconnected network parts N 1 ,N 2 ,…N K .

[0127] The hypergraph model H can be partitioned to achieve load balancing constraints Minimize the number of cut nets where part V m The weight of The average part weight is And the maximum allowed imbalance ratio is ∈. Each vertex v i The weight can be determined as w(v i )=1. The purpose of the load balancing constraint may be to produce equally sized block arrays that can each fit in the coefficient buffer of a neural network accelerator (NNA), e.g. Figure 3 In the coefficient buffer 330 of the neural network accelerator (NNA) 302 shown in .

[0128] K-way partition Π(H)={V 1 ,V 2 ,…V K}={N 1 ,N 2 ,…N k ; N S} may cause a partial ordering of the rows and columns of the coefficient matrix W. In this ordering, in the example where the hypergraph model is formed as a row-net hypergraph model, m+1 The columns associated with the vertices in V m (where m = 1, 2, ... K-1) and then sort the columns associated with the vertices. Similarly, use part V m+1 Intranet m+1 The lines that represent this can be in the same m (where m = 1, 2, ... K-1) internal network N m The associated rows are then sorted. S The associated rows are finally sorted into a bounding array. In other words, the vertex v i ∈V m This means arranging the column W(:,i) into the mth column slice, with internal network n j ∈N m This means arranging the row W(j,:) into the mth row slice, and the external network n j ∈N S This means aligning the row W(j,:) to the boundary matrix.

[0129] In the example described herein, where the hypergraph model is formed as a row-net hypergraph model, partitioning the hypergraph model as described herein will rearrange the matrix into Figure 7 , where the border array is arranged along the bottom of the rearranged matrix 710. In an alternative example described herein, where the hypergraph model is formed as a column-net hypergraph model, partitioning the hypergraph model as described herein will cause the matrix to be rearranged into a single-bordered block-diagonal form, where the border array is arranged along the right side of the rearranged matrix (e.g., see Fig.10 The example rearranged matrix 1010 shown in FIG.

[0130] It should be understood that it is not necessary to use a hypergraph model in step S506. There are other methods for rearranging the rows and / or columns of a matrix so as to cluster multiple elements of the matrix representing non-zero values ​​into one or more sub-matrices. For example, a hypergraph clustering algorithm or a graph partitioning algorithm may be used for this purpose instead.

[0131] Return to Figure 5, in step S508, a compressed neural network including a compression layer is output, the compression layer being arranged to perform a compression operation according to one or more sub-matrices formed in step S506. The compression layer is arranged to perform the same type of operation as the layer selected in step S504 is arranged to perform. That is, the compression layer is arranged to perform the type of operation according to the one or more sub-matrices formed in step S506, for example, rather than being arranged to perform the type of operation according to the matrix determined in step S504. In other words, only a subset of the set of coefficients represented by the elements of the one or more sub-matrices formed in step S506 can be used by the compression layer. This applies to both the first example and the second example described herein. Other coefficients not included in the one or more sub-matrices formed in step S506 in the set of coefficients represented by the elements of the rearranged matrix are not used by the compression layer. The other coefficients are only zero coefficients, so that the element-by-element multiplication between the input activation value and any of those other coefficients will inevitably produce a zero output value, regardless of the value of the input activation value. Therefore, the compression layer that does not use the other coefficients does not lose "information". In other words, the received neural network includes layers arranged to perform operations using coefficient sets. One or more sub-matrices represent subsets of the coefficient sets of the layers of the received neural network. The compression layer can be arranged to perform compression operations using subsets of the coefficient sets of the layers of the received neural network. The subsets of the coefficient sets of the layers of the received neural network include all non-zero coefficients of the coefficient sets of the layers of the received neural network. That is, during the method of compressing a neural network described herein, none of the non-zero coefficients of the coefficient sets of the layers of the received neural network are set to zero or discarded. The other coefficients in the coefficient set not included in the subset are only zero coefficients. Therefore, the compression layer does not lose any information (e.g., non-zero information), and the compression layer is arranged to perform compression operations using subsets of the coefficient sets of the layers of the received neural network, and does not use other coefficients of the coefficient sets not included in the subset. In other words, the method of compressing a neural network described herein is a lossless method of compressing a neural network. Figure 4 The at least one processor 404 shown in FIG. 4 may be configured to perform step S508 .

[0132] For example, in a first example defined herein, a selected layer of the received neural network is arranged to perform a fully connected operation by performing a matrix multiplication using a matrix including a set of coefficients for the layer and an input matrix including a set of input activation values ​​for the layer. Specifically, as described herein, a matrix multiplication WX=Y may be performed by the selected layer, where: W is a coefficient matrix (e.g., matrix 600) including a set of coefficients; X is an input matrix including a set of input activation values; and Y is an output matrix including a set of output values. Alternatively, also as described herein, a matrix multiplication XW=Y may be performed by the selected layer.

[0133] In a first example, in step S508, the compressed neural network is configured such that the compressed layer is arranged to perform a compressed fully connected operation by performing one or more matrix multiplications using one or more sub-matrices, wherein the one or more sub-matrices include a subset of the coefficient set of the selected layer and one or more input sub-matrices, each input sub-matrix including a corresponding subset of the input activation value set of the selected layer.

[0134] For example, Fig. 9 It is shown how compression operations can be performed using multiple sub-matrices. Fig. 9 A "compressed" version of the matrix multiplication WX=Y is shown. Fig. 9 A rearranged coefficient matrix 910 is shown comprising a plurality of sub-matrices, labeled B 1 902-1, B 2 902-2, B 3 902-3, R 1 902-4, R 2 902-5 and R 3 902-6. Fig. 9 The rearranged matrix 910 shown in FIG. Figure 7 The rearranged matrix 710 shown in FIG. 7 has the same characteristics as described herein. Fig. 9 As shown, each block array of the boundary block matrix 910 can be marked as a sub-matrix B, and the boundary array of the boundary block matrix 910 can be divided into a plurality of sub-matrices R. The boundary array of the boundary block matrix 910 can be divided according to the number of columns included in each block array. That is, the sub-matrix R 1 can have the same submatrix B 1 Same number of columns; submatrix R 2 can have the same submatrix B 2 Same number of columns; submatrix R 3 can have the same submatrix B 3 In the first example, the multiple sub-matrices B of the compression layer 1 , B 2 , B 3 , R 1 , R 2 , R 3 A subset of the set of coefficients included in the coefficient matrix W (eg, matrix 600) of the selected layer is included.

[0135] Fig. 9 Also shown are the 1 904-1, X 2 904-2, X 3 904-3. In the first example, the multiple input sub-matrices X of the compression layer 1 , X2 , X 3 Each includes a corresponding subset of the set of input activation values ​​included in the input matrix X of the selected layer. In a first example, the rows of the input matrix X can be rearranged (e.g., reordered or arranged) to correspond to the rearranged columns of the rearranged coefficient matrix. The multiple input sub-matrices X 1 , X 2 , X 3 Each of them may only include the input activation values ​​of the input matrix X, which will be multiplied by the multiple sub-matrices B of the compressed layer. 1 , B 2 , B 3 , R 1 , R 2 , R 3 The coefficients included are explained further below. 1 , X 2 , X 3 The number of rows included can be equal to the number of submatrices B 1 and R 1 , B 2 and R 2 and B 3 and R 3 The number of columns included respectively. That is to say: the input submatrix X 1 The number of rows can be equal to the submatrix B 1 and R 1 The number of columns; Input submatrix X 2 The number of rows can be equal to the submatrix B 2 and R 2 The number of columns; and the input submatrix X 3 The number of rows can be equal to the submatrix B 3 and R 3 The rows of the input matrix X may be rearranged (eg, reordered or permuted) to correspond to the rearranged columns of the rearranged coefficient matrix such that: the input submatrix X 1 Includes input activation values ​​from the rows of X that correspond to (e.g., will be multiplied by) the submatrix B 1 and R 1 The coefficient values ​​of the columns of 2 Includes input activation values ​​from the rows of X that correspond to (e.g., will be multiplied by) the submatrix B 2 and R 2 The coefficient values ​​of the columns of 3 Includes input activation values ​​from the rows of X that correspond to (e.g., will be multiplied by) the submatrix B 3 and R 3In an example where each column of the rearranged coefficient matrix contains at least one element representing a non-zero coefficient, the rows of the input matrix X may be rearranged (e.g., reordered or permuted) to form an input submatrix X 1 , X 2 , X 3 without discarding any input activations from the input matrix X. That is, all input submatrices X 1 , X 2 , X 3 The sum of the number of input activation values ​​in can be equal to the number of input activation values ​​in the input matrix X.

[0136] Outputting the compressed neural network in step S508 may also include adding an aggregation layer before the compression layer in the compressed neural network. In a first example, the aggregation layer may be configured to form one or more input sub-matrices (e.g., input sub-matrices X ) by aggregating corresponding subsets of output activation values ​​formed by previous layers of the compressed neural network into one or more input sub-matrices. 1 , X 2 , X 3 ). In the case where the previous layer or operation of the compressed neural network is not compressed (e.g., remains configured to output data as a single output matrix, rather than one or more output sub-matrices of the structure or size required by the compressed layer), or in the case where the previous layer of the compressed neural network is based on Figure 5 The clustering layer may be used in situations where the method is independent of the selected layer being compressed (e.g., such that a previous layer of compression is configured to output data with one or more input sub-matrices having a structure or size that is different from the structure or size of one or more output sub-matrices required by the selected layer of compression).

[0137] Fig. 9 Also shown is a 1 906-1, Y 2 906-2, Y 3 906-3 and Y 4 906-4. In the first example, the multiple output sub-matrices Y of the compression layer 1 , Y 2 , Y 3 , Y 4 Each of the input sub-matrices X includes a corresponding subset of the set of output values ​​included in the output matrix Y of the selected layer. In the first example, when a "compressed" version of the matrix multiplication WX=Y is to be performed, the following equations (1) and (2) can be used, using multiple input sub-matrices X 1 , X 2 , X 3 and multiple sub-matrices B 1 , B 2 , B3 , R 1 , R 2 , R 3 To form (for example, a compressed full connection operation can be performed) multiple output sub-matrices Y 1 , Y 2 , Y 3 , Y 4 :

[0138] Y i =B i X i , where i=1,2…K (1)

[0139]

[0140] It should be understood that equations (1) and (2) are general formulas that can be used to perform compressed full-connect operations (e.g., a "compressed" version of the matrix multiplication WX=Y) using submatrices of any K-way partitioned single-boundary block-diagonal matrix rearranged according to the rownet hypergraph model. In this specific example, where K=3: 1 =B 1 X 1 To form the output sub-matrix Y 1 ; can be done by performing matrix multiplication Y 2 =B 2 X 2 To form the output sub-matrix Y 2 ; can be done by performing matrix multiplication Y 3 =B 3 X 3 To form the output sub-matrix Y 3 ; and can be obtained by performing matrix multiplication Y 4 =R 1 X 1 +R 2 X 2 +R 3 X 3 To form the output sub-matrix Y 4 .

[0141] Outputting the compressed neural network in step S508 may also include adding a scattering layer after the compression layer in the compressed neural network. In a first example, the scattering layer may be configured to output the compressed neural network by adding one or more output sub-matrices (e.g., output sub-matrices Y 1 , Y 2 , Y 3 , Y 4) are scattered into a single output matrix to form an output matrix (e.g., output matrix Y). The single output matrix may have the same number of elements as the sum of the number of elements in each of the one or more output sub-matrices. Alternatively, the single output matrix may have a greater number of elements than the sum of the number of elements of the one or more output sub-matrices (e.g., if one or more rows or columns of input activation values ​​are discarded when forming the one or more input sub-matrices), in which case zero values ​​(i.e., "0") may be added as additional elements. A scattering layer may be used where subsequent layers or operations of the compressed neural network are not compressed. That is, where subsequent layers (e.g., fully connected layers) or operations (e.g., summation operations) of the compressed neural network are configured to receive and process input activation data in a format that would have been generated by a (uncompressed) selected layer of the received neural network, such as a single input matrix, rather than one or more input sub-matrices output by the compression layer in the first example.

[0142] As described in this paper, the row network hypergraph model can be used to form Fig. 9 910. For completeness, Fig.10 Shown is a rearranged matrix 1010 that may be formed according to a column-net hypergraph model, as described herein.

[0143] Fig.10 Another example of how a compression operation can be performed using multiple sub-matrices is shown. Fig.10 Another "compressed" version of the matrix multiplication WX=Y is shown. Fig.10 A rearranged coefficient matrix 1010 is shown comprising a plurality of sub-matrices, labeled B 1 1002-1, B 2 1002-2, B 3 1002-3, C 1 1002-4, C 2 1002-5 and C 3 1002-6. Fig.10 As shown, each block array of the boundary block matrix 1010 can be marked as a sub-matrix B, and the boundary array of the boundary block matrix 1010 can be divided into a plurality of sub-matrices C. The boundary array of the boundary block matrix 1010 can be divided according to the number of rows included in each block array. That is, the sub-matrix C 1 can have the same submatrix B 1 Same number of rows; submatrix C 2 can have the same submatrix B 2 Same number of rows; submatrix C 3 can have the same submatrix B 3In the first example, the multiple sub-matrices B of the compression layer 1 , B 2 , B 3 , C 1 , C 2 , C 3 A subset of the set of coefficients included in the coefficient matrix W (eg, matrix 600) of the selected layer is included.

[0144] Fig.10 Also shown are the 1 1004-1, X 2 1004-2, X 3 1004-3 and X 4 In a first example, the rows of the input matrix X may be rearranged (eg, reordered or arranged) to correspond to the rearranged columns of the rearranged coefficient matrix. 1 , X 2 , X 3 , X 4 Each of the input sub-matrices X comprises a corresponding subset of the set of input activation values ​​included in the input matrix X of the selected layer. In a first example, the plurality of input sub-matrices X 1 , X 2 , X 3 , X 4 Each of them may only include the input activation values ​​of the input matrix X, which will be multiplied by the multiple sub-matrices B of the compressed layer. 1 , B 2 , B 3 , C 1 , C 2 , C 3 The coefficients included. Multiple input sub-matrices X 1 , X 2 , X 3 , X 4 The number of rows included can be equal to the number of submatrices B 1 and B 2 , B 3 and C 1 and C 2 and C 3 The number of columns included respectively. That is to say: the input submatrix X 1 The number of rows can be equal to the submatrix B 1 The number of columns; Input submatrix X 2 The number of rows can be equal to the submatrix B 2 The number of columns; Input submatrix X 3 The number of rows can be equal to the submatrix B 3 The number of columns; and the input submatrix X4 The number of rows can be equal to the submatrix C 1 , C 2 and C 3 The rows of the input matrix X may be rearranged (eg, reordered or permuted) to correspond to the rearranged columns of the rearranged coefficient matrix such that: the input submatrix X 1 Includes activation values ​​from the rows of X that correspond to (e.g., will be multiplied by) the submatrix B 1 The coefficient values ​​of the columns of 2 Includes activation values ​​from the rows of X that correspond to (e.g., will be multiplied by) the submatrix B 2 The coefficient values ​​of the columns of 3 Includes activation values ​​from the rows of X that correspond to (e.g., will be multiplied by) the submatrix B 3 The coefficient values ​​of the columns of 4 Includes activation values ​​from the rows of X that correspond to (e.g., will be multiplied by) the submatrix C 1 , C 2 and C 3 In an example where each column of the rearranged coefficient matrix contains at least one element representing a non-zero coefficient, the rows of the input matrix X may be rearranged (e.g., reordered or permuted) to form an input submatrix X 1 , X 2 , X 3 , X 4 without discarding any input activations from the input matrix X. That is, all input submatrices X 1 , X 2 , X 3 , X 4 The sum of the number of input activation values ​​in can be equal to the number of input activation values ​​in the input matrix X.

[0145] Fig.10 Also shown is a 1 1006-1, Y 2 1006-2 and Y 3 In the first example, the multiple output sub-matrices Y of the compression layer 1 , Y 2 , Y 3 Each of the input sub-matrices X includes a corresponding subset of the set of output values ​​included in the output matrix Y of the selected layer. In the first example, when a "compressed" version of the matrix multiplication WX=Y is to be performed, the following equations (3) and (4) can be used, using multiple input sub-matrices X 1 , X 2 , X 3, X 4 and multiple sub-matrices B 1 , B 2 , B 3 , C 1 , C 2 , C 3 To form (for example, a compressed full connection operation can be performed) multiple output sub-matrices Y 1 , Y 2 , Y 3 :

[0146] Y i , where i=1,2…K (3)

[0147] Y i =B i X i +C i X K+1 (4)

[0148] It should be understood that equations (3) and (4) are general formulas that can be used to perform compressed fully connected operations (e.g., a "compressed" version of the matrix multiplication WX=Y) using submatrices of any K-way partitioned single-boundary block-diagonal matrix rearranged according to the column-net hypergraph model. In this particular example, where K=3: 1 =B 1 X 1 +C 1 X 4 To form the output sub-matrix Y 1 ; can be done by performing matrix multiplication Y 2 =B 2 X 2 +C 2 X 4 To form the output sub-matrix Y 2 ; and can be obtained by performing matrix multiplication Y 3 =B 3 X 3 +C 3 X 4 To form the output sub-matrix Y 3 .

[0149] It should be understood that technicians will have no difficulty in applying the references herein Fig. 9 and / or Fig.10 The described principles implement a compression layer arranged to perform a compressed fully connected operation by performing a "compressed" version of the matrix multiplication XW=Y using a sub-matrix of the rearranged coefficient matrix as output from step S506, as described herein.

[0150] In a second example defined herein, the selected layer of the received neural network is a convolutional layer, the convolutional layer being arranged to perform a convolution operation by convolving a set of input activation values ​​of the convolutional layer with a set of coefficients of the convolutional layer. Figure 2a and Figure 2b As understood from the description herein, to perform a convolution operation, the first input channel of the input activation data is convolved with the first input channel of each filter in the coefficient set, the second input channel of the input activation data is convolved with the second input channel of each filter in the coefficient set, and so on, until the Nth input channel of the input activation data is convolved with the Nth input channel of each filter in the coefficient set.

[0151] In a second example, each of the one or more sub-matrices formed in step S506 includes a plurality of elements representing a corresponding subset of the input channels of the filter of the coefficient set of the convolutional layer. Figure 6 and Figure 7 , Figure 6 The matrix 600 shown in comprises 196 elements representing 196 filter input channels (i.e., 14 input channels for 14 filters). In contrast, Figure 7 The plurality of sub-matrices 702-1, 702-2, 702-3, and 702-4 shown in FIG. 7 include 103 elements representing 103 filter input channels (ie, a subset of the 14 input channels of the 14 filters).

[0152] In step S508, in a second example, the compressed neural network is configured such that the compression layer is arranged to perform a compressed convolution operation by convolving one or more subsets of the input activation values ​​of the convolution layer with a subset of the set of coefficients of the convolution layer included in one or more subsets of the input channels of the filter represented by elements of the one or more sub-matrices. As will be appreciated by those skilled in the art, the compressed convolution operation may be performed with any stride, padding and / or dilation parameters as desired.

[0153] For example, Fig.16b It shows how compression operations can be performed using multiple subsets of the coefficient set of a convolutional layer. Fig.16b According to the reference Fig.16a The described principle uses a rearranged set of coefficients of a three-dimensional convolutional layer of a cuboid 1610. out ×C in The plane shows the Figure 7 The rearranged matrix 710 and Fig. 9The rearranged matrix 910 shown in FIG. 1 is a rearranged matrix 910 having equivalent characteristics. According to the row and column order of the rearranged matrix 910 (for example, after the rearrangement of the matrix 600 shown on the rectangular parallelepiped 1600), the C of the rearranged rectangular parallelepiped 1610 is out ×C in In the plane, there is (C in , C out ) coordinates of the one-dimensional coefficient sequence H w W w Each one of them.

[0154] As described herein, the rearranged coefficient matrix 910 includes the coefficients labeled B 1 , B 2 , B 3 , R 1 , R 2 and R 3 Multiple sub-matrices of . Fig.16b The multiple sub-matrices B shown in 1 , B 2 , B 3 , R 1 , R 2 and R 3 References to this article Fig. 9 Multiple sub-matrices B described 1 , B 2 , B 3 , R 1 , R 2 and R 3 The multiple sub-matrices B 1 , B 2 , B 3 , R 1 , R 2 and R 3 Each of comprises a plurality of elements representing a corresponding subset of the input channels of the filter of the coefficient set of the convolutional layer.

[0155] As described in the paper, the input activation dataset of the convolutional layer can have dimension C in ×H a ×W a In the second example, the input activation data C of the convolutional layer in The dimensions may be rearranged (e.g., reordered or permuted) so as to correspond to the rearranged set of coefficients of the convolutional layer. in Dimension. Fig.16b The rearranged input activation data set 1604 of the convolutional layer is shown. Fig.16b As shown, the rearranged input activation data set can be divided into input activation data X 1 , X2 , X 3 Input activation data X 1 , X 2 , X 3 The multiple subsets include the input channels (C in ) can be equal to the number of sub-matrices B 1 and R 1 , B 2 and R 2 and B 3 and R 3 The elements of the convolutional layer represent the subset of the filter input channels of the coefficient set of the convolutional layer, respectively, including the input channels (C in ) number. That is to say: input activation data X 1 The subset of has input channels (C in ) can be equal to the number of submatrices B 1 and R 1 The elements of the convolutional layer represent the number of input channels in the subset of input channels of the filter coefficient set (C in );Enter activation data X 2 The subset of has input channels (C in ) can be equal to the number of submatrices B 2 and R 2 The elements of the convolutional layer represent the subset of input channels of the filter coefficient set (C in ) number; and input activation data X 3 The subset of has input channels (C in ) can be equal to the number of submatrices B 3 and R 3 The elements of the convolutional layer represent the subset of input channels of the filter coefficient set (C in ). The number of input activation data of the convolutional layer is C in The dimensions may be rearranged (e.g., reordered or permuted) so as to correspond to the rearranged set of coefficients of the convolutional layer. in Dimensions such that: Input activation data X 1 The subset of includes the input channels from the input activation data (C in ), which corresponds to the activation value of the submatrix B 1 and R 1 The elements of the convolutional layer represent the subset of the filter input channels of the coefficient set (i.e., to be convolved with it); the input activation data X 2 The subset of includes the input channels from the input activation data (C in ), which corresponds to the activation value of the submatrix B 2 and R2 The elements of represent the subset of the input channels of the filter of the convolutional layer's coefficient set (e.g., to be convolved with it); and the input activation data X 3 The subset of includes the input channels from the input activation data (C in ), which corresponds to the activation value of the submatrix B 3 and R 3 The elements of represent the subset of the input channels of the filter of the coefficient set of the convolutional layer (e.g., to be convolved with it). In the example where each column of the rearranged matrix contains at least one element representing a non-zero value, the C of the input activation data of the convolutional layer in The dimensions may be rearranged (eg, reordered or permuted) to form the input activation data X 1 , X 2 , X 3 without discarding any input activation values ​​from the input activation data. That is, the input activation data X 1 , X 2 , X 3 The sum of the number of input activation values ​​in all subsets of can be equal to the number of values ​​in the input activation data.

[0156] As described herein, outputting the compressed neural network in step S508 may also include adding an aggregation layer before the compression layer in the compressed neural network. In a second example, the aggregation layer may be configured to aggregate corresponding subsets of output activation values ​​formed by previous layers of the compressed neural network to form one or more subsets of input activation data to be operated on in the compressed convolutional layer (e.g., Fig.16b The input activation data X shown in the example shown 1 , X 2 , X 3 multiple subsets of ). This can be done when the previous layers or operations of the compressed neural network are not compressed, or when the previous layers of the compressed neural network are based on Figure 5 The method is independent of the selected layer being compressed using the clustered layer.

[0157] Fig.16b Also shown is a 1 , Y 2 , Y 3 and Y 4 1606. In the second example, the output activation data Y of the compressed layer 1 , Y 2 , Y 3 , Y 4 Each of the plurality of subsets of includes a corresponding subset of the output value set included in the output data of the selected layer. In a second example, the following equations (5) and (6) can be used, using the input activation data X1 , X 2 , X 3 The sum of multiple subsets of the submatrix B 1 , B 2 , B 3 , R 1 , R 2 , R 3 The elements of the convolutional layer represent the coefficient sets of the filters of multiple subsets of the input channels to form the output activation data Y (for example, a compressed convolution operation can be performed). 1 , Y 2 , Y 3 , Y 4 Multiple subsets of:

[0158] Where i=1,2…K(5)

[0159]

[0160] symbol represents the convolution operation. That is, Indicates that the input activates the data X i The subset of i The elements of represent the subset of input channels of the filter of the coefficient set of the convolutional layer. It should be understood that equations (5) and (6) are general formulas that can be used to perform compressed convolution operations using any K-way partitioned single-boundary block-diagonal matrix submatrices rearranged according to the row network hypergraph model. In this specific example, where K = 3: the convolution can be performed by To form the output activation data Y 1 A subset of ; can be obtained by performing a convolution To form the output activation data Y 2 A subset of ; can be obtained by performing a convolution To form the output activation data Y 3 subset of ; and can be obtained by performing a convolution To form the output activation data Y 4 A subset of .

[0161] As described herein, outputting the compressed neural network in step S508 may also include adding a scattering layer after the compression layer in the compressed neural network. In a second example, the scattering layer may be configured to form an output activation value set by scattering a subset of output activation values ​​formed by the compressed convolution layer into a single output activation value set. The single output activation value set may have the same number of output activation values ​​as the sum of the number of output activation values ​​in each of the one or more subsets of output activation values. Alternatively, the single output activation value set may have a greater number of output activation values ​​than the number of output activation values ​​of the output activation value set formed by the compressed convolution layer (e.g., if one or more input channels of the input activation values ​​are discarded when forming the one or more subsets of input activation data), in which case a zero value (i.e., "0") may be added as an additional value. A scattering layer may be used in the case where a subsequent layer or operation of the compressed neural network is not compressed, for example, in the case where a subsequent layer (e.g., a convolution layer) or operation (e.g., a summation operation) of the compressed neural network is configured to receive and process input activation data in a format generated by a (uncompressed) selected layer of the received neural network.

[0162] Based on the principles described herein, it should also be understood that, although not shown in the figures or described in detail for the sake of brevity, the following equations (7) and (8) are general formulas that can be used to perform compressed convolution operations based on any K-way partitioned single-boundary block-diagonal matrix submatrix rearranged according to the column-net hypergraph model.

[0163] Y i , where i=1,2…K(7)

[0164]

[0165] For example, in the specific example where K=3 (e.g., as Fig.10 , as described herein): the convolution can be performed by To form the output activation data Y 1 A subset of ; can be obtained by performing a convolution To form the output activation data Y 2 subset of ; and can be obtained by performing a convolution To form the output activation data Y 3 A subset of .

[0166] Step S508 may include storing the compressed neural network for subsequent implementation. Figure 4, at least one processor 404 may write the compressed neural network into the memory 406 for subsequent implementation. Step S508 may include outputting a computer-readable description of the compressed neural network, which is useful when used in a system for implementing the neural network (e.g., Figure 3 When implemented at the system 300 shown in FIG. 5 , the compressed neural network is executed. Step S508 may include configuring the system for implementing the neural network to implement the compressed neural network.

[0167] Step S508 may include configuring hardware logic to implement the compressed neural network. The hardware logic may include a neural network accelerator. For example, referring to Figure 4 , at least one processor 404 can be configured to configure hardware logic included in the system 300 for implementing a neural network to implement a compressed neural network.

[0168] The compressed neural network output in step S508 may be used. The compressed neural network output in step S508 may be used to perform image processing. As a non-limiting example, the compressed neural network may be used to perform one or more of image super-resolution processing, semantic image segmentation processing, and object detection. For example, performing image super-resolution processing involves the compressed neural network processing a lower resolution image input to the neural network so as to output a higher resolution image.

[0169] According to this article reference Figure 5 The described method of compressing the received neural network is advantageous because the compression layer of the compressed neural network uses only a subset of the coefficient set of the selected layer of the received neural network. That is, the compression layer uses only a subset of the coefficient set represented by the elements of the one or more sub-matrices formed in step S506. This applies to both the first and second examples described herein. Other coefficients in the coefficient set represented by the elements of the rearranged matrix that are not included in the one or more sub-matrices formed in step S506 are not used by the compression layer. This means that when, for example, Figure 3When the compressed neural network is implemented at the neural network accelerator 302 shown in , those other coefficients do not need to be stored in a memory such as the off-chip memory 304. This reduces the memory footprint of the compressed neural network relative to the received neural network. In addition, this means that, relative to the selected layer, fewer coefficients need to be read into the neural network accelerator at runtime to implement the compressed layer. This reduces the weight bandwidth required to implement the compressed layer relative to the selected layer. For the same reason, the time spent reading from the memory in the coefficient set of the compressed layer is also reduced relative to the selected layer. This can reduce the latency of the compressed neural network relative to the received neural network. Similarly, the smaller coefficient set used by the compressed layer relative to the larger coefficient set used by the selected layer also reduces the computational requirements of the processing elements of the neural network accelerator that implements the compressed neural network by reducing the number of multiplication and accumulation operations performed by these processing elements to perform compression operations. Finally, as defined herein, each subset of the coefficient set only needs to operate on a subset of the input activation value set of the selected layer. This means that, in each cycle of runtime, in order to implement the compressed layer, fewer input activation values ​​are generally required to be read into the on-chip memory and / or input buffer of the neural network accelerator relative to the selected layer. This reduces the activation bandwidth required to implement the compressed layer relative to the selected layer.

[0170] Fig.11 A second method of compressing a neural network is shown. Fig.11 The method of compressing a neural network shown in is a computer-implemented method. Figure 4 The processing system 400 for compressing a neural network shown in FIG. 4 may be configured to perform Fig.11 The method for compressing neural networks shown in .

[0171] In step S1102, a neural network is received. Step S1102 may be performed in a manner similar to step S502 described herein. The neural network (e.g., a software model defining the neural network) may be Figure 4 The data is received at the processing system 400 and stored in the memory 406 .

[0172] In step S1104, two or more adjacent layers of the received neural network are selected. The two or more adjacent layers include a first layer and a second subsequent layer of the received neural network. The first layer is arranged to perform a first operation. The set of activation values ​​output by the first layer (e.g., due to performing the first operation) is the set of activation values ​​input to the second subsequent layer. The second layer is arranged to perform a second operation. Both the first layer and the second layer can be arranged to perform the same type of operation. In a first example, both the first layer and the second layer can be fully connected layers. In a second example, both the first layer and the second layer can be convolutional layers. Alternatively, the first layer and the second layer can be arranged to perform different types of operations. For example, the first layer can be a convolutional layer, and the second layer can be a fully connected layer.

[0173] Each of the selected two or more adjacent layers has one or more input channels and one or more output channels. The one or more output channels of the first layer correspond to the one or more input channels of the second subsequent layer. In other words, for 1 to N, the Nth output channel of the coefficient set of the first layer can be responsible for forming a channel of output activation data to be operated by the Nth input channel of the coefficient set of the second layer.

[0174] In a first example, both the first layer and the second layer may be fully connected layers arranged to perform matrix multiplication. The first layer may be configured to perform matrix multiplication W 0 X 0 =Y 0 , where: W 0 is the set of coefficients comprising the first layer and has dimension The first matrix of 0 is the set of input activations that includes the first layer and has dimension M 0 ×N 0 The first input matrix is And Y 0 is the set of output values ​​that includes the first layer and has dimensions As described in this paper, the set of activation values ​​output by the first layer (i.e., Y 0 ) is the set of activation values ​​input to the second subsequent layer. Therefore, the second layer can be configured to perform matrix multiplication W 1 Y 0 =Y 1 , where: W 1 is the set of coefficients comprising the first layer and has dimension The first matrix of Y 1 is a second output matrix comprising the output value set of the second layer. As will be appreciated by the skilled person, in order to perform the matrix multiplication W 1 Y 0 =Y 1 , W 1 The number of columns must be equal to Y0 Therefore: Therefore, when the first and second layers are both fully connected layers, it can be said that one or more output channels of the first layer in two or more adjacent layers One or more input channels corresponding to a second subsequent layer in two or more adjacent layers

[0175] Alternatively, in a first example, the first layer may be configured to perform the matrix multiplication X 0 W 0 =Y 0 , where: X 0 is the set of input activations that includes the first layer and has dimension M 0 ×N 0 The first input matrix of 0 is the set of coefficients comprising the first layer and has dimension The first matrix of And Y 0 is the set of output values ​​that includes the first layer and has dimensions As described in this paper, the set of activation values ​​output by the first layer (i.e., Y 0 ) is the set of activation values ​​that are input to the second subsequent layer. Therefore, the second layer can be configured to perform the matrix multiplication Y 0 W 1 =Y 1 , where: W 1 is the set of coefficients comprising the first layer and has dimension The first matrix of Y 1 is a second output matrix comprising the set of output values ​​of the second layer. As will be appreciated by the skilled person, in order to perform the matrix multiplication Y 0 W 1 =Y 1 , Y 0 The number of columns must be equal to W 1 Therefore: Therefore, in this alternative to the first example, when both the first layer and the second layer are fully connected layers, we can also say that one or more output channels of the first layer in two or more adjacent layers are One or more input channels corresponding to a second subsequent layer in two or more adjacent layers

[0176] In the second example, both the first layer and the second layer can be convolutional layers. Figure 2a and Figure 2bAs described herein, the number of output channels (e.g., filters) in a coefficient set of a layer (e.g., a first layer) corresponds to (e.g., is equal to) the number of channels in an output activation value set formed by the layer. As described herein, the activation value set output by the first layer is the activation value set input to the second subsequent layer. As described herein with reference to Figure 2a and Figure 2b As described, the number of input channels in the coefficient set of a layer (e.g., the second layer) corresponds to (e.g., is equal to) the number of input channels in the activation value set input to the layer. It can therefore be seen that the number of output channels (e.g., filters) in the coefficient set of the first layer will be equal to the number of input channels in the coefficient set of the second subsequent layer. Each filter in the coefficient set of the first layer is responsible for forming a channel of output activation data that will be operated on by the corresponding input channel of the coefficient set of the second layer. Therefore, when the first layer and the second layer are both convolutional layers, it can be said that one or more output channels of the first layer in two or more adjacent layers correspond to one or more input channels of the second subsequent layer in two or more adjacent layers.

[0177] In step S1106, a first matrix (e.g., W 0 ). The first matrix includes a plurality of elements representing non-zero values ​​and a plurality of elements representing zero values. One or more rows or columns of the first matrix represent one or more output channels of the first layer, and one or more other rows or columns of the first matrix represent one or more input channels of the first layer. For example, one or more rows of the first matrix may represent one or more output channels of the first layer, and one or more columns of the first matrix may represent one or more input channels of the first layer. Alternatively, one or more columns of the first matrix may represent one or more output channels of the first layer, and one or more rows of the first matrix may represent one or more input channels of the first layer. Step S1106 may be performed for the first layer in a manner similar to the manner in which step S504 is performed for the selected layer as described herein. The first layer may be a fully connected layer or a convolutional layer. Figure 4 The at least one processor 404 shown in may be configured to perform step S1106.

[0178] In step S1108, a second matrix (e.g., W 1). The second matrix includes a plurality of elements representing non-zero values ​​and a plurality of elements representing zero values. One or more rows or columns of the second matrix represent one or more output channels of the second layer, and one or more other rows or columns of the second matrix represent one or more input channels of the second layer. For example, one or more rows of the second matrix may represent one or more output channels of the second layer, and one or more columns of the second matrix may represent one or more input channels of the second layer. Alternatively, one or more columns of the second matrix may represent one or more output channels of the second layer, and one or more rows of the second matrix may represent one or more input channels of the second layer. Step S1108 may be performed for the second layer in a manner similar to the manner in which step S504 is performed for the selected layer as described herein. The second layer may be a fully connected layer or a convolutional layer. Figure 4 The at least one processor 404 shown in may be configured to perform step S1108.

[0179] In step S1110, an array is formed. In some examples, in steps S1106 and S1108, the first matrix and the second matrix are determined in a "consistent" manner, for example, so that rows or columns of both the first matrix and the second matrix represent the same type of channels (e.g., input channels or output channels). For example, one or more rows of the first matrix may represent one or more output channels of the first layer, one or more columns of the first matrix may represent one or more input channels of the first layer, one or more rows of the second matrix may represent one or more output channels of the second layer, and one or more columns of the second matrix may represent one or more input channels of the second layer. Alternatively, one or more columns of the first matrix may represent one or more output channels of the first layer, one or more rows of the first matrix may represent one or more input channels of the first layer, one or more columns of the second matrix may represent one or more output channels of the second layer, and one or more rows of the second matrix may represent one or more input channels of the second layer.

[0180] In these "consistent matrix" examples, an array including a transposed first matrix and a second matrix can be formed by transposing the first matrix and by aligning columns or rows of the transposed first matrix representing one or more output channels of the first layer with columns or rows of the second matrix representing one or more input channels of the second layer, thereby forming an array. For example, for 1 to N, the Nth column of the first matrix representing the transposed Nth output channel of the first layer can be aligned with (e.g., included in the same column of the array as) the Nth column of the second matrix representing the Nth input channel of the second layer, where the Nth output channel of the first layer corresponds to the Nth input channel of the second layer (e.g., the channel responsible for forming output activation data to be operated by the input channel). Alternatively, for 1 to N, the Nth row of the first matrix representing the transposed Nth output channel of the first layer can be aligned with (e.g., included in the same row of the array as) the Nth row of the second matrix representing the Nth input channel of the second layer, where the Nth output channel of the first layer corresponds to the Nth input channel of the second layer (e.g., the channel responsible for forming output activation data to be operated by the input channel). In other words, within the array, each output channel of the first layer is aligned with the corresponding input channel of the second layer.

[0181] Alternatively, in these "consistent matrix" examples, an array including a transposed second matrix and a first matrix may be formed by transposing the second matrix and by aligning rows or columns of the transposed second matrix representing one or more input channels of the second layer with rows or columns of the first matrix representing one or more output channels of the first layer, thereby forming an array. For example, for 1 to N, the Nth row of the first matrix representing the Nth output channel of the first layer may be aligned with (e.g., placed in the same row of the array as) the Nth row of the second matrix representing the transposed Nth input channel of the second layer, where the Nth output channel of the first layer corresponds to the Nth input channel of the second layer (e.g., the channel responsible for forming output data to be operated by the input channel). Alternatively, for 1 to N, the Nth column of the first matrix representing the Nth output channel of the first layer may be aligned with (e.g., placed in the same column of the array as) the Nth column of the second matrix representing the transposed Nth input channel of the second layer, where the Nth output channel of the first layer corresponds to the Nth input channel of the second layer (e.g., the channel responsible for forming output data to be operated by the input channel). In other words, within the array, each output channel of the first layer is aligned with a corresponding input channel of the second layer.

[0182] Typically, in these "consistent matrix" examples, to form an array according to the principles described herein, the matrices determined for each other layer in a series of adjacent layers can be transposed so that corresponding output channels and input channels of adjacent layers within the series can be aligned. The first matrix to be transposed can be the matrix determined for the first layer in the series of adjacent layers, or the matrix determined for the second layer in the series of adjacent layers, where the matrices determined for each other layer in the series of adjacent layers are then transposed. Figure 4 The at least one processor 404 shown in can be configured to perform step S1110.

[0183] Step S1110 can refer to Fig.12 It is understood that this figure shows an example array 1200. By transposing the first matrix W 0 And by replacing the first matrix (W) representing the transpose of one or more output channels of the first layer 0 ) T The columns of 1201-1 and the second matrix W representing one or more input channels of the second layer 1 1200-2 columns are aligned to form a first matrix (W) including a transposed 0 ) T 1201-1 and the second matrix W 1 1200-2 array 1200, thereby forming Fig.12 An array of 1200.

[0184] To help readers understand, Fig.12 The transposed third matrix (W 2 ) T 1201-3, the transposed third matrix has been determined by transposing a third matrix determined for a third layer of the received neural network that is adjacent to the second layer and after the second layer (e.g., as described in reference Figure 5 In array 1200, a second matrix W representing one or more output channels of the second layer is 1 The rows of 1200-2 and the third matrix (W 2 ) T 1201-3 row alignment. Fig.12 A fourth matrix W is also shown in 3 1201-4, the fourth matrix has been determined for a fourth layer of the received neural network that is adjacent to and after the third layer (e.g., as shown in reference Figure 5 In array 1200, a third matrix (W) representing the transpose of one or more output channels of the third layer is 2 ) TThe columns of 1201-3 and the fourth matrix W representing one or more input channels of the fourth layer 3 1201-4 column alignment. Fig.12 The array 1200 shows the transposed third matrix (W 2 ) T 1201-3 and the fourth matrix W 3 1201-4, to illustrate as herein referenced Fig.11 The described method of compressing a neural network can be applied to a sequence of more than two adjacent layers of the received neural network. Fig.11 The described method of compressing a neural network can be applied to any number of adjacent layers of a received neural network.

[0185] Fig.12 The array 1200 of FIG. 1 shows a plurality of matrices arranged in a progressively downward, progressively leftward pattern from the upper right corner of the array 1200 to the lower left corner of the array 1200. It should be understood that the same plurality of matrices (e.g., W 0 ) T , W 1 ,(W 2 ) T , W 3 ): from the upper left corner of the array to the lower right corner of the array in a step-down, step-right pattern; from the lower left corner of the array to the upper right corner of the array in a step-up, step-right pattern; or from the lower right corner of the array to the upper left corner of the array in a step-up, step-left pattern. 1 In an alternative example of forming an array, multiple matrices (eg, W) may be arranged as follows 0 ,(W 1 ) T , W 2 ,(W 3 ) T ): from the upper right corner of the array to the lower left corner of the array in a step-by-left, step-by-down pattern; from the upper left corner of the array to the lower right corner of the array in a step-by-right, step-by-down pattern; from the lower left corner of the array to the upper right corner of the array in a step-by-right, step-by-up pattern; or from the lower right corner of the array to the upper left corner of the array in a step-by-left, step-by-up pattern.

[0186] It should be understood that in steps S1106 and S1108, the first matrix and the second matrix may alternatively be determined in an "inconsistent" manner, for example, such that the rows or columns of the first matrix and the second matrix represent different types of channels (e.g., input channels or output channels). For example, when both the first layer and the second layer are convolutional layers, when executing Fig.11When the method is used, it can be predetermined that the matrices representing the coefficient sets of adjacent layers will be defined "inconsistently", and then filled accordingly by evaluating whether the input channels of the filters represented by each element in these matrices include non-zero coefficients. Specifically, a first matrix can be determined so that one or more rows of the first matrix represent one or more output channels of the first layer, and one or more columns of the first matrix represent one or more input channels of the first layer; and ("inconsistently") a second matrix can be determined so that one or more columns of the second matrix represent one or more output channels of the second layer, and one or more rows of the second matrix represent one or more input channels of the second layer. Alternatively, a first matrix can be determined so that one or more columns of the first matrix represent one or more output channels of the first layer, and one or more rows of the first matrix represent one or more input channels of the first layer; and ("inconsistently") a second matrix can be determined so that one or more rows of the second matrix represent one or more output channels of the second layer, and one or more columns of the second matrix represent one or more input channels of the second layer. It should be understood that in these "inconsistent matrix" examples, forming the array does not need to include transposing either the first matrix or the second matrix (or the third or fourth matrix, where the method is performed for a series of four adjacent layers). That is, an array including a first matrix and a second matrix may be formed by aligning rows or columns of a first matrix representing one or more output channels of a first layer with rows or columns of a second matrix representing one or more input channels of a second layer.

[0187] It should also be understood that in the example where the first layer is a convolutional layer and the second layer is a fully connected layer, forming the array may also include including an intermediate flattened matrix (not shown in the figure) between the first matrix or the transposed first matrix representing the first convolutional layer and the second matrix or the transposed second matrix representing the second fully connected layer. The intermediate flattened matrix should connect the rows or columns representing the output channels of the first convolutional layer to the rows or columns representing the input channels of the second fully connected layer in the array by taking into account the receiving window of the shape of the input tensor.

[0188] In step S1112, the rows and / or columns of the array are rearranged (e.g., reordered). By performing step S1112 on the array, each matrix included in the array can be rearranged simultaneously. By performing step S1112, the first matrix or the transposed first matrix including multiple elements representing non-zero values ​​(depending on the way the array has been formed) are aggregated into the first one or more sub-matrices, and the first one or more sub-matrices have a ratio of the number of elements representing non-zero values ​​to the total number of elements of the first one or more sub-matrices that is greater than the ratio of the number of elements representing non-zero values ​​to the total number of elements of the first matrix. In other words, the "non-zero density" of the first one or more sub-matrices as a whole is greater than the "non-zero density" of the first matrix. Similarly, the second matrix or the transposed second matrix including multiple elements representing non-zero values ​​(depending on the way the array has been formed) are aggregated into the second one or more sub-matrices, and the second one or more sub-matrices have a ratio of the number of elements representing non-zero values ​​to the total number of elements of the second matrix that is greater than the ratio of the number of elements representing non-zero values ​​to the total number of elements of the second matrix. In other words, the "non-zero density" of the second one or more sub-matrices as a whole is greater than the "non-zero density" of the second matrix. In addition, the plurality of elements representing non-zero values ​​included in the third matrix or the transposed third matrix (depending on the way the array has been formed) are clustered into a third one or more sub-matrices, the third one or more sub-matrices having a ratio of the number of elements representing non-zero values ​​to the total number of elements of the third one or more sub-matrices greater than the ratio of the number of elements representing non-zero values ​​to the total number of elements of the third matrix. In other words, the "non-zero density" of the third one or more sub-matrices as a whole is greater than the "non-zero density" of the third matrix. Similarly, the plurality of elements representing non-zero values ​​included in the fourth matrix or the transposed fourth matrix (depending on the way the array has been formed) are clustered into a fourth one or more sub-matrices, the fourth one or more sub-matrices having a ratio of the number of elements representing non-zero values ​​to the total number of elements of the fourth matrix greater than the ratio of the number of elements representing non-zero values ​​to the total number of elements of the fourth matrix. In other words, the "non-zero density" of the fourth one or more sub-matrices as a whole is greater than the "non-zero density" of the fourth matrix.

[0189] Each of the first one or more sub-matrices may have a ratio of the number of elements representing non-zero values ​​to the total number of elements of the first sub-matrix that is greater than the ratio of the number of elements representing non-zero values ​​to the total number of elements of the first matrix. In other words, the "non-zero density" of each of the first one or more sub-matrices may be greater than the "non-zero density" of the first matrix, but it is not necessary. Each of the second one or more sub-matrices may have a ratio of the number of elements representing non-zero values ​​to the total number of elements of the second sub-matrix that is greater than the ratio of the number of elements representing non-zero values ​​to the total number of elements of the second matrix. In other words, the "non-zero density" of each of the second sub-matrices in the second one or more sub-matrices may be greater than the "non-zero density" of the second matrix, but it is not necessary. Each of the third one or more sub-matrices may have a ratio of the number of elements representing non-zero values ​​to the total number of elements of the third sub-matrix that is greater than the ratio of the number of elements representing non-zero values ​​to the total number of elements of the third matrix. In other words, the "non-zero density" of each of the third one or more sub-matrices in the third one or more sub-matrices may be greater than the "non-zero density" of the third matrix, but it is not necessary. Each of the fourth one or more sub-matrices may have a ratio of the number of elements representing non-zero values ​​to the total number of elements of the fourth sub-matrix that is greater than the ratio of the number of elements representing non-zero values ​​to the total number of elements of the fourth matrix. In other words, the "non-zero density" of each of the fourth one or more sub-matrices may be greater than the "non-zero density" of the fourth matrix, but need not be.

[0190] Step S1112 may be performed for the array in a manner similar to the manner in which step S506 is performed for the matrix as described herein. That is, step S1112 may include rearranging the rows and / or columns of the array according to a hypergraph model. Using the principles described herein, a hypergraph model may be formed according to the corresponding row and column positions of each of the multiple elements representing non-zero values ​​in the array. The hypergraph model may be a row-net hypergraph model. The hypergraph model may be a column-net hypergraph model. The hypergraph model of the array may be partitioned using the principles described herein. The rows and / or columns of the array may be rearranged according to a partitioned hypergraph model. It should be understood that it is not necessary to use a hypergraph model in step S1112. For example, a hypergraph clustering algorithm or a graph partitioning algorithm may be used instead in step S1112. The rearrangement of the rows and / or columns of the array may be constrained so that the rows and / or columns of each matrix in the array may only be rearranged to row or column positions within the range of the rows and / or columns of the array that the matrix originally spanned. Figure 4 The at least one processor 404 shown in can be configured to perform step S1112.

[0191] Step S1112 can refer to Fig.13 It is understood that this figure shows an example rearranged array 1310. Fig.12The rows and columns of the array 1200 shown in FIG. Fig.13 The rearranged array 1310 shown in FIG. That is, Fig.13 Shown in different order Fig.12 By rearranging the rows and columns of array 1200 in this manner, the rows and columns of each of the following are rearranged (e.g., simultaneously): the transposed first matrix (W 0 ) T 1201-1; second matrix W 1 1200-2; the transposed third matrix (W 2 ) T 1201-3; and the fourth matrix W 3 1200-4. The rearranged matrix (ie, (W 0 ) T The rearranged transposed first matrix 1311-1, W 1 The rearranged second matrix 1310-2, (W 2 ) T The rearranged transposed third matrix 1311-3 and W 3 Each of the rearranged fourth matrices 1310-4) includes a plurality of sub-matrices, into which a plurality of its elements representing non-zero values ​​have been aggregated. Fig.13 The matrix shown in (i.e., (W 0 ) T , W 1 ,(W 2 ) T , W 3 ) has been converted into a boundary block matrix form. Specifically, Fig.13 In the matrix (i.e., (W 0 ) T , W 1 ,(W 2 ) T , W 3 ) has been converted into a single-boundary block-diagonal matrix form. This can be done by rearranging according to the column-net hypergraph model as described herein Fig.12 The rows and columns of the array 1200 are implemented. Fig.12 A column-net hypergraph model formed by array 1200 and partitioned using the principles described herein may be used to individually convert each of the matrices into a single-boundary block-diagonal matrix form.

[0192] As described herein, in step S1110, an array is formed so that a series of elements (e.g., rows or columns) representing one or more output channels of a first layer is aligned with a series of elements (e.g., rows or columns) representing one or more corresponding input channels of a second subsequent layer (i.e., contained in the same row or column of the array), which means that in step S1112, when the rows and / or columns of the array are rearranged (e.g., reordered), the series of elements (e.g., rows or columns) representing each output channel of the first layer remains aligned with a series of elements (e.g., rows or columns) representing the corresponding, corresponding input channels of the second subsequent layer (i.e., contained in the same row or column of the array). This enables the output-input dependency between the first layer and the second layer to be maintained through step S1112. More generally, by applying these principles to form and rearrange arrays, it is possible to maintain the reference operations performed thereon. Fig.11 The method described herein has output-input dependencies between each pair of adjacent layers in a series of two or more adjacent layers.

[0193] Back to Fig.11 In step S1114, a compressed neural network is output, wherein the compressed neural network includes a first compression layer arranged to perform a first compression operation based on a first one or more sub-matrices and a second subsequent compression layer arranged to perform a second compression operation based on a second one or more sub-matrices. Figure 4 The at least one processor 404 shown in can be configured to perform step S1114. The compressed neural network may also include a third compression layer, the third compression layer being arranged to perform a third compression operation based on the third one or more sub-matrices. The compressed neural network may also include a fourth compression layer, the fourth compression layer being arranged to perform a fourth compression operation based on the fourth one or more sub-matrices.

[0194] The first compression layer is arranged to perform the same type of operation as the operation that the first layer is arranged to perform. That is, the first compression layer is arranged to perform the type of operation according to the first one or more sub-matrices, for example, instead of performing the type of operation according to the first matrix. The manner in which the first compression layer can be arranged to perform the first compression operation according to the first one or more sub-matrices can be understood with reference to the description of step S508 herein. The second compression layer is arranged to perform the same type of operation as the operation that the second layer is arranged to perform. That is, the second compression layer is arranged to perform the type of operation according to the second one or more sub-matrices, for example, instead of performing the type of operation according to the second matrix. The manner in which the second compression layer can be arranged to perform the second compression operation according to the second one or more sub-matrices can be understood with reference to the description of step S508 herein. Similarly, the (optional) third compression layer can be arranged to perform the same type of operation as the operation that the third layer is arranged to perform. That is, the third compression layer can be arranged to perform the type of operation according to the third one or more sub-matrices, for example, instead of performing the type of operation according to the third matrix. The manner in which the third compression layer can be arranged to perform the third compression operation according to the third one or more sub-matrices can be understood with reference to the description of step S508 herein. Similarly, the (optional) fourth compression layer may be arranged to perform the same type of operation as the operation that the fourth layer is arranged to perform. That is, the fourth compression layer may be arranged to perform the type of operation based on the fourth one or more sub-matrices, for example rather than performing the type of operation based on the fourth matrix. The manner in which the fourth compression layer may be arranged to perform the fourth compression operation based on the fourth one or more sub-matrices may be understood with reference to the description herein of step S508.

[0195] For reference in this article Figure 5 For all the same reasons as the method described for compressing neural networks, the paper referenced Fig.11 The described method of compressing a neural network is advantageous. Fig.11 The described method of compressing a neural network is further advantageous because the output of a first compressed layer of two or more adjacent layers can be directly (e.g., without any intermediate rearrangement, such as a gather or scatter operation) input into a second subsequent compressed layer of the pair of adjacent layers. This is because the output-input dependencies between those adjacent layers are maintained by performing steps S1100 and S1112 described herein. Similarly, for the output of a first compressed layer of two or more adjacent layers according to Fig.11 In a series of more than four layers compressed by the method, the output of the second compression layer of two or more adjacent layers can be directly (e.g., without any intermediate rearrangement, such as a gathering or scattering operation) input into a third subsequent compression layer; and the output of the third compression layer of two or more adjacent layers can be directly (e.g., without any intermediate rearrangement, such as a gathering or scattering operation) input into a fourth subsequent compression layer; and so on.

[0196] For example, considering only the first and second layers from now on, in a first example, both the first and second layers may be fully connected layers or any other type of layers arranged to perform matrix multiplication.

[0197] In a first example, a first layer of the received neural network may be arranged to perform a first operation by performing matrix multiplication using a first matrix including a set of coefficients of the first layer and a first input matrix including a set of input activation values ​​of the first layer. The compression neural network may be configured such that the first compression layer is arranged to perform a first compression operation by performing one or more matrix multiplications using a first one or more sub-matrices and one or more subsets of the set of coefficients of the first layer included by the one or more first input sub-matrices, the one or more first input sub-matrices each including a corresponding subset of the set of input activation values ​​of the first layer. Similarly, in the first example, a second layer of the received neural network may be arranged to perform a second operation by performing matrix multiplication using a second matrix including a set of coefficients of the second layer and a second input matrix including a set of input activation values ​​of the second layer. The compression neural network is configured such that the second compression layer is arranged to perform a second compression operation by performing one or more matrix multiplications using a second one or more sub-matrices and one or more subsets of the set of coefficients of the second layer included by the one or more second input sub-matrices, the one or more second input sub-matrices each including a corresponding subset of the set of input activation values ​​of the second layer.

[0198] In a first example, in step S1114, the first compression layer may be arranged to perform a first compression operation to form one or more first output sub-matrices including a set of output activation values ​​of the first compression layer, wherein the one or more first output sub-matrices of the first compression layer are one or more second input sub-matrices of the second compression layer. That is, in step S1114, it may not be necessary to include a scattering layer after the first compression layer (e.g., as described herein) or a gathering layer before the second compression layer (e.g., as described herein) within the compression neural network. This is because, by performing steps S1100 and S1112 as described herein to maintain output-input dependencies between adjacent layers, the output of the first compression layer may be directly (i.e., without any intermediate rearrangement) input into the second compression layer.

[0199] This can refer to Fig.14 Understand that this figure shows the output activation values ​​formed by a first compression layer from multiple sub-matrices used as input activation values ​​for a second subsequent compression layer. Fig.14 A first rearranged matrix 1410-1 including a first plurality of sub-matrices is shown. The first rearranged matrix 1410-1 can be obtained by transposing Fig.13To perform the first compression operation, a set of matrix multiplications may be performed between the sub-matrices of the first rearranged matrix 1410-1 and the plurality of input sub-matrices 1404-A to form a plurality of output sub-matrices 1404-B, for example, as described herein with reference to Fig. 9 As described. Fig.14 As shown in , the plurality of output sub-matrices 1404-B output by the first compression layer are in the correct format to be directly input (ie, without any intermediate gather or scatter operations) to the second compression layer as the plurality of input sub-matrices 1404-B. Fig.14 A second rearranged matrix 1410-2 including a second plurality of sub-matrices is shown. The second rearranged matrix 1410-2 may be Fig.13 To perform the second compression operation, a set of matrix multiplications may be performed between the sub-matrices of the second rearranged matrix 1410-2 and the plurality of input sub-matrices 1404-B to form a plurality of output sub-matrices 1404-C, for example, as described herein with reference to Fig.10 described.

[0200] Alternatively, from now on again only the first and the second layer are considered, which in the second example can both be convolutional layers.

[0201] In a second example, a first convolutional layer of the received neural network may be arranged to perform a first operation by convolving a set of input activation values ​​of the first convolutional layer with a set of coefficients of the first convolutional layer. Each of the first one or more sub-matrices includes a plurality of elements representing a corresponding subset of input channels of filters of the coefficient set of the first convolutional layer. The compression neural network may be configured such that the first compression layer is arranged to perform a first compression operation by convolving one or more subsets of input activation values ​​of the first convolutional layer with a subset of the coefficient set of the first convolutional layer included by one or more subsets of input channels of filters represented by elements in the first one or more sub-matrices. Similarly, in a second example, a second convolutional layer of the received neural network may be arranged to perform a second operation by convolving a set of input activation values ​​of the second convolutional layer with a set of coefficients of the second convolutional layer. Each of the second one or more sub-matrices includes a plurality of elements representing a corresponding subset of input channels of filters of the coefficient set of the second convolutional layer. The compression neural network can be configured such that the second compression layer is arranged to perform a second compression operation by convolving one or more subsets of the input activation values ​​of the second convolutional layer with a subset of the set of coefficients of the second convolutional layer included in one or more subsets of the input channels of the filters represented by elements in the one or more sub-matrices.

[0202] In a second example, in step S1114, the first compression layer may be arranged to perform a first compressed convolution operation to form one or more subsets of output activation data of the first compression layer, wherein the one or more subsets of output activation data of the first compression layer are one or more subsets of input activation data of the second compression layer. That is, in step S1114, it may not be necessary to include a scattering layer after the first compression layer (e.g., as described herein) or a gathering layer before the second compression layer (e.g., as described herein) within the compressed neural network. This is because, by performing steps S1100 and S1112 as described herein to maintain output-input dependencies between adjacent layers, the output of the first compression layer may be directly (i.e., without any intermediate rearrangement) input into the second compression layer.

[0203] For example, the first compressed convolution layer can be arranged to perform a first compressed convolution operation according to equations (5) and (6) as described herein to form multiple subsets of output activation data (e.g., output activation data Y 1 , Y 2 , Y 3 , Y 4 The second compressed convolutional layer may be arranged to use multiple subsets of the output activation data output by the first compressed convolutional layer (e.g., output activation data Y 1 , Y 2 , Y 3 , Y 4 As a subset of the input activation data of the second compressed convolutional layer, multiple subsets of the input activation data (for example, the input activation data X 1 , X 2 , X 3 , X 4 A subset of ), performing a second compressed convolution operation according to equations (7) and (8) as described herein.

[0204] Step S1114 may include storing the compressed neural network for subsequent implementation. Figure 4 , at least one processor 404 may write the compressed neural network into the memory 406 for subsequent implementation. Step S1114 may include outputting a computer-readable description of the compressed neural network, which is useful when used in a system for implementing the neural network (e.g., Figure 3 When implemented at the system 300 shown in FIG. 11 , the compressed neural network is executed. Step S1114 may include configuring the system for implementing the neural network to implement the compressed neural network.

[0205] Step S1114 may include configuring hardware logic to implement the compressed neural network. The hardware logic may include a neural network accelerator. For example, referring to Figure 4, at least one processor 404 can be configured to configure hardware logic included in the system 300 for implementing a neural network to implement a compressed neural network.

[0206] The compressed neural network output in step S1114 may be used. The compressed neural network output in step S1114 may be used to perform image processing. As a non-limiting example, the compressed neural network may be used to perform one or more of image super-resolution processing, semantic image segmentation processing, and object detection. For example, performing image super-resolution processing involves compressing the neural network to process a lower resolution image input to the neural network so as to output a higher resolution image.

[0207] exist Fig.12 and Fig.13 In the example shown, a series of four adjacent layers are executed Fig.11 The method of claim 1, wherein the series of four adjacent layers comprises: a third layer of the received neural network adjacent to and after the second layer; and a fourth layer of the received neural network adjacent to and after the third layer. This is not necessarily the case. Typically, the larger the number of matrices contained in the array to be rearranged in step S1112, the more restricted the rearrangement process, and therefore the smaller the "non-zero density" gain for each matrix included in the array. Therefore, it may be as referred to herein Fig.11 The method described herein selects one or more groups of adjacent layers of a received neural network for compression by forming and rearranging a separate array for each group of adjacent layers, wherein the layers are compressed according to Fig.11 The result of step S1112 is to "intersperse" other layers between those groups of adjacent layers in an alternative rearrangement, as described below.

[0208] In the following, two examples are given, in which Fig.11 The method is performed on the first and second layers as described herein, but not on the third and fourth layers as described herein. Fig.11 The method of rearranging the first spreading layer after the second layer (for example, the third layer described previously) and the second spreading layer before the first layer can be based on Fig.11 As a result of step S1112 , the first scattering layer and / or the second scattering layer are rearranged in an alternative manner with respect to the first layer and the second layer.

[0209] For example, a first distributed layer of the received neural network may be selected, the first distributed layer being after and adjacent to a second layer within the received neural network. The first distributed layer of the received neural network may be arranged to perform a first distributed operation. The first distributed layer may have one or more input channels corresponding to one or more output channels of the second layer. The second layer and the first distributed layer may both be arranged to perform the same type of operation. In a first example, the second layer and the first distributed layer may both be fully connected layers. In a second example, the second layer and the first distributed layer may both be convolutional layers.

[0210] A first scatter matrix representing a set of coefficients for a first scatter layer may be determined. The first scatter matrix may include a plurality of elements representing non-zero values ​​and a plurality of elements representing zero values. Figure 5 A first scatter matrix is ​​determined for the first scatter layer in a similar manner as the matrix is ​​determined for the selected layer in step S504. One or more rows or columns of the first scatter matrix may represent one or more input channels of the first scatter layer.

[0211] Can be based on Fig.11 The rearranged second matrix is ​​determined from the rearranged array formed in step S1112. The rearranged second matrix may be determined by any of the following: identifying the rearranged second matrix within the rearranged array (e.g., in an example of forming an array including the second matrix); or identifying the transposed second matrix rearranged within the rearranged array (e.g., in an example of forming an array including the transposed second matrix), and transposing the rearranged transposed second matrix.

[0212] The rows or columns of the first scatter matrix may be rearranged (e.g., reordered) so that the order of the one or more rows or columns of the first scatter matrix representing one or more input channels of the first scatter layer corresponds to the order of the one or more rows or columns of the rearranged second matrix representing one or more output channels of the second layer. That is, in order to aggregate the plurality of elements representing non-zero values ​​included in the first scatter matrix into one or more sub-matrices, it is not necessary to rearrange the rows or columns of the first scatter matrix. Instead, the rows or columns of the first scatter matrix may be rearranged (e.g., reordered) so that for 1 to N, the Nth row or column of the first scatter matrix represents an input channel of the coefficient set of the first scatter layer, and the input channel is responsible for operating on a channel of output activation data formed by an output channel of the coefficient set of the second layer represented by the Nth row or column of the rearranged second matrix.

[0213] The compressed neural network may be output, the output comprising a first scatter layer arranged to perform a first scatter operation according to a rearranged first scatter matrix. Rearranging the rows or columns of the first scatter matrix as described herein is advantageous because the output of the second compressed layer may be directly (e.g., without requiring any intermediate rearrangement, such as a gathering or scattering operation) input into the first scatter layer. This is because by rearranging the rows or columns of the first scatter matrix as described herein, the output-input dependency between the second scatter layer and the first scatter layer is maintained.

[0214] The first scatter layer of the received neural network may be selected for this alternative rearrangement based on the number of elements representing non-zero values ​​out of the total number of elements of the first scatter matrix exceeding a threshold. In other words, when the "non-zero density" of the first scatter matrix exceeds a "non-zero density" threshold. This is because one or more sub-matrices formed by aggregating multiple elements representing non-zero values ​​within a matrix that already has a high "non-zero density" may not have a significantly higher "non-zero value density" than the matrix itself (if any).

[0215] Alternatively, for this alternative rearrangement, one or more layers of the received neural network (e.g., including the first distributed layer) may be randomly selected or selected according to a predetermined pattern (e.g., every Nth layer) to reduce the number of layers. Fig.11 The constraints of rearranging step S1112.

[0216] Similarly, a second distributed layer of the received neural network may be selected that precedes the first layer within the received neural network and is adjacent to the first layer. The second distributed layer of the received neural network may be arranged to perform a second distributed operation. The second distributed layer may have one or more output channels corresponding to one or more input channels of the first layer. The second distributed layer and the first layer may both be arranged to perform the same type of operation. In a first example, the second distributed layer and the first layer may both be fully connected layers. In a second example, the second distributed layer and the first layer may both be convolutional layers.

[0217] A second scatter matrix representing a set of coefficients for a second scatter layer may be determined. The second scatter matrix may include a plurality of elements representing non-zero values ​​and a plurality of elements representing zero values. Figure 5 A second scatter matrix is ​​determined for the second scatter layer in a similar manner as the matrix was determined for the selected layer in step S504. One or more rows or columns of the second scatter matrix may represent one or more output channels of the second scatter layer.

[0218] Can be based on Fig.11The rearranged first matrix is ​​determined from the rearranged array formed in step S1112. The rearranged first matrix may be determined by any of the following: identifying the rearranged first matrix within the rearranged array (e.g., in an example of forming an array including the first matrix); or identifying the transposed first matrix rearranged within the rearranged array (e.g., in an example of forming an array including the transposed first matrix), and transposing the rearranged transposed first matrix.

[0219] The rows or columns of the second scatter matrix may be rearranged (e.g., reordered) so that the order of the one or more rows or columns of the second scatter matrix representing one or more output channels of the second scatter layer corresponds to the order of the one or more columns or rows of the rearranged first matrix representing one or more input channels of the first layer. That is, in order to aggregate the plurality of elements representing non-zero values ​​included in the second scatter matrix into one or more sub-matrices, it is not necessary to rearrange the rows or columns of the second scatter matrix. Instead, the rows or columns of the second scatter matrix may be rearranged (e.g., reordered) so that for 1 to N, the Nth row or column of the second scatter matrix represents an output channel of the coefficient set of the second scatter layer, the output channel being responsible for forming a channel of output activation data that operates on the input channel of the coefficient set of the first layer represented by the Nth column or row of the rearranged first matrix.

[0220] The compressed neural network may be output, the output comprising a second scatter layer arranged to perform a second scatter operation according to a rearranged second scatter matrix. Rearranging the rows or columns of the second scatter matrix as described herein is advantageous because the output of the second scatter layer may be directly (e.g., without requiring any intermediate rearrangement, such as a gathering or scattering operation) input into the first compressed layer. This is because by rearranging the rows or columns of the second scatter matrix as described herein, the output-input dependency between the second scatter layer and the first layer is maintained.

[0221] The second scatter layer of the received neural network may be selected for this alternative rearrangement based on the number of elements representing non-zero values ​​out of the total number of elements of the second scatter matrix exceeding a threshold. In other words, when the "non-zero density" of the second scatter matrix exceeds a "non-zero density" threshold. This is because one or more sub-matrices formed by aggregating multiple elements representing non-zero values ​​within a matrix that already has a high "non-zero density" may not have a significantly higher "non-zero value density" than the matrix itself (if any).

[0222] Alternatively, for this alternative rearrangement, one or more layers of the received neural network (e.g., including the second distributed layer) may be randomly selected or selected according to a predetermined pattern (e.g., every Nth layer) to reduce the number of layers. Fig.11The constraints of rearranging step S1112.

[0223] It should be understood that a layer of the received neural network may rearrange its input channels (e.g., as described herein with reference to the first scatter matrix) so as to correspond to the input channels of the received neural network. Fig.11 or Figure 5 and rearranges its output channels (e.g., as described herein with reference to the second scatter matrix) so as to correspond to the output channels of the previously adjacent layers of the method that have been performed. Fig.11 or Figure 5 In this way, the input channels of the subsequent adjacent layers of the method can be as described in this article. Fig.11 The method described herein selects one or more groups of adjacent layers of a received neural network for compression by forming and rearranging a separate array for each group of adjacent layers, wherein the layers are compressed according to Fig.11 The result of step S1112 (and / or according to Figure 5 Other layers are "interspersed" between those groups of adjacent layers that are rearranged in an alternative manner described herein (as a result of step S506), as described below.

[0224] For example, Fig.15 The fully connected operations performed by three adjacent layers of a compressed neural network according to the principles described herein are shown. The first layer is arranged to perform a first compression operation, the second (e.g., spread) layer is arranged to perform a second spread operation, and the third layer is arranged to perform a third compression operation. Fig.15 A first rearranged matrix 1510-A is shown (eg, according to Figure 5 Step S506 or Fig.11 To perform the first compression operation, a set of matrix multiplications may be performed between the sub-matrices of the first rearranged matrix 1510-A and the plurality of input sub-matrices 1504-A to form a plurality of output sub-matrices 1504-B, for example, as described herein with reference to Fig. 9 The plurality of output sub-matrices 1504-B output by the first compression layer are in the correct format to be used as Fig.15 Marked with X 1 , X 2 and X 3 The multiple input sub-matrices 1504-B are directly input (ie, without any intermediate gathering or scattering operations) to the second (eg, scattering) layer. Fig.15A second rearranged matrix 1510-B is shown, which has its columns representing one or more input channels of the second layer reordered to correspond to the row order of the first rearranged matrix 1510-A representing one or more output channels of the first layer, and has its rows representing one or more output channels of the second layer reordered to correspond to the column order of the third rearranged matrix 1510-C representing one or more input channels of the third layer. To perform the second scatter operation, it can be Fig.15 Marked as W 1,1 , W 1,2 , W 1,3 , W 2,1 , W 2,2 , W 2,3 , W 3,1 , W 3,2 and W 3,3 The coefficient blocks of the second rearranged matrix 1510-B are the same as in Fig.15 Marked with X 1 , X 2 and X 3 A set of matrix multiplications is performed between the plurality of input sub-matrices 1504-B to form Fig.15 Marked as Y 1 , Y 2 and Y 3 Multiple output sub-matrices 1504-C of coefficient block W 1,1 , W 2,1 , W 3,1 The number of columns included can be equal to the input submatrix X 1 The number of rows included. Coefficient block W 1,2 , W 2,2 , W 3,2 The number of columns included can be equal to the input submatrix X 2 The number of rows included. Coefficient block W 1,3 , W 2,3 , W 3,3 The number of columns included can be equal to the input submatrix X 3 The number of rows included. Coefficient block W 1,1 , W 1,2 , W 1,3 The number of rows included can be equal to the output submatrix Y 1 The number of rows included. Coefficient block W 2,1 , W 2,2 , W 2,3 The number of rows included can be equal to the output submatrix Y 2 The number of rows included. Coefficient block W 3,1 , W 3,2 , W 3,3 The number of rows included can be equal to the output submatrix Y 3For example, the second scatter operation can be performed according to the following general formula, where K X represents the number of subsets of the input data X, and K Y Represents the number of subsets in the output data Y:

[0225] where each i=1,2…K X (9)

[0226] exist Fig.15 Marked as Y 1 , Y 2 , Y 3 The multiple output sub-matrices 1504-C output by the second (e.g., scattering) layer are in the correct format to be directly input (i.e., without any intermediate gathering or scattering operations) to the third compression layer as the multiple input sub-matrices 1504-C. Fig.15 A third rearranged matrix 1510-C is shown (eg, according to Figure 5 Step S506 or Fig.11 To perform the third compression operation, a set of matrix multiplications may be performed between the sub-matrices of the third rearranged matrix 1510-C and the plurality of input sub-matrices 1504-C to form a plurality of output sub-matrices 1504-D, for example, as described herein with reference to Fig.10 described.

[0227] It should also be understood that technicians will have no difficulty referring to this article Fig.15 The described principles are applied to intersperse the convolutional layers rearranged in the alternative way described in this paper over the layers that have been trained according to Figure 5 or Fig.11 The method compresses the two convolutional layers.

[0228] There is a synergy between the method of compressing a neural network described herein and implementing a compressed neural network in hardware, i.e., by configuring the hardware logic including a neural network accelerator (NNA) to implement the compressed neural network. This is because the method of compressing a neural network is intended to improve the implementation of a compressed neural network at such a system, where the coefficient sets will be stored in an off-chip memory, and the layers of the compressed neural network will be executed by reading those coefficient sets from the off-chip memory into the hardware logic including the neural network accelerator (NNA) at runtime. That is, the method described herein is particularly advantageous when used for compressing a neural network to be implemented in hardware.

[0229] Fig.17 17 shows a computer system that can implement the neural network system described herein. The computer system includes a CPU 1702, a GPU 1704, a memory 1706 (e.g., equivalent to Figure 3), a neural network accelerator (NNA) 1708 (e.g., equivalent to Figure 3 ) and other devices 1714, such as a display 1716, a speaker 1718, and a camera 1722 (e.g., equivalent to Figure 3 1708). In some examples, camera 1722 may input activation data directly into neural network accelerator (NNA) 1708. In some examples, the NNA may be a GPU with one or more hardware accelerators for accelerating one or more neural network operations. It should be understood that one or more of the depicted components may be omitted from the system. The components of the computer system may communicate with each other via communication bus 1720.

[0230] Figure 3 , Figure 4 and Fig.17 The system is shown as including a plurality of functional blocks. This is merely schematic and is not intended to define a strict division between different logical elements of such entities. Each functional block may be provided in any suitable manner. It should be understood that the intermediate values ​​described herein as being formed by at least one processor need not be physically generated by at least one processor at any point, and may merely represent logical values ​​that conveniently describe the processing performed by at least one processor between its input and output.

[0231] The processing system described herein can be embodied in hardware on an integrated circuit. The processing system described herein can be configured to perform any of the methods described herein. In general, any of the functions, methods, techniques or components described above can be implemented in software, firmware, hardware (e.g., fixed logic circuits) or any combination thereof. The terms "module", "functionality", "component", "element", "unit", "block" and "logic" can be used herein to generally represent software, firmware, hardware or any combination thereof. In the case of software implementation, a module, functionality, component, element, unit, block or logic represents a program code, which performs a specified task when executed on a processor. The algorithms and methods described herein can be performed by one or more processors that execute code, which prompts the processor to perform the algorithm / method. Examples of computer-readable storage media include random access memory (RAM), read-only memory (ROM), optical disk, flash memory, hard disk storage, and other memory devices that can use magnetic, optical and other technologies to store instructions or other data and can be accessed by a machine.

[0232] As used herein, the terms computer program code and computer readable instructions refer to any kind of executable code for a processor, including code expressed in machine language, interpreted language, or scripting language. Executable code includes binary code, machine code, byte code, code that defines an integrated circuit (such as a hardware description language or netlist), and code expressed in programming language code such as C, Java, or OpenCL. Executable code can be, for example, any kind of software, firmware, script, module, or library that, when properly executed, processed, interpreted, compiled, run in a virtual machine or other software environment, causes a processor of a computer system supporting the executable code to perform the tasks specified by the code.

[0233] A processor, computer or computer system may be any kind of device, machine or special purpose circuit, or a collection or part thereof, which has processing capabilities so that instructions can be executed. A processor may be or include any kind of general or special purpose processor, such as a CPU, GPU, NNA, system on chip, state machine, media processor, application specific integrated circuit (ASIC), programmable logic array, field programmable gate array (FPGA), etc. A computer or computer system may include one or more processors.

[0234] The present invention is also intended to cover software (e.g., HDL (Hardware Description Language) software) that defines the configuration of hardware as described herein, which is used to design integrated circuits or to configure programmable chips to perform desired functions. That is, a computer-readable storage medium may be provided on which is encoded a computer-readable program code in the form of an integrated circuit definition data set, which, when processed (i.e., run) in an integrated circuit manufacturing system, configures the system to manufacture a processing system configured to perform any of the methods described herein, or to manufacture a processing system that includes any of the devices described herein. The integrated circuit definition data set may be, for example, an integrated circuit description.

[0235] Thus, a method of manufacturing a processing system as described herein at an integrated circuit manufacturing system may be provided. Furthermore, an integrated circuit definition data set may be provided which, when processed in an integrated circuit manufacturing system, causes the method of manufacturing a processing system to be performed.

[0236] The integrated circuit definition data set may be in the form of computer code, for example as a netlist, code for configuring a programmable chip, as a hardware description language that defines hardware suitable for fabrication at any level in an integrated circuit, including as register transfer level (RTL) code, as a high-level circuit representation such as Verilog or VHDL, and as a low-level circuit representation such as OASIS (RTM) and GDSII. The higher-level representation that logically defines hardware suitable for fabrication in an integrated circuit (e.g., RTL) may be processed at a computer system configured to generate a manufacturing definition of the integrated circuit in the context of a software environment that includes definitions of circuit elements and rules for combining those elements to generate a manufacturing definition of the integrated circuit so defined by the representation. As is typically the case with software executed at a computer system to define a machine, one or more intermediate user steps (e.g., providing commands, variables, etc.) may be required to configure the computer system to generate a manufacturing definition of the integrated circuit to execute code that defines the integrated circuit to generate the manufacturing definition of the integrated circuit.

[0237] Now refer to Fig.18 An example of processing an integrated circuit definition data set at an integrated circuit manufacturing system to configure the system into a manufacturing processing system is described.

[0238] Fig.18 An example of an integrated circuit (IC) manufacturing system 1802 is shown, which is configured to manufacture a processing system as described in any of the examples herein. Specifically, the IC manufacturing system 1802 includes a layout processing system 1804 and an integrated circuit production system 1806. The IC manufacturing system 1802 is configured to receive an IC definition data set (e.g., defining a processing system as described in any of the examples herein), process the IC definition data set, and produce an IC (e.g., which embodies a processing system as described in any of the examples herein) based on the IC definition data set. The processing of the IC definition data set configures the IC manufacturing system 1802 to manufacture an integrated circuit embodying a processing system as described in any of the examples herein.

[0239] The layout processing system 1804 is configured to receive and process an IC definition data set to determine a circuit layout. Methods for determining a circuit layout based on an IC definition data set are known in the art and may, for example, involve synthesizing RTL code to determine a gate-level representation of a circuit to be generated, for example, in terms of logic components (e.g., NAND, NOR, AND, OR, MUX, and FLIP-FLOP components). By determining location information for the logic components, the circuit layout may be determined by the gate-level representation of the circuit. This may be done automatically or with user involvement in order to optimize the circuit layout. When the layout processing system 1804 has determined the circuit layout, the layout processing system may output the circuit layout definition to the IC generation system 1806. The circuit layout definition may be, for example, a circuit layout description.

[0240] As is known in the art, IC generation system 1806 generates an IC according to the circuit layout definition. For example, IC generation system 1806 can implement a semiconductor device manufacturing process to generate an IC, which process can involve a multi-step sequence of photolithography and chemical processing steps during which electronic circuits are gradually formed on a wafer made of semiconductor material. The circuit layout definition can be in the form of a mask that can be used in a photolithography process to generate an IC according to the circuit definition. Alternatively, the circuit layout definition provided to IC generation system 1806 can be in the form of a computer readable code that IC generation system 1806 can use to form a suitable mask for generating an IC.

[0241] The different processes performed by the IC manufacturing system 1802 may all be implemented in one location, for example, by one party. Alternatively, the IC manufacturing system 1802 may be a distributed system such that some processes may be performed at different locations and may be performed by different parties. For example, some of the following stages may be performed at different locations and / or by different parties: (i) synthesizing RTL code representing an IC definition data set to form a gate-level representation of a circuit to be produced; (ii) generating a circuit layout based on the gate-level representation; (iii) forming a mask based on the circuit layout; and (iv) using the mask to manufacture the integrated circuit.

[0242] In other examples, processing of an integrated circuit definition data set at an integrated circuit manufacturing system may configure the system to manufacture a processing system without processing the IC definition data set to determine a circuit layout. For example, the integrated circuit definition data set may define a configuration of a reconfigurable processor such as an FPGA, and processing of the data set may configure the IC manufacturing system to produce a reconfigurable processor having the defined configuration (e.g., by loading the configuration data into the FPGA).

[0243] In some embodiments, when processed in an integrated circuit manufacturing system, the integrated circuit manufacturing definition data set may cause the integrated circuit manufacturing system to produce an apparatus as described herein. Fig.18 An integrated circuit manufacturing system configured in the manner described may manufacture the apparatus as described herein.

[0244] In some examples, the integrated circuit definition data set may include software that runs on, or in combination with, hardware defined at the data set. Fig.18 In the example shown, the IC production system can be further configured by the integrated circuit definition dataset to load firmware onto the integrated circuit according to the program code defined at the integrated circuit definition dataset when manufacturing the integrated circuit, or otherwise provide the integrated circuit with program code for use with the integrated circuit.

[0245] When compared to known implementations, the implementation of the concepts set forth in this application in devices, equipment, modules and / or systems (and in the methods implemented herein) may result in performance improvements. Performance improvements may include one or more of improved computing performance, reduced latency, increased throughput and / or reduced power consumption. During the manufacture of such devices, equipment, modules and systems (e.g., in integrated circuits), a trade-off may be made between performance improvements and physical implementations to improve manufacturing methods. For example, a trade-off may be made between performance improvements and layout area to match the performance of known implementations, but using less silicon. For example, this may be accomplished by reusing functional blocks in a serial manner or sharing functional blocks between elements of devices, equipment, modules and / or systems. In contrast, the concepts set forth in this application that result in improvements in the physical implementations of devices, equipment, modules and systems (e.g., reduced silicon area) may be weighed against performance improvements. This may be accomplished, for example, by manufacturing multiple instances of a module within a predefined area budget.

[0246] Applicants hereby independently disclose each individual feature described herein and any combination of two or more such features, to the extent that such feature or combination can be implemented based on the specification as a whole in view of the common general knowledge of a person skilled in the art, regardless of whether such feature or combination of features solves any problem disclosed herein. In view of the foregoing description, it will be clear to a person skilled in the art that various modifications can be made within the scope of the invention.

Claims

1. A computer-implemented method of compressing a neural network, the method comprising: receiving neural network; selecting two or more adjacent layers of the received neural network, each of the two or more adjacent layers having one or more input channels and one or more output channels, the one or more output channels of a first layer of the two or more adjacent layers corresponding to the one or more input channels of a second subsequent layer of the two or more adjacent layers, the first layer being arranged to perform a first operation and the second layer being arranged to perform a second operation; determining a first matrix representing a set of coefficients of the first layer of the received neural network, the first matrix comprising a plurality of elements representing non-zero values ​​and a plurality of elements representing zero values, one or more rows or columns of the first matrix representing the one or more output channels of the first layer, and one or more other rows or columns of the rows or columns of the first matrix representing the one or more input channels of the first layer; determining a second matrix representing a set of coefficients of the second layer of the received neural network, the second matrix comprising a plurality of elements representing non-zero values ​​and a plurality of elements representing zero values, one or more rows or columns of the second matrix representing the one or more output channels of the second layer, and one or more other rows or columns of the rows or columns of the second matrix representing the one or more input channels of the second layer; Form an array in one of the following ways: transposing the first matrix and forming the array comprising the transposed first matrix and the second matrix by aligning the columns or rows of the transposed first matrix representing the one or more output channels of the first layer with the columns or rows of the second matrix representing the one or more input channels of the second layer; or transposing the second matrix and forming the array comprising the transposed second matrix and the first matrix by aligning the row or column of the transposed second matrix representing the one or more input channels of the second layer with the row or column of the first matrix representing the one or more output channels of the first layer; or forming the array including the first matrix and the second matrix by aligning the rows or columns of the first matrix representing the one or more output channels of the first layer with the rows or columns of the second matrix representing the one or more input channels of the second layer; Rearranging the rows and / or columns of the array so as to: aggregating the plurality of elements representing non-zero values ​​included in the first matrix or the transposed first matrix into first one or more sub-matrices, the first one or more sub-matrices having a ratio of the number of elements representing non-zero values ​​to the total number of elements of the first one or more sub-matrices that is greater than a ratio of the number of elements representing non-zero values ​​to the total number of elements of the first matrix; as well as aggregating the plurality of elements representing non-zero values ​​included in the second matrix or the transposed second matrix into a second one or more sub-matrices, the second one or more sub-matrices having a ratio of the number of elements representing non-zero values ​​to the total number of elements of the second one or more sub-matrices that is greater than a ratio of the number of elements representing non-zero values ​​to the total number of elements of the second matrix; as well as Outputting a compressed neural network, the compressed neural network comprising a first compression layer arranged to perform a first compression operation based on the first one or more sub-matrices, and a second subsequent compression layer arranged to perform a second compression operation based on the second one or more sub-matrices.

2. The method according to claim 1, wherein: each of the first one or more sub-matrices has a ratio of the number of elements representing non-zero values ​​to the total number of elements of the first sub-matrix that is greater than a ratio of the number of elements representing non-zero values ​​to the total number of elements of the first matrix; and / or Each of the second one or more sub-matrices has a ratio of the number of elements representing non-zero values ​​to the total number of elements of the second sub-matrix that is greater than a ratio of the number of elements representing non-zero values ​​to the total number of elements of the second matrix.

3. The method according to claim 1 or 2, wherein: The first matrix comprises the coefficient set of the first layer, the plurality of elements representing non-zero values ​​are a plurality of non-zero coefficients, the plurality of elements representing zero values ​​are a plurality of zero coefficients, and the one or more first sub-matrices comprise a subset of the coefficient set of the first layer; and / or The second matrix includes the coefficient set of the second layer, the multiple elements representing non-zero values ​​are multiple non-zero coefficients, the multiple elements representing zero values ​​are multiple zero coefficients, and the one or more second sub-matrices include subsets of the coefficient set of the second layer.

4. The method according to claim 3, wherein: the first layer of the received neural network being arranged to perform the first operation by performing matrix multiplications using the first matrix comprising the set of coefficients of the first layer and a first input matrix comprising a set of input activation values ​​of the first layer, and the compressed neural network being configured such that the first compressed layer is arranged to perform the first compressed operation by performing one or more matrix multiplications using one or more subsets of the set of coefficients of the first layer comprised by the first one or more sub-matrices and one or more first input sub-matrices, each of the one or more first input sub-matrices comprising a respective subset of the set of input activation values ​​of the first layer; and / or The second layer of the received neural network is arranged to perform the second operation by performing matrix multiplication using the second matrix including the set of coefficients of the second layer and a second input matrix including the set of input activation values ​​of the second layer, and the compressed neural network is configured such that the second compressed layer is arranged to perform the second compressed operation by performing one or more matrix multiplications using the second one or more sub-matrices and one or more subsets of the set of coefficients of the second layer included by one or more second input sub-matrices, each of the one or more second input sub-matrices including a corresponding subset of the set of input activation values ​​of the second layer.

5. A method according to claim 4, wherein the first compression layer is arranged to perform the first compression operation to form one or more first output sub-matrices, each of which includes a corresponding subset of the output activation values ​​of the first compression layer, and the one or more first output sub-matrices of the first compression layer are the one or more second input sub-matrices.

6. The method of claim 1 or 2, wherein the first layer of the received neural network is a first convolutional layer comprising a set of coefficients arranged in one or more filters, each of the one or more filters being arranged in one or more input channels, each input channel of each filter comprising a respective subset of the set of coefficients of the first convolutional layer, and wherein determining the first matrix comprises: For each input channel of each filter: determining whether the input channel of the filter includes non-zero coefficients; as well as In response to determining that the input channel of the filter includes at least one non-zero coefficient, representing the input channel of the filter with elements in the first matrix representing non-zero values; or In response to determining that the input channels of the filter include only zero coefficients, the input channels of the filter are represented by elements in the first matrix representing zero values.

7. The method according to claim 6, wherein: the first convolutional layer of the received neural network being arranged to perform the first operation by convolving a set of input activation values ​​of the first convolutional layer with the set of coefficients of the first convolutional layer; The first one or more sub-matrices include a plurality of elements representing a subset of the input channels of the filter of the coefficient set of the first convolutional layer; and The compression neural network is configured such that the first compression layer is arranged to perform the first compression operation by convolving one or more subsets of the input activation values ​​of the first convolutional layer with the subset of the set of coefficients of the first convolutional layer represented by elements in the first one or more sub-matrices comprised by the one or more subsets of the input channels of the filter.

8. The method of claim 1 or 2, wherein the second layer of the received neural network is a second convolutional layer comprising a set of coefficients arranged in one or more filters, each of the one or more filters being arranged in one or more input channels, each input channel of each filter comprising a respective subset of the set of coefficients of the second convolutional layer, and wherein determining the second matrix comprises: For each input channel of each filter: determining whether the input channel of the filter includes non-zero coefficients; as well as In response to determining that the input channel of the filter includes at least one non-zero coefficient, representing the input channel of the filter with elements in the second matrix representing non-zero values; or In response to determining that the input channels of the filter include only zero coefficients, the input channels of the filter are represented by elements in the second matrix representing zero values.

9. The method according to claim 8, wherein: the second convolutional layer of the received neural network being arranged to perform the second operation by convolving the set of input activation values ​​of the second convolutional layer with the set of coefficients of the second convolutional layer; the second one or more sub-matrices comprising a plurality of elements representing a subset of the input channels of the filters of the set of coefficients of the second convolutional layer; and The compression neural network is configured such that the second compression layer is arranged to perform the second compression operation by convolving one or more subsets of the input activation values ​​of the second convolutional layer with the subset of the set of coefficients of the second convolutional layer represented by elements in the one or more sub-matrices comprised by the one or more subsets of the input channels of the filter.

10. The method according to claim 9, wherein: The first layer of the received neural network is a first convolutional layer comprising a set of coefficients arranged in one or more filters, each of the one or more filters being arranged in one or more input channels, each input channel of each filter comprising a respective subset of the set of coefficients of the first convolutional layer, and determining the first matrix comprises: For each input channel of each filter: determining whether the input channel of the filter includes non-zero coefficients; and In response to determining that the input channel of the filter includes at least one non-zero coefficient, representing the input channel of the filter with an element in the first matrix representing a non-zero value; or In response to determining that the input channels of the filter include only zero coefficients, representing the input channels of the filter with elements in the first matrix representing zero values; the first convolutional layer of the received neural network being arranged to perform the first operation by convolving a set of input activation values ​​of the first convolutional layer with the set of coefficients of the first convolutional layer; The first one or more sub-matrices include a plurality of elements representing a subset of the input channels of the filter of the coefficient set of the first convolutional layer; The compression neural network is configured such that the first compression layer is arranged to perform the first compression operation by convolving one or more subsets of input activation values ​​of the first convolutional layer with the subset of the set of coefficients of the first convolutional layer represented by elements in the first one or more sub-matrices comprised by the one or more subsets of input channels of the filter; and The first compression layer is arranged to perform the first compression operation to form one or more subsets of output activation values ​​of the first compression layer, the one or more subsets of output activation values ​​of the first compression layer being the one or more subsets of input activation values ​​of the second compression layer.

11. The method according to claim 1 or 2, further comprising: selecting a third layer of the received neural network, the third layer being arranged to perform a third operation, the third layer being after and adjacent to the second layer within the received neural network, the third layer having one or more input channels corresponding to the one or more output channels of the second layer; Determine a third matrix representing a coefficient set of the third layer, the third matrix comprising a plurality of elements representing non-zero values ​​and a plurality of elements representing zero values, one or more rows or columns of the third matrix representing the one or more output channels of the third layer, and one or more other rows or columns of the rows or columns of the first matrix representing the one or more input channels of the third layer; Wherein forming the array further comprises any of the following: transposing the third matrix and forming the array including the transposed third matrix by aligning the columns or rows of the transposed third matrix representing the one or more input channels of the third layer with the columns or rows of the second matrix or the transposed second matrix representing the one or more output channels of the second layer; or forming the array including the third matrix by aligning the columns or rows of the third matrix representing the one or more input channels of the third layer with the columns or rows of the second matrix or the transposed second matrix representing the one or more output channels of the second layer; Wherein rearranging the rows and / or columns of the array further comprises: aggregating the plurality of elements representing non-zero values ​​included in the third matrix or the transposed third matrix into third one or more sub-matrices, the third one or more sub-matrices having a ratio of the number of elements representing non-zero values ​​to the total number of elements of the third one or more sub-matrices that is greater than a ratio of the number of elements representing non-zero values ​​to the total number of elements of the third matrix; as well as The compressed neural network is output, wherein the compressed neural network further comprises a third compression layer, wherein the third compression layer is arranged to perform a third compression operation based on the third one or more sub-matrices.

12. The method according to claim 1 or 2, comprising: selecting a first scatter layer of the received neural network, the first scatter layer being arranged to perform a first scatter operation, the first scatter layer being after and adjacent to the second layer within the received neural network, the first scatter layer having one or more input channels corresponding to the one or more output channels of the second layer; determining a first scatter matrix representing a set of coefficients of the first scatter layer, the one or more columns or rows of the first scatter matrix representing the one or more input channels of the first scatter layer; The rearranged second matrix is ​​determined from the rearranged array by any of the following means: identifying the rearranged second matrix within the rearranged array; or identifying a rearranged, transposed second matrix within the rearranged array, and transposing the rearranged, transposed second matrix; rearranging the columns or rows of the first scatter matrix so that an order of the one or more columns or rows of the first scatter matrix representing the one or more input channels of the first scatter layer corresponds to an order of the one or more rows or columns of the rearranged second matrix representing the one or more output channels of the second layer; as well as The compressed neural network is outputted to include a first scatter layer arranged to perform the first scatter operation according to a rearranged first scatter matrix.

13. The method according to claim 1 or 2, comprising: selecting a second scatter layer of the received neural network, the second scatter layer being arranged to perform a second scatter operation, the second scatter layer being before and adjacent to the first layer within the received neural network, the second scatter layer having one or more output channels corresponding to the one or more input channels of the first layer; determining a second scatter matrix representing a set of coefficients for the second scatter layer, the one or more rows or columns of the second scatter matrix representing the one or more output channels of the second scatter layer; The rearranged first matrix is ​​determined from the rearranged array by any of the following means: identifying the rearranged first matrix within the rearranged array; or identifying a rearranged, transposed first matrix within the rearranged array, and transposing the rearranged, transposed first matrix; rearranging the rows or columns of the second scatter matrix so that an order of the one or more rows or columns of the second scatter matrix representing the one or more output channels of the second scatter layer corresponds to an order of the one or more columns or rows of the rearranged first matrix representing the one or more input channels of the first layer; as well as The compressed neural network is outputted to include a second scatter layer arranged to perform the second scatter operation according to a rearranged second scatter matrix.

14. The method of claim 1 or 2, further comprising storing the compressed neural network for subsequent implementation.

15. The method of claim 1 or 2, further comprising outputting a computer-readable description of the compressed neural network, the computer-readable description causing the compressed neural network to be executed when implemented at a system for implementing a neural network.

16. The method of claim 1 or 2, further comprising configuring hardware logic to implement the compressed neural network, optionally wherein the hardware logic comprises a neural network accelerator.

17. The method of claim 1 or 2, comprising using the compressed neural network to perform image processing.

18. The method according to claim 1 or 2, wherein: The one or more rows of the first matrix represent the one or more output channels of the first layer, the one or more columns of the first matrix represent the one or more input channels of the first layer, the one or more rows of the second matrix represent the one or more output channels of the second layer, and the one or more columns of the second matrix represent the one or more input channels of the second layer; or The one or more columns of the first matrix represent the one or more output channels of the first layer, the one or more rows of the first matrix represent the one or more input channels of the first layer, the one or more columns of the second matrix represent the one or more output channels of the second layer, and the one or more rows of the second matrix represent the one or more input channels of the second layer; or The one or more rows of the first matrix represent the one or more output channels of the first layer, the one or more columns of the first matrix represent the one or more input channels of the first layer, the one or more columns of the second matrix represent the one or more output channels of the second layer, and the one or more rows of the second matrix represent the one or more input channels of the second layer; or The one or more columns of the first matrix represent the one or more output channels of the first layer, the one or more rows of the first matrix represent the one or more input channels of the first layer, the one or more rows of the second matrix represent the one or more output channels of the second layer, and the one or more columns of the second matrix represent the one or more input channels of the second layer.

19. A processing system for compressing a neural network, the processing system comprising at least one processor, the at least one processor being configured to: receiving neural network; selecting two or more adjacent layers of the received neural network, each of the two or more adjacent layers having one or more input channels and one or more output channels, the one or more output channels of a first layer of the two or more adjacent layers corresponding to the one or more input channels of a second subsequent layer of the two or more adjacent layers, the first layer being arranged to perform a first operation and the second layer being arranged to perform a second operation; determining a first matrix representing a set of coefficients of the first layer of the received neural network, the first matrix comprising a plurality of elements representing non-zero values ​​and a plurality of elements representing zero values, one or more rows or columns of the first matrix representing the one or more output channels of the first layer, and one or more other rows or columns of the rows or columns of the first matrix representing the one or more input channels of the first layer; determining a second matrix representing a set of coefficients of the second layer of the received neural network, the second matrix comprising a plurality of elements representing non-zero values ​​and a plurality of elements representing zero values, one or more rows or columns of the second matrix representing the one or more output channels of the second layer, and one or more other rows or columns of the rows or columns of the second matrix representing the one or more input channels of the second layer; Form an array in one of the following ways: transposing the first matrix and forming the array comprising the transposed first matrix and the second matrix by aligning the columns or rows of the transposed first matrix representing the one or more output channels of the first layer with the columns or rows of the second matrix representing the one or more input channels of the second layer; or transposing the second matrix and forming the array comprising the transposed second matrix and the first matrix by aligning the row or column of the transposed second matrix representing the one or more input channels of the second layer with the row or column of the first matrix representing the one or more output channels of the first layer; or forming the array including the first matrix and the second matrix by aligning the rows or columns of the first matrix representing the one or more output channels of the first layer with the rows or columns of the second matrix representing the one or more input channels of the second layer; Rearranging the rows and / or columns of the array so as to: aggregating the plurality of elements representing non-zero values ​​included in the first matrix or the transposed first matrix into first one or more sub-matrices, the first one or more sub-matrices having a ratio of the number of elements representing non-zero values ​​to the total number of elements of the first one or more sub-matrices that is greater than a ratio of the number of elements representing non-zero values ​​to the total number of elements of the first matrix; as well as aggregating the plurality of elements representing non-zero values ​​included in the second matrix or the transposed second matrix into a second one or more sub-matrices, the second one or more sub-matrices having a ratio of the number of elements representing non-zero values ​​to the total number of elements of the second one or more sub-matrices that is greater than a ratio of the number of elements representing non-zero values ​​to the total number of elements of the second matrix; as well as Outputting a compressed neural network, the compressed neural network comprising a first compression layer arranged to perform a first compression operation based on the first one or more sub-matrices, and a second subsequent compression layer arranged to perform a second compression operation based on the second one or more sub-matrices.

20. A non-transitory computer-readable storage medium having stored thereon computer-readable instructions that, when executed at a computer system, cause the computer system to perform a computer-implemented method of compressing a neural network, the method comprising: receiving neural network; selecting two or more adjacent layers of the received neural network, each of the two or more adjacent layers having one or more input channels and one or more output channels, the one or more output channels of a first layer of the two or more adjacent layers corresponding to the one or more input channels of a second subsequent layer of the two or more adjacent layers, the first layer being arranged to perform a first operation and the second layer being arranged to perform a second operation; determining a first matrix representing a set of coefficients of the first layer of the received neural network, the first matrix comprising a plurality of elements representing non-zero values ​​and a plurality of elements representing zero values, one or more rows or columns of the first matrix representing the one or more output channels of the first layer, and one or more other rows or columns of the rows or columns of the first matrix representing the one or more input channels of the first layer; determining a second matrix representing a set of coefficients of the second layer of the received neural network, the second matrix comprising a plurality of elements representing non-zero values ​​and a plurality of elements representing zero values, one or more rows or columns of the second matrix representing the one or more output channels of the second layer, and one or more other rows or columns of the rows or columns of the second matrix representing the one or more input channels of the second layer; Form an array in one of the following ways: transposing the first matrix and forming the array comprising the transposed first matrix and the second matrix by aligning the columns or rows of the transposed first matrix representing the one or more output channels of the first layer with the columns or rows of the second matrix representing the one or more input channels of the second layer; or transposing the second matrix and forming the array comprising the transposed second matrix and the first matrix by aligning the row or column of the transposed second matrix representing the one or more input channels of the second layer with the row or column of the first matrix representing the one or more output channels of the first layer; or forming the array including the first matrix and the second matrix by aligning the rows or columns of the first matrix representing the one or more output channels of the first layer with the rows or columns of the second matrix representing the one or more input channels of the second layer; Rearranging the rows and / or columns of the array so as to: aggregating the plurality of elements representing non-zero values ​​included in the first matrix or the transposed first matrix into first one or more sub-matrices, the first one or more sub-matrices having a ratio of the number of elements representing non-zero values ​​to the total number of elements of the first one or more sub-matrices that is greater than a ratio of the number of elements representing non-zero values ​​to the total number of elements of the first matrix; as well as aggregating the plurality of elements representing non-zero values ​​included in the second matrix or the transposed second matrix into a second one or more sub-matrices, the second one or more sub-matrices having a ratio of the number of elements representing non-zero values ​​to the total number of elements of the second one or more sub-matrices that is greater than a ratio of the number of elements representing non-zero values ​​to the total number of elements of the second matrix; as well as Outputting a compressed neural network, the compressed neural network comprising a first compression layer arranged to perform a first compression operation based on the first one or more sub-matrices, and a second subsequent compression layer arranged to perform a second compression operation based on the second one or more sub-matrices.

Images