Matrix operations for processing rate-limited systems

Abstract

The data to be processed includes vector element values of an input vector and matrix element values of a model matrix associated with a neural network model. The vector-matrix multiplication module receives a set of matrix element values for performing a vector-matrix multiplication operation. Processing the data includes computing a plurality of intermediate vectors based on element-wise vector multiplication between different subsets of the vector element values and different respective pre-processing vectors. The vector-matrix multiplication module is loaded with a kernel matrix, and the input vector is multiplied by the model matrix based on multiplying each of the intermediate vectors by the loaded kernel matrix, respectively.

Description

[0001] Cross-references to related applications

[0002] This application claims priority and benefit from U.S. Provisional Patent Application No. 62 / 799,849, filed on February 1, 2019. Technical Field

[0003] This disclosure relates to matrix operations for processing rate-limited systems. Background Technology

[0004] Various techniques used in machine learning include computationally intensive steps for performing vector-matrix multiplication, where vectors are multiplied by matrices. For example, in neural network models, the size of the matrices used in neural network-based computations can be larger than the size of the matrices that can be loaded into modules of a computing system configured to perform the vector-matrix multiplication portion of the computation. Therefore, performing a portion of the computation may involve dividing the matrices and vectors into smaller parts that can be provided separately to modules. Some systems have limitations on the rate at which different elements of the matrices can be loaded for different parts of the computation. Summary of the Invention

[0005] On one hand, the data to be processed typically includes the vector element values ​​of the input vector and the matrix element values ​​of the model matrix associated with the neural network model. The vector-matrix multiplication module receives a set of matrix element values ​​used to perform vector-matrix multiplication operations. Data processing involves computing multiple intermediate vectors based on element-wise vector multiplications between different subsets of the vector element values ​​and different corresponding preprocessed vectors. The vector-matrix multiplication module loads a core matrix and multiplies the input vector by the model matrix by multiplying each intermediate vector by the loaded core matrix.

[0006] On the other hand, typically, an apparatus for processing data including vector element values ​​of an input vector and matrix element values ​​of a model matrix associated with a neural network model includes: a clock module configured to provide a clock signal associated with the data; a vector-matrix multiplication module configured to receive a set of matrix element values ​​for performing vector-matrix multiplication operations, wherein the vector-matrix multiplication module has a rate limit characterizing the maximum rate at which the set of matrix element values ​​can be loaded between consecutive vector-matrix multiplication operations; and a computation subsystem configured to process the data based on the clock signal, the processing including: computing a plurality of intermediate vectors based on element-wise vector multiplications between different subsets of vector element values ​​and different corresponding preprocessed vectors, wherein the number of cycles of the clock signal used to compute the plurality of intermediate vectors is less than the minimum number of clock cycles associated with the rate limit; loading a core matrix into the vector-matrix multiplication module; and multiplying the input vector by the model matrix based on multiplying each intermediate vector by the loaded core matrix respectively.

[0007] An aspect may include one or more of the following features.

[0008] The loaded core matrix is ​​multiplied by different intermediate vectors at different periods of the clock signal.

[0009] The length of the intermediate vector is k, and the size of the core matrix is ​​k×k.

[0010] The value k is greater than 31 and less than 257.

[0011] The speed limit is less than 5 GHz.

[0012] The vector-matrix multiplication module includes an optical waveguide and an optical amplitude modulator, and at least some vector element values ​​are encoded on the corresponding optical signals guided by the optical waveguide.

[0013] Based on the amplitude modulation levels of one or more optical amplitude modulators, the core matrix is ​​loaded into the vector-matrix multiplication module.

[0014] Rate limiting is based on the rate at which the amplitude modulation level can be set.

[0015] The vector-matrix multiplication module includes a computing device that operates on electrical signals, and at least some vector element values ​​are encoded on corresponding electrical signals received by the computing device.

[0016] Based on electrical signals obtained from storage devices coupled to computing devices, the core matrix is ​​loaded into the vector-matrix multiplication module.

[0017] Rate limiting is based on the rate at which electrical signals can be obtained from the storage device.

[0018] On the other hand, typically, a method for processing data including vector element values ​​of an input vector and matrix element values ​​of a model matrix associated with a neural network model includes: providing a clock signal associated with the data; receiving a set of matrix element values ​​for performing vector-matrix multiplication operations at a vector-matrix multiplication module, wherein the vector-matrix multiplication module has a rate limit characterizing the maximum rate at which the set of matrix element values ​​can be loaded between consecutive vector-matrix multiplication operations; and processing the data based on the clock signal in a computational subsystem, the processing including: computing multiple intermediate vectors based on element-wise vector multiplications between different subsets of vector element values ​​and different corresponding preprocessed vectors, wherein the number of cycles of the clock signal used to compute the multiple intermediate vectors is less than the minimum number of clock cycles associated with the rate limit; loading a core matrix into the vector-matrix multiplication module; and multiplying the input vector by the model matrix based on multiplying each intermediate vector by the loaded core matrix, respectively.

[0019] On the other hand, a typical method for performing inference computation using a model matrix associated with a neural network model typically includes: receiving a core matrix comprising a set of matrix elements determined based on training of the neural network model; loading the core matrix into a vector-matrix multiplication module; computed multiple intermediate vectors based on element-wise vector multiplication between different subsets of the vector element values ​​of the input vector and different corresponding preprocessed vectors determined based on training of the neural network model; and performing vector-matrix multiplication of the input vector and the model matrix by multiplying each intermediate vector by the loaded core matrix.

[0020] In some respects, the model matrix is ​​related to the core matrix and each preprocessing vector such that: the model matrix is ​​formed by two or more submatrices joined together, for each submatrix there exists a different corresponding diagonal matrix with diagonal elements equal to the vector element values ​​of the different preprocessing vectors, and each submatrix is ​​equal to the core matrix multiplied by the corresponding diagonal matrix.

[0021] On the other hand, typically, an apparatus for performing inference computation using a model matrix associated with a neural network model includes: a vector-matrix multiplication module configured to receive a core matrix comprising a set of matrix elements determined based on training of the neural network model; circuitry configured to load the core matrix into the vector-matrix multiplication module; and circuitry configured to compute a plurality of intermediate vectors based on element-wise vector multiplication between different subsets of vector element values ​​of an input vector and different corresponding preprocessed vectors determined based on training of the neural network model; wherein the vector-matrix multiplication module is configured to perform vector-matrix multiplication of the input vector and the model matrix by multiplying each intermediate vector by the loaded core matrix respectively.

[0022] On the other hand, typically, an apparatus for processing data including vector element values ​​of an input vector and matrix element values ​​of a model matrix associated with a neural network model includes: a vector-matrix multiplication module configured to receive a set of matrix element values ​​for performing vector-matrix multiplication operations, wherein the vector-matrix multiplication module is configured to perform optical amplitude modulation on values ​​encoded in an optical signal based on the set of matrix element values; and a computational subsystem configured to process data encoded in an electrical signal, the processing including: computing multiple intermediate vectors based on element-wise vector multiplication between different subsets of vector element values ​​and different corresponding preprocessed vectors, wherein the vector element values ​​and the preprocessed vectors are encoded electrical signals; loading a core matrix into the vector-matrix multiplication module; and multiplying the input vector by the model matrix based on multiplying each intermediate vector by the loaded core matrix respectively.

[0023] On the other hand, typically, a method for processing data including vector element values ​​of an input vector and matrix element values ​​of a model matrix associated with a neural network model includes: receiving a set of matrix element values ​​at a vector-matrix multiplication module for performing vector-matrix multiplication operations, wherein the vector-matrix multiplication module is configured to perform optical amplitude modulation on values ​​encoded in an optical signal based on the set of matrix element values; and processing the data encoded in an electrical signal in a computational subsystem, the processing including: computing multiple intermediate vectors based on element-wise vector multiplication between different subsets of vector element values ​​and different corresponding preprocessed vectors, wherein the vector element values ​​and the preprocessed vectors are encoded electrical signals; loading a core matrix into the vector-matrix multiplication module; and multiplying the input vector by the model matrix based on multiplying each intermediate vector by the loaded core matrix, respectively.

[0024] This aspect can have one or more of the following advantages.

[0025] The described techniques enable vector-matrix multiplication for various machine learning processes, including those using deep neural networks for natural language processing (NLP) applications (e.g., neural machine translation (NMT) or dialogue systems), image classification, or object detection. Significant speedups can be achieved in the inference process without significantly sacrificing accuracy by reducing the number of times different submatrices need to be loaded into the vector-matrix multiplication module. In some implementations, the speedup is based on avoiding the slowdown associated with accessing data from memory (also known as the "memory wall"). In other implementations, the speedup is based on avoiding the slowdown associated with loading values ​​for optical modulation. While the number of times submatrices need to be loaded can be reduced, these techniques also allow the underlying matrix formed by the submatrices to change dynamically during inference (e.g., in contrast to techniques where matrix weights can be fixed for a longer period during inference).

[0026] Other features and advantages will become apparent from the following description, drawings and claims. Attached Figure Description

[0027] This disclosure is best understood from the following detailed description when read in conjunction with the accompanying drawings. It should be emphasized that, in accordance with common practice, the various features in the drawings are not to scale. Rather, for clarity, the dimensions of the various features have been arbitrarily enlarged or reduced.

[0028] Figure 1 This is a flowchart and diagram illustrating an example of a vector-vector-matrix processing architecture.

[0029] Figure 2This is a schematic diagram of an example computing system. Detailed Implementation

[0030] Some machine learning systems, such as those used in NLP applications, improve performance accuracy by increasing the size of the neural network model (or simply "model"). However, larger models typically require a significant amount of hardware computation, which can significantly degrade system speed due to increased latency during operations (e.g., during inference). The vector-vector-matrix architecture (VVMA) described herein reduces latency during operations in various systems using VVMA. VVMA leverages both efficient model design to compress neural networks to reduce hardware computation requirements and accelerated computations performed by hardware due to hardware design. For example, some systems using VVMA utilize hardware designs that support relatively low-latency vector-vector operations (e.g., element-wise vector multiplication) compared to higher-latency vector-matrix operations. For some models that rely on efficient matrix multiplication, the techniques described herein can reduce the number of parameters and floating-point operations per second (FLOPs) without significantly impacting accuracy.

[0031] The inherent sequential nature of many NLP tasks increases inference latency. Limited by memory bandwidth and footprint, some systems rely on large batch sizes to avoid underutilization. However, increasing batch size is not always feasible when conclusions must be inferred quickly, such as when inference is performed in real time. For example, systems using vector-matrix architecture (VMA) with hardware configured to perform vector-matrix multiplication, or matrix-matrix architecture (MMA) with hardware configured to perform matrix-matrix multiplication, may exhibit long latency during inference for a given mini-batch task (e.g., when translating a single sentence). Systems using hardware configured for VVMA can reduce inference latency, even for mini-batch tasks.

[0032] Efficient model configuration can reduce the complexity of neural networks. For example, techniques such as shuffling data between neural network layers or compressing model matrices can reduce the amount of computation required while maintaining similar performance and accuracy. VVMA can also reduce computational complexity without significantly sacrificing performance or accuracy. However, unlike some techniques that make models more efficient, VVMA focuses on low-level execution, such as underlying matrix multiplication. Therefore, VVMA is generally suitable for a variety of neural network inference tasks that utilize matrix multiplication.

[0033] VVMA can provide significant speedups in systems with hardware configured to perform relatively low-latency vector-to-vector operations (e.g., element-wise vector multiplication). Reference Figure 1 The vector-vector-matrix processing procedure 100 can be used, for example, in a neural network-based computational system. This procedure takes two vectors and a matrix, then multiplies the first vector element-wise by the second vector, and then multiplies the resulting vector by the matrix. In preparation step 102, procedure 100 determines the data associated with the model matrix 112 to be used to process the input vector 110. For example, some data may include information already determined based on a previously performed training process. The input vector 110 may include vector element values ​​that represent the data used during inference using the neural network model.

[0034] In some implementations, matrix loading step 104 and vector processing step 106 can be performed simultaneously, and some of these operations can be performed in parallel. Matrix loading step 104 loads the core matrix 114 into the vector-matrix multiplication module as a set of matrix element values ​​to be used for vector-matrix multiplication operations. Since this matrix loading step 104 can be time-consuming (e.g., due to rate limitations associated with the vector-matrix multiplication module), vector processing step 106 can use that loading time for useful operations as part of the computation. For example, different subsets 115 of the vector element values ​​of the input vector 110 can be multiplied with different corresponding preprocessed vectors 116 using element-wise vector multiplication (where corresponding elements of each vector are multiplied together to form the elements of the resulting vector). Then, in matrix processing step 108, which is performed after matrix loading step 104 is complete, each resulting intermediate vector 118 can be multiplied by the same core matrix 114. The rate limit in different implementations can vary depending on the time required to perform actions such as fetching data from memory (e.g., resulting in rate limits of approximately 100 kHz, 1 MHz, 10 MHz, 100 MHz, 1 GHz, or 5 GHz).

[0035] refer to Figure 2An example computing system 200 for executing process 100 is shown. Clock module 202 can be used, for example, to distribute clock signals to various modules that will perform operations synchronously using digital and / or analog electrical signals, and in some embodiments, optical signals are used for certain modules capable of performing certain operations optically (e.g., optical amplitude modulation that can be used for vector-matrix multiplication). Interface module 204 can be used to provide the data determined in preparation step 102. Interface module 204 can be coupled to a storage device, for example, storing data based on a neural network model. Preprocessing module 206 can perform various preprocessing calculations, including preparation or training of data determined based on previous training.

[0036] The various steps of process 100 can be performed using modules of system 200. Loading module 208 can be configured to perform matrix loading step 104. Element-wise vector multiplication module 212 can be configured to perform element-wise vector multiplication as part of vector processing step 106. The vector-matrix multiplication operation of matrix processing step 108 can be performed by vector-matrix multiplication module 210. In some embodiments, the operation is performed electronically on electrical signals (e.g., in digital operation, or in analog operation using a modulator array that attenuates or absorbs power from electrical signals). In other embodiments, the operation is performed optically on optical signals (e.g., using techniques as described in U.S. Patent Publication 2007 / 0351293, entitled "Apparatus and Methods for Optical Neural Network," which is incorporated herein by reference).

[0037] The element-wise vector multiplication module 212 and the loading module 208 can be configured to operate simultaneously. When the loading operation occurs, multiple element-wise vector multiplication operations can occur between different subsets 115 of the vector element values ​​of the input vector 110 and different corresponding preprocessed vectors 116. For example, in some embodiments, each of these operations between a specific subset of vector element values ​​and a specific preprocessed vector can be performed within a short period (e.g., one clock cycle), such that all operations are completed before the loading is finished. Then, after the loading is finished, the common core matrix can be multiplied by all the computed intermediate vectors.

[0038] The following example of steps that can be performed on a computing system such as computing system 200 using a process such as vector-vector-matrix processing procedure 100 is described in the context of a shrinking loop that includes matrix multiplication using model matrices with different weights. In some embodiments, the weights can be reused across multiple batches of data, while matrix multiplication can be performed extremely quickly using a shrinking loop. In this example, A is n 2The model has an n×n weight matrix, and x is an n-dimensional input vector. To perform vector-matrix multiplication Wx on a computational system that supports vector-matrix multiplication of matrices of size k×k (where k < n), the matrix W can first be decomposed into multiple submatrices of size k×k, and the vector x can be decomposed into multiple subvectors of size k, as shown below.

[0039]

[0040] Here W i，j It is a k×k submatrix of W, x j It is a k-dimensional subvector of x, and the computation system calculates each submatrix W i，j Multiply by subvector x j Previously loaded into the matrix multiplication hardware. Loading a k×k submatrix may require O(k) clock cycles in some computing systems. Then, the submatrix W... i，j After being loaded into the matrix multiplication hardware, it may take O(2k+t) clock cycles to multiply t k-dimensional subvectors by the submatrix W using VMA (instead of VVMA). i，j Therefore, in a computing system using VMA, the total number of clock cycles required to multiply t n-dimensional vectors by the model matrix W is as follows.

[0041]

[0042] Note that for single-batch inference (i.e., for t=1), the process has a relatively large delay (measured on the order of clock cycles).

[0043] To reduce latency, the model matrix W can be adjusted so that the computational system using VVMA only needs to load a single k×k core matrix M into the matrix multiplication hardware (e.g., the vector-matrix multiplication module 210). Then, the computational system performs multiplication on each subvector x. j Element-wise vector multiplication is performed to compensate for the additional parameters lost by reusing the same k×k core matrix M. With this VVMA adjustment, the total number of clock cycles required to multiply t n-dimensional vectors by the model matrix W in a VVMA-based computational system is as follows.

[0044]

[0045] Note that this adjusted process, such as vector-vector-matrix processing 100, has lower latency for single-batch inference (i.e., for t=1). In some implementations, k can be chosen as a power of 2, and some values ​​of k can have sufficiently small and sufficiently large magnitudes (e.g., 32, 64, 128, or 256) to balance the various trade-offs in any of the different system configurations.

[0046] In some implementations, the element-wise vector multiplication operation performed before the vector-matrix multiplication by the core matrix M requires no additional clock cycles. This can be achieved by pipelined the element-wise vector multiplication operation as the input vector data is fed into the matrix multiplication hardware. Furthermore, the element-wise vector multiplication operation can be initialized when the weights of the core matrix M are loaded into the matrix multiplication hardware.

[0047] This is an example of the process for adjusting the model matrix W and calculating the associated core matrix M. The initial step is to tile copies of the core matrix M into a larger matrix, making its size greater than or equal to the size of the model matrix W. Each copy of the core matrix M is multiplied by its corresponding diagonal matrix D. i，j This allows the model matrix W to be replaced with a modified matrix having the following structure:

[0048]

[0049] The core matrix M is a shared k×k weight matrix, and D i，j It is a diagonal k×k weight matrix. Introducing the diagonal matrix D... i，j This is to provide variations for each part of the modified model matrix, allowing the modified model matrix to replace the original model matrix W without causing a significant decrease in accuracy. While in these examples both the model matrix W and the core matrix M are square matrices, in other examples, either or both of these matrices may not be square, having different numbers of rows and columns.

[0050] Each element of the core matrix M is used to compute each submatrix of the modified model matrix, thus limiting the total number of free parameters compared to the original model matrix W. Each element of the core matrix M and each diagonal matrix D... i，j Each element of the matrix D is learned as part of the training process for the machine learning algorithm. i，j These can be mathematically represented as matrices, but they can also be represented as k-dimensional vectors v. i，j Then, as part of the inference computation, vector-matrix multiplication D... i， j x j It can be used as an element-wise vector multiplication v i，j ⊙x j The process involves multiplying the vector obtained from element-wise vector multiplication with the core matrix M using vector-matrix multiplication.

[0051] In some implementations, to efficiently implement a modified model matrix as a trainable matrix instead of constructing the entire matrix representation, the computational system can utilize broadcasting. This allows the system to perform element-wise multiplication on tensors of different shapes. Similarly, broadcasting can be used to multiply an input vector x by a large diagonal tensor D. The system then performs matrix multiplication using the broadcast vector and the core matrix M. Thus, the system is able to construct a single k×k core matrix M at a time, instead of constructing the core matrix multiple times during the tiling of a larger matrix.

[0052] When executing different types of machine learning algorithms, various other techniques can also be used in conjunction with some of the implementation methods described here. For example, to train a seq2seq model using VVMA, performing more aggressive gradient pruning can be useful. If the value used to prune the gradient is too large, the gradient may become very large and eventually overflow. This overflow can occur because the gradient accumulates as it propagates back to a single small core matrix M. Several techniques can accelerate the training or use of neural networks, such as quantization and pruning techniques.

[0053] Some systems may include hardware that can be configured or modified to use VVMA to optimize neural network inference (e.g., reduce latency during inference). For example, some systems using vector-matrix multiplication modules can be modified by adding element-wise vector multiplication modules that feed the vector-matrix multiplication modules. The element-wise vector multiplication modules can operate in a pipelined manner and are initialized simultaneously with the loading of the vector-matrix multiplication modules. Some systems using matrix-matrix multiplication modules can also be modified by adding element-wise vector multiplication modules that feed the matrix-matrix multiplication modules.

[0054] As described above, some systems can be configured to perform element-wise vector multiplication and / or vector-matrix multiplication using optical amplitude modulation. The potentially large latency associated with loading matrix weights from memory into the optical modulator can be mitigated by using an optical or electrical element-wise vector multiplication module prior to the optical vector-matrix multiplication module. This module can be pipelined and initialized simultaneously with the loading of the optical vector-matrix multiplication module. The electrical element-wise vector multiplication module does not significantly increase the overall latency of a system using an optical vector-matrix multiplication module, as the overall latency is likely dominated by memory-related operations (e.g., fetching input vector data from digital electronic memory).

[0055] While this disclosure has been described in conjunction with certain embodiments, it should be understood that this disclosure is not limited to the disclosed embodiments. Rather, this disclosure is intended to cover various modifications and equivalent arrangements included within the scope of the appended claims, which will be interpreted in the broadest sense to include all such modifications and equivalent structures permitted by law.

Claims

1. An apparatus for processing data including vector element values ​​of an input vector and matrix element values ​​of a model matrix associated with a neural network model, the apparatus comprising: A clock module is configured to provide a clock signal associated with the data; A vector-matrix multiplication module is configured to receive a set of matrix element values ​​for performing vector-matrix multiplication operations, wherein the vector-matrix multiplication module has a rate limit, the rate limit representing the maximum rate at which the set of matrix element values ​​can be loaded between consecutive vector-matrix multiplication operations; as well as A computing subsystem is configured to process data based on the clock signal, the processing including: Multiple intermediate vectors are calculated based on element-wise vector multiplications between different subsets of the vector element values ​​and different corresponding preprocessed vectors, wherein the number of cycles of the clock signal used to calculate the multiple intermediate vectors is less than the minimum number of clock cycles associated with the rate limit. The core matrix is ​​loaded into the vector-matrix multiplication module. The input vector is multiplied by the model matrix by multiplying each of the intermediate vectors by the loaded core matrix.

2. The apparatus according to claim 1, wherein, The loaded core matrix is ​​multiplied by different intermediate vectors at different periods of the clock signal.

3. The apparatus according to claim 1, wherein, The length of the intermediate vector is k And the size of the core matrix is k×k .

4. The apparatus according to claim 3, wherein, The k Greater than 31 and less than 257.

5. The apparatus according to claim 1, wherein, The rate limit is less than 5 GHz.

6. The apparatus according to claim 1, wherein, The vector-matrix multiplication module includes an optical waveguide and an optical amplitude modulator, and at least some of the vector element values ​​are encoded on corresponding optical signals guided by the optical waveguide.

7. The apparatus according to claim 6, wherein, Based on the amplitude modulation levels of one or more of the optical amplitude modulators, the core matrix is ​​loaded into the vector-matrix multiplication module.

8. The apparatus according to claim 7, wherein, The rate limit is based on the rate at which the amplitude modulation level can be set.

9. The apparatus according to claim 1, wherein, The vector-matrix multiplication module includes a computing device that operates on electrical signals, and at least some of the vector element values ​​are encoded on corresponding electrical signals received by the computing device.

10. The apparatus according to claim 9, wherein, Based on the electrical signal obtained from the storage device coupled to the computing device, the core matrix is ​​loaded into the vector-matrix multiplication module.

11. The apparatus according to claim 10, wherein, The rate limit is based on the rate at which the electrical signal can be obtained from the storage device.

12. A method for processing data, said data including vector element values ​​of an input vector and matrix element values ​​of a model matrix associated with a neural network model, the method comprising: Provide a clock signal associated with the data; The vector-matrix multiplication module receives a set of matrix element values ​​for performing vector-matrix multiplication operations, wherein the vector-matrix multiplication module has a rate limit characterizing the maximum rate at which the set of matrix element values ​​can be loaded between consecutive vector-matrix multiplication operations; and The data is processed based on the clock signal in the computing subsystem, and the processing includes: Multiple intermediate vectors are calculated based on element-wise vector multiplications between different subsets of the vector element values ​​and different corresponding preprocessed vectors, wherein the number of cycles of the clock signal used to calculate the multiple intermediate vectors is less than the minimum number of clock cycles associated with the rate limit. The core matrix is ​​loaded into the vector-matrix multiplication module. The input vector is multiplied by the model matrix by multiplying each of the intermediate vectors by the loaded core matrix.

13. A method for performing inference computation using a model matrix associated with a neural network model, the method comprising: Receive a core matrix, the core matrix comprising a set of matrix elements determined based on training of the neural network model; Load the core matrix into the vector-matrix multiplication module; Multiple intermediate vectors are computed by element-wise vector multiplication between different subsets of the vector element values ​​of the input vector and different corresponding preprocessed vectors determined based on the training of the neural network model. as well as Vector-matrix multiplication of the input vector and the model matrix is ​​performed by multiplying each of the intermediate vectors by the loaded core matrix.

14. The method according to claim 13, wherein, The model matrix is ​​related to the core matrix and each preprocessed vector, such that: The model matrix is ​​formed by connecting two or more sub-matrices together. For each submatrix, there exists a distinct corresponding diagonal matrix, which has diagonal elements equal to the vector element values ​​of the distinct preprocessed vectors, and Each submatrix is ​​equal to the core matrix multiplied by the corresponding diagonal matrix.

15. An apparatus for performing inference computation using a model matrix associated with a neural network model, the apparatus comprising: The vector-matrix multiplication module is configured to receive a core matrix, which includes a set of matrix elements determined based on training of the neural network model; A circuit configured to load the core matrix into a vector-matrix multiplication module; as well as A circuit configured to compute multiple intermediate vectors by element-wise vector multiplication between different subsets of the vector element values ​​of the input vector and different corresponding preprocessed vectors determined based on the training of the neural network model; The vector-matrix multiplication module is configured to perform vector-matrix multiplication between the input vector and the model matrix by multiplying each of the intermediate vectors by the loaded core matrix.

16. An apparatus for processing data including vector element values ​​of an input vector and matrix element values ​​of a model matrix associated with a neural network model, the apparatus comprising: A vector-matrix multiplication module is configured to receive a set of matrix element values ​​for performing vector-matrix multiplication operations, wherein the vector-matrix multiplication module is configured to perform optical amplitude modulation on values ​​encoded in an optical signal based on the set of matrix element values; and A computing subsystem is configured to process data encoded on electrical signals, the processing including: Multiple intermediate vectors are calculated based on element-wise vector multiplication between different subsets of the vector element values ​​and different corresponding preprocessed vectors, wherein the vector element values ​​and the preprocessed vectors are encoded electrical signals. The core matrix is ​​loaded into the vector-matrix multiplication module. The input vector is multiplied by the model matrix by multiplying each of the intermediate vectors by the loaded core matrix.

17. A method for processing data including vector element values ​​of an input vector and matrix element values ​​of a model matrix associated with a neural network model, the method comprising: The vector-matrix multiplication module receives a set of matrix element values ​​for performing vector-matrix multiplication operations, wherein the vector-matrix multiplication module is configured to perform optical amplitude modulation on values ​​encoded in an optical signal based on the set of matrix element values; and The computational subsystem processes data encoded on electrical signals, the processing including: Multiple intermediate vectors are calculated based on element-wise vector multiplication between different subsets of the vector element values ​​and different corresponding preprocessed vectors, wherein the vector element values ​​and the preprocessed vectors are encoded electrical signals. The core matrix is ​​loaded into the vector-matrix multiplication module. The input vector is multiplied by the model matrix by multiplying each of the intermediate vectors by the loaded core matrix.

Images