Time-space-wavelength-multiplexed photonic matrix multiplier

Abstract

A photonic matrix multiplier multiplies an MxP first matrix with a PxN second matrix. The multiplier includes a splitter that receives light including M different wavelengths into M separate light beams. M first optical modulators modulate the M light beams with time-multiplexed row vectors of the first matrix. An arrayed waveguide grating router (AWGR) processes the output signals of the M first optical modulators and outputs N processed light signals. N second optical modulators modulate the N processed light signals with time-multiplexed column vectors of the second matrix. N demultiplexers separate the N second optical modulator output signals in light beams at each of the M different wavelengths. Integrators integrate demultiplexer output signals to obtain multiplied and accumulated signals.

Description

Time-Space-Wavelength-Multiplexed Photonic Matrix MultiplierBACKGROUNDTECHNICAL FIELD

[0001] The disclosed implementations relate generally to systems and methods used to provide multiply-accumulate operations, such as used in neural networks. The systems presented herein may use a combination of photonic and electrical operations to obtain high performance results at modest power dissipation.CONTEXT

[0002] Neural networks use parallel processing to perform massive matrix multiplications, for both training and inference. Typically, processing is performed by digital or analog electronic circuits. However, these systems have a high power consumption and require considerable execution time to train large networks.

[0003] The subject matter discussed in this section should not be assumed to be prior art merely as a result of its mention in this section. Similarly, a problem mentioned in this section or associated with the subject matter provided as background should not be assumed to have been previously recognized in the prior art. The subject matter in this section merely represents different approaches, which in and of themselves can also correspond to implementations of the claimed technology.SUMMARYMatrix Multiplication

[0004] A first implementation provides a photonic matrix multiplier that performs multiplication of a first matrix with M rows and P columns with a second matrix with P rows and N columns. The photonic matrix multiplier includes a splitter configured to receive light that includes components at M different wavelengths. The splitter has M splitter outputs that are coupled with M first optical modulators, each configured to receive a first time-multiplexed electric signal representing a row vector of the first matrix. An arrayed waveguide grating router (AWGR) with M inputs is photonically coupled with outputs of the M first optical modulators. N second optical modulators are photonically coupled with N outputs of the AWGR, and each is configured to receive a second time-multiplexed electric signal representing a column vector of the second matrix. N demultiplexers with an input and M outputs are coupled with outputs of the N second optical modulators and configured to separate light received at their input into Mcomponents at the M different wavelengths, provi ing one of the M components at one of the M outputs. M x N integrators are coupled with outputs of the N demultiplexers.

[0005] A second implementation provides a method to multiply and accumulate parameters from a first matrix with M rows and P columns and a second matrix with P rows and N columns. The method includes splitting light with M wavelengths into M light beams, each carrying the M different wavelengths; in M first optical modulators, modulating light from the M light beams with signals representing time- division-multiplexed row vectors of the first matrix to obtain M first modulated light signals; in an arrayed waveguide grating router (AWGR), processing the M first modulated light signals to obtain N processed light signals; in N second optical modulators, modulating the N processed light signals with signals representing time-division-multiplexed column vectors of the second matrix to obtain N second modulated light signals; in N demultiplexers, separating the N second modulated light signals into MxN light beams each including one of the M different wavelengths; and in an integrator, integrating a signal from one of the M light beams from one of the N demultiplexers to obtain an accumulated multiplied signal.BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The technology will be described with reference to the drawings, in which:

[0007] FIG. 1 illustrates an example system including a photonic integrated circuit (PIC) and an electronic integrated circuit (EIC) to perform a vector-vector multiplication.

[0008] FIG. 2 illustrates examples of four basic variations of the system in FIG. 1 to perform the vector multiplication.

[0009] FIG. 3 illustrates an example system including PIC hardware to perform a matrix-vector multiplication.

[0010] FIG. 4 illustrates an example matrix-matrix multiplication and time-division multiplexing of the matrix components.

[0011] FIG. 5 illustrates an example system that performs the matrix-matrix multiplication of FIG. 4.

[0012] FIG. 6 illustrates an example method to multiply and accumulate parameters from a first matrix and a second matrix.

[0013] FIG. 7 illustrates an example tensor-matrix multiplication and associated time-division multiplexing of the tensor and matrix components.

[0014] FIG. 8 illustrates an example photonic tensor multiplier to perform the tensor-matrix multiplication of FIG. 7.

[0015] FIG. 9 illustrates an example method to multiply and accumulate parameters from a first matrix with P rows and N columns and a second tensor with K slices of M rows and P columns, where K, M, N, and P are positive integer numbers.

[0016] In the figures, like reference numbers may indicate functionally similar elements. The systems and methods illustrated in the figures— and described in the Detailed Description below— may be arranged and designed in a wide variety of different implementations. Neither the figures nor the Detailed Description are intended to limit the scope as claimed. Instead, they merely represent examples of different implementations.DETAILED DESCRIPTION

[0017] Neural networks use parallel processing to perform massive matrix multiplications. Typically, processing is performed by digital or analog electronic circuits. However, these consume very high power. The technology disclosed herein uses photonics systems to perform the multiply-accumulate (MAC) operations needed, and do so at modest power levels.

[0018] Implementations of the systems disclosed herein use various forms of multiplexing to efficiently combine operations and to allow for massive parallelism.TERMINOLOGY

[0019] As used herein, the phrase "one of" should be interpreted to mean exactly one of the listed items. For example, the phrase "one of A, B, and C" should be interpreted to mean any of: only A, only B, or only C.

[0020] As used herein, the phrases at least one of and one or more of should be interpreted to mean one or more items. For example, the phrase "at least one of A, B, or C" or the phrase "one or more of A, B, or C" should be interpreted to mean any combination of A, B, and / or C. The phrase "at least one of A, B, and C" means at least one of A and at least one of B and at least one of C.

[0021] Unless otherwise specified, the use of ordinal adjectives first, second, third, etc., to describe an object, merely refers to different instances or classes of the object and does not imply any ranking or sequence.

[0022] The terms "comprising" and "consisting" have different meanings in this patent document. An apparatus, method, or product "comprising" (or "including") certain features means that it includes those features but does not exclude the presence of other features. On the other hand, if the apparatus, method, or product "consists of" certain features, the presence of any additional features is excluded.

[0023] The term "coupled" is used in an operational sense and is not limited to a direct or an indirect coupling. "Coupled to" is generally used in the sense of directly coupled, whereas "coupled with" is generally used in the sense of directly or indirectly coupled. Coupled in an electronic system may refer to a configuration that allows a flow of information, signals, data, or physical quantities such as electrons between two elements coupled to or coupled with each other. In some cases, the flow may be unidirectional, in other cases the flow may be bidirectional or multidirectional. Coupling may be galvanic (in this context meaning that a direct electrical connection exists), capacitive, inductive, electromagnetic, optical, or through any other process allowed by physics.

[0024] The term "connected" is used to indicate a direct connection, such as electrical, optical, electromagnetic, or mechanical, between the things that are connected, without any intervening things or devices.

[0025] The term "configured" to perform a task or tasks is a broad recitation of structure generally meaning having circuitry that performs the task or tasks during operation. As such, the described item can be configured to perform the task even when the unit / circuit / component is not currently on or active. In general, the circuitry that forms the structure corresponding to configured to may include hardware circuits, and may further be controlled by switches, fuses, bond wires, metal masks, firmware, and / or software. Similarly, various items may be described as performing a task or tasks, for convenience in the description. Such descriptions should be interpreted as including the phrase configured to.

[0026] As used herein, the term "based on" is used to describe one or more factors that affect a determination. This term does not foreclose the possibility that additional factors may affect the determination. That is, a determination may be solely based on specified factors or based on the specified factors as well as other, unspecified factors. Consider the phrase "determine A based on B". This phrase specifies that B is a factor that is used to determine A or that affects the determination of A. This phrase does not foreclose that the determination of A may also be based on some other factor, such as C. This phrase is also intended to cover an implementation in which A is determined based solely on B.The phrase based on is thus synonymous with the phrase based at least in part on.

[0027] The terms "substantially", "close", "approximately", "near", and "about" refer to being within minus or plus 10% of an indicated value, unless explicitly specified otherwise.

[0028] The following terms or acronyms used herein are defined at least in part as follows:

[0029] "ADC" - analog-to-digital converter

[0030] "ASIC" - application-specific integrated circuit

[0031] "AWGR" - arrayed waveguide grating router

[0032] "EAM" - electro-absorption modulator

[0033] "EIC" - electronic integrated circuit - a monolithic circuit that integrates multiple electronic devices that process electrical signals to perform operations and / or transport information.

[0034] "FPGA" - field-programmable gate array

[0035] "LED" - light-emitting diode

[0036] "MAC" - multiply and accumulate - an operation performed in, for example, neural networks.

[0037] "MCM" - multi-chip module

[0038] "PCB" - printed circuit board

[0039] "PIC" - photonic integrated circuit

[0040] "PIC" - photonic integrated circuit - a monolithic circuit that integrates multiple photonic functions and devices that process light to perform operations and / or transport information.

[0041] "Q.CSE" - quantum confined stark effect

[0042] "SDM" - space-division multiplex

[0043] "SLD" - superluminescent diode

[0044] "TDM" - time-division multiplex

[0045] "TIA" - transimpedance amplifier

[0046] "VCSEL" - vertical-cavity surface-emitting laser

[0047] "WDM" - wavelength-division multiplex.IMPLEMENTATIONS - Vector Multiplication

[0048] FIG. 1 illustrates an example system 120 including a photonic integrated circuit (PIC 130) and an electronic integrated circuit (EIC 140) to perform a vector-vector multiplication 100.

[0049] Vector-vector multiplication 100 multiplies an example weight vector [W] that includes P weights wl ... wP with an example input data vector [x] that includes P input values xl ... xP to obtain a weighted sum Y. In mathematics the result is known as the dot product of the two vectors. This operation is common in neural network processing and known as a multiply-accumulate (MAC) operation. For example, a neural network cell may have P input data values and a bias value. Each of the P input values is multiplied with a predetermined weight and included in the weighted sum Y. The bias is then added to the weighted sum Y.

[0050] To perform the MAC operation with limited photonic hardware, both vectors are timedivision multiplexed (TDM). The result is shown as TDM 110, where the input data takes the form of a time-discrete electric signal x(t), whose amplitude at time tj for i = 1 ... P equals the input data value Xj. The weights take the form of time-discrete electric signal w(t), whose amplitude at time tj equals the weight value Wj.

[0051] The electric signals x(t) and w(t) may be applied to PIC 130 which performs the P serialized multiplications and converts an optical signal to an electric signal for EIC 140, which may convert the multiplication result to a digital value. An implementation performs accumulation of the P serialized multiplications in either electrical integrator 141 which may be included in EIC 140, or in all-optical integrator 142, which may be included in PIC 130. PIC 130 may include a light source coupled with an optical input 131, and a light source may include for example a laser diode or a superluminescent diode (SLD). Optical input 131 receives light ata wavelength X from the light source and couples it into a photonic path 132. Different implementations may use different types of light sources, including laser diodes— these are highly coherent light sources that produce a narrow beam of light, and that are widely used in photonic chips for data communication and sensing applications; light-emitting diodes (LEDs) — a type of diode that emits light when a current passes through it (they are low-cost, compact, and have a long lifetime, making them a popular choice for photonic chips); superluminescent diodes (SLDs)— SLDs are similar to LEDs, but they emit a wider, broadband light spectrum (SLDs are used for applications such as optical amplification, wavelength division multiplexing, and fiber optic sensing); and vertical-cavity surface-emitting lasers (VCSELs)— these are lasers that emit light perpendicular to the chip surface, making them ideal for applications in photonics (VCSELs are widely used in data communication and sensing applications, such as 3D sensing, LiDAR, and data center interconnects). The light source may be or include a comb-generating laser source and / or a multiwavelength laser bank.

[0052] Photonic path 132 is optically coupled with first optical modulator 133 (MODI). Implementations may use any type of optical modulator known in the art, for example a LiNbO3 Mach- Zehnder modulator, or an electro-absorption modulator (EAM) which uses the Franz-Keldysh effect for an electrically induced charge in an optical absorption. The optical modulator may include materials selected from a group including germanium, silicon, an alloy of germanium, an alloy of silicon, a 11 l-V material based on indium phosphide (InP), and a lll-V material based on gallium arsenide (GaAs). In one or more implementations, the thermally stable optical modulator uses a quantum confined stark effect (Q.CSE) for an electrically induced change in optical absorption.

[0053] First optical modulator 133 receives the light X from optical input 131 and the time-discrete electric signal x(t) and modulates the light with x(t) to output a first modulated light signal x(t)X into photonic path 134 which is optically coupled between first optical modulator 133 and second opticalmodulator 135 (M0D2). Second optical modulator 135 is configured to receive the time-discrete signal w(t), and it modulates the first modulated light signal x(t)X with w(t), to output the second modulated light signal w(t)x(t)X into optical path 136. Optical path 136 is optically coupled between second optical modulator 135 and photodetector 137 which may be, or include, a photo diode that converts the light it receives into an electric signal, for example, an electric current. The amplitude (or another property) of the electric signal depends on the amplitude of the light received, and therefore it includes the information from the second modulated light signal w(t)x(t)X. PIC 130 passes the electric signal from photodetector 137 on to EIC 140 via an electric connection (or coupling), where it is received by electrical integrator 141 which may include a transimpedance amplifier (TIA) to convert current to voltage. Electrical integrator 141 sums its received signal over time, and therefore it performs an analog accumulation of the weighted input signals. Its output signal is representative for the result [W][x] of the MAC operation and, if desired, it may be converted to a digital signal using analog-to digital converter ADC 143.

[0054] Elements shown in FIG. 1 to perform a vector multiplication are used in all implementations described in this patent document. Elements may be integrated in a single PIC and a single EIC, as shown in FIG. 1, or in multiple PICs and / or multiple EICs. A PIC may include a light source and / or an all-optical integrator, and an EIC may include an electrical integrator.

[0055] FIG. 2 illustrates examples of four basic variations of the system in FIG. 1 to perform the vector multiplication. The variations can be applied to each of the example systems presented herein, and are fully within the scope and the ambit of the disclosed technology. The first implementation 200 first modulates the light with wavelength X with the time-multiplexed values of the column vector x(t) in first optical modulator 133, then modulates the modulated light with the time-multiplexed values of the row vector w(t) in second optical modulator 135. These two actions account for serialized multiplication of the values of the row vector with the values of the column vector. The result of the serialized multiplication is carried in an optical signal w(t)x(t)X, which is converted to an electrical signal by photodetector 137. Electrical integrator 141 performs accumulation of the serialized multiplication results to yield the dot product Y = [W][x],

[0056] In second implementation 225, the modulations are reversed. First optical modulator 133 first modulates the light with wavelength X with the time-multiplexed values of the row vector w(t), and second optical modulator 135 modulates the modulated light with the time-multiplexed values of the column vector x(t). Again, these two actions account for serialized multiplication of the values of the row vector with the values of the column vector. The result of the serialized multiplication is carried in an optical signal w(t)x(t)X, which is converted to an electrical signal by photodetector 137. Electrical integrator 141 performs accumulation of the serialized multiplication results to yield the dot product Y = [W][x],

[0057] In third implementation 250, the order of the multiplications is the same as in first implementation 200, but all-optical integrator 142 accumulates the results of the serialized multiplication of the values of the row vector with the values of the column vector before photodetector 137 converts the optical output signal of all-optical integrator 142 to an electrical signal that represents the dot productY = [W][x],

[0058] In fourth implementation 275, the order of the multiplications is the same as in second implementation 225, but all-optical integrator 142 accumulates the results of the serialized multiplication of the values of the row vector with the values of the column vector before photodetector 137 converts the optical output signal of all-optical integrator 142 to an electrical signal that represents the dot productY = [W][x],

[0059] Some example implementations presented later in this document are based on the topology of first implementation 200 or second implementation 225, but may be rearranged, mutatis mutandis, according to any of the topologies in first implementation 200, second implementation 225, third implementation 250, or fourth implementation 275. All such rearrangements are fully within the scope and ambit of the disclosed technology.

[0060] Some further implementations may add a non-linear function coupled with an output of the integrator. The non-linear function can act as an activation function and turns the example implementations in FIG. 3, FIG. 5, and FIG. 8 into complete neural cells that can be used in a photonic neural network. In the topologies of first implementation 200 and second implementation 225, the nonlinear function is an electronic function that can be added after electrical integrator 141 or that can be combined with electrical integrator 141. In the topologies of third implementation 250 and fourth implementation 275, the non-linear function is an electronic function that can be added after photodetector 137 or that can be combined with photodetector 137, or the non-linear function is a photonic function that can be coupled between all-optical integrator 142 and photodetector 137, or that can be combined with all-optical integrator 142.IMPLEMENTATIONS - Matrix-Vector Multiplication

[0061] FIG. 3 illustrates part of an example system 320 that includes PIC 330 to perform a matrixvector multiplication 300. System 320 also includes EIC hardware, which is similar to EIC 140 described with reference to FIG. 1, and not separately drawn. Matrix-vector multiplication 300 includes an example weights matrix [W] of dimensions MxP which is multiplied with an example input vector [x] of dimension P to obtain a weighted sum vector [Y] of dimension M. Note that the weighted sums Y may also be obtained by parallelizing M instances of PIC 130. However, the implementation of PIC 330 rather than M instances of PIC 130 saves M-l optical modulators, so that it has a lower cost to manufacture and operate.

[0062] The elements of both the weights matrix [W] and the input vector [x] are time-division multiplexed, as shown in TDM 310, where, as before, the input data takes the form of a time-discrete electric signal x(t), whose amplitude at time tj for j = 1 ... P equals the input data value Xj. The weights take the form of M time-discrete electric signals Wj(t), for i = 1 ... M, whose amplitude at time tj for j = 1 ... P equals the weight value Wjj.

[0063] The electric signals x(t) and Wj(t) may be applied to PIC 330 which performs the serialized multiplications and outputs those as electric signals to one or more EICs, which may accumulate the multiplication results and convert the accumulation result to a digital value. In case the electric signals carry their information in the form of currents, a single instance of EIC 140 may suffice to perform the integrations, if needed conversion to voltages, and the conversion to digital signals. In this implementation, PIC 330 includes a multiple-wavelengths light source 331, which may include M laser diodes (or other light sources, as discussed before) emitting light at wavelengths Xj to X^, and an optical multiplexer which may combine the M light beams in a single photonic multiplex X^ ... X[yj output into photonic path 132. Photonic path 132 is optically coupled with first optical modulator 133 (MODI). In other implementations, multiple-wavelengths light source 331 may be external to PIC 330, and more generally, any or all of the optical components shown in FIG. 3 may be internal or external to PIC 330. Implementations may use any type of optical modulator known in the art, as discussed before. Second optical modulator 135 receives the light X^ ... X|y| from multiple-wavelengths light source 331 and the timediscrete electric signal x(t) and modulates the light with x(t) to output a first modulated light signal x(t)[X ... X|y|] into photonic path 134 which is optically coupled between first optical modulator 133 and demultiplexer 335, which outputs signals x(t)Xj for i = 1 ... M. Each of these signals is coupled into a second optical modulator 135 that modulates it with the time-discrete electrical signal w;(t) to obtain second modulated signals Wj(t)x(t)Xj. Each first optical modulator 133-i couples the second modulated signal Wj(t)x(t)Xj into a photodetector, which outputs an electric signal for an EIC on electric signal output 338-i for integration and optional conversion to a digital signal.IMPLEMENTATIONS - Matrix-Matrix Multiplication

[0064] FIG. 4 illustrates an example matrix-matrix multiplication 400 and time-division multiplexing 410 of the matrix components. In this example, the weights matrix [W] has M rows and P columns (MxP size) and the input matrix [x] has P rows and N columns (PxN size). As a result, the output matrix [Y] has M rows and N columns (MxN size). The output matrix [Y] has MxN elements yjj, each representing a MAC operation. To perform the multiplications, implementations perform time-multiplexing of values in rows in the weights matrix and values in columns in the input matrix.

[0065] FIG. 5 illustrates an example system 500 that performs matrix-matrix multiplication 400 of FIG. 4. System 500 receives the time-division-multiplexed signals wj(t) and Xj(t), where i = 1 ... M and j = 1 ... N. Each time-division-multiplexed signal has P successive values. System 500 performs the serialized multiplications and accumulations, and outputs those as M x N electric signals. System 500 optionally includes a multiple-wavelengths light source (not drawn) which emits light at at least M wavelengths X to A|y| and provides its photonic multiplex [A^ ... A^j] to splitter input 531. The splitter 532 outputs the multi-wavelength light beam into M photonic paths, each coupled with a first optical modulator 133 that receives the time-division-multiplexed electric signal Wj(t) and that modulates the photonic multiplex [A ... A|y|] with Wj(t) to obtain a first modulated optical signal Wj(t) [A^ ... A|y|]. The first modulated optical signals are coupled into M inputs of an arrayed waveguide grating router (AWGR 535).

[0066] AWGR 535 has at least M inputs and M outputs if M is greater than N, or N inputs and N outputs if N is greater than M. Each of N of the AWGR 535 outputs is coupled with an input of a second optical modulator 135. A second optical modulator 135-j multiplies the received signal with time-division- multiplexed input signals vector Xj(t) to obtain a second modulated optical signal. Second optical modulator 135-j is coupled with a demultiplexer 538-j via optical path 136-j. Demultiplexer 538-i has M outputs for the M wavelengths A^ ... A| , where an output k feeds into a photodetector 539-ki which provides an electric signal for an EIC that takes care of integration to obtain the value y of the output matrix [Y].

[0067] Although some examples in FIGS. 1 through 5 show all photonic devices as being integrated in a single PIC, in some cases only part of the photonic devices may be integrated, whereas another part of the photonic devices may be external or embedded in other circuits. Implementations may include further photonic devices as desired for further functionality or enhanced performance.

[0068] FIG. 6 illustrates an example method 600 to multiply and accumulate parameters from a first matrix with M rows and P columns and a second matrix with P rows and M columns, where N, M, and P are positive integer numbers. Method 600 comprises the following steps:

[0069] Step 610 - splitting light with at least M different wavelengths into at least M light beams, each carrying the at least M different wavelengths.

[0070] Step 620 - in each of M first optical modulators, modulating light from one of the at least M light beams with a signal representing a time-division-multiplexed row vector of the first matrix to obtain a first modulated light signal.

[0071] Step 630 - in an arrayed waveguide grating router (AWGR), processing the first modulated light signal to obtain a processed light signal. The AWGR has at least M inputs and M outputs if M > N orat least N inputs and N outputs if N > M. M of the AWGR inputs receive output signals from the M first optical modulators.

[0072] Step 640 - in each of N second optical modulators, modulating a processed light signal received from the AWGR with a signal representing a time-division-multiplexed row vector of the second matrix to obtain a second modulated light signal.

[0073] Step 650 - in each of N demultiplexers, separating the second modulated light signal into N light beams each including one of M different wavelengths... A|y| .

[0074] Step 660 - in each of MxN integrators, integrating the signal from a light beam from one of the N demultiplexers to obtain an accumulated multiplied signal. In some implementations, the integrator is preceded by a photodetector to convert a signal from a light beam from the multiplexer into an electric signal, and the integrator is an electronic circuit. In some implementations, there are MxN integrators to integrate signals from N demultiplexer outputs of each N second optical modulators.

[0075] While in some implementations there are exactly M wavelengths, M first optical modulators, N second optical modulators, N demultiplexers, MxN photodetectors, and MxN integrators, in some other implementations there may be fewer or more of these devices if some of the coefficients of either the weights matrix, the input matrix, or the output matrix are known to be unused, or if additional functionality must be supported.IMPLEMENTATIONS - Tensor-Matrix Multiplication

[0076] FIG. 7 illustrates an example tensor-matrix multiplication 700 and associated time-division multiplexing 710 of the tensor and matrix components. In this example, the tensor (weights tensor [W]) has dimensions K x M x P, and the matrix (input matrix [x]) has dimensions P x N. K, M, N, and P are positive integers. Thus, there are K x M time-division-multiplexed vector signals for the weights matrices that are included in the weights tensor and N time-division-multiplexed vector signals for the input matrix. All time-division-multiplexed signals have a length of P discrete timesteps.

[0077] FIG. 8 illustrates an example photonic tensor multiplier 800 to perform tensor-matrix multiplication 700 of FIG. 7. Photonic tensor multiplier 800, parts or all of which may be integrated into one or more PICs and / or one or more EICs, receives the time-division-multiplexed signals Xj(t) and w^(t), where i = 1 ... M, j = 1 ... N, and k = 1 ... K. Photonic tensor multiplier 800 performs the serialized multiplications and may output the results to K x M x N instances of electrical integrator 141, included in the one or more PICs or in the one or more EICs. The integrators may accumulate the multiplication results and convert the accumulation result to digital values for the weighted sum tensor [Y], In case the electric signals carry their information in the form of currents, a single instance of EIC 140 may suffice to performthe integrations, if needed conversion to voltages, and the conversion to digital signals. Photonic tensor multiplier 800 may include or may be photonically coupled with a multiple-wavelengths light source, which may include laser diodes or other light sources, as discussed elsewhere in this document, emitting light at N different wavelengths A^ to A^, and an optional optical multiplexer or combiner (not separately drawn) which may combine light beams at separate wavelengths in a single photonic multiplex [A^ ... A^] coupled into input 831 of splitter 832. Splitter 832 is configured to split the light into N separate light beams, each carrying the light at the N different wavelengths (i.e., photonic multiplex [A^ ... A[sj]). The N outputs of splitter 832 are each coupled with a first optical modulator 133-j. The first optical modulator 133-j receives a time-division-multiplexed electric signal Xj(t) that represents a column vector of the matrix and modulates the photonic multiplex [A^ ... A j] with Xj(t) to obtain a first modulated optical signal Xj(t) [A-[ ... A|\j]. The N first modulated optical signals are coupled into N inputs of AWGR 834. If N is greater than M, then AWGR 834 may have N inputs and N outputs. If M is greater than N, then AWGR 834 may have M inputs and M outputs. An AWGR output 835-i carries a signal [xj(t) ... x^(t) x j(t) ... xj+i(t)] [A^ ... A^]. Up to this point, Photonic tensor multiplier 800 is similar or identical to system 500.

[0078] M of the AWGR 834 outputs are coupled with M splitters, e.g., AWGR output 835-i is coupled with splitter 836-i, which splits the optical signal into K separate light beams. Thus, there are K x M instances of a photonic path 837-ki coupled between an output (k) of a splitter 836-i and a second optical modulator 135-ki. Second optical modulator 135-ki modulates the signal [xj(t) ... x t) X[\j(t) ... Xj+i(t)] [A^ ... A^|] with the signal w^j(t) which represents a row vector included in the tensor to obtain a second modulated signal w^(t) [xj(t) ... xj(t) x^t) ... x;+^(t)] [A^ ... A^]. Second optical modulator 135-ki is coupled with demultiplexer 839-ki which separates the signal into its N individual wavelength A components. Each demultiplexer (839-ki) output signal includes a demultiplexed second modulated optical signal carried on one of the N different wavelengths. A demultiplexer output kil, with I = 1...N, may be coupled with a photodetector 137-kil to convert the demultiplexed second modulated optical signal to an electrical signal, which can be coupled into an electrical integrator 141, and optionally a transimpedance amplifier and / or ADC. The integrators, transimpedance amplifiers, and ADCs may be included in one or more EICs. Alternatively, an implementation may use an all-optical integrator photonically coupled with a demultiplexer output, and a photodetector may be coupled with an output of the all-optical integrator.

[0079] While the example in FIGS. 7 and 8 multiplies a weights tensor with an input signal matrix, the same architecture may be used to multiply an input signal tensor with a weights matrix (or any other combination of a matrix and a tensor).

[0080] FIG. 9 illustrates an example method 900 to multiply and accumulate parameters from a first tensor with K slices of M rows and P columns with a second matrix with P rows and N columns, where K, M, N, and P are positive integer numbers. Method 900 comprises the following steps:

[0081] Step 910 - splitting light with N different wavelengths into N light beams, each carrying the N different wavelengths.

[0082] Step 920 - in N first optical modulators, modulating light from the N light beams with signals representing time-division-multiplexed column vectors of the second matrix to obtain N first modulated light signals.

[0083] Step 930 - in an arrayed waveguide grating router (AWGR), processing the N first modulated light signals to obtain M processed light signals. The AWGR may have N inputs and N outputs if N is greater than M, or M inputs and M outputs if M is greater than N. N of the AWGR inputs receive output signals from the N first optical modulators.

[0084] Step 940 - in M splitters, splitting light from M AWGR outputs each in K light beams. As a result, there are K x M splitter outputs, each outputting a processed light signal.

[0085] Step 950 - in K x M second optical modulators, modulating K x M processed light signals received from the M splitters with signals representing time-division-multiplexed row vectors of a matrix included in the first tensor to obtain K x M second modulated light signals.

[0086] Step 960 - in K x M demultiplexers, separating the K x M second modulated light signals into K x M x N light beams each including one of the N different wavelengths... Xfj.

[0087] Step 970 - in K x M x N integrators, integrating the signals from the K x M x N light beams to obtain K x M x N accumulated multiplied signals. In some implementations, an integrator is preceded by a photodetector to convert a signal from a light beam from the multiplexer into an electric signal, and the integrator is an electronic circuit. In other implementations, an all-optical integrator is coupled with a demultiplexer output and may be followed by a photodetector to convert a signal from a light beam into an electric signal.PARTICULAR IMPLEMENTATIONSClause 1. A photonic matrix multiplier to perform multiplication of a first matrix with M rows and P columns with a second matrix with P rows and N columns, wherein M, N, and P are positive integers, the photonic matrix multiplier comprising: a splitter configured to receive light from a light source, the light including components at M different wavelengths, wherein the splitter has M splitter outputs;M first optical modulators, each photonically coupled with one of the M splitter outputs, and each configured to receive a first time-multiplexed electric signal representing a row vector of the first matrix; an arrayed waveguide grating router (AWGR) with M inputs photonically coupled with outputs of the M first optical modulators;N second optical modulators, each photonically coupled with one of N outputs of the AWGR, and each configured to receive a second time-multiplexed electric signal representing a column vector of the second matrix;N demultiplexers with an input and M outputs, wherein the input is coupled with an output of one of the N second optical modulators, and wherein a demultiplexer is configured to separate light received at its input into M components at the M different wavelengths, providing one of the M components at one of the M outputs; andM x N integrators, each one coupled with one of the M outputs of one of the N demultiplexers.Clause 2. The photonic matrix multiplier of clause 1, wherein the AWGR has M inputs and M outputs if M is greater than N, and N inputs and N outputs if N is greater than or equal to M.Clause 3. The photonic matrix multiplier of clause 1, wherein the first time-multiplexed electric signal and the second time-multiplexed electric signal each include P successive values.Clause 4. The photonic matrix multiplier of clause 1, wherein the first matrix represents weights and the second matrix includes input signals.Clause 5. The photonic matrix multiplier of clause 1, wherein the first matrix includes input signals and the second matrix represents weights.Clause 6. The photonic matrix multiplier of clause 1, further comprising the light source.Clause 7. The photonic matrix multiplier of clause 6, wherein the light source includes a laser diode.Clause 8. The photonic matrix multiplier of clause 6, wherein the light source includes a superluminescent diode (SLD).Clause 9. The photonic matrix multiplier of clause 6, wherein the light source includes one of a combgenerating laser source and / or a multiwavelength laser bank.Clause 10. The photonic matrix multiplier of clause 1, wherein the splitter, the M first optical modulators, the N second optical modulators, and the N demultiplexers are included in one or more photonic integrated circuits (PICs).Clause 11. The photonic matrix multiplier of clause 1, wherein an integrator is an all-optical device.Clause 12. The photonic matrix multiplier of clause 1, further comprising M x N photodetectors, each coupled between one of the M outputs of the N demultiplexers and an input of the M x N integrators, wherein the integrators are electrical integrators.Clause 13. The photonic matrix multiplier of clause 1, further comprising a transimpedance amplifier (TIA).Clause 14. The photonic matrix multiplier of clause 1, further comprising an analog-to-digital converter (ADC).Clause 15. The photonic matrix multiplier of clause 1, further comprising a non-linear function coupled after or combined with an integrator to act as an activation function.Clause 16. A method to multiply and accumulate parameters from a first matrix with M rows and P columns and a second matrix with P rows and N columns, wherein M, N, and P are positive integer numbers, comprising: splitting light with M wavelengths into M light beams, each carrying the M different wavelengths; in M first optical modulators, modulating light from the M light beams with signals representing time-division-multiplexed row vectors of the first matrix to obtain M first modulated light signals; in an arrayed waveguide grating router (AWGR), processing the M first modulated light signals to obtain N processed light signals; in N second optical modulators, modulating the N processed light signals with signals representing time-division-multiplexed column vectors of the second matrix to obtain N second modulated light signals; in N demultiplexers, separating the N second modulated light signals into MxN light beams each including one of the M different wavelengths; and in an integrator, integrating a signal from one of the M light beams from one of the N demultiplexers to obtain an accumulated multiplied signal.CONSIDERATIONS

[0088] Although the description has been described with respect to distinct implementations thereof, these distinct implementations are merely illustrative, and not restrictive. The description may reference specific structural implementations and methods and does not intend to limit the technology to the specifically disclosed implementations and methods. The technology may be practiced using other features, elements, methods and implementations. Implementations are described to illustrate thepresent technology, not to limit its scope, which is defined by the claims. Those of ordinary skill in the art recognize a variety of equivalent variations on the description above.

[0089] All features disclosed in the specification, including the claims, abstract, and drawings, and all the steps in any method or process disclosed, may be combined in any combination, except combinations where at least some of such features and / or steps are mutually exclusive. Each feature disclosed in the specification, including the claims, abstract, and drawings, can be replaced by alternative features serving the same, equivalent, or similar purpose, unless expressly stated otherwise.

[0090] Although the description has been described with respect to distinct implementations thereof, these particular implementations are merely illustrative, and not restrictive. For instance, implementations may be as a single chip PIC and a single-chip EIC, or as multiple dies that may be packaged a multi-chip module (MCM) that interconnects the various components. All such variations and modifications are to be considered within the ambit of the disclosed technology the nature of which is to be determined from the foregoing description.

[0091] Any suitable technology for manufacturing electronic devices can be used to implement the circuits of distinct implementations, including CMOS, FinFET, GAAFET, BiCMOS, bipolar, JFET, MOS, NMOS, PMOS, HBT, MESFET, etc. Different semiconductor materials can be employed for the EIC and the PIC, such as silicon, germanium, SiGe, GaAs, InP, GaN, SiC, graphene, etc. Circuits may have single-ended or differential inputs, and single-ended or differential outputs. Terminals to circuits may function as inputs, outputs, both, or be in a high-impedance state, or they may function to receive supply power, a ground reference, a reference voltage, a reference current, or other. Although the physical processing of signals may be presented in a specific order, this order may be changed in different particular implementations. In some distinct implementations, multiple elements, devices, or circuits shown as sequential in this specification can be operating in parallel.

[0092] Different integration technologies can be used for the PIC, such as silicon photonics, InP, silicon nitride, hybrid and heterogeneous integration technologies, etc. Implementations may use different technologies for optical modulators, including Mach-Zehnder, microring resonator, SiGe electroabsorption modulator, Franz-Keldysh electroabsorption modulators, a quantum-confined Start effect (Q.CSE) electroabsorption modulator, a quantum well modulator (CQ.W), etc. Different alternative layouts can be employed for an AWGR implementation, such as a combination of interconnected multiplexers and demultiplexers, echelle grating structures, etc. PICs may operate at any wavelength band. Any type of laser source can be employed, like distributed feedback lasers, laser diodes, hybrid integrated InP lasers, multiwavelength laser sources, microcomb light generator sources, etc. Different technologies can be implemented for photodiodes, such as SiGe electro-absorption photodiodes, InP photodiodes, avalanche photodiodes, etc.

[0093] It will also be appreciated that one or more of the elements depicted in the drawings / figures can also be implemented in a more separated or integrated manner, or even removed or rendered as inoperable in certain cases, as is useful in accordance with a particular application.

[0094] Thus, while distinct implementations have been described herein, latitudes of modification, various changes, and substitutions are intended in the foregoing disclosures, and it will be appreciated that in some instances some features of particular implementations will be employed without a corresponding use of other features without departing from the scope and spirit as set forth. Therefore, many modifications may be made to adapt a particular situation or material to the essential scope and spirit.

Claims

CLAIMS1. A photonic matrix multiplier to perform multiplication of a first matrix with M rows and P columns with a second matrix with P rows and N columns, wherein M, N, and P are positive integers, the photonic matrix multiplier comprising: a splitter configured to receive light from a light source, the light including components at M different wavelengths, wherein the splitter has M splitter outputs;M first optical modulators, each photonically coupled with one of the M splitter outputs, and each configured to receive a first time-multiplexed electric signal representing a row vector of the first matrix; an arrayed waveguide grating router (AWGR) with M inputs photonically coupled with outputs of the M first optical modulators;N second optical modulators, each photonically coupled with one of N outputs of the AWGR, and each configured to receive a second time-multiplexed electric signal representing a column vector of the second matrix;N demultiplexers with an input and M outputs, wherein the input is coupled with an output of one of the N second optical modulators, and wherein a demultiplexer is configured to separate light received at its input into M components at the M different wavelengths, providing one of the M components at one of the M outputs; andM x N integrators, each one coupled with one of the M outputs of one of the N demultiplexers.

2. The photonic matrix multiplier of claim 1, wherein the AWGR has M inputs and M outputs if M is greater than N, and N inputs and N outputs if N is greater than or equal to M.

3. The photonic matrix multiplier of claim 1 or claim 2, wherein the first time-multiplexed electric signal and the second time-multiplexed electric signal each include P successive values.

4. The photonic matrix multiplier of any of the preceding claims, wherein the first matrix represents weights and the second matrix includes input signals.

5. The photonic matrix multiplier of any of the preceding claims, wherein the first matrix includes input signals and the second matrix represents weights.

6. The photonic matrix multiplier of any of the preceding claims, further comprising the light source.

7. The photonic matrix multiplier of claim 6, wherein the light source includes a laser diode.

8. The photonic matrix multiplier of claim 6, wherein the light source includes a superluminescent diode (SLD).

9. The photonic matrix multiplier of claim 6, wherein the light source includes one of a combgenerating laser source and / or a multiwavelength laser bank.

10. The photonic matrix multiplier of any of the preceding claims, wherein the splitter, the M first optical modulators, the N second optical modulators, and the N demultiplexers are included in one or more photonic integrated circuits (PICs).

11. The photonic matrix multiplier of any of the preceding claims, wherein an integrator is an all-optical device.

12. The photonic matrix multiplier of any of claim 1 to claim 10, further comprising M x N photodetectors, each coupled between one of the M outputs of the N demultiplexers and an input of the M x N integrators, wherein the integrators are electrical integrators.

13. The photonic matrix multiplier of any of the preceding claims, further comprising: a transimpedance amplifier (TIA).

14. The photonic matrix multiplier of any of the preceding claims, further comprising: an analog-to-digital converter (ADC).

15. The photonic matrix multiplier of any of the preceding claims, further comprising a non-linear function coupled after or combined with an integrator to act as an activation function.

16. A method to multiply and accumulate parameters from a first matrix with M rows and P columns and a second matrix with P rows and N columns, wherein M, N, and P are positive integer numbers, comprising: splitting light with M different wavelengths into M light beams, each carrying the M different wavelengths; in M first optical modulators, modulating light from the M light beams with signals representing time-division-multiplexed row vectors of the first matrix to obtain M first modulated light signals; in an arrayed waveguide grating router (AWGR), processing the M first modulated light signals to obtain N processed light signals; in N second optical modulators, modulating the N processed light signals with signals representing time-division-multiplexed column vectors of the second matrix to obtain N second modulated light signals; in N demultiplexers, separating the N second modulated light signals into MxN light beams each including one of the M different wavelengths; andin an integrator, integrating a signal from one of the M light beams from one of the N demultiplexers to obtain an accumulated multiplied signal.

Images