Computing system and computing device

Abstract

A computing system includes: a first unit generating a plurality of modulator control signals; a processor unit comprising: a light source or port providing a plurality of light outputs; a first set of light modulators coupled to a light source or port and the first unit, light modulators of the first set of light modulators configured to modulate the plurality of light outputs provided by the light source or port based on digital input values corresponding to a first set of modulator control signals of the plurality of modulator control signals, generating an optical input vector, the optical input vector comprising a plurality of optical signals; the matrix multiplication unit comprises a second group of light modulators, and the matrix multiplication unit is coupled to the first unit; and the matrix multiplication unit is configured to transform the optical input vector to an analog output vector based on a plurality of digital weight values corresponding to a second set of modulator control signals of the plurality of modulator control signals applied to the second set of optical modulators.

Description

[0001] This patent application is a divisional application of the following invention patent applications:

[0002] Application number: 201980066434.2

[0003] Application date: December 4, 2019

[0004] Invention Name: Photoelectric Computing System

[0005] CROSS-REFERENCE TO RELATED APPLICATIONS

[0006] This application claims priority to U.S. Provisional Application No. 62 / 792,144, filed January 14, 2019, U.S. Provisional Application No. 62 / 820,562, filed March 19, 2019, and U.S. Application No. 16 / 431,167, filed June 4, 2019. The entire disclosures of the above applications are incorporated herein by reference. Technical Field

[0007] The present disclosure relates to an optoelectronic computing system. Background Art

[0008] Neuromorphic computing is an approach to electronics that mimics the brain's operations. A prominent approach to neuromorphic computing is the artificial neural network (ANN), a collection of artificial neurons interconnected in a specific pattern that processes information in a manner similar to the brain's functions. ANNs have found use in a variety of applications, including artificial intelligence, speech recognition, text recognition, natural language processing, and various forms of pattern recognition.

[0009] An ANN has an input layer, one or more hidden layers, and an output layer. Each layer has nodes or artificial neurons, and the nodes are interconnected between the layers. Each node in a hidden layer performs a weighted sum of the signals received from the nodes in the previous layer and performs a nonlinear transformation ("activation") of the weighted sum to produce an output. The weighted sum can be calculated by performing a matrix multiplication step. Therefore, calculating an ANN typically involves multiple matrix multiplication steps, which are typically performed using electronic integrated circuits.

[0010] Computations performed on electronic data encoded in analog or digital form on electronic signals (e.g., voltage or current) are typically implemented using electronic computing hardware, such as analog or digital electronics implemented in integrated circuits (e.g., processors, application-specific integrated circuits (ASICs), or systems on a chip (SoCs)), electronic circuit boards, or other electronic circuits. Optical signals have been used to transmit data over long distances and shorter distances (e.g., within data centers). Operations performed on such optical signals are typically performed in the context of optical data transmission, such as within devices used to switch or filter optical signals in a network. The use of optical signals in computing platforms has been more limited. Various components and systems for all-optical computing have been proposed. Such systems may include conversions to and from electrical signals at input and output, respectively, but both types of signals (electrical and optical) cannot be used for important operations performed in computing. Summary of the Invention

[0011] In general, in a first aspect, a system includes: a first unit configured to generate a plurality of modulator control signals; and a processing unit. The processing unit includes: a light source or port configured to provide a plurality of light outputs; and a first group of light modulators coupled to the light source or port and the first unit. The light modulators in the first group of light modulators are configured to modulate the plurality of light outputs provided by the light source or port based on digital input values ​​corresponding to a first group of modulator control signals in a plurality of modulator control signals to generate a light input vector, the light input vector including a plurality of light signals. The processing unit also includes a matrix multiplication unit, the matrix multiplication unit including a second group of light modulators. The matrix multiplication unit is coupled to the first unit and configured to convert the light input vector into an analog output vector based on a plurality of digital weight values ​​corresponding to a second group of modulator control signals in a plurality of modulator control signals applied to the second group of light modulators. At least one optical modulator of at least one of the first group of optical modulators or the second group of optical modulators is configured to modulate an optical signal based on a first modulator control signal of the plurality of modulator control signals, and the first unit is configured to shape the first modulator control signal to include a bandwidth enhancement associated with amplitude variations associated with corresponding variations in consecutive digital values ​​corresponding to the first modulator control signal.

[0012] Embodiments of the system may include one or more of the following features. The system may include a second unit coupled to the matrix multiplication unit and configured to convert an analog output vector into a digital output vector; and a controller. The controller may include an integrated circuit configured to perform operations including: receiving an artificial neural network computation request, the artificial neural network computation request including an input data set including a first digital input vector; receiving a first plurality of neural network weights; and generating, via the first unit, a first plurality of modulator control signals based on the first digital input vector and a first plurality of weight control signals based on the first plurality of neural network weights.

[0013] The first unit may include a digital to analog converter (DAC).

[0014] The system may include a storage unit configured to store a data set and a plurality of neural network weights.

[0015] The integrated circuit of the controller may also be configured to perform operations including storing the input data set and the first plurality of neural network weights in the memory unit.

[0016] The controller may include an application specific integrated circuit (ASIC), and receiving the artificial neural network computation request may include receiving the artificial neural network computation request from a general purpose data processor.

[0017] The first unit, the processing unit, the second unit, and the controller may be arranged on at least one of a multi-chip module or an integrated circuit. Receiving the artificial neural network computation request may include receiving the artificial neural network computation request from a second data processor, wherein the second data processor is external to the multi-chip module or the integrated circuit, the second data processor is coupled to the multi-chip module or the integrated circuit via a communication channel, and the processing unit can process data at a data rate that is at least one order of magnitude greater than a data rate of the communication channel.

[0018] The first unit, the processing unit, the second unit, and the controller can be used to perform a photoelectric processing cycle that is repeated in multiple iterations. The photoelectric processing cycle includes: (1) at least a first light modulation operation based on at least one modulator control signal, and at least a second light modulation operation based on at least one weight control signal, and (2) at least one of (a) an electrical summing operation or (b) an electrical storage operation.

[0019] The photoelectric processing cycle may include an electrical storage operation, and the electrical storage operation may be performed using a memory unit coupled to the controller. The operations performed by the controller may also include storing the input data set and the first plurality of neural network weights in the memory unit.

[0020] The photoelectric processing cycle may include an electrical summation operation, and the electrical summation operation may be performed using an electrical summation module within the matrix multiplication unit. The electrical summation module may be configured to generate a current corresponding to an element of the analog output vector, the current representing the sum of the corresponding element of the photoinput vector multiplied by the corresponding neural network weight.

[0021] The first modulator control signal may include an analog signal associated with a plurality of predetermined amplitude levels, and each of the amplitude levels is associated with a different corresponding digital value.

[0022] The first modulator control signal may include an analog signal associated with two predetermined amplitude levels, and each of the amplitude levels is associated with a different corresponding binary value.

[0023] The continuous digital value may include a plurality of consecutive binary values ​​in a series of binary values.

[0024] The controller may be configured to shape the first modulator control signal to include bandwidth enhancement for an initial portion of the second time interval by increasing the magnitude of an amplitude variation between a first predetermined amplitude level associated with the first time interval and a second predetermined amplitude level associated with the second time interval.

[0025] A series of binary values ​​may be used to determine an amplitude level of a first modulator control signal used to modulate an optical signal according to a non-return-to-zero (NRZ) modulation scheme.

[0026] The first unit may be configured to shape the first modulator control signal to include bandwidth enhancement by pumping a current between a diode structure of a first modulator in the second group of optical modulators and a capacitor connected in series between the diode structure and a circuit that provides the first modulator control signal, and an amount of charge transferred by the pumping current may be determined at least in part based on a constant voltage over a time period that provides continuous digital values.

[0027] In another general aspect, an apparatus includes: a plurality of optical waveguides coupled to a first set of optical amplitude modulators, wherein a plurality of input values ​​are encoded on respective optical signals carried by the optical waveguides using the first set of optical amplitude modulators. The apparatus includes a plurality of replication modules, and for each of at least two subsets of the one or more optical signals, a corresponding set of the one or more replication modules is configured to split the subset of the one or more optical signals into two or more copies of the optical signals. The apparatus includes a plurality of multiplication modules, each of the multiplication modules including an optical amplitude modulator from a second set of optical amplitude modulators, and for each of the at least two copies of a first subset of the one or more optical signals, a corresponding one of the multiplication modules is configured to multiply the one or more optical signals of the first subset by one or more matrix element values ​​using the optical amplitude modulator from the second set of optical amplitude modulators. The apparatus includes one or more summation modules, and for results of two or more multiplication modules, a corresponding one of the summation modules is configured to generate an electrical signal representing the sum of the results of the two or more multiplication modules. At least one optical amplitude modulator of at least one of the first group of optical amplitude modulators or the second group of optical amplitude modulators is configured to modulate the optical signal by the modulation value using a power that increases monotonically with respect to an absolute value of the modulation value.

[0028] Embodiments of the apparatus may include one or more of the following features: At least one optical amplitude modulator of at least one of the first set of optical amplitude modulators or the second set of optical amplitude modulators may include a coherence-sensitive optical amplitude modulator configured to modulate the optical signal by a modulation value based on interference between light waves, the light waves having a coherence length that is at least as long as a propagation distance through the coherence-sensitive optical amplitude modulator.

[0029] The coherence-sensitive optical amplitude modulator may include a Mach-Zehnder Interferometer (MZI) that distributes a light wave guided by an input optical waveguide into a first optical waveguide arm and a second optical waveguide arm of the MZI. The first optical waveguide arm may include an active phase shifter that generates a relative phase shift relative to the phase delay of the second optical waveguide arm, and the MZI may combine the light waves from the first optical waveguide arm and the second optical waveguide arm into at least one output optical waveguide.

[0030] The power used to modulate the optical signal by the modulation value may include power applied to an active phase shifter.

[0031] An input value in the set of multiple input values ​​encoded on the respective optical signal may represent an element of an input vector multiplied by a matrix comprising one or more matrix element values.

[0032] A set of multiple output values ​​may be encoded on a plurality of corresponding electrical signals generated by the one or more summing modules, and the output values ​​in the set of multiple output values ​​may represent elements of an output vector generated by multiplying the input vector by the matrix.

[0033] Each of the optical signals carried by the optical waveguide may comprise a light wave having a common wavelength, the common wavelength being substantially the same for all of the optical signals.

[0034] The replication module may include at least one replication module having an optical splitter that transmits a predetermined proportion of the power of the lightwave at the input port of the replication module to a first output port of the replication module and transmits a remaining proportion of the power of the lightwave at the input port of the replication module to a second output port of the replication module.

[0035] The optical splitter may include a waveguide splitter that transmits a predetermined proportion of the power of the light waves guided by the input optical waveguide of the replication module to a first output optical waveguide of the replication module, and transmits a remaining proportion of the power of the light waves guided by the input optical waveguide of the replication module to a second output optical waveguide of the replication module.

[0036] The guided mode of the input optical waveguide may be adiabatically coupled to the plurality of guided modes of each of the first and second output optical waveguides.

[0037] The optical splitter may include a beam splitter including at least one surface that transmits a predetermined proportion of the power of the light waves at the input port and reflects a remaining proportion of the power of the light waves at the input port.

[0038] At least one of the plurality of optical waveguides may include an optical fiber coupled to an optical coupler that couples a guided mode of the optical fiber to a free-space propagation mode.

[0039] The multiplication module may include at least one coherence-sensitive optical amplitude modulator configured to multiply the one or more optical signals of the first subset by one or more matrix element values ​​based on interference between optical waves, the optical waves having a coherence length that is at least as long as a propagation distance through the coherence-sensitive optical amplitude modulator.

[0040] The coherence-sensitive optical amplitude modulator may include a Mach-Zehnder interferometer (MZI) that distributes a light wave guided by an input optical waveguide to a first optical waveguide arm and a second optical waveguide arm of the MZI. The first optical waveguide arm may include a phase shifter that generates a relative phase shift relative to a phase delay of the second optical waveguide arm, and the MZI may combine multiple light waves from the first optical waveguide arm and the second optical waveguide arm into at least one output optical waveguide.

[0041] The Mach-Zehnder interferometer can combine lightwaves from the first optical waveguide arm and the second optical waveguide arm into each of the first output optical waveguide and the second output optical waveguide. The first photodetector can receive the lightwave from the first output optical waveguide to generate a first photocurrent, the second photodetector can receive the lightwave from the second output optical waveguide to generate a second photocurrent, and the result of the coherence-sensitive optical amplitude modulator can include the difference between the first photocurrent and the second photocurrent.

[0042] The coherence-sensitive optical amplitude modulator may include one or more ring resonators, including at least one ring resonator coupled to the first optical waveguide and at least one ring resonator coupled to the second optical waveguide.

[0043] The first photodetector may receive a light wave from the first optical waveguide to generate a first photocurrent, the second photodetector may receive a light wave from the second optical waveguide to generate a second photocurrent, and a result of the coherent sensitive optical amplitude modulator may include a difference between the first photocurrent and the second photocurrent.

[0044] The multiplication module may include at least one coherent insensitive optical amplitude modulator configured to multiply the one or more optical signals of the first subset by one or more matrix element values ​​based on energy absorption within the optical wave.

[0045] The coherent insensitive optical amplitude modulator may comprise an electro-absorption modulator.

[0046] The one or more summing modules may include at least one summing module having the following components: (1) two or more input conductors, each of the input conductors carrying an electrical signal in the form of an input current, the magnitude of the input current representing a respective result of a respective one of the multiplication modules, and (2) at least one output conductor, the output conductor carrying an electrical signal in the form of an output current representing a sum of the respective results, the output current being proportional to the sum of the input currents.

[0047] The two or more input conductors and the output conductor may include wires that touch at one or more junctions between the wires, and the output current may be substantially equal to the sum of the input currents.

[0048] At least a first one of the input currents may be provided in the form of at least one photocurrent generated by at least one photodetector receiving the optical signal generated by a first one of the multiplication modules.

[0049] The first input current may be provided in the form of a difference between two photocurrents, the two photocurrents being generated by different corresponding photodetectors that receive different corresponding light signals generated by the first multiplication module.

[0050] One of the copies of the first subset of the one or more optical signals may consist of a single optical signal on which one of the input values ​​is encoded.

[0051] The multiplication modules corresponding to the replicas of the first subset may multiply the encoded input value by a single matrix element value.

[0052] One of the copies of the first subset of the one or more optical signals may include more than one, and less than all, of the optical signals on which the plurality of input values ​​are encoded.

[0053] The multiplication modules corresponding to the replicas of the first subset may multiply the encoded input values ​​by different corresponding matrix element values.

[0054] Different multiplication modules corresponding to different respective copies of the first subset of the one or more optical signals may be included in different devices that are in optical communication to transmit one of the copies of the first subset of the one or more optical signals between the different devices.

[0055] Two or more of the plurality of optical waveguides, two or more of the plurality of replica modules, two or more of the plurality of multiplication modules, and at least one of the one or more summation modules may be arranged on a substrate of a common device.

[0056] The apparatus may perform vector-matrix multiplication, where an input vector may be provided as a set of optical signals and an output vector may be provided as a set of electrical signals.

[0057] The device may also include an accumulator that integrates input electrical signals corresponding to outputs of the multiplication module or the summation module, wherein the input electrical signal is encoded using time domain coding, the time domain coding uses switching amplitude modulation within each of a plurality of time slots, and the accumulator generates an output electrical signal, the output electrical signal is encoded with more than two amplitude levels, the amplitude levels corresponding to different duty cycles of the time domain coding over the plurality of time slots.

[0058] Each of the two or more multiplication modules may correspond to a different subset of the one or more optical signals.

[0059] The apparatus may further include: a multiplication module for each copy of a second subset of one or more optical signals different from the optical signals in the first subset of one or more optical signals, configured to multiply the one or more optical signals of the second subset by one or more matrix element values ​​using optical amplitude modulation.

[0060] In another general aspect, a method includes encoding a set of a plurality of input values ​​on respective optical signals using a first set of optical amplitude modulators; for each of at least two subsets of the one or more optical signals, using a corresponding set of one or more replication modules to split the subset of the one or more optical signals into two or more replicas of the optical signals; for each of the at least two replicas of the first subset of the one or more optical signals, using a corresponding multiplication module to multiply the one or more optical signals of the first subset by one or more matrix element values ​​using an optical amplitude modulator of a second set of optical amplitude modulators; and for results of the two or more multiplication modules, using a summation module configured to generate an electrical signal representing a sum of the results of the two or more multiplication modules. At least one optical amplitude modulator of at least one of the first set of optical amplitude modulators or the second set of optical amplitude modulators is configured to modulate the optical signal by a modulation value using a power that monotonically increases with respect to an absolute value of the modulation value.

[0061] In another general aspect, a system includes: a storage unit configured to store a data set and a plurality of neural network weights; a digital-to-analog conversion (DAC) unit configured to generate a plurality of modulator control signals and to generate a plurality of weight control signals; an optical processor including a laser unit configured to generate a plurality of optical outputs; a plurality of optical modulators coupled to the laser unit and the DAC unit, the plurality of optical modulators configured to generate an optical input vector by modulating the plurality of optical outputs generated by the laser unit based on the plurality of modulator control signals; an optical matrix multiplication unit coupled to the plurality of optical modulators and the DAC unit, the optical matrix multiplication unit configured to convert the optical input vector into an optical output vector based on the plurality of weight control signals; and a photodetection unit. coupled to the optical matrix multiplication unit and configured to generate a plurality of output voltages corresponding to the optical output vector; an analog-to-digital conversion (ADC) unit, coupled to the photodetection unit and configured to convert the plurality of output voltages into a plurality of digital optical outputs; a controller, comprising an integrated circuit, configured to perform the following operations: receive an artificial neural network calculation request including an input data set and a first plurality of neural network weights from a computer, wherein the input data set includes a first digital input vector; store the input data set and the first plurality of neural network weights in a storage unit; and generate, through a DAC unit, a first plurality of modulator control signals based on the first digital input vector, and generate a first plurality of weight control signals based on the first plurality of neural network weights.

[0062] Embodiments of the system may include one or more of the following features. For example, the operations may further include: obtaining a first plurality of digital light outputs corresponding to the light output vector of the light matrix multiplication unit from the ADC unit, the first plurality of digital light outputs forming a first digital output vector; performing a nonlinear transformation on the first digital output vector to generate a first transformed digital output vector; and storing the first transformed digital output vector in a storage unit.

[0063] The system may have a first cycle period defined as the time elapsed between the step of storing the input data set and the first plurality of neural network weights in the memory unit and the step of storing the first transformed digital output vector in the memory unit. The first cycle period may be less than or equal to 1 ns.

[0064] In some embodiments, the operations may further include outputting an artificial neural network output generated based on the first transformed digital output vector.

[0065] In some embodiments, the operations may further include generating, by a DAC unit, a second plurality of modulator control signals based on the first transformed digital output vector.

[0066] In some embodiments, the artificial neural network calculation request may further include a second plurality of neural network weights, and the operation may further include: generating, by a DAC unit, a second plurality of weight control signals based on the second plurality of neural network weights based on the acquisition of the first plurality of digital light outputs. The first plurality of neural network weights and the second plurality of neural network weights may correspond to different layers of the artificial neural network.

[0067] In some embodiments, the input data set may further include a second digital input vector, and the operation may further include: generating, by the DAC unit, a second plurality of modulator control signals based on the second digital input vector; obtaining, from the ADC unit, a second plurality of digital optical outputs corresponding to the optical output vector of the optical matrix multiplication unit, the second plurality of digital optical outputs forming a second digital output vector; performing a nonlinear transformation on the second digital output vector to generate a second transformed digital output vector; storing the second transformed digital output vector in a storage unit; and outputting an artificial neural network output generated based on the first transformed digital output vector and the second transformed digital output vector. The optical output vector of the optical matrix multiplication unit is generated by the second optical input vector generated based on the second plurality of modulator control signals, the second optical input vector being transformed by the optical matrix multiplication unit based on the plurality of weight control signals mentioned first.

[0068] In some embodiments, the system may further include: an analog nonlinear unit, arranged between the photodetection unit and the ADC unit, the analog nonlinear unit being configured to receive multiple output voltages from the photodetection unit, apply a nonlinear transfer function, and output multiple conversion output voltages to the ADC unit, and the operation further includes: obtaining a first plurality of converted digital output voltages corresponding to the multiple conversion output voltages from the ADC unit, the first plurality of converted digital output voltages forming a first converted digital output vector; and storing the first converted digital output vector in a storage unit.

[0069] In some embodiments, the integrated circuit of the controller may be configured to generate the first plurality of modulator control signals at a rate greater than or equal to 8 GHz.

[0070] In some embodiments, the system may further include: an analog storage unit disposed between the DAC unit and the plurality of optical modulators, the analog storage unit configured to store analog voltages and output the stored analog voltages; and an analog nonlinear unit disposed between the photodetection unit and the ADC unit, the analog nonlinear unit configured to receive the plurality of output voltages from the photodetection unit, apply a nonlinear transfer function, and output a plurality of converted output voltages. The analog storage unit may include a plurality of capacitors.

[0071] In some embodiments, the analog storage unit may be configured to receive and store multiple conversion output voltages of the analog nonlinear unit, and output the stored multiple conversion output voltages to multiple optical modulators, and the operation may further include: storing the multiple conversion output voltages of the analog nonlinear unit in the analog storage unit based on generating a first plurality of modulator control signals and a first plurality of weight control signals; outputting the stored conversion output voltages through the analog storage unit; obtaining a second plurality of conversion digital output voltages from the ADC unit, the second plurality of conversion digital output voltages forming a second transformed digital output vector; and storing the second transformed digital output vector in the storage unit.

[0072] In some embodiments, the input data set requested by the artificial neural network for computation may include multiple digital input vectors. The laser unit may be configured to generate multiple wavelengths. The multiple optical modulators may include: a bank of optical modulators configured to generate multiple optical input vectors, each optical modulator bank corresponding to one of the multiple wavelengths and generating a corresponding optical input vector having the corresponding wavelength; and an optical multiplexer configured to combine the multiple optical input vectors into a combined optical input vector including the multiple wavelengths. The photodetector unit may be further configured to demultiplex the multiple wavelengths and generate multiple demultiplexed output voltages. Operations may include: obtaining multiple digital demultiplexed optical outputs from the ADC unit, the multiple digital demultiplexed optical outputs forming multiple first digital output vectors, wherein each of the multiple first digital output vectors corresponds to one of the multiple wavelengths; performing a nonlinear transformation on each of the multiple first digital output vectors to generate multiple transformed first digital output vectors; and storing the multiple transformed first digital output vectors in a storage unit. Each of the multiple digital input vectors may correspond to one of the multiple optical input vectors.

[0073] In some embodiments, the artificial neural network computation request may include multiple digital input vectors. The laser unit may be configured to generate multiple wavelengths. The multiple optical modulators may include: an optical modulator group configured to generate multiple optical input vectors, each optical modulator group corresponding to one of the multiple wavelengths and generating a corresponding optical input vector having the corresponding wavelength; and an optical multiplexer configured to combine the multiple optical input vectors into a combined optical input vector including multiple wavelengths. Operations may include: obtaining a first plurality of digital optical outputs corresponding to the optical output vector from the ADC unit, the optical output vector including the multiple wavelengths, the first plurality of digital optical outputs forming a first digital output vector; performing a nonlinear transformation on the first digital output vector to generate a first transformed digital output vector; and storing the first transformed digital output vector in a storage unit.

[0074] In some embodiments, the DAC unit may include: a 1-bit DAC subunit configured to generate a plurality of 1-bit modulator control signals. The resolution of the ADC unit may be 1 bit. The resolution of the first digital input vector may be N bits. The operations may include: decomposing the first digital input vector into N 1-bit input vectors, each of the N 1-bit input vectors corresponding to one of the N bits of the first digital input vector; generating a sequence of N 1-bit modulator control signals corresponding to the N 1-bit input vectors through the 1-bit DAC subunit; obtaining a sequence of N digital 1-bit optical outputs corresponding to the sequence of N 1-bit modulator control signals from the ADC unit; constructing an N-bit digital output vector from the sequence of N digital 1-bit optical outputs; performing a nonlinear transformation on the constructed N-bit digital output vector to generate a transformed N-bit digital output vector; and storing the transformed N-bit digital output vector in a storage unit.

[0075] In some embodiments, the storage unit may include: a digital input vector memory configured to store a first digital input vector and including at least one SRAM; and a neural network weight memory configured to store a plurality of neural network weights and including at least one DRAM.

[0076] In some embodiments, the DAC unit may include: a first DAC subunit configured to generate a plurality of modulator control signals; and a second DAC subunit configured to generate a plurality of weight control signals, wherein the first DAC subunit and the second DAC subunit are different.

[0077] In some embodiments, the laser unit may include a laser source configured to generate light; and an optical power splitter configured to split the light generated by the laser source into a plurality of light outputs, wherein each of the plurality of light outputs has substantially the same power.

[0078] In some embodiments, the plurality of optical modulators may include one of an MZI modulator, a ring resonator modulator, or an electro-absorption modulator.

[0079] In some embodiments, the photodetection unit may include: a plurality of photodetectors; and a plurality of amplifiers configured to convert photocurrents generated by the photodetectors into a plurality of output voltages.

[0080] In some embodiments, the integrated circuit may be an application specific integrated circuit.

[0081] In some embodiments, the optical matrix multiplication unit may include: an input waveguide array for receiving an optical input vector; an optical interference unit in optical communication with the input waveguide array, for performing a linear transformation to convert the optical input vector into a second optical signal array; and an output waveguide array in optical communication with the optical interference unit, for guiding the second optical signal array, wherein at least one input waveguide in the input waveguide array is in optical communication with each output waveguide in the output waveguide array through the optical interference unit.

[0082] In some embodiments, the optical interference unit may include: a plurality of interconnected Mach-Zehnder interferometers (MZIs), each of the plurality of interconnected MZIs including: a first phase shifter configured to change the splitting ratio of the MZI; and a second phase shifter configured to shift the phase of one output of the MZI, wherein the first phase shifter and the second phase shifter are coupled to a plurality of weight control signals.

[0083] On the other hand, a system includes: a storage unit configured to store a data set and a plurality of neural network weights; a driver unit configured to generate a plurality of modulator control signals and to generate a plurality of weight control signals; an optical processor including: a laser unit configured to generate a plurality of optical outputs; a plurality of optical modulators coupled to the laser unit and the driver unit, the plurality of optical modulators configured to generate an optical input vector by modulating the plurality of optical outputs generated by the laser unit based on the plurality of modulator control signals; an optical matrix multiplication unit coupled to the plurality of optical modulators and the driver unit, the optical matrix multiplication unit configured to convert the optical input vector into an optical output vector based on the plurality of weight control signals; and a photodetection unit coupled to the optical matrix multiplication unit and configured to generate a plurality of output voltages corresponding to the optical output vectors; a comparator unit coupled to the photodetection unit and configured to convert the plurality of output voltages into a plurality of digital 1-bit optical outputs; and a controller including An integrated circuit is configured to perform the following operations: receive an artificial neural network calculation request including an input data set and a first plurality of neural network weights from a computer, wherein the input data set includes a first digital input vector having an N-bit resolution; store the input data set and the first plurality of neural network weights in a storage unit; decompose the first digital input vector into N 1-bit input vectors, each of the N 1-bit input vectors corresponding to one of the N bits of the first digital input vector; generate a sequence of N 1-bit modulator control signals corresponding to the N 1-bit input vectors through a driver unit; obtain a sequence of N digital 1-bit optical outputs corresponding to the sequence of N 1-bit modulator control signals from a comparator unit; construct an N-bit digital output vector from the sequence of N digital 1-bit optical outputs; perform a nonlinear transformation on the constructed N-bit digital output vector to generate a transformed N-bit digital output vector; and store the transformed N-bit digital output vector in the storage unit.

[0084] On the other hand, a method for performing artificial neural network calculations in a system having an optical matrix multiplication unit, the optical matrix multiplication unit being configured to convert an optical input vector into an optical output vector based on a plurality of weight control signals, the method comprising: receiving an artificial neural network calculation request including an input data set and a first plurality of neural network weights from a computer, wherein the input data set includes a first digital input vector; storing the input data set and the first plurality of neural network weights in a storage unit; generating, through a digital-to-analog conversion (DAC) unit, a first plurality of modulator control signals based on the first digital input vector, and generating a first plurality of weight control signals based on the first plurality of neural network weights; obtaining, from an analog-to-digital conversion (ADC) unit, a first plurality of digital optical outputs corresponding to the optical output vector of the optical matrix multiplication unit, the first plurality of digital optical outputs forming a first digital output vector; performing, by a controller, a nonlinear transformation on the first digital output vector to generate a first transformed digital output vector; storing the first transformed digital output vector in the storage unit; and outputting, by the controller, an artificial neural network output generated based on the first transformed digital output vector.

[0085] On the other hand, a method includes: providing input information in an electronic format; converting at least a portion of the electronic input information into an optical input vector; optically converting the optical input vector into an optical output vector based on optical matrix multiplication; converting the optical output vector into an electronic format; and electronically applying a nonlinear transformation to the electronically converted optical output vector to provide output information in an electronic format.

[0086] Embodiments of the method may include one or more of the following features. For example, the method may further include repeating the electronic-to-optical conversion, optical transformation, optical-to-electronic conversion, and electrically applied nonlinear transformation for new electronic input information corresponding to output information provided in an electronic format.

[0087] In some embodiments, the optical matrix multiplication for the initial optical transformation and the optical matrix multiplication for the repeated optical transformation may be the same and may correspond to the same layer of the artificial neural network.

[0088] In some embodiments, the optical matrix multiplication for the initial optical transformation and the optical matrix multiplication for the repeated optical transformation may be different and may correspond to different layers of the artificial neural network.

[0089] In some embodiments, the method may further include: repeating electro-optical conversion, optical transformation, optoelectronic conversion, and electrically applied nonlinear transformation for different parts of the electronic input information, wherein the optical matrix multiplication for the initial optical transformation and the optical matrix multiplication for the repeated optical transformation are the same and correspond to the first layer of the artificial neural network.

[0090] In some embodiments, the method may further include: providing intermediate information in an electronic format based on electronic output information for multiple parts of the electronic input information generated by the first layer of the artificial neural network; and repeating the electro-optical conversion, optical transformation, optoelectronic conversion and electrically applied nonlinear transformation for each different part of the electronic intermediate information, wherein the optical matrix multiplication for the initial optical transformation and the optical matrix multiplication for the repeated optical transformation associated with the different parts of the electronic intermediate information are the same and correspond to the second layer of the artificial neural network.

[0091] In another aspect, a system includes an optical processor comprising a passive diffractive optical element, wherein the passive diffractive optical element is configured to transform an optical input vector or matrix into an optical output vector or matrix representing a result of matrix processing applied to the optical input vector or matrix and a predetermined vector defined by an arrangement of the diffractive optical element.

[0092] Embodiments of the system may include one or more of the following features.For example, matrix processing may include matrix multiplication between a light input vector or matrix and a predetermined vector defined by the arrangement of the diffractive optical components.

[0093] In some embodiments, the optical processor may include an optical matrix processing unit, which includes: an input waveguide array for receiving an optical input vector; an optical interference unit including a passive diffraction optical component, wherein the optical interference unit is in optical communication with the input waveguide array and is configured to perform a linear transformation to convert the optical input vector into a second optical signal array; and an output waveguide array in optical communication with the optical interference unit for guiding the second optical signal array, wherein at least one input waveguide of the input waveguide array is in optical communication with each output waveguide in the output waveguide array through the optical interference unit.

[0094] In some embodiments, the light interference unit may include a substrate having at least one of a hole or a stripe, the hole having a size in a range of 100 nm to 10 μm, and the stripe having a width in a range of 100 nm to 10 μm.

[0095] In some embodiments, the light interference unit may include a substrate having passive diffractive optical components arranged in a two-dimensional configuration, and the substrate includes at least one of a planar substrate or a curved substrate.

[0096] In some embodiments, the substrate may include a planar substrate that is parallel to the direction of light propagation from the input waveguide array to the output waveguide array.

[0097] In some embodiments, the optical processor may include an optical matrix processing unit, which includes: an input waveguide matrix for receiving an optical input matrix; an optical interference unit including a passive diffraction optical component, wherein the optical interference unit is in optical communication with the input waveguide matrix and is configured to perform a linear transformation to transform the optical input matrix into a second optical signal matrix; and an output waveguide matrix in optical communication with the optical interference unit for guiding the second optical signal matrix, wherein at least one input waveguide of the input waveguide matrix is ​​in optical communication with each output waveguide in the output waveguide matrix through the optical interference unit.

[0098] In some embodiments, the light interference unit may include a substrate having at least one of holes or stripes, the size of the holes being in a range of 100 nm to 10 μm, and the width of the stripes being in a range of 100 nm to 10 μm.

[0099] In some embodiments, the light interference unit may include a substrate having passive diffractive optical components arranged in a three-dimensional configuration.

[0100] In some embodiments, the substrate may have a shape of at least one of a cube, a column, a prism, or an irregular volume.

[0101] In some embodiments, the light processor may include an optical interferometer unit comprising a hologram having a passive diffractive optical component, the light processor being configured to receive modulated light representing a light input matrix and to continuously convert the light as the light passes through the hologram until the light is emitted from the hologram as a light output matrix.

[0102] In some embodiments, the light interference unit may include a substrate having a passive diffractive optical component, and the substrate includes at least one of silicon, silicon oxide, silicon nitride, quartz, lithium niobate, a phase change material, or a polymer.

[0103] In some embodiments, the light interference unit may include a substrate having a passive diffractive optical component, and the substrate includes at least one of a glass substrate or an acrylic substrate.

[0104] In some embodiments, the passive diffractive optical component may be formed in part by a dopant.

[0105] In some embodiments, matrix processing may represent processing of input data by a neural network, where the input data is represented by an optical input vector.

[0106] In some embodiments, the optical processor may include: a laser unit configured to generate multiple light outputs; a plurality of optical modulators coupled to the laser unit and configured to generate a light input vector by modulating the multiple light outputs generated by the laser unit based on a plurality of modulator control signals; an optical matrix processing unit coupled to the multiple optical modulators, the optical matrix processing unit including a passive diffraction optical component configured to convert the light input vector into a light output vector based on a plurality of weights defined by the passive diffraction optical component; and a photodetection unit coupled to the optical matrix processing unit and configured to generate a plurality of output electrical signals corresponding to the light output vectors.

[0107] In some embodiments, the passive diffractive optical component may be arranged in a three-dimensional configuration, the plurality of light modulators comprises a two-dimensional light modulator array, and the photodetection unit comprises a two-dimensional photodetector array.

[0108] In some embodiments, the optical matrix processing unit may include a housing module to support and protect the input waveguide array, the optical interference unit and the output waveguide array, the optical processor includes a receiving module, the receiving module is configured to receive the optical matrix processing unit, the receiving module includes a first interface, enabling the optical matrix processing unit to receive optical input vectors from multiple optical modulators, and a second interface, enabling the optical matrix processing unit to transmit the optical output vectors to the photoelectric detection unit.

[0109] In some embodiments, the plurality of output electrical signals may include at least one of a plurality of voltage signals or a plurality of current signals.

[0110] In some embodiments, the system may include: a storage unit; a digital-to-analog conversion (DAC) unit configured to generate a plurality of modulator control signals; an analog-to-digital conversion (ADC) unit coupled to the photodetection unit and configured to convert a plurality of output electrical signals into a plurality of digital outputs; and a controller including an integrated circuit configured to perform the following operations: receive an artificial neural network calculation request including an input data set from a computer, wherein the input data set includes a first digital input vector; store the input data set in the storage unit; and generate a first plurality of modulator control signals based on the first digital input vector through the DAC unit.

[0111] In another aspect, a method includes 3D printing a light matrix processing unit comprising a passive diffractive optical component, wherein the passive diffractive optical component is configured to transform a light input vector or matrix into a light output vector or matrix representing a result of matrix processing applied to the light input vector or matrix and a predetermined vector defined by an arrangement of the diffractive optical component.

[0112] In another aspect, a method includes generating a hologram using one or more laser beams that includes a passive diffractive optical component, wherein the passive diffractive optical component is configured to transform a light input vector or matrix into a light output vector or matrix representing a result of a matrix processing applied to the light input vector or matrix and a predetermined vector defined by an arrangement of the diffractive optical component.

[0113] In another aspect, a system includes: a light processor comprising passive diffractive optical components arranged in a one-dimensional manner, wherein the passive diffractive optical components are configured to convert a light input into a light output representing a result of matrix processing applied to the light input and a predetermined vector defined by the arrangement of the diffractive optical components.

[0114] Embodiments of the system may include one or more of the following features.For example, the matrix processing may include a matrix multiplication between the optical input and a predetermined vector defined by the arrangement of the diffractive optical components.

[0115] In some embodiments, the optical processor may include an optical matrix processing unit, which includes: an input waveguide for receiving an optical input; an optical interference unit including a passive diffraction optical component, wherein the optical interference unit is in optical communication with the input waveguide and is configured to perform a linear transformation on the optical input; and an output waveguide in optical communication with the optical interference unit for guiding the optical output.

[0116] In some embodiments, the light interference unit may include a substrate having at least one of a hole or a grating, and the hole or grating may have a size in a range of 100 nm to 10 μm.

[0117] On the other hand, a system includes: a storage unit; a digital-to-analog conversion (DAC) unit configured to generate multiple modulator control signals; and an optical processor including: a laser unit configured to generate multiple optical outputs; multiple optical modulators coupled to the laser unit and the DAC unit, the multiple optical modulators configured to generate an optical input vector by modulating the multiple optical outputs generated by the laser unit based on the multiple modulator control signals; an optical matrix processing unit coupled to the multiple optical modulators, the optical matrix processing unit including a passive diffraction optical component configured to convert the optical input vector into an optical output vector based on multiple weights defined by the passive diffraction optical component; and a photodetection unit coupled to the optical matrix processing unit and configured to generate multiple output electrical signals corresponding to the optical output vectors. The system further includes: an analog-to-digital conversion (ADC) unit coupled to the photodetection unit and configured to convert the plurality of output electrical signals into a plurality of digital light outputs; and a controller including an integrated circuit configured to perform the following operations: receive an artificial neural network calculation request including an input data set from a computer, wherein the input data set includes a first digital input vector; store the input data set in a storage unit; and generate, through a DAC unit, a first plurality of modulator control signals based on the first digital input vector.

[0118] Embodiments of the system may include one or more of the following features. For example, the matrix processing unit may include a passive diffractive optical component configured to convert a light input vector into a light output vector, the light output vector representing the product of a matrix multiplication between the light input vector and a predetermined vector defined by the passive diffractive optical component.

[0119] In some embodiments, the operation further includes: obtaining a first plurality of digital light outputs corresponding to the light output vector of the light matrix processing unit from the ADC unit, the first plurality of digital light outputs forming a first digital output vector; performing a nonlinear transformation on the first digital output vector to generate a first transformed digital output vector; and storing the first transformed digital output vector in a storage unit.

[0120] In some embodiments, the system may have a first cycle period defined as the time elapsed between storing the input data set in the memory unit and storing the first transformed digital output vector in the memory unit, and wherein the first cycle period may be less than or equal to 1 ns.

[0121] In some embodiments, the operations may further include outputting an artificial neural network output generated based on the first transformed digital output vector.

[0122] In some embodiments, the operations may further include generating, by a DAC unit, a second plurality of modulator control signals based on the first transformed digital output vector.

[0123] In some embodiments, the input data set may further include a second digital input vector, and wherein the operation may further include: generating a second plurality of modulator control signals based on the second digital input vector through a DAC unit; obtaining a second plurality of digital light outputs corresponding to the light output vector of the light matrix processing unit from the ADC unit, the second plurality of digital light outputs forming a second digital output vector; performing a nonlinear transformation on the second digital output vector to generate a second transformed digital output vector; storing the second transformed digital output vector in a storage unit; and outputting an artificial neural network output generated based on the first transformed digital output vector and the second transformed digital output vector, wherein the light output vector of the light matrix processing unit is generated by the second light input vector generated based on the second plurality of modulator control signals, and the second light input vector is converted by the light matrix processing unit based on multiple weights defined by the passive diffraction optical component.

[0124] In some embodiments, the system may further include: an analog nonlinear unit, arranged between the photodetection unit and the ADC unit, the analog nonlinear unit being configured to receive a plurality of output electrical signals from the photodetection unit, apply a nonlinear transfer function, and output a plurality of converted output electrical signals to the ADC unit, wherein the operation may further include: obtaining a first plurality of converted digital output electrical signals corresponding to the plurality of converted output electrical signals from the ADC unit, the first plurality of converted digital output electrical signals forming a first converted digital output vector; and storing the first converted digital output vector in a storage unit.

[0125] In some embodiments, the integrated circuit of the controller may be configured to generate the first plurality of modulator control signals at a rate greater than or equal to 8 GHz.

[0126] In some embodiments, the system may further include: an analog storage unit arranged between the DAC unit and the plurality of optical modulators, the analog storage unit being configured to store an analog voltage and output the stored analog voltage; and an analog nonlinear unit arranged between the photodetection unit and the ADC unit, the analog nonlinear unit being configured to receive a plurality of output electrical signals from the photodetection unit, apply a nonlinear transfer function, and output a plurality of converted output electrical signals.

[0127] In some embodiments, the analog memory cell may include multiple capacitors.

[0128] In some embodiments, the analog storage unit may be configured to receive and store multiple conversion output electrical signals of the analog nonlinear unit, and output the stored multiple conversion output electrical signals to multiple optical modulators, and wherein the operation may further include: storing the multiple conversion output electrical signals of the analog nonlinear unit in the analog storage unit based on generating a first plurality of modulator control signals; outputting the stored conversion output electrical signals through the analog storage unit; obtaining a second plurality of conversion digital output electrical signals from the ADC unit, the second plurality of conversion digital output electrical signals forming a second transformed digital output vector; and storing the second transformed digital output vector in the storage unit.

[0129] In some embodiments, an input data set requested by an artificial neural network for computation may include a plurality of digital input vectors, wherein the laser unit may be configured to generate a plurality of wavelengths, and wherein the plurality of optical modulators may include: an optical modulator group configured to generate a plurality of optical input vectors, each optical modulator group corresponding to one of the plurality of wavelengths and generating a corresponding optical input vector having a corresponding wavelength; and an optical multiplexer configured to combine the plurality of optical input vectors into a combined optical input vector comprising the plurality of wavelengths. The photodetector unit may be further configured to demultiplex the plurality of wavelengths and generate a plurality of demultiplexed output electrical signals, and the operations may include: obtaining a plurality of digital demultiplexed optical outputs from the ADC unit, the plurality of digital demultiplexed optical outputs forming a plurality of first digital output vectors, wherein each of the plurality of first digital output vectors corresponds to one of the plurality of wavelengths; performing a nonlinear transformation on each of the plurality of first digital output vectors to generate a plurality of transformed first digital output vectors; and storing the plurality of transformed first digital output vectors in a storage unit, wherein each of the plurality of digital input vectors corresponds to one of the plurality of optical input vectors.

[0130] In some embodiments, an artificial neural network computation request may include a plurality of digital input vectors, wherein the laser unit is configured to generate a plurality of wavelengths, and wherein the plurality of optical modulators may include: an optical modulator group configured to generate a plurality of optical input vectors, each optical modulator group corresponding to one of the plurality of wavelengths and generating a corresponding optical input vector having the corresponding wavelength; and an optical multiplexer configured to combine the plurality of optical input vectors into a combined optical input vector including the plurality of wavelengths. Operations may include: obtaining a first plurality of digital optical outputs corresponding to the optical output vectors from the ADC unit, the optical output vectors including the plurality of wavelengths, the first plurality of digital optical outputs forming a first digital output vector; performing a nonlinear transformation on the first digital output vector to generate a first transformed digital output vector; and storing the first transformed digital output vector in a storage unit.

[0131] In some embodiments, the DAC unit may include: a 1-bit DAC unit configured to generate a plurality of 1-bit modulator control signals, wherein the resolution of the ADC unit may be 1 bit, and wherein the resolution of the first digital input vector may be N bits. Operations may include: decomposing the first digital input vector into N 1-bit input vectors, each of the N 1-bit input vectors corresponding to one of the N bits of the first digital input vector; generating a sequence of N 1-bit modulator control signals corresponding to the N 1-bit input vectors through the 1-bit DAC unit; obtaining a sequence of N digital 1-bit optical outputs corresponding to the sequence of N 1-bit modulator control signals from the ADC unit; constructing an N-bit digital output vector from the sequence of N digital 1-bit optical outputs; performing a nonlinear transformation on the constructed N-bit digital output vector to generate a transformed N-bit digital output vector; and storing the transformed N-bit digital output vector in a storage unit.

[0132] In some embodiments, the storage unit may include a digital input vector memory configured to store the first digital input vector and including at least one SRAM.

[0133] In some embodiments, the laser unit may include a laser source configured to generate light; and an optical power splitter configured to split the light generated by the laser source into a plurality of light outputs, wherein each of the plurality of light outputs has substantially the same power.

[0134] In some embodiments, the plurality of optical modulators include one of an MZI modulator, a ring resonance modulator, or an electro-absorption modulator.

[0135] In some embodiments, the photodetection unit may include: a plurality of photodetectors; and a plurality of amplifiers configured to convert photocurrents generated by the photodetectors into a plurality of output electrical signals.

[0136] In some embodiments, the integrated circuit may comprise an application specific integrated circuit.

[0137] In some embodiments, the optical matrix processing unit may include: an input waveguide array for receiving an optical input vector; an optical interference unit in optical communication with the input waveguide array, for performing a linear transformation to convert the optical input vector into a second optical signal array, wherein the optical interference unit includes a passive diffraction optical component; and an output waveguide array in optical communication with the optical interference unit, for guiding the second optical signal array, wherein at least one input waveguide in the input waveguide array optically communicates with each output waveguide in the output waveguide array through the optical interference unit.

[0138] On the other hand, a system includes: a storage unit; a driver unit configured to generate multiple modulator control signals; an optical processor including: a laser unit configured to generate multiple optical outputs; multiple optical modulators coupled to the laser unit and the driver unit, the multiple optical modulators configured to generate an optical input vector by modulating the multiple optical outputs generated by the laser unit based on the multiple modulator control signals; an optical matrix processing unit coupled to the multiple optical modulators and the driver unit, the optical matrix processing unit including a passive diffraction optical component, configured to convert the optical input vector into an optical output vector based on multiple weight control signals defined by the passive diffraction optical component; and a photodetection unit coupled to the optical matrix processing unit and configured to generate multiple output electrical signals corresponding to the optical output vectors. The system also includes a comparator unit coupled to the photodetection unit and configured to convert multiple output electrical signals into multiple digital 1-bit optical outputs; and a controller including an integrated circuit configured to perform the following operations: receive an artificial neural network calculation request including an input data set from a computer, wherein the input data set includes a first digital input vector with an N-bit resolution; store the input data set in a storage unit; decompose the first digital input vector into N 1-bit input vectors, each of the N 1-bit input vectors corresponding to one of the N bits of the first digital input vector; generate a sequence of N 1-bit modulator control signals corresponding to the N 1-bit input vectors through the driver unit; obtain a sequence of N digital 1-bit optical outputs corresponding to the sequence of N 1-bit modulator control signals from the comparator unit; construct an N-bit digital output vector from the sequence of N digital 1-bit optical outputs; perform a nonlinear transformation on the constructed N-bit digital output vector to generate a transformed N-bit digital output vector; and store the transformed N-bit digital output vector in the storage unit.

[0139] Embodiments of the system may include one or more of the following features. For example, the optical matrix processing unit may include an optical matrix multiplication unit configured to convert an optical input vector into an optical output vector, the optical output vector representing the product of a matrix multiplication between an input vector represented by the optical input vector and a predetermined vector defined by the passive diffractive optical component.

[0140] On the other hand, a method for performing artificial neural network calculations in a system having an optical matrix processing unit includes: receiving an artificial neural network calculation request including an input data set from a computer, the input data set including a first digital input vector; storing the input data set in a storage unit; generating a first plurality of modulator control signals based on the first digital input vector by a digital-to-analog conversion (DAC) unit; converting the optical input vector into an optical output vector by using an optical matrix processing unit including an arrangement of passive diffractive optical components, wherein the optical output vector represents a result of matrix processing applied to the optical input vector and a predetermined vector defined by the arrangement of the diffractive optical components; obtaining a first plurality of digital optical outputs corresponding to the optical output vector of the optical matrix processing unit from the analog-to-digital conversion (ADC) unit, the first plurality of digital optical outputs forming a first digital output vector; performing a nonlinear transformation on the first digital output vector by a controller to generate a first transformed digital output vector; storing the first transformed digital output vector in the storage unit; and outputting the artificial neural network output generated based on the first transformed digital output vector by the controller.

[0141] Embodiments of the method may include one or more of the following features. For example, converting the light input vector to the light output vector may include converting the light input vector to a light output vector that represents a product of a matrix multiplication between the digital input vector and a predetermined vector defined by the arrangement of the diffractive optical components.

[0142] On the other hand, a method includes: providing input information in an electronic format; converting at least a portion of the electronic input information into an optical input vector; optically converting the optical input vector into an optical output vector based on optical matrix processing by an optical processor including a passive diffractive optical component; converting the optical output vector into an electronic format; and electronically applying a nonlinear transformation to the electronically converted optical output vector to provide output information in an electronic format.

[0143] Embodiments of the method may include one or more of the following features. For example, optically converting the light input vector to the light output vector may include optically converting the light input vector to the light output vector based on an optical matrix multiplication between a digital input vector represented by the light input vector and a predetermined vector defined by the passive diffractive optical component.

[0144] In some embodiments, the method may further include repeating the electrical-to-optical conversion, the optical conversion, the optical-to-electrical conversion, and the electrically applied non-linear conversion for new electronic input information corresponding to the output information provided in the electronic format.

[0145] In some embodiments, the light matrix processing for the initial light transformation and the light matrix processing for the repeated light transformation may be the same and may correspond to the same layer of the artificial neural network.

[0146] In some embodiments, the method may further include: repeating electro-optical conversion, optical transformation, optoelectronic conversion, and electrically applied nonlinear transformation for different parts of the electronic input information, wherein the optical matrix processing for the initial optical transformation and the optical matrix processing for the repeated optical transformation may be the same and correspond to a layer of the artificial neural network.

[0147] In another aspect, a system includes an optical matrix processing unit configured to process an input vector of length N, wherein the optical matrix processing unit includes N+2 layers of directional couplers and N layers of phase shifters, and N is a positive integer.

[0148] Embodiments of the system may include one or more of the following features: For example, the optical matrix processing unit may include no more than N+2 layers of directional couplers.

[0149] In some embodiments, the optical matrix processing unit may include an optical matrix multiplication unit.

[0150] In some embodiments, the optical matrix processing unit may include a substrate and interconnected interferometers arranged on the substrate, wherein each interferometer includes an optical waveguide arranged on the substrate, and the directional coupler and the phase shifter are part of the interconnected interferometers.

[0151] In some embodiments, the optical matrix processing unit may include a layer of attenuators after the last layer of directional couplers.

[0152] In some embodiments, a layer of attenuators may include N attenuators.

[0153] In some embodiments, the system may include one or more homodyne detectors for detecting the output from the attenuator.

[0154] In some embodiments, N=3, and the optical matrix processing unit may include: an input terminal configured to receive an input vector; a first layer of directional couplers coupled to the input terminal; a first layer of phase shifters coupled to the first layer of directional couplers; a second layer of directional couplers coupled to the first layer of phase shifters; a second layer of phase shifters coupled to the second layer of directional couplers; a third layer of directional couplers coupled to the second layer of phase shifters; a third layer of phase shifters coupled to the third layer of directional couplers; a fourth layer of directional couplers coupled to the third layer of phase shifters; and a fifth layer of directional couplers coupled to the fourth layer of directional couplers.

[0155] In some embodiments, N=4, and the optical matrix processing unit may include: an input end configured to receive an input vector; a first-layer, a second-layer, a third-layer, and a fourth-layer directional coupler, each layer of directional couplers being followed by a layer of phase shifters, wherein the first-layer directional coupler is coupled to the input end; a second-to-last layer of directional couplers coupled to the fourth layer of phase shifters; and a final layer of directional couplers coupled to the second-to-last layer of directional couplers.

[0156] In some embodiments, N=8, and the optical matrix processing unit may include: an input end configured to receive an input vector; eight layers of directional couplers, each layer of directional couplers followed by a layer of phase shifters, wherein a first layer of directional couplers is coupled to the input end; a second-to-last layer of directional couplers is coupled to an eighth layer of phase shifters; and a final layer of directional couplers is coupled to the second-to-last layer of directional couplers.

[0157] In some embodiments, the optical matrix multiplication unit may include: an input end configured to receive an input vector; N layers of directional couplers, each layer of directional couplers followed by a layer of phase shifters, wherein a first layer of directional couplers is coupled to the input end; a second-to-last layer of directional couplers is coupled to the Nth layer of directional couplers; and a final layer of directional couplers is coupled to the second-to-last layer of directional couplers.

[0158] In some embodiments, N is an even number.

[0159] In some embodiments, each i-th layer of directional couplers includes N / 2 directional couplers, where i is an odd number, and each j-th layer of directional couplers includes N / 2-1 directional couplers, where j is an even number.

[0160] In some embodiments, for each i-th layer of directional couplers where i is an odd number, the k-th directional coupler may be coupled to the (2k-1)-th and 2k-th outputs of the previous layer, where k is an integer from 1 to N / 2.

[0161] In some embodiments, for each jth layer of directional couplers where j is an even number, the mth directional coupler may be coupled to the (2m)th and (2m+1)th outputs of the previous layer, where m is an integer from 1 to N / 2-1.

[0162] In some embodiments, each i-th layer of phase shifters may include N phase shifters, where i is an odd number, and each j-th layer of phase shifters may include N-2 phase shifters, where j is an even number.

[0163] In some embodiments, N may be an odd number.

[0164] In some embodiments, each layer of directional couplers may include (N-1) / 2 directional couplers.

[0165] In some embodiments, each layer of phase shifters may include N-1 phase shifters.

[0166] On the other hand, a system includes: a generator configured to generate a first data set, wherein the generator includes an optical matrix processing unit; and a discriminator configured to receive a second data set including data from the first data set and data from a third data set, the data in the first data set having similar characteristics to the data in the third data set, and classify the data in the second data set as data from the first data set or data from the third data set.

[0167] Embodiments of the method may include one or more of the following features: For example, the optical matrix processing unit may include at least one of the following: (i) the optical matrix multiplication unit described above, (ii) the passive diffractive optical component described above, or (iii) the optical matrix processing unit described above.

[0168] In some embodiments, the third data set may include real data, the generator is configured to generate synthesized data similar to the real data, and the discriminator is configured to classify the data as real data or synthesized data.

[0169] In some embodiments, the generator may be configured to generate a data set for training at least one of an autonomous vehicle, a medical diagnostic system, a fraud detection system, a weather forecasting system, a financial forecasting system, a facial recognition system, a speech recognition system, or a product defect detection system.

[0170] In some embodiments, the generator may be configured to generate an image that resembles an image of at least one of a real object or a real scene, and the discriminator is configured to classify a received image as (i) an image of a real object or a real scene, or (ii) a synthetic image generated by the generator.

[0171] In some embodiments, the real object may include at least one of a person, an animal, a cell, a tissue, or a product, and the real scene includes a scene encountered by a vehicle.

[0172] In some embodiments, the discriminator may be configured to classify a received image as being (i) an image of a real person, a real animal, a real cell, a real tissue, a real product, or a real scene encountered by a vehicle, or (ii) a synthetic image produced by the generator.

[0173] In some embodiments, the vehicle may include at least one of a motorcycle, a car, a truck, a train, a helicopter, an airplane, a submarine, a ship, or a drone.

[0174] In some embodiments, the generator may be configured to generate images of tissue or cells associated with at least one of a human disease, an animal disease, or a plant disease.

[0175] In some embodiments, the generator may be configured to generate images of tissue or cells associated with a human disease, and the disease includes at least one of cancer, Parkinson's disease, sickle cell anemia, heart disease, cardiovascular disease, diabetes, chest disease, or skin disease.

[0176] In some embodiments, the generator may be configured to generate images of tissue or cells associated with cancer, and the cancer may include at least one of skin cancer, breast cancer, lung cancer, liver cancer, prostate cancer, or brain cancer.

[0177] In some embodiments, the system may further include a random noise generator configured to generate random noise that is input to the generator, and the generator is configured to generate the first data set based on the random noise.

[0178] In another aspect, a system includes a random noise generator configured to generate random noise; and a generator configured to generate data based on the random noise, wherein the generator includes an optical matrix processing unit.

[0179] Embodiments of the system may include one or more of the following features: For example, the optical matrix processing unit may include at least one of (i) the optical matrix multiplication unit described above, (ii) the passive diffractive optical component described above, or (iii) the optical matrix processing unit described above.

[0180] In another aspect, a system includes an optical circuit configured to perform a logic function on two input signals, the optical circuit comprising: a first directional coupler having two inputs and two outputs, the two inputs configured to receive the two input signals; a first pair of phase shifters configured to modify the phases of the signals at the two outputs of the first directional coupler; a second directional coupler having two inputs and two outputs, the two inputs configured to receive signals from the first pair of phase shifters; and a second pair of phase shifters configured to modify the phases of the signals at the two outputs of the second directional coupler.

[0181] Embodiments of the method may include one or more of the following features. For example, the phase shifter may be configured to cause the optical circuit to perform a rotation:

[0182]

[0183] In some embodiments, when input signals x1 and x2 are provided to two input terminals of the first directional coupler, the phase shifter can be configured to cause the optical circuit to perform the operation:

[0184]

[0185] In some embodiments, the optical circuit may include a first photodetector configured to generate an absolute value of a signal from the second pair of phase shifters to cause the optical circuit to perform the operation:

[0186]

[0187] In some embodiments, the optical circuit may include a comparator configured to compare the output signal of the first photodetector with a threshold value to generate a binary value to cause the optical circuit to generate an output:

[0188]

[0189] In some embodiments, the optical circuit may include a feedback mechanism configured such that an output signal of the photodetector is fed back to an input of the first directional coupler, passes through the first directional coupler, the first pair of phase shifters, the second directional coupler, and the second pair of phase shifters, and is detected by the photodetector, causing the optical circuit to perform the following operation:

[0190]

[0191] It produces the outputs AND(x1,x2) and OR(x1,x2).

[0192] In some embodiments, the optical circuit may include: a third directional coupler having two inputs and two outputs, the two inputs configured to receive signals from the second pair of phase shifters; a third pair of phase shifters configured to modify the phases of signals at the two outputs of the third directional coupler; a fourth directional coupler having two inputs and two outputs, the two inputs configured to receive signals from the third pair of phase shifters; a fourth pair of phase shifters configured to modify the phases of signals at the two outputs of the fourth directional coupler; and a second photodetector configured to generate an absolute value of the signal from the fourth pair of phase shifters, so as to cause the optical circuit to perform the operation:

[0193]

[0194] It produces the outputs AND(x1,x2) and OR(x1,x2).

[0195] In some embodiments, a system may include a bitonic sorter configured to perform a sorting function of the bitonic sorter using an optical circuit.

[0196] In some embodiments, a system may include an apparatus configured to perform a hashing function using an optical circuit.

[0197] In some embodiments, the hash function may include Secure Hash Algorithm 2 (SHA-2).

[0198] Generally speaking, systems for performing computations use different types of operations to produce computational results, each of which is performed on a signal (e.g., an electrical signal or an optical signal) that best suits the underlying physical properties of the operation (e.g., in terms of energy consumption and / or speed). For example, three such operations are: copying, summation, and multiplication. Copying can be performed using optical power splitting, summation can be performed using electrical current-based summation, and multiplication can be performed using optical amplitude modulation, as described in more detail below. An example of a computation that can be performed using these three types of operations is multiplying a vector by a matrix (e.g., as employed in artificial neural network computations). Various other computations can be performed using these operations, which represent a set of general linear operations that can perform various computations, including but not limited to: vector-vector dot product, vector-vector element-wise multiplication, vector-scalar element-wise multiplication, or matrix-matrix element-wise multiplication. Some of the examples described herein illustrate techniques and configurations for vector-matrix multiplication, but the corresponding techniques and configurations can be used for any of these types of computations.

[0199] Aspects can have one or more of the following advantages.

[0200] The optoelectronic computing systems described herein that use electrical and optical signals can promote increased flexibility and / or efficiency. In the past, there may have been potential challenges associated with combining optical (or photonic) integrated devices with electrical (or electronic) integrated devices on a common platform (e.g., a common semiconductor die, or multiple semiconductor dies combined in a controlled collapsed chip connection or a "flip-chip" arrangement). For example, such potential challenges may include input / output (I / O) packaging or temperature control. For those systems described herein, potential challenges may be increased when used with a relatively large number of optical input ports and a relatively large number of electrical output ports (e.g., 4 or more optical I / O ports, 200 or more electrical I / O ports). These potential challenges can be mitigated using appropriate system design. For example, the system may utilize a high-density packaging arrangement that uses temperature control (e.g., thermoelectric cooling) to control thermal expansion between different material types (e.g., semiconductor materials (e.g., silicon), glass materials (silicon dioxide or "silica"), ceramic materials, etc.), and / or utilizes an enclosing housing as a heat sink and provides a degree of sealing. Utilizing such temperature stabilization techniques, different coefficients of thermal expansion (CTE) and the resulting misalignment between system ports and ports of a packaged high-density optical fiber array can be limited.

[0201] For the replication operation, since optical power splitting is passive, no power is consumed to perform the operation. Additionally, the frequency bandwidth of an electrical splitter has a limit related to the RC time constant. In contrast, the frequency bandwidth of an optical splitter is essentially unlimited. Different types of optical power splitters can be used, including waveguide optical splitters or free-space beam splitters, as described in more detail below.

[0202] For multiplication operations, one value can be encoded as an optical signal, and the other value can be encoded as an amplitude scaling coefficient (e.g., multiplication by a value in the range of 0 to 1). After setting the scaling coefficient, multiplication operations in the optical domain require less (or no) conditioning of the electrical signal, thereby reducing constraints due to electrical noise, power consumption, and bandwidth limitations. By appropriately selecting the detection scheme, signed results (e.g., multiplication by a value between -1 and +1) can be obtained, as described in more detail below.

[0203] For summation operations, different techniques can be used to achieve a result in which the magnitude of the current in a conductor is determined based on the sum of different contributions. In the case of input current signals, when two or more conductors carrying those input current signals are combined at a junction, the single conductor carrying the output current signal represents the sum of those input current signals. In the case of input optical signals, when two or more light waves of different wavelengths impinge on a detector, the current signal carried by the photocurrent generated by the detector represents the sum of the powers in the input optical signals. Both generate an electrical signal (e.g., current) as an output representing the sum, but one uses current as input (current-input-based summation, also known as "electrical summation" performed in the "electrical domain"), while the other uses light waves as input (optical-input-based summation, also known as "optoelectronic summation" performed in the "optoelectronic domain"). However, in some embodiments, summing based on current inputs is used rather than summing based on light inputs, which enables a single optical wavelength to be used in the system, avoiding potentially complex components of the system that may need to be provided and maintaining multiple wavelengths.

[0204] Combinations of these basic operations performed by these modules can be arranged to provide a device that performs linear operations (e.g., vector-matrix multiplication with arbitrary matrix element magnitudes). Other implementations of matrix multiplication using optical signals and interferometers for combining signals using optical interference have been limited to providing vector-matrix multiplication with certain restrictions (e.g., unitary matrices or diagonal matrices). In addition, some other embodiments may rely on large-scale phase alignment of multiple optical signals as they propagate through a relatively large number of optical components (e.g., optical modulators). Alternatively, the embodiments described herein may relax such phase alignment constraints by converting the optical signals to electrical signals after propagating through fewer optical components (e.g., after propagating through no more than a single optical amplitude modulator), which allows the use of optical signals with reduced coherence, or even the use of incoherent optical signals that do not rely on constructive / destructive interference optical modulators.

[0205] For time-domain coding of optical and electrical signals, as described in more detail below, analog electronic circuits can be optimized for operation at specific power levels, which can be helpful if the circuits are operating at high speeds. Such time-domain coding is useful in reducing any challenges that may be associated with precisely controlling a relatively large number of clearly distinguishable intensity levels for each symbol. Conversely, when precise control of the duty cycle is applied in the time domain over multiple time slots within a single symbol duration, a relatively constant amplitude can be used (with zero or near-zero amplitude for the "on" level and at the "off" level).

[0206] By integrating photonics and electronics on a common substrate, such as a silicon chip, modules can be easily manufactured on a large scale. Routing signals on the substrate as optical rather than electrical signals and grouping the photodetectors in a portion of the substrate can help avoid long electronic routings and their associated challenges (e.g., parasitic capacitance, inductance, and crosstalk).

[0207] For embodiments of systems using sub-matrix multiplication, different devices (e.g., different cores, different processors, different computers, different servers) can be used to simultaneously compute each element of the output vector, helping to alleviate certain potential limitations (e.g., memory walls) and helping the overall system scale to very large matrices. In some embodiments, a different device can be used to multiply each sub-matrix by the corresponding sub-vector. The sum can then be calculated by collecting or accumulating the summands from the different devices. Intermediate results in the form of optical signals can be conveniently transmitted between devices, even if the devices are separated by relatively large distances.

[0208] Other aspects include other combinations of the above features and other features expressed as methods, apparatus, systems, program products, and otherwise.

[0209] Particular embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages: ANN computation throughput, latency, or both can be improved. ANN computation power efficiency can be improved.

[0210] In another aspect, an apparatus includes: a plurality of optical waveguides, wherein a set of a plurality of input values ​​is encoded on respective optical signals carried by the optical waveguides; a plurality of replicating modules, and for each of at least two subsets of the one or more optical signals, a respective set of the one or more replicating modules is configured to split the subset of the one or more optical signals into two or more copies of the optical signals; a plurality of multiplying modules, and for each of the at least two copies of a first subset of the one or more optical signals, a respective multiplying module is configured to multiply the one or more optical signals of the first subset by one or more matrix element values ​​using optical amplitude modulation, wherein at least one of the multiplying modules includes an optical amplitude modulator, the optical amplitude modulator including one input port and two output ports, and providing a pair of correlated optical signals from the two output ports such that a difference between amplitudes of the correlated optical signals corresponds to a result of multiplying the input value by the signed matrix element value; and one or more summing modules, and for the results of two or more multiplying modules, a respective one of the summing modules is configured to generate an electrical signal, the electrical signal representing a sum of the results of the two or more multiplying modules.

[0211] Embodiments of the apparatus may include one or more of the following features: For example, an input value of a set of multiple input values ​​encoded on a corresponding optical signal may represent an element of an input vector multiplied by a matrix comprising one or more matrix element values.

[0212] In some embodiments, a set of multiple output values ​​may be encoded on corresponding electrical signals generated by one or more summing modules, and the output values ​​in the set of multiple output values ​​may represent elements of an output vector generated by multiplying the input vector by a matrix.

[0213] In some embodiments, each optical signal carried by the optical waveguide may include a light wave having a common wavelength, the common wavelength being substantially the same for all optical signals.

[0214] In some embodiments, the replication module may include at least one replication module having an optical splitter that transmits a predetermined proportion of the power of the lightwave at the input port to the first output port and transmits a remaining proportion of the power of the lightwave at the input port to the second output port.

[0215] In some embodiments, the optical splitter may include a waveguide splitter that transmits a predetermined proportion of the power of the light waves guided by the input optical waveguide to the first output optical waveguide and transmits the remaining proportion of the power of the light waves guided by the input optical waveguide to the second output optical waveguide.

[0216] In some embodiments, a guided mode of the input optical waveguide can be adiabatically coupled to a guided mode of each of the first and second output optical waveguides.

[0217] In some embodiments, the optical splitter may include a beam splitter including at least one surface that transmits a predetermined proportion of the power of the light wave at the input port and reflects a remaining proportion of the power of the light wave at the input port.

[0218] In some embodiments, at least one of the plurality of optical waveguides may include an optical fiber coupled to an optical coupler that couples a guided mode of the optical fiber to a free-space propagation mode.

[0219] In some embodiments, the multiplication module may include at least one coherence-sensitive multiplication module configured to multiply one or more optical signals of the first subset by one or more matrix element values ​​using optical amplitude modulation based on interference between optical waves, where the optical waves have a coherence length that is at least as long as a propagation distance through the coherence-sensitive multiplication module.

[0220] In some embodiments, the coherence-sensitive multiplication module may include a Mach-Zehnder interferometer (MZI), which separates the light wave guided by the input optical waveguide into a first optical waveguide arm and a second optical waveguide arm of the MZI, the first optical waveguide arm including a phase shifter that produces a relative phase shift relative to the phase delay of the second optical waveguide arm, and the MZI combines the light waves from the first optical waveguide arm and the second optical waveguide arm into at least one output optical waveguide.

[0221] In some embodiments, the MZI may combine lightwaves from the first optical waveguide arm and the second optical waveguide arm into each of the first output optical waveguide and the second output optical waveguide, the first photodetector may receive the lightwave from the first output optical waveguide to generate a first photocurrent, the second photodetector may receive the lightwave from the second output optical waveguide to generate a second photocurrent, and the result of the coherence-sensitive multiplication module may include a difference between the first photocurrent and the second photocurrent.

[0222] In some embodiments, the coherence-sensitive multiplication module may include one or more ring resonators, including at least one ring resonator coupled to the first optical waveguide and at least one ring resonator coupled to the second optical waveguide.

[0223] In some embodiments, the first photodetector may receive a light wave from the first optical waveguide to generate a first photocurrent, the second photodetector may receive a light wave from the second optical waveguide to generate a second photocurrent, and the result of the coherence-sensitive multiplication module may include a difference between the first photocurrent and the second photocurrent.

[0224] In some embodiments, the multiplication module may include at least one coherence-insensitive multiplication module configured to multiply the one or more optical signals of the first subset by one or more matrix element values ​​using optical amplitude modulation based on energy absorption within the optical wave.

[0225] In some embodiments, the coherence-insensitive multiplication module may include an electro-absorption modulator.

[0226] In some embodiments, the one or more summing modules may include at least one summing module having the following components: (1) two or more input conductors, each input conductor carrying an electrical signal in the form of an input current, the magnitude of the input current representing a respective result of a respective one of the multiplication modules, and (2) at least one output conductor, the output conductor carrying an electrical signal representing a sum of the respective results in the form of an output current, the output current being proportional to the sum of the input currents.

[0227] In some embodiments, the two or more input conductors and the output conductor may include a plurality of wires that are in contact at one or more junctions between the wires, and the output current is substantially equal to the sum of the input currents.

[0228] In some embodiments, at least a first one of the input currents may be provided in the form of at least one photocurrent generated by at least one photodetector that receives the optical signal generated by a first multiplication module of the multiplication modules.

[0229] In some embodiments, the first input current may be provided in the form of a difference between two photocurrents, the two photocurrents being generated by different corresponding photodetectors that receive different corresponding optical signals generated by the first multiplication module.

[0230] In some embodiments, one of the copies of the first subset of the one or more optical signals may consist of a single optical signal, wherein one of the input values ​​is encoded on the single optical signal.

[0231] In some embodiments, the multiplication modules corresponding to the replicas of the first subset may multiply the encoded input value by a single matrix element value.

[0232] In some embodiments, one of the copies of the first subset of one or more optical signals may include more than one, and less than all, of the optical signals on which the plurality of input values ​​are encoded.

[0233] In some embodiments, the multiplication modules corresponding to the replicas of the first subset may multiply the encoded input values ​​by different corresponding matrix element values.

[0234] In some embodiments, different multiplication modules corresponding to different respective copies of the first subset of one or more optical signals may be included by different devices that are in optical communication to transmit one of the copies of the first subset of one or more optical signals between the different devices.

[0235] In some embodiments, two or more of the plurality of optical waveguides, two or more of the plurality of replica modules, two or more of the plurality of multiplication modules, and at least one of the one or more summation modules may be arranged on a substrate of a common device.

[0236] In some embodiments, the apparatus performs a vector-matrix multiplication, where an input vector may be provided as a set of optical signals and an output vector may be provided as a set of electrical signals.

[0237] In some embodiments, the device may further include an accumulator that combines input electrical signals corresponding to the outputs of the multiplication module or the summation module, wherein the input electrical signal may be encoded using time domain encoding, the time domain encoding using on-off amplitude modulation within each of a plurality of time slots, and the accumulator may generate an output electrical signal that is encoded with more than two amplitude levels, the amplitude levels corresponding to different duty cycles of the time domain encoding over the plurality of time slots.

[0238] In some embodiments, each of the two or more multiplication modules corresponds to a different subset of the one or more optical signals.

[0239] In some embodiments, the apparatus may further include a multiplication module for each copy of a second subset of one or more optical signals different from the optical signals in the first subset of one or more optical signals, configured to multiply the one or more optical signals of the second subset by one or more matrix element values ​​using optical amplitude modulation.

[0240] In another aspect, a method includes encoding a set of multiple input values ​​on respective optical signals; for each of at least two subsets of the one or more optical signals, using a respective set of one or more replication modules to split the subset of the one or more optical signals into two or more copies of the optical signals; for each of the at least two copies of a first subset of the one or more optical signals, using a respective multiplication module to multiply the one or more optical signals of the first subset by one or more matrix element values ​​using optical amplitude modulation, wherein at least one of the multiplication modules includes an optical amplitude modulator including one input port and two output ports and providing a pair of correlated optical signals from the two output ports such that a difference between amplitudes of the correlated optical signals corresponds to a result of multiplying the input value by the signed matrix element value; and for the results of the two or more multiplication modules, using a summation module configured to generate an electrical signal representing a sum of the results of the two or more multiplication modules.

[0241] In another aspect, a method includes encoding a set of input values ​​representing elements of an input vector on corresponding optical signals; encoding a set of coefficients representing matrix elements as amplitude modulation levels of a set of optical amplitude modulators coupled to the optical signal, wherein at least one optical amplitude modulator including one input port and two output ports provides a pair of correlated optical signals from the two output ports such that a difference between the amplitudes of the correlated optical signals corresponds to a result of multiplying the input value by a signed matrix element value; and encoding a set of output values ​​representing elements of an output vector on corresponding electrical signals, wherein at least one electrical signal is in the form of a current having an amplitude corresponding to a sum of corresponding elements of the input vector multiplied by corresponding elements of a row of the matrix.

[0242] Embodiments of the method may include one or more of the following features. For example, at least one optical signal may be provided by a first optical waveguide, and the first optical waveguide may be coupled to an optical splitter that transmits a predetermined proportion of the power of the optical wave guided by the first optical waveguide to a second output optical waveguide and transmits the remaining proportion of the power of the optical wave guided by the first optical waveguide to a third optical waveguide.

[0243] In another aspect, an apparatus includes: a plurality of optical waveguides encoding a set of input values ​​representing elements of an input vector on corresponding optical signals carried by the optical waveguides; a set of optical amplitude modulators coupled to the optical signals encoding a set of coefficients representing matrix elements as amplitude modulation levels, wherein at least one optical amplitude modulator including one input port and two output ports provides a pair of correlated optical signals from the two output ports such that a difference between the amplitudes of the correlated optical signals corresponds to a result of multiplying the input value by a signed matrix element value; and a plurality of summing modules encoding a set of output values ​​representing elements of an output vector on corresponding electrical signals, wherein at least one electrical signal is in the form of a current having an amplitude corresponding to a sum of the corresponding element of the input vector multiplied by the corresponding element of a row of the matrix.

[0244] In another aspect, a method for multiplying an input vector by a given matrix includes: encoding a set of input values ​​representing elements of the input vector on corresponding optical signals of a set of optical signals; coupling a first set of one or more devices to a first set of one or more waveguides providing a first subset of the set of optical signals and generating a result of multiplying a first submatrix of the given matrix by the values ​​encoded on the first subset of the set of optical signals; coupling a second set of one or more devices to a second set of one or more waveguides providing a second subset of the set of optical signals and generating a result of multiplying a second submatrix of the given matrix by the values ​​encoded on the second subset of the set of optical signals; coupling a third set of one or more devices to a first set of one or more waveguides providing a copy of the first subset of the set of optical signals generated by a first optical splitter; coupling a fourth group of one or more devices to a fourth group of one or more waveguides that provide a copy of the second subset of the group of optical signals generated by the second optical splitter, and producing a result of multiplying a fourth submatrix of the given matrix by the values ​​encoded on the second subset of the group of optical signals; wherein the first, second, third, and fourth submatrices, concatenated together, form the given matrix; and wherein at least one output value representing an element of an output vector corresponding to the multiplication of the input vector by the given matrix is ​​encoded on an electrical signal generated by a device in communication with the first group of one or more devices and the second group of one or more devices.

[0245] Embodiments of the method may include one or more of the following features: For example, each pair of the first group of one or more devices, the second group of one or more devices, the third group of one or more devices, and the fourth group of one or more devices may be mutually exclusive.

[0246] In another aspect, an apparatus includes: a first set of one or more apparatuses configured to receive a first set of optical signals and generate a result of multiplying a first matrix by a value encoded on the first set of optical signals; a second set of one or more apparatuses configured to receive a second set of optical signals and generate a result of multiplying the second matrix by the value encoded on the second set of optical signals; a third set of one or more apparatuses configured to receive a third set of optical signals and generate a result of multiplying the third matrix by the value encoded on the third set of optical signals; a fourth set of one or more apparatuses configured to receive a fourth set of optical signals and generate a result of multiplying the fourth matrix by the value encoded on the fourth set of optical signals; and a configurable connection path between two or more of the first set of one or more apparatuses, the second set of one or more apparatuses, the third set of one or more apparatuses, or the fourth set of one or more apparatuses, wherein a first configuration of the configurable connection path is configured to (1) provide a copy of the first set of optical signals as at least one of the second set of optical signals, the third set of optical signals, or the fourth set of optical signals, and (2) provide one or more signals from the first set of one or more apparatuses and one or more signals from the second set of one or more apparatuses to a summing module, the summing module configured to generate an electrical signal representing a sum of the values ​​encoded on the signals received by the summing module.

[0247] In another aspect, an apparatus includes: a first group of one or more apparatuses configured to receive a first group of optical signals and generate a result based on optical amplitude modulation of one or more optical signals of the first group of optical signals; a second group of one or more apparatuses configured to receive a second group of optical signals and generate a result based on optical amplitude modulation of one or more optical signals of the second group of optical signals; a third group of one or more apparatuses configured to receive a third group of optical signals and generate a result based on optical amplitude modulation of one or more optical signals of the third group of optical signals; a fourth group of one or more apparatuses configured to receive a fourth group of optical signals and generate a result based on optical amplitude modulation of one or more optical signals of the fourth group of optical signals; and a configurable connection path between two or more of the first group of one or more apparatuses, the second group of one or more apparatuses, the third group of one or more apparatuses, or the fourth group of one or more apparatuses, wherein a first configuration of the configurable connection path is configured to (1) provide a copy of the first group of optical signals as a third group of optical signals, or (2) provide one or more signals from the first group of one or more apparatuses and one or more signals from the second group of one or more apparatuses to a summing module, the summing module configured to generate an electrical signal representing a sum of values ​​encoded on the signals received by the summing module.

[0248] Embodiments of the apparatus may include one or more of the following features: For example, each pair of the first set of one or more apparatuses, the second set of one or more apparatuses, the third set of one or more apparatuses, and the fourth set of one or more apparatuses may be mutually exclusive.

[0249] In some embodiments, the first configuration of the configurable connection path is configured to (1) provide a copy of the first set of optical signals as a third set of optical signals, and (2) provide one or more signals from the first set of one or more devices and one or more signals from the second set of one or more devices to a summing module configured to generate an electrical signal representing a sum of values ​​encoded on at least two different signals received by the summing module.

[0250] In some embodiments, a first configuration of the configurable connection paths may be configured to provide a copy of the first set of optical signals as a third set of optical signals, and a second configuration of the configurable connection paths may be configured to provide one or more signals from the first set of one or more devices and one or more signals from the second set of one or more devices to a summing module configured to generate an electrical signal representing a sum of values ​​encoded on the signals received by the summing module.

[0251] In another aspect, an apparatus includes: a plurality of optical waveguides, wherein a set of a plurality of input values ​​is encoded on respective optical signals carried by the optical waveguides; a plurality of replication modules, the plurality of replication modules comprising, for each of at least two subsets of the one or more optical signals, a respective set of one or more replication modules configured to split the subset of the one or more optical signals into two or more copies of the optical signals; a plurality of multiplication modules, the plurality of multiplication modules comprising, for each of the at least two copies of a first subset of the one or more optical signals, a respective multiplication module configured to multiply the one or more optical signals of the first subset by one or more values ​​using optical amplitude modulation; and one or more summation modules, the one or more summation modules comprising, for a result of two or more multiplication modules, a summation module configured to generate an electrical signal, the electrical signal representing a sum of the results of the two or more multiplication modules, wherein the result comprises at least one result encoded on the electrical signal and the result is derived from one copy of the optical signal that propagated through no more than a single optical amplitude modulator before being converted to the electrical signal.

[0252] In another aspect, a system includes: a first unit configured to generate a plurality of modulator control signals; and a processor including: a light source configured to provide a plurality of light outputs; a plurality of light modulators coupled to the light source and the first unit, the plurality of light modulators configured to generate a light input vector by modulating the plurality of light outputs provided by the light source based on the plurality of modulator control signals, the light input vector comprising a plurality of light signals; and a matrix multiplication unit coupled to the plurality of light modulators and the first unit, the matrix multiplication unit configured to convert the light input vector into an analog output vector based on a plurality of weight control signals. The computing system also includes a second unit coupled to the matrix multiplication unit and configured to convert the analog output vector into a digital output vector; and a controller including an integrated circuit configured to perform the following operations: receive an artificial neural network computation request including an input data set including a first digital input vector; receive a first plurality of neural network weights; and generate, by the first unit, a first plurality of modulator control signals based on the first digital input vector and a first plurality of weight control signals based on the first plurality of neural network weights.

[0253] Embodiments of the system may include one or more of the following features.For example, the first unit may include a digital-to-analog converter (DAC).

[0254] In some embodiments, the second unit may include an analog-to-digital converter (ADC).

[0255] In some embodiments, the system may include a storage unit configured to store a data set and a plurality of neural network weights.

[0256] In some embodiments, the integrated circuit of the controller may be further configured to perform operations including storing the input data set and the first plurality of neural network weights in the memory unit.

[0257] In some embodiments, the first unit may be configured to generate a plurality of weight control signals.

[0258] In some embodiments, the controller may include an application specific integrated circuit (ASIC), and receiving the artificial neural network computation request may include receiving the artificial neural network computation request from a general purpose data processor.

[0259] In some embodiments, the first unit, the processing unit, the second unit, and the controller may be arranged on at least one of a multi-chip module or an integrated circuit. Receiving the artificial neural network computation request may include receiving the artificial neural network computation request from a second data processor, wherein the second data processor may be external to the multi-chip module or the integrated circuit, the second data processor may be coupled to the multi-chip module or the integrated circuit via a communication channel, and the processing unit may process data at a data rate at least one order of magnitude greater than a data rate of the communication channel.

[0260] In some embodiments, the first unit, the processing unit, the second unit, and the controller can be used in a photoelectric processing cycle that is repeated in multiple iterations, and the photoelectric processing cycle includes: (1) at least a first photomodulation operation based on at least one of a plurality of modulator control signals, and at least a second photomodulation operation based on at least one of the weight control signals, and (2) at least one of (a) an electrical summing operation or (b) an electrical storage operation.

[0261] In some embodiments, the photoelectric processing cycle may include an electrical storage operation, and the electrical storage operation is performed using a memory unit coupled to the controller, wherein the operations performed by the controller may further include storing the input data set and the first plurality of neural network weights in the memory unit.

[0262] In some embodiments, the photoelectric processing cycle may include an electrical summation operation, and the electrical summation operation may be performed using an electrical summation module within the matrix multiplication unit, wherein the electrical summation module may be configured to generate currents corresponding to elements of an analog output vector, the analog output vector representing the sum of corresponding elements of the photoinput vector multiplied by corresponding neural network weights.

[0263] In some embodiments, the optoelectronic processing loop may include at least one signal path on which no more than one first optical modulation operation is performed in a single loop iteration based on at least one of the plurality of modulator control signals, and no more than one second optical modulation operation is performed in a single loop iteration based on at least one of the weight control signals.

[0264] In some embodiments, the first light modulation operation may be performed by one of a plurality of light modulators coupled to the light source of the light output and the matrix multiplication unit, and the second light modulation operation may be performed by a light modulator included in the matrix multiplication unit.

[0265] In some embodiments, the optoelectronic processing cycle may include at least one signal path on which no more than one electrical storage operation is performed in a single cycle iteration.

[0266] In some embodiments, the light source may include a laser unit configured to generate a plurality of light outputs.

[0267] In some embodiments, the matrix multiplication unit may include: an input waveguide array for receiving an optical input vector, and the optical input vector includes a first optical signal array; an optical interference unit in optical communication with the input waveguide array, for performing a linear transformation to convert the optical input vector into a second optical signal array; and an output waveguide array in optical communication with the optical interference unit, for guiding the second optical signal array, wherein at least one input waveguide in the input waveguide array is in optical communication with each output waveguide in the output waveguide array through the optical interference unit.

[0268] In some embodiments, the optical interference unit may include: a plurality of interconnected Mach-Zehnder interferometers (MZIs), each of the plurality of interconnected MZIs including: a first phase shifter configured to change the splitting ratio of the MZI; and a second phase shifter configured to shift the phase of one output of the MZI, wherein the first phase shifter and the second phase shifter are coupled to a plurality of weight control signals.

[0269] In some embodiments, the matrix multiplication unit may include: a plurality of replication modules, wherein each replication module corresponds to a subset of one or more optical signals of an optical input vector and is configured to split the subset of the one or more optical signals into two or more copies of the optical signals; a plurality of multiplication modules, wherein each multiplication module corresponds to a subset of the one or more optical signals and is configured to multiply the one or more optical signals of the subset by one or more matrix element values ​​using optical amplitude modulation; and one or more summation modules, wherein each summation module is configured to generate an electrical signal, the electrical signal representing the sum of the results of two or more of the multiplication modules.

[0270] In some embodiments, at least one multiplication module includes an optical amplitude modulator, which includes one input port and two output ports and can provide a pair of correlated optical signals from the two output ports such that a difference between the amplitudes of the correlated optical signals corresponds to a result of multiplying the input value by the signed matrix element value.

[0271] In some embodiments, the matrix multiplication unit may be configured to multiply the light input vector by a matrix comprising one or more matrix element values.

[0272] In some embodiments, a set of multiple output values ​​may be encoded on corresponding electrical signals generated by one or more summing modules, and the output values ​​in the set of multiple output values ​​may represent elements of an output vector generated by multiplying the optical input vector by the matrix.

[0273] In some embodiments, the system may include a storage unit configured to store an input data set and a neural network weight, the second unit may include an analog-to-digital conversion (ADC) unit, and the operation may further include: obtaining a first plurality of digital outputs of an analog output vector corresponding to the matrix multiplication unit from the ADC unit, the first plurality of digital outputs forming a first digital output vector; performing a nonlinear transformation on the first digital output vector to generate a first transformed digital output vector; and storing the first transformed digital output vector in the storage unit.

[0274] In some embodiments, the system has a first cycle period, the first cycle period being defined as the time elapsed between the step of storing the input data set and the first plurality of neural network weights in the memory unit and the step of storing the first transformed digital output vector in the memory unit, and wherein the first cycle period is less than or equal to 1 ns.

[0275] In some embodiments, the operations may further include outputting an artificial neural network output generated based on the first transformed digital output vector.

[0276] In some embodiments, the first unit may include a digital-to-analog conversion (DAC) unit, and the operations may further include generating, by the DAC unit, a second plurality of modulator control signals based on the first transformed digital output vector.

[0277] In some embodiments, the first unit may include a digital-to-analog conversion (DAC) unit, the artificial neural network calculation request may further include a second plurality of neural network weights, and wherein the operation may further include: based on obtaining the first plurality of digital outputs, generating a second plurality of weight control signals based on the second plurality of neural network weights through the DAC unit.

[0278] In some embodiments, the first plurality of neural network weights and the second plurality of neural network weights may correspond to different layers of the artificial neural network.

[0279] In some embodiments, the first unit may include a digital-to-analog converter (DAC) unit, and the input data set may further include a second digital input vector. The operation may further include: generating a second plurality of modulator control signals based on the second digital input vector by the DAC unit; obtaining a second plurality of digital outputs corresponding to the analog output vector of the matrix multiplication unit from the ADC unit, the second plurality of digital outputs forming a second digital output vector; performing a nonlinear transformation on the second digital output vector to generate a second transformed digital output vector; storing the second transformed digital output vector in a storage unit; and outputting an artificial neural network output generated based on the first transformed digital output vector and the second transformed digital output vector. The analog output vector of the matrix multiplication unit may be generated by a second optical input vector generated based on the second plurality of modulator control signals, the second optical input vector being transformed by the matrix multiplication unit based on the plurality of weight control signals mentioned first.

[0280] In some embodiments, the system may include a storage unit configured to store an input data set and a neural network weight, and the second unit may include an analog-to-digital conversion (ADC) unit. The system may further include: an analog nonlinear unit disposed between the matrix multiplication unit and the ADC unit, the analog nonlinear unit may be configured to receive a plurality of output voltages from the matrix multiplication unit, apply a nonlinear transfer function, and output a plurality of converted output voltages to the ADC unit. The operations performed by the integrated circuit of the controller may further include: obtaining a first plurality of converted digital output voltages corresponding to the plurality of converted output voltages from the ADC unit, the first plurality of converted digital output voltages forming a first transformed digital output vector; and storing the first transformed digital output vector in the storage unit.

[0281] In some embodiments, the integrated circuit of the controller may be configured to generate the first plurality of modulator control signals at a rate greater than or equal to 8 GHz.

[0282] In some embodiments, the first unit may include a digital-to-analog conversion (DAC) unit, and the second unit may include an analog-to-digital conversion (ADC) unit. The matrix multiplication unit may include: an optical matrix multiplication unit coupled to the plurality of optical modulators and the DAC unit, the optical matrix multiplication unit configured to convert an optical input vector into an optical output vector based on a plurality of weight control signals; and a photodetection unit coupled to the optical matrix multiplication unit and configured to generate a plurality of output voltages corresponding to the optical output vectors.

[0283] In some embodiments, the system may further include: an analog storage unit arranged between the DAC unit and the plurality of optical modulators, the analog storage unit being configured to store an analog voltage and output the stored analog voltage; and an analog nonlinear unit arranged between the photodetection unit and the ADC unit, the analog nonlinear unit being configured to receive a plurality of output voltages from the photodetection unit, apply a nonlinear transfer function, and output a plurality of converted output voltages.

[0284] In some embodiments, the analog memory cell may include multiple capacitors.

[0285] In some embodiments, the analog storage unit may be configured to receive and store a plurality of converted output voltages of the analog nonlinear unit, and output the stored plurality of converted output voltages to the plurality of optical modulators. The operation may further include: storing the plurality of converted output voltages of the analog nonlinear unit in the analog storage unit based on generating a first plurality of modulator control signals and a first plurality of weight control signals; outputting the stored converted output voltages via the analog storage unit; obtaining a second plurality of converted digital output voltages from the ADC unit, the second plurality of converted digital output voltages forming a second transformed digital output vector; and storing the second transformed digital output vector in the storage unit.

[0286] In some embodiments, the system may include a storage unit configured to store an input data set and neural network weights, and the input data set requested by the artificial neural network calculation may include multiple digital input vectors. The light source may be configured to generate multiple wavelengths. The multiple optical modulators may include: an optical modulator group configured to generate multiple optical input vectors, each optical modulator group corresponding to one of the multiple wavelengths and generating a corresponding optical input vector having the corresponding wavelength; and an optical multiplexer configured to combine the multiple optical input vectors into a combined optical input vector including the multiple wavelengths. The photodetector unit may further be configured to demultiplex the multiple wavelengths and generate multiple demultiplexed output voltages. Operations may include: obtaining multiple digital demultiplexed optical outputs from the ADC unit, the multiple digital demultiplexed optical outputs forming multiple first digital output vectors, wherein each of the multiple first digital output vectors corresponds to one of the multiple wavelengths; performing a nonlinear transformation on each of the multiple first digital output vectors to generate multiple transformed first digital output vectors; and storing the multiple transformed first digital output vectors in the storage unit. Each of the multiple digital input vectors corresponds to one of the multiple optical input vectors.

[0287] In some embodiments, the system may include a storage unit configured to store an input data set and neural network weights, the second unit may include an analog-to-digital conversion (ADC) unit, and the artificial neural network computation request may include multiple digital input vectors. The light source may be configured to generate multiple wavelengths. The multiple optical modulators may include: an optical modulator group configured to generate multiple optical input vectors, each optical modulator group corresponding to one of the multiple wavelengths and generating a corresponding optical input vector having a corresponding wavelength; and an optical multiplexer configured to combine the multiple optical input vectors into a combined optical input vector including multiple wavelengths. Operations may include: obtaining a first plurality of digital optical outputs corresponding to the optical output vector from the ADC unit, the optical output vector including multiple wavelengths, the first plurality of digital optical outputs forming a first digital output vector; performing a nonlinear transformation on the first digital output vector to generate a first transformed digital output vector; and storing the first transformed digital output vector in the storage unit.

[0288] In some embodiments, the first unit may include a digital-to-analog conversion (DAC) unit, the second unit may include an analog-to-digital conversion (ADC) unit, and the DAC unit may include: a 1-bit DAC subunit configured to generate a plurality of 1-bit modulator control signals. The resolution of the ADC unit may be 1 bit, and the resolution of the first digital input vector may be N bits. Operations may include: decomposing the first digital input vector into N 1-bit input vectors, each of the N 1-bit input vectors corresponding to one of the N bits of the first digital input vector; generating a sequence of N 1-bit modulator control signals corresponding to the N 1-bit input vectors through the 1-bit DAC subunit; obtaining a sequence of N digital 1-bit optical outputs corresponding to the sequence of N 1-bit modulator control signals from the ADC unit; constructing an N-bit digital output vector from the sequence of N digital 1-bit optical outputs; performing a nonlinear transformation on the constructed N-bit digital output vector to generate a transformed N-bit digital output vector; and storing the transformed N-bit digital output vector in a storage unit.

[0289] In some embodiments, the system may include a storage unit configured to store an input data set and a neural network weight. The storage unit may include: a digital input vector memory configured to store a first digital input vector and comprising at least one SRAM; and a neural network weight memory configured to store a plurality of neural network weights and comprising at least one DRAM.

[0290] In some embodiments, the first unit may include a digital-to-analog conversion (DAC) unit, the digital-to-analog conversion unit including: a first DAC subunit configured to generate a plurality of modulator control signals; and a second DAC subunit configured to generate a plurality of weight control signals, wherein the first DAC subunit and the second DAC subunit are different.

[0291] In some embodiments, the light source may include a laser source configured to generate light; and an optical power splitter configured to split the light generated by the laser source into a plurality of light outputs, wherein each of the plurality of light outputs has substantially the same power.

[0292] In some embodiments, the plurality of optical modulators include one of an MZI modulator, a ring resonance modulator, or an electro-absorption modulator.

[0293] In some embodiments, the photodetection unit may include: a plurality of photodetectors; and a plurality of amplifiers configured to convert photocurrents generated by the photodetectors into a plurality of output voltages.

[0294] In some embodiments, the integrated circuit may be an application specific integrated circuit.

[0295] In some embodiments, a system may include a plurality of optical waveguides coupled between an optical modulator and a matrix multiplication unit, wherein an optical input vector may include a set of multiple input values, the set of multiple input values ​​being encoded on a corresponding optical signal carried by the optical waveguide, and each optical signal carried by one of the optical waveguides may include an optical wave having a common wavelength, the common wavelength being substantially the same for all optical signals.

[0296] In some embodiments, the replication module may include at least one replication module having an optical splitter that transmits a predetermined proportion of the power of the lightwave at the input port to the first output port and transmits a remaining proportion of the power of the lightwave at the input port to the second output port.

[0297] In some embodiments, the optical splitter may include a waveguide splitter that transmits a predetermined proportion of the power of the light waves guided by the input optical waveguide to the first output optical waveguide and transmits the remaining proportion of the power of the light waves guided by the input optical waveguide to the second output optical waveguide.

[0298] In some embodiments, a guided mode of the input optical waveguide can be adiabatically coupled to a guided mode of each of the first and second output optical waveguides.

[0299] In some embodiments, the optical splitter may include a beam splitter including at least one surface that transmits a predetermined proportion of the power of the light wave at the input port and reflects a remaining proportion of the power of the light wave at the input port.

[0300] In some embodiments, at least one of the plurality of optical waveguides may include an optical fiber coupled to an optical coupler that couples a guided mode of the optical fiber to a free-space propagating mode.

[0301] In some embodiments, the multiplication module may include at least one coherence-sensitive multiplication module configured to multiply the one or more optical signals of the first subset by one or more matrix element values ​​using optical amplitude modulation based on interference between optical waves, the optical waves having a coherence length that is at least as long as a propagation distance through the coherence-sensitive multiplication module.

[0302] In some embodiments, the coherence-sensitive multiplication module may include a Mach-Zehnder interferometer (MZI), which separates the light wave guided by the input optical waveguide into a first optical waveguide arm of the MZI and a second optical waveguide arm of the MZI, the first optical waveguide arm includes a phase shifter, the phase shifter produces a relative phase shift relative to the phase delay of the second optical waveguide arm, and the MZI can combine the light waves from the first optical waveguide arm and the second optical waveguide arm into at least one output optical waveguide.

[0303] In some embodiments, the MZI may combine lightwaves from the first optical waveguide arm and the second optical waveguide arm into each of the first output optical waveguide and the second output optical waveguide, the first photodetector may receive the lightwave from the first output optical waveguide to generate a first photocurrent, the second photodetector may receive the lightwave from the second output optical waveguide to generate a second photocurrent, and the result of the coherence-sensitive multiplication module may include a difference between the first photocurrent and the second photocurrent.

[0304] In some embodiments, the coherence-sensitive multiplication module may include one or more ring resonators, including at least one ring resonator coupled to the first optical waveguide and at least one ring resonator coupled to the second optical waveguide.

[0305] In some embodiments, the first photodetector may receive a light wave from the first optical waveguide to generate a first photocurrent, the second photodetector may receive a light wave from the second optical waveguide to generate a second photocurrent, and the result of the coherence-sensitive multiplication module may include a difference between the first photocurrent and the second photocurrent.

[0306] In some embodiments, the multiplication module may include at least one coherence insensitive multiplication module configured to multiply the one or more optical signals of the first subset by one or more matrix element values ​​using optical amplitude modulation based on energy absorption within the optical wave.

[0307] In some embodiments, the coherent insensitive multiplication module may include an electro-absorption modulator.

[0308] In some embodiments, the one or more summing modules may include at least one summing module having the following components: (1) two or more input conductors, each input conductor carrying an electrical signal in the form of an input current, the magnitude of the input current representing a respective result of a respective one of the multiplication modules, and (2) at least one output conductor, the output conductor carrying an electrical signal representing a sum of the respective results in the form of an output current, the output current being proportional to the sum of the input currents.

[0309] In some embodiments, the two or more input conductors and the output conductor may comprise wires that are in contact at one or more junctions between the wires, and the output current is substantially equal to the sum of the input currents.

[0310] In some embodiments, at least a first one of the input currents may be provided in the form of at least one photocurrent generated by at least one photodetector that receives the optical signal generated by a first multiplication module of the multiplication modules.

[0311] In some embodiments, the first input current may be provided in the form of a difference between two photocurrents, the two photocurrents being generated by different corresponding photodetectors that receive different corresponding optical signals generated by the first multiplication module.

[0312] In some embodiments, one of the copies of the first subset of the one or more optical signals may consist of a single optical signal, wherein one of the input values ​​is encoded on the single optical signal.

[0313] In some embodiments, the multiplication modules corresponding to the replicas of the first subset may multiply the encoded input value by a single matrix element value.

[0314] In some embodiments, one of the copies of the first subset of one or more optical signals may include more than one, and less than all, of the optical signals on which the plurality of input values ​​are encoded.

[0315] In some embodiments, the multiplication modules corresponding to the replicas of the first subset may multiply the encoded input values ​​by different corresponding matrix element values.

[0316] In some embodiments, different multiplication modules corresponding to different respective copies of the first subset of one or more optical signals may be included by different devices that are in optical communication to transmit one of the copies of the first subset of one or more optical signals between the different devices.

[0317] In some embodiments, two or more of the plurality of optical waveguides, two or more of the plurality of replica modules, two or more of the plurality of multiplication modules, and at least one of the one or more summation modules may be arranged on a substrate of a common device.

[0318] In some embodiments, the apparatus performs a vector-matrix multiplication, where an input vector may be provided as a set of optical signals and an output vector may be provided as a set of electrical signals.

[0319] In some embodiments, the device may further include an accumulator that combines input electrical signals corresponding to the outputs of the multiplication module or the summation module, wherein the input electrical signal may be encoded using time domain coding, the time domain coding using switching amplitude modulation within each of a plurality of time slots, and the accumulator may generate an output electrical signal, the output electrical signal being encoded with more than two amplitude levels, the amplitude levels corresponding to different duty cycles of the time domain coding over the plurality of time slots.

[0320] In some embodiments, each of the two or more multiplication modules corresponds to a different subset of the one or more optical signals.

[0321] In some embodiments, the apparatus may further include a multiplication module for each copy of a second subset of one or more optical signals different from the optical signals in the first subset of one or more optical signals, configured to multiply the one or more optical signals of the second subset by one or more matrix element values ​​using optical amplitude modulation.

[0322] In another aspect, a system includes: a storage unit configured to store a data set and a plurality of neural network weights; and a driver unit configured to generate a plurality of modulator control signals. The system includes an optoelectronic processor, the optoelectronic processor including: a light source configured to provide a plurality of light outputs; a plurality of optical modulators coupled to the light source and the driver unit, the plurality of optical modulators configured to generate a light input vector by modulating the plurality of light outputs generated by the light source based on the plurality of modulator control signals; a matrix multiplication unit coupled to the plurality of optical modulators and the driver unit, the matrix multiplication unit configured to convert the light input vector into an analog output vector based on the plurality of weight control signals; and a comparator unit coupled to the matrix multiplication unit and configured to convert the analog output vector into a plurality of digital 1-bit outputs. The system includes a controller, which includes an integrated circuit and is configured to perform the following operations: receive an artificial neural network calculation request including an input data set and a first plurality of neural network weights, wherein the input data set includes a first digital input vector with an N-bit resolution; store the input data set and the first plurality of neural network weights in a storage unit; decompose the first digital input vector into N 1-bit input vectors, each of the N 1-bit input vectors corresponding to one of the N bits of the first digital input vector; generate a sequence of N 1-bit modulator control signals corresponding to the N 1-bit input vectors through a driver unit; obtain a sequence of N digital 1-bit outputs corresponding to the sequence of N 1-bit modulator control signals from a comparator unit; construct an N-bit digital output vector from the sequence of N digital 1-bit outputs; perform a nonlinear transformation on the constructed N-bit digital output vector to generate a transformed N-bit digital output vector; and store the transformed N-bit digital output vector in the storage unit.

[0323] Embodiments of the system may include one or more of the following features: For example, receiving the artificial neural network computation request may include receiving the artificial neural network computation request from a general purpose computer.

[0324] In some embodiments, the driver unit may be configured to generate a plurality of weight control signals.

[0325] In some embodiments, the matrix multiplication unit may include: an optical matrix multiplication unit, coupled to a plurality of optical modulators and a driver unit, the optical matrix multiplication unit being configured to convert an optical input vector into an optical output vector based on a plurality of weight control signals; and a photodetection unit, coupled to the optical matrix multiplication unit and configured to generate a plurality of output voltages corresponding to the optical output vectors.

[0326] In some embodiments, the matrix multiplication unit may include: an input waveguide array for receiving an optical input vector; an optical interference unit in optical communication with the input waveguide array, for performing a linear transformation to convert the optical input vector into a second optical signal array; and an output waveguide array in optical communication with the optical interference unit, for guiding the second optical signal array, wherein at least one input waveguide in the input waveguide array is in optical communication with each output waveguide in the output waveguide array through the optical interference unit.

[0327] In some embodiments, the optical interference unit may include: a plurality of interconnected Mach-Zehnder interferometers (MZIs), each of the plurality of interconnected MZIs including: a first phase shifter configured to change the splitting ratio of the MZI; and a second phase shifter configured to shift the phase of one output of the MZI, wherein the first phase shifter and the second phase shifter may be coupled to a plurality of weight control signals.

[0328] In some embodiments, the matrix multiplication unit may include: a plurality of replication modules, for each of at least two subsets of one or more optical signals of the optical input vector, the plurality of replication modules including a corresponding group of one or more replication modules configured to split the subset of the one or more optical signals into two or more copies of the optical signals; a plurality of multiplication modules, for each of at least two copies of a first subset of the one or more optical signals, the plurality of multiplication modules including a corresponding multiplication module configured to multiply the one or more optical signals of the first subset by one or more matrix element values ​​using optical amplitude modulation; and one or more summation modules, for results of the two or more multiplication modules, the one or more summation modules including a summation module configured to generate an electrical signal, the electrical signal representing the sum of the results of the two or more multiplication modules.

[0329] In some embodiments, at least one multiplication module may include an optical amplitude modulator, which includes one input port and two output ports, and can provide a pair of correlated optical signals from the two output ports such that a difference between the amplitudes of the correlated optical signals corresponds to a result of multiplying the input value by the signed matrix element value.

[0330] In some embodiments, the matrix multiplication unit may be configured to multiply the light input vector by a matrix comprising one or more matrix element values.

[0331] In some embodiments, a set of multiple output values ​​may be encoded on corresponding electrical signals generated by one or more summing modules, and the output values ​​in the set of multiple output values ​​may represent elements of an output vector generated by multiplying the optical input vector by the matrix.

[0332] In another aspect, a method is provided for performing artificial neural network calculations in a system having a matrix multiplication unit configured to convert an optical input vector into an analog output vector based on a plurality of weight control signals. The method includes: receiving an artificial neural network calculation request including an input data set and a first plurality of neural network weights, wherein the input data set includes a first digital input vector; storing the input data set and the first plurality of neural network weights in a memory unit; generating a first plurality of modulator control signals based on the first digital input vector and generating a first plurality of weight control signals based on the first plurality of neural network weights; obtaining a first plurality of digital outputs corresponding to an output vector of the matrix multiplication unit, the first plurality of digital outputs forming a first digital output vector; performing a nonlinear transformation on the first digital output vector by a controller to generate a first transformed digital output vector; storing the first transformed digital output vector in the memory unit; and outputting, by the controller, an artificial neural network output generated based on the first transformed digital output vector.

[0333] Embodiments of the method may include one or more of the following features: For example, receiving the artificial neural network computation request may include receiving the artificial neural network computation request from a computer via a communication channel.

[0334] In some embodiments, generating the first plurality of modulator control signals may include generating the first plurality of modulator control signals via a digital-to-analog conversion (DAC) unit.

[0335] In some embodiments, obtaining the first plurality of digital outputs may include obtaining the first plurality of digital outputs from an analog-to-digital conversion (ADC) unit.

[0336] In some embodiments, the method may include: applying a first plurality of modulator control signals to a plurality of optical modulators coupled to a light source and a DAC unit; and generating a light input vector using the plurality of optical modulators by modulating a plurality of light outputs generated by a laser unit based on the plurality of modulator control signals.

[0337] In some embodiments, a matrix multiplication unit may be coupled to the plurality of optical modulators and the DAC unit, and the method may include converting the optical input vector into an analog output vector based on the plurality of weight control signals using the matrix multiplication unit.

[0338] In some embodiments, the ADC unit may be coupled to the matrix multiplication unit, and the method may include converting the analog output vector into a first plurality of digital outputs using the ADC unit.

[0339] In some embodiments, the matrix multiplication unit may include an optical matrix multiplication unit coupled to a plurality of optical modulators and a DAC unit. Converting the optical input vector into an analog output vector may include using the optical matrix multiplication unit to convert the optical input vector into an optical output vector based on a plurality of weight control signals. The method may include generating a plurality of output voltages corresponding to the optical output vectors using a photodetection unit coupled to the optical matrix multiplication unit.

[0340] In some embodiments, a method may include: receiving an optical input vector at an input waveguide array; performing a linear transformation to convert the optical input vector into a second optical signal array using an optical interference unit in optical communication with the input waveguide array; and guiding the second optical signal array using an output waveguide array in optical communication with the optical interference unit, wherein at least one input waveguide in the input waveguide array is in optical communication with each output waveguide in the output waveguide array through the optical interference unit.

[0341] In some embodiments, an optical interferometer may include: a plurality of interconnected Mach-Zehnder interferometers (MZIs), each of the plurality of interconnected MZIs may include a first phase shifter and a second phase shifter, and the first phase shifter and the second phase shifter may be coupled to a plurality of weight control signals. A method may include: using the first phase shifter to change the splitting ratio of the MZI, and using the second phase shifter to shift the phase of one output of the MZI.

[0342] In some embodiments, a method may include: for each of at least two subsets of one or more optical signals of an optical input vector, using a corresponding set of one or more replication modules to split the subset of the one or more optical signals into two or more copies of the optical signals; for each of the at least two copies of a first subset of the one or more optical signals, using a corresponding multiplication module to multiply the one or more optical signals of the first subset by one or more matrix element values ​​using optical amplitude modulation; and for results of the two or more multiplication modules, using a summation module configured to generate an electrical signal, the electrical signal representing the sum of the results of the two or more multiplication modules.

[0343] In some embodiments, at least one multiplication module may include an optical amplitude modulator, which includes one input port and two output ports, and can provide a pair of correlated optical signals from the two output ports such that a difference between the amplitudes of the correlated optical signals corresponds to a result of multiplying the input value by the signed matrix element value.

[0344] In some embodiments, the method may include multiplying the light input vector by a matrix comprising one or more matrix element values ​​using a matrix multiplication unit.

[0345] In some embodiments, the method may include encoding a set of multiple output values ​​on corresponding electrical signals generated by one or more summing modules, and using output values ​​from the set of multiple output values ​​to represent elements of an output vector, the output vector being generated by multiplying the optical input vector by the matrix.

[0346] On the other hand, a method includes: providing input information in an electronic format; converting at least a portion of the electronic input information into an optical input vector; optoelectronically converting the optical input vector into an analog output vector based on matrix multiplication; and electronically applying a nonlinear transformation to the analog output vector to provide output information in an electronic format.

[0347] Embodiments of the method may include one or more of the following features.For example, the method may further include repeating the electro-optical conversion, the opto-electrical conversion, and the electrically applied non-linear transformation for new electronic input information corresponding to the output information provided in the electronic format.

[0348] In some embodiments, the matrix multiplication for the initial photoelectric conversion and the matrix multiplication for the repeated photoelectric conversion may be the same and may correspond to the same layer of the artificial neural network.

[0349] In some embodiments, the matrix multiplication for the initial photoelectric conversion and the matrix multiplication for the repeated photoelectric conversion may be different and may correspond to different layers of the artificial neural network.

[0350] In some embodiments, the method may further include: repeating electro-optical conversion, photoelectric conversion, and electrically applied nonlinear transformation for different parts of the electronic input information, wherein the matrix multiplication for the initial photoelectric conversion and the matrix multiplication for the repeated photoelectric conversion are the same and correspond to the first layer of the artificial neural network.

[0351] In some embodiments, the method may further include: providing intermediate information in an electronic format based on electronic output information for multiple parts of the electronic input information generated by the first layer of the artificial neural network; and repeating the electro-optical conversion, the optoelectronic conversion, and the electrically applied nonlinear transformation for each of the different parts of the electronic intermediate information, wherein the matrix multiplication for the initial optoelectronic conversion and the matrix multiplication for the repeated optoelectronic conversion associated with the different parts of the electronic intermediate information are the same and correspond to the second layer of the artificial neural network.

[0352] In another aspect, a system for performing artificial neural network calculations is provided. The system includes: a first unit configured to generate a plurality of vector control signals and to generate a plurality of weight control signals; a second unit configured to provide an optical input vector based on the plurality of vector control signals; and a matrix multiplication unit coupled to the second unit and the first unit, the matrix multiplication unit configured to convert the optical input vector into an output vector based on the plurality of weight control signals. The system includes a controller, the controller including an integrated circuit configured to perform the following operations: receive an artificial neural network calculation request including an input data set and a first plurality of neural network weights, wherein the input data set includes a first digital input vector; and generate, by the first unit, a first plurality of vector control signals based on the first digital input vector and a first plurality of weight control signals based on the first plurality of neural network weights; wherein the first unit, the second unit, the matrix multiplication unit, and the controller are used for a photoelectric processing cycle that is repeated in multiple iterations, and the photoelectric processing cycle includes: (1) at least two optical modulation operations, and (2) at least one of (a) an electrical summation operation or (b) an electrical storage operation.

[0353] In another aspect, a method for performing artificial neural network computations is provided. The method includes providing input information in an electronic format; converting at least a portion of the electronic input information into an optical input vector; and converting the optical input vector into an output vector based on matrix multiplication using a set of neural network weights. The providing and converting are performed in an optical processing cycle, the optical processing cycle being repeated in multiple iterations using different corresponding sets of neural network weights and different corresponding input information, and the optical processing cycle includes: (1) at least two optical modulation operations, and (2) at least one of (a) an electrical summing operation or (b) an electrical storage operation.

[0354] On the other hand, a computing system is provided, comprising: a first unit configured to generate a plurality of modulator control signals; a processor unit comprising: a light source or port configured to provide a plurality of light outputs; a first group of light modulators coupled to the light source or port and the first unit, the light modulators in the first group of light modulators being configured to generate a light input vector by modulating the plurality of light outputs provided by the light source or port based on digital input values ​​corresponding to a first group of modulator control signals in the plurality of modulator control signals, the light input vector comprising a plurality of light signals; and a matrix multiplication unit comprising a second group of light modulators, wherein the matrix multiplication unit is coupled to the first unit, and the matrix multiplication unit is configured to generate a light input vector by modulating the plurality of light outputs provided by the light source or port based on digital input values ​​corresponding to a second group of modulator control signals in the plurality of modulator control signals applied to the second group of light modulators. The optical input vector is transformed into an analog output vector by using a plurality of digital weight values ​​corresponding to the optical signals of the optical input vector, wherein the matrix multiplication unit comprises: a plurality of replication modules, and for each of the at least two subsets of the one or more optical signals of the optical input vector, the plurality of replication modules comprises a corresponding group of one or more replication modules configured to split the subset of the one or more optical signals into two or more copies of the optical signals, at least one optical modulator of at least one of the first group of optical modulators or the second group of optical modulators is configured to modulate the optical signal based on a first modulator control signal of the plurality of modulator control signals, and the first unit is configured to shape the first modulator control signal to include a bandwidth enhancement associated with an amplitude variation associated with a corresponding variation in the continuous digital values ​​corresponding to the first modulator control signal.

[0355] In another aspect, a computing device is provided, comprising: a plurality of optical waveguides coupled to a first set of optical amplitude modulators, wherein a set of a plurality of input values ​​is encoded on respective optical signals carried by the optical waveguides using the first set of optical amplitude modulators; a plurality of replication modules, wherein for each of at least two subsets of one or more optical signals, a corresponding set of the one or more replication modules is configured to split the subset of the one or more optical signals into two or more copies of the optical signals; a plurality of multiplication modules, each multiplication module comprising an optical amplitude modulator from a second set of optical amplitude modulators, wherein for each of the at least two copies of the first subset of the one or more optical signals, a corresponding set of the one or more replication modules is configured to split the subset of the one or more optical signals into two or more copies of the optical signals; a corresponding multiplication module configured to multiply one or more optical signals in the first subset by one or more matrix element values ​​using an optical amplitude modulator in the second group of optical amplitude modulators; and one or more summation modules, and for the results of two or more multiplication modules, a corresponding one summation module configured to generate an electrical signal, the electrical signal representing the sum of the results of the two or more multiplication modules; wherein at least one optical amplitude modulator of at least one of the first group of optical amplitude modulators or the second group of optical amplitude modulators is configured to modulate the optical signal by the modulation value using a power that increases monotonically with respect to the absolute value of the modulation value.

[0356] The details of one or more embodiments of the subject matter described in this disclosure are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the disclosure will become apparent from the description and drawings.

[0357] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. In the event of a conflict with an incorporated herein by reference patent application or patent application publication, the present disclosure, including definitions, will control. BRIEF DESCRIPTION OF THE DRAWINGS

[0358] The present disclosure will be best understood from the following detailed description when read in conjunction with the accompanying drawings. It is emphasized that, according to common practice, the various features of the drawings are not to scale. On the contrary, the dimensions of the various features have been arbitrarily expanded or reduced for clarity.

[0359] Figure 1A is a schematic diagram of an example of an artificial neural network (ANN) computing system.

[0360] Figure 1B is a schematic diagram of an example of an optical matrix multiplication unit.

[0361] Figure 1C and Figure 1D is a schematic diagram of an example configuration of interconnected Mach-Zehnder interferometers (MZIs).

[0362] Figure 1E is a schematic diagram of an example of an MZI.

[0363] Figure 1F is a schematic diagram of an example of a wavelength division multiplexed ANN (WDM) computing system.

[0364] Figure 2A is a flow chart illustrating an example of a method for performing ANN calculations.

[0365] Figure 2B It shows Figure 2A FIG. 1 is a diagram of one aspect of a method.

[0366] Figure 3A and Figure 3B is a schematic diagram of an example of an ANN computing system.

[0367] Figure 4A is a schematic diagram of an example of an ANN computing system with 1-bit internal resolution.

[0368] Figure 4B yes Figure 4A A mathematical representation of the operation of an ANN computing system.

[0369] Figure 5 is a schematic diagram of an example of an artificial neural network (ANN) computing system.

[0370] Figure 6 is a schematic diagram of an example of an optical matrix multiplication unit.

[0371] Figure 7 is a schematic diagram of an example of an artificial neural network (ANN) computing system.

[0372] Figure 8 is a diagram of an example of a light matrix multiplication unit.

[0373] Figure 9 is a schematic diagram of an example of an artificial neural network (ANN) computing system.

[0374] Figure 10 is a diagram of an example of a light matrix multiplication unit.

[0375] Figure 11 is a diagram of an example of a compact matrix multiplier unit.

[0376] Figure 12A A diagram comparing photonic matrix multiplier units is shown.

[0377] Figure 12B is a diagram of a compact interconnected interferometer.

[0378] Figure 13 is a diagram of a compact matrix multiplier unit.

[0379] Figure 14 This is a diagram of an optical generative adversarial network.

[0380] Figure 15 This is a diagram of a Mach-Zehnder interferometer.

[0381] Figure 16 、 Figure 17A as well as Figure 17B is a diagram of a photonic circuit.

[0382] Figure 18 is a schematic diagram of an example of an optoelectronic computing system.

[0383] Figure 19A and Figure 19B is a diagram of an example system configuration.

[0384] Figure 20A is a schematic diagram of an example of a symmetric differential configuration.

[0385] Figure 20B and Figure 20C This is a circuit diagram of an example of a system module.

[0386] Figure 21A is a schematic diagram of an example of a symmetric differential configuration.

[0387] Figure 21B is a diagram showing an example of a system configuration.

[0388] Figure 22A is a schematic diagram of an example optical amplitude modulator.

[0389] FIG. 22B to FIG. 22D is a schematic diagram of an example of an optical amplitude modulator using optical detection in a symmetric differential configuration.

[0390] Figures 23A to 23C is a photoelectric circuit diagram of an example system configuration.

[0391] Figures 24A to 24E is a schematic diagram of an example computing system using multiple optoelectronic systems.

[0392] Figure 25 is a flow chart illustrating an example of a method for performing ANN calculations.

[0393] Figure 26 and Figure 27 is a schematic diagram of an example of an ANN computing system.

[0394] Figure 28 is a schematic diagram of an example of a neural network computing system using a passive 2D optical matrix multiplication unit.

[0395] Figure 29 is a schematic diagram of an example of a neural network computing system using a passive 3D optical matrix multiplication unit.

[0396] Figure 30 is a schematic diagram of an example of an artificial neural network computing system with 1-bit internal resolution, where the system uses a passive 2D optical matrix multiplication unit.

[0397] Figure 31 is a schematic diagram of an example of an artificial neural network computing system with 1-bit internal resolution, where the system uses a passive 3D optical matrix multiplication unit.

[0398] Figure 32A is a schematic diagram of an example of an artificial neural network (ANN) computing system.

[0399] Figure 32B is a schematic diagram of an example of an optoelectronic matrix multiplication unit.

[0400] Figure 33 is a flow chart illustrating an example of a method for performing ANN calculations using an optoelectronic processor.

[0401] Figure 34 It shows Figure 33 FIG. 1 is a diagram of one aspect of a method.

[0402] Figure 35A is a schematic diagram of an example of a wavelength division multiplexing ANN computing system using an optoelectronic processor.

[0403] Figure 35B and Figure 35C is a schematic diagram of an example of a wavelength division multiplexing optoelectronic matrix multiplication unit.

[0404] Figure 36 and Figure 37 is a schematic diagram of an example of an ANN computing system using an optoelectronic matrix multiplication unit.

[0405] Figure 38 is a schematic diagram of an example of an artificial neural network computing system with 1-bit internal resolution, where the system uses an optoelectronic matrix multiplication unit.

[0406] Figure 39Ais a schematic diagram of an example of a Mach-Zehnder modulator.

[0407] Figure 39B It shows Figure 39A A graph of the intensity-voltage curve of a Mach-Zehnder modulator.

[0408] Figure 40 is a schematic diagram of a homodyne detector.

[0409] Figure 41 is a schematic diagram of a computing system including optical fibers, each of which carries signals having multiple wavelengths.

[0410] Figure 42 is a graph of a probability distribution of modulation values ​​and an example relationship between modulator power and modulation value.

[0411] Figure 43 Diagram of an example of a Mach-Zehnder modulator.

[0412] Figure 44 Diagram of an example of a charge-pump bandwidth enhancement circuit.

[0413] Like reference numbers and names in the various drawings refer to like components. DETAILED DESCRIPTION

[0414] Figure 1A A diagram illustrates an example of an artificial neural network (ANN) computing system 100. The system 100 includes a controller 110, a memory unit 120, a digital-to-analog conversion (DAC) unit 130, an optical processor 140, and an analog-to-digital conversion (ADC) unit 160. The controller 110 is coupled to a computer 102, the memory unit 120, the DAC unit 130, and the ADC unit 160. The controller 110 includes an integrated circuit configured to control the operation of the ANN computing system 100 to perform ANN computations.

[0415] The integrated circuit of controller 110 may be an application-specific integrated circuit specifically configured to perform the steps of the ANN computation process. For example, the integrated circuit may implement microcode or firmware specific to performing the ANN computation process. As such, controller 110 may have a reduced instruction set relative to a general-purpose processor used in a conventional computer (e.g., computer 102). In some embodiments, the integrated circuit of controller 110 may include two or more circuits configured to perform different steps of the ANN computation process.

[0416] In an example operation of the ANN computing system 100, the computer 102 may issue an artificial neural network computing request to the ANN computing system 100. The ANN computing request may include neural network weights defining the ANN and an input data set to be processed by the provided ANN. The controller 110 receives the ANN computing request and stores the input data set and neural network weights in the storage unit 120.

[0417] The input dataset can correspond to a variety of digital information to be processed by the ANN. Examples of input datasets include image files, audio files, LiDAR point clouds, and GPS coordinate sequences. The operation of the ANN computing system 100 will be described based on receiving image files as input datasets. Generally speaking, the size of an input dataset can vary greatly, from hundreds of data points to millions of data points or more. For example, a digital image file with a resolution of 1 megapixel has approximately one million pixels, and each of the one million pixels can be a data point processed by the ANN. Due to the large number of data points in a typical input dataset, the input dataset is often divided into multiple digital input vectors of smaller size to be processed separately by the optical processor 140. As an example, for a grayscale digital image, the elements of the digital input vector may be 8-bit values ​​representing image intensity, and the digital input vector may have a length ranging from tens of elements (e.g., 32 elements, 64 elements) to hundreds of elements (e.g., 256 elements, 512 elements). In general, an input data set of any size can be divided into digital input vectors of a size suitable for processing by the optical processor 140. In the event that the number of elements in the input data set is not evenly divisible by the length of the digital input vector, zero padding can be used to fill the data set so that it is evenly divisible by the length of the digital input vector. The processed outputs of the respective digital input vectors can be processed to reconstruct the complete output, which is the result of processing the input data set by the ANN. In some embodiments, block matrix multiplication techniques can be used to implement the division of the input data set into multiple input vectors and subsequent vector-level processing.

[0418] Neural network weights are a set of values ​​that define the connectivity of the artificial neurons of the ANN, including the relative importance or weights of those connections. The ANN may include one or more hidden layers with corresponding sets of nodes. In the case of an ANN with a single hidden layer, the ANN may be defined by two sets of neural network weights, one set corresponding to the connectivity between input nodes and nodes of the hidden layer, and a second set corresponding to the connectivity between the hidden layer and output nodes. Each set of neural network weights describing the connectivity corresponds to a matrix implemented by the optical processor 140. For an ANN with two or more hidden layers, additional sets of neural network weights are required to define the connectivity between the additional hidden layers. As such, in general, the neural network weights included in the ANN computation request may include multiple sets of neural network weights that represent the connectivity between corresponding layers of the ANN.

[0419] Because the input data set to be processed is typically divided into multiple smaller digital input vectors for separate processing, the input data set is typically stored in digital memory. However, the speed of storage operations between the computer 102's memory and the processor is significantly slower than the rate at which the ANN computing system 100 can perform ANN computations. For example, the ANN computing system 100 may perform dozens to hundreds of ANN computations during a typical memory read period of the computer 102. As a result, during the process of processing an ANN computation request, if the ANN computation of the ANN computing system 100 involves multiple data transfers between the system 100 and the computer 102, the rate at which the ANN computation can be performed by the ANN computing system 100 may be limited to below its overall processing rate. For example, if the computer 102 were to access the input data set from its own memory and provide the digital input vectors to the controller 110 upon request, the operation of the ANN computing system 100 could be significantly slowed by the time required for the series of data transfers required between the computer 102 and the controller 110. It is worth noting that the memory access latency of the computer 102 is generally non-deterministic, which further complicates and reduces the speed at which the digital input vector can be provided to the ANN computing system 100. In addition, processor cycles of the computer 102 may be wasted in managing data transfers between the computer 102 and the ANN computing system 100.

[0420] In contrast, in some embodiments, the ANN computing system 100 stores the entire input data set in a memory unit 120 that is part of and dedicated to the ANN computing system 100. The dedicated memory unit 120 allows transactions between the memory unit 120 and the controller 110, which are particularly well-suited to allowing smooth and uninterrupted data flow between the memory unit 120 and the controller 110. This uninterrupted data flow can significantly improve the overall throughput of the ANN computing system 100 by allowing the optical processor 140 to perform matrix multiplications at its full processing rate without being limited by the slow memory operations of conventional computers (e.g., computer 102). Furthermore, because all data required to perform the ANN computation is provided to the ANN computing system 100 by computer 102 in a single transaction, the ANN computing system 100 can perform its ANN computations in a manner uniquely independent of computer 102. This unique operation of the ANN computing system 100 reduces the computational burden on the computer 102 and eliminates external dependencies in the operation of the ANN computing system 100 , improving the performance of the system 100 and the computer 102 .

[0421] The internal operation of the ANN computing system 100 will now be described. The optical processor 140 includes a laser unit 142, a modulator array 144, a detection unit 146, and an optical matrix multiplication (OMM) unit 150. The optical processor 140 operates by encoding a digital input vector of length N onto an optical input vector of length N and propagating the optical input vector through the OMM unit 150. The OMM unit 150 receives an optical input vector of length N and performs an N×N matrix multiplication on the received optical input vector in the optical domain. The N×N matrix multiplication performed by the OMM unit 150 is determined by the internal configuration of the OMM unit 150. The internal configuration of the OMM unit 150 can be controlled by an electrical signal, such as the electrical signal generated by the DAC unit 130.

[0422] The OMM unit 150 may be implemented in various ways. Figure 1BA diagram illustrates an example of an OMM cell 150. The OMM cell 150 may include an array of input waveguides 152 to receive an optical input vector; an optical interferometer 154 in optical communication with the array of input waveguides 152; and an array of output waveguides 156 in optical communication with the optical interferometer 154. The optical interferometer 154 linearly transforms the optical input vector into a second optical signal array. The array of output waveguides 156 guides the second optical signal array output by the optical interferometer 154. At least one input waveguide in the array of input waveguides 152 is in optical communication with each output waveguide in the array of output waveguides 156 via the optical interferometer 154. For example, for an optical input vector of length N, the OMM cell 150 may include N input waveguides 152 and N output waveguides 156.

[0423] The optical interference unit may include a plurality of interconnected Mach-Zehnder interferometers (MZIs). Figure 1C and Figure 1D A diagram shows examples of example configurations 157 and 158 of interconnected MZIs. The MZIs can be interconnected in various ways (eg, in configurations 157 or 158) to achieve a linear transformation of the light input vector received by the array of input waveguides 152.

[0424] Figure 1EA diagram illustrates an example of an MZI 170. MZI 170 includes a first input waveguide 171, a second input waveguide 172, a first output waveguide 178, and a second output waveguide 179. Furthermore, each of the plurality of interconnected MZIs 170 includes a first phase shifter 174 configured to change the splitting ratio of MZI 170, and a second phase shifter 176 configured to shift the phase of one output of MZI 170, such as light exiting MZI 170 through second output waveguide 179. The first and second phase shifters 174, 176 of MZI 170 are coupled to a plurality of weight control signals generated by DAC unit 130. The first and second phase shifters 174, 176 are examples of reconfigurable components of OMM unit 150. Examples of reconfigurable components include thermo-optic phase shifters or electro-optic phase shifters. Thermo-optic phase shifters operate by heating the waveguide to change the refractive index of the waveguide and cladding materials, which translates into a change in phase. Electro-optic phase shifters operate by applying an electric field (e.g., lithium niobate (LiNbO3), reverse biasing a PN junction) or current (e.g., forward biasing a PIN junction), which changes the refractive index of the waveguide material. By changing the weight control signal, the phase delay of the first phase shifter 174 and the second phase shifter 176 of each interconnected MZI 170 can be changed, which reconfigures the optical interferometer unit 154 of the OMM unit 150 to achieve a specific matrix multiplication determined by the phase delays arranged across the optical interferometer unit 154. Additional embodiments of the OMM unit 150 and the optical interference unit 154 are disclosed in U.S. Patent Publication No. US2017 / 0351293A1, entitled “APPARATUS AND METHODS FOR OPTICAL NEURAL NETWORK,” which is incorporated herein by reference in its entirety.

[0425] An optical input vector is generated by laser unit 142 and modulator array 144. The optical input vector of length N has N independent optical signals, each having an intensity corresponding to the value of a corresponding element of the digital input vector of length N. As an example, laser unit 142 can generate N optical outputs. The N optical outputs have the same wavelength and are optically coherent. The optical coherence of the optical outputs allows the optical outputs to optically interfere with each other, a property exploited by OMM unit 150 (e.g., in the operation of an MZI). Furthermore, the optical outputs of laser unit 142 can be substantially identical to each other. For example, the N optical outputs can be substantially uniform in their intensities (e.g., within 5%, within 3%, within 1%, within 0.5%, within 0.1%, or within 0.01%) and their relative phases (e.g., within 10 degrees, within 5 degrees, within 3 degrees, within 1 degree, or within 0.1 degrees). The uniformity of the light output can improve the faithfulness of the optical input vector to the digital input vector, thereby improving the overall accuracy of the optical processor 140. In some embodiments, the light output of the laser unit 142 can have an optical power of 0.1 mW to 50 mW per output, a wavelength in the near-infrared range (e.g., between 900 nm and 1600 nm), and a linewidth of less than 1 nm. The light output of the laser unit 142 can be a single transverse-mode light output.

[0426] In some embodiments, the laser unit 142 includes a single laser source and an optical power splitter. The single laser source is configured to generate laser light. The optical power splitter is configured to split the light generated by the laser source into N light outputs having substantially the same intensity and phase. By splitting the single laser output into multiple outputs, optical coherence of the multiple light outputs can be achieved. For example, the single laser source can be a semiconductor laser diode, a vertical-cavity surface-emitting laser (VCSEL), a distributed feedback (DFB) laser, or a distributed Bragg reflector (DBR) laser. For example, the optical power splitter can be a 1:N multimode interference (MMI) splitter, a multi-stage splitter including multiple 1:2 MMI splitters or directional couplers, or a star coupler. In some other embodiments, a master-slave laser configuration may be used, in which a slave laser is injection locked to a master laser to have a stable phase relationship to the master laser.

[0427] The optical output of the laser unit 142 is coupled to the modulator array 144. The modulator array 144 is configured to receive the optical input from the laser unit 142 and modulate the intensity of the received optical input based on the modulator control signal (which is an electrical signal). Examples of modulators include Mach-Zehnder interferometer (MZI) modulators, ring resonator modulators, and electro-absorption modulators. The modulator array 144 has N modulators, each of which receives one of the N optical outputs of the laser unit 142. The modulator receives a control signal corresponding to an element of a digital input vector and modulates the intensity of the light. The control signal can be generated by the DAC unit 130.

[0428] DAC unit 130 is configured to generate multiple modulator control signals and multiple weight control signals under the control of controller 110. For example, DAC unit 130 receives a first DAC control signal from controller 110, corresponding to a digital input vector to be processed by optical processor 140. Based on the first DAC control signal, DAC unit 130 generates modulator control signals, which are analog signals suitable for driving modulator array 144 and OMM 150. For example, the analog signal can be a voltage or a current, depending on the technology and design of the modulators in array 144. The voltage can have an amplitude ranging from ±0.1V to ±10V, and the current can have an amplitude ranging from 100μA to 100mA. In some embodiments, DAC unit 130 may include a modulator driver configured to buffer, amplify, or condition the analog signal so that the modulators in array 144 and OMM 150 can be adequately driven. For example, certain types of modulators can be driven using differential control signals. In this case, the modulator driver can be a differential driver that produces a differential electrical output based on a single-ended input signal. As another example, certain types of modulators may have a 3dB bandwidth that is less than the desired processing rate of the optical processor 140. In this case, the modulator driver can include a pre-emphasis circuit or other bandwidth enhancement circuit designed to extend the operating bandwidth of the modulator. For example, such bandwidth enhancement can be useful for modulators based on a PIN diode structure, which is forward biased to use carrier injection to modulate the refractive index of a portion of the waveguide that guides the modulated light wave. For example, if the modulator is an MZI modulator, the PIN diode structure can be used to implement a phase shifter in one or both waveguide arms of the MZI modulator. Configuring the phase shifter for forward biased operation facilitates a shorter modulator length and a more compact overall design, which can be useful for an OMM cell 150 having a large number of modulators.

[0429] For example, in a bandwidth-enhanced pre-emphasis form, the analog electrical signal (e.g., voltage or current) driving the modulator can be shaped to include transient pulses that overshoot changes in the analog signal level representing a given digital data value of the DAC control signal within a range of digital data values. Each digital data value can have any number of bits, including a single 1-bit data value, as assumed in the remainder of this example. Thus, if the bit value is the same as the previous value, the analog electrical signal driving the modulator is maintained at a steady-state level (e.g., signal level X0 for a bit value of 0 and a higher signal level X1 for a bit value of 1). However, if the bit changes from 0 to 1, the corresponding analog electrical signal driving the modulator may include a transient pulse having a peak value of X1+(X1-X0) at the beginning of the bit transition before settling to the steady-state value X1. Likewise, if a bit changes from 1 to 0, the corresponding analog electrical signal driving the modulator may include a transient pulse having a peak value of X0 + (X0 - X1) at the beginning of the bit transition before settling to a steady-state value of X0. The size and length of the transient pulse may be selected to optimize bandwidth enhancement (e.g., to maximize the open area of ​​an eye diagram for a non-return-to-zero (NRZ) modulation scheme).

[0430] In a bandwidth-enhanced form of charge pumping, the analog current signal driving the modulator can be shaped into a momentary pulse that moves a precisely determined amount of charge. Figure 44 A charge pumping bandwidth enhancement circuit is shown that uses a capacitor connected in series between the voltage source and the modulator to precisely control the charge flow. Figure 44 A portion of the circuit shown may be included in the modulator driver described above. In this embodiment, the modulator is represented by a modulator circuit 4400 that models the electrical characteristics of the modulator's phase shifter as a PIN diode. The modulator circuit 4400 includes an ideal diode, a capacitor C d The pump capacitor 4402 has a capacitance C and a parallel connection of a resistor having a resistance R. pThe control voltage waveform 4404 is provided to an inverter circuit 4405 to generate a drive voltage waveform 4406, the amplitude of which can be precisely calibrated to shift a predetermined amount of charge into or out of the modulator circuit 4400 through the pumping capacitor 4402. The PIN diode modeled by the modulator circuit 4400 is forward biased by applying a constant voltage VDD_IO at terminal 4408. A charge pump control voltage VCP is applied at terminal 4410 of the inverter 4405 to control the amount of charge pumped at the transitions of the drive voltage waveform 4406, and the corresponding optical phase shift applied by the modulator.

[0431] The value of the charge pump control voltage VCP can be adjusted before operation so that the nominal charge Q stored in the charge pump capacitor 4402 is based on the capacitance C p The measured value (e.g., there may be some variability due to manufacturing uncertainties) is precisely calibrated. For example, the voltage VCP may be equal to the nominal charge Q divided by the capacitance C p The induced change in the refractive index of the portion of the waveguide that intersects the PIN diode can then provide a phase shift in the guided lightwave that is proportional to the amount of charge Q moved between the PIN diode and the charge pump capacitor 4402 (e.g., via internal capacitance C d If the drive voltage changes from a low value to a high value, the current flowing from the charge pump capacitor 4402 into the PIN diode transfers a predetermined amount of charge in a short period of time (i.e., the integral of the positive current over time). If the drive voltage changes from a high value to a low value, the current flowing from the PIN diode into the charge pump capacitor 4402 removes a predetermined amount of charge in a short period of time (i.e., the integral of the negative current over time). After this relatively short switching time, a steady-state current is provided by the current source 4412, which is controlled by the switch 4414 to replace the charge lost due to the internal capacitor losing current through the internal resistor R while maintaining the drive voltage (e.g., during the hold time of a particular digital value). Using this charge pumping configuration can have advantages, such as better accuracy than other techniques (including some pre-emphasis techniques), because the amount of charge moved in the short switching time depends on a constant physical parameter (C p ) and the steady-state control value (VCP), and is therefore precisely controllable and repeatable.

[0432] In some embodiments, reduced power consumption can be achieved by designing the modulators of modulator array 144 and / or OMM unit 150 such that less power is consumed when the modulators are operated to generate modulation values ​​representing more frequently occurring coefficients, and more power is consumed when the modulators are operated to generate modulation values ​​representing less frequently occurring coefficients. For example, power consumption can be reduced for certain data sets known to have certain characteristics. Figure 42 Graph 4200 (dashed line) shows a modulation value probability distribution graph for a particular design of a modulator and / or OMM unit 150 for modulator array 144, superimposed on graph 4202 (solid line) of modulator power. Both graphs are functions of modulation value (on the horizontal axis) expressed in normalized units to represent coefficients between -1 and 1. In this embodiment, a dataset includes various coefficients (e.g., vector coefficients and / or matrix coefficients) for artificial neural network calculations such that the coefficient probability distribution function (PDF) yields higher probabilities (and therefore more frequent occurrences) for smaller coefficients (i.e., coefficients with relatively smaller absolute values). For such datasets ("low-coefficient weight datasets"), reduced power consumption can be achieved by designing the modulator such that the modulator operates in a lower power state to perform calculations using smaller coefficients (which occur more frequently in the dataset) and in a higher power state to perform calculations using larger coefficients (which occur less frequently in the dataset).

[0433] Some optical amplitude modulators use relatively high power to modulate optical signals with small modulation values. For example, for coherence-insensitive optical amplitude modulators, modulation values ​​close to zero may require relatively high modulator power, such as for electro-absorption modulators, which require driving a diode-based absorber with a relatively high current for large absorbed optical powers to reduce the optical amplitude of the modulated optical signal. For coherence-sensitive optical amplitude modulators, modulation values ​​close to zero may require relatively high modulator power, such as for MZI modulators, which require driving a diode-based phase shifter with a relatively high current to provide a relative phase shift between the two MZI arms for destructive optical interference, thereby reducing the optical amplitude of the modulated signal.

[0434] Optical amplitude modulators can be configured to overcome this power relationship and achieve Figure 42 The modulator power shown in FIG, which assigns the low power modulator state to a modulation value close to zero. For example, Figure 43As shown, an MZI modulator 4300 can be configured with asymmetric arms that provide built-in passive relative phase shifts (e.g., a phase shift of approximately 180 degrees), requiring only a small active relative phase shift (and therefore low modulator power) for destructive optical interference. The MZI modulator 4300 includes an input optical splitter 4302, which splits the incoming optical signal to provide 50% of the power to the first arm and 50% to the second arm. An active phase shifter 4304 in the first arm provides a method for using a variable phase shift to vary the modulation value within a range of possible values ​​(in this embodiment, for unsigned modulation values ​​between 0 and 1). The variable phase shift is determined based on the magnitude of the applied electrical signal, requiring a certain amount of supplied electrical power (e.g., a diode-based phase shifter formed from doped semiconductor material within or near the waveguide of the first arm). A passive phase shifter 4306 in the second arm provides a relative phase shift between the first and second arms even when no power is applied to the MZI modulator 4300. For example, an optical material with a high refractive index can be configured to impart a 180-degree relative phase shift between the arms, such that the output optical combiner 4308 provides optical interference such that no significant optical power is coupled to its output. A variety of alternative configurations of active and passive phase shifters can be implemented, including (but not limited to): both active and passive phase shifters can be in one arm, with no modulator or phase shifter in the other arm; both arms can have active and passive phase shifters (in a push-pull arrangement); or both arms can have active phase shifters, with one arm having a passive phase shifter.

[0435] Alternatively, an MZI modulator configured according to the symmetrical differential configuration described herein can be used to provide near-zero coefficients using only a small active relative phase shift (and therefore low modulator power). Figure 22A An optical amplitude modulator constructed using an MZI configured according to a symmetric differential configuration is shown, where Figure 22BDetected light output is shown. Low modulation power is used to perform multiplication (using optical amplitude modulation) of modulation values ​​with low amplitude (i.e., absolute value). Specifically, the low power applied to the phase modulator 2204 corresponds to modulation of the low amplitude modulation value, thereby producing a corresponding nearly equal separation (e.g., nearly 50% / 50%) in the output of the coupler 2206 and generating a low amplitude current at the node 2216, representing the result of the multiplication. The symmetrical differential configuration also has the advantage of being able to provide signed modulation values ​​between -1 and +1 (as described in more detail below). Although this implementation uses a phase modulator in a single arm of the MZI, other implementations may have other configurations, such as a push-pull arrangement with phase modulators in both arms to provide phase shifts of opposite signs.

[0436] exist Figure 42 The example power profiles shown in show that zero modulation power is used to achieve zero modulation value, but in other embodiments there may be a residual low but non-zero modulation power at the zero modulation value. For these low coefficient weight data sets, reduced power consumption can often be achieved by using a modulator that is designed such that the modulator modulates the optical signal with the modulation value using a power that increases with respect to the absolute value of the modulation value. The exact shape of the modulation power that varies with the modulation value may vary for different implementations and is not necessarily a linear increase as the magnitude of the modulation value increases. There may be different power consuming elements in the optical amplitude modulator that contribute to the overall power consumption. In some embodiments, the modulators are designed such that they modulate the optical signal with the modulation value using a power that increases monotonically with respect to the absolute value of the modulation value.

[0437] In some cases, the modulators of the array 144 and / or the OMM 150 may have a nonlinear transfer function. For example, an MZI optical modulator may have a nonlinear relationship (e.g., a sinusoidal dependence) between the applied control voltage and its transmission. In this case, the first DAC control signal may be adjusted or compensated based on the nonlinear transfer function of the modulator so that a linear relationship between the digital input vector and the generated optical input vector can be maintained. Maintaining this linearity is generally important to ensure that the input to the OMM unit 150 is an accurate representation of the digital input vector. In some embodiments, compensation of the first DAC control signal may be performed by the controller 110 using a lookup table that maps the values ​​of the digital input vector to the values ​​to be output by the DAC unit 130 so that the resulting modulated optical signal is linearly proportional to the elements of the digital input vector. The lookup table may be generated by characterizing the nonlinear transfer function of the modulator and calculating the inverse function of the nonlinear transfer function.

[0438] In some embodiments, the nonlinearity of the modulator and the resulting nonlinearity in the generated optical input vector can be compensated by an ANN computation algorithm.

[0439] The optical input vector generated by modulator array 144 is input to OMM unit 150. The optical input vector can be N spatially separated optical signals, each having an optical power corresponding to an element of the digital input vector. For example, the optical power of the optical signals typically ranges from 1 μW to 10 mW. OMM unit 150 receives the optical input vector and performs an N×N matrix multiplication based on its internal configuration. This internal configuration is controlled by an electrical signal generated by DAC unit 130. For example, DAC unit 130 receives a second DAC control signal from controller 110, corresponding to the neural network weights to be implemented by OMM unit 150. Based on the second DAC control signal, DAC unit 130 generates a weight control signal, which is an analog signal suitable for controlling a reconfigurable component within OMM unit 150. For example, the analog signal can be a voltage or a current, depending on the type of reconfigurable component in OMM unit 150. The voltage can have an amplitude ranging from 0.1 V to 10 V, and the current can have an amplitude ranging from 100 μA to 10 mA.

[0440] The modulator array 144 can operate at a modulation rate that is different from the reconfiguration rate of the reconfigurable OMM cells 150. The optical input vectors generated by the modulator array 144 propagate through the OMM cells at a roughly proportional fraction of the speed of light (e.g., 80%, 50%, or 25% of the speed of light), depending on the optical properties of the OMM cells 150 (e.g., the effective refractive index). For a typical OMM cell 150, the propagation time of the optical input vectors is in the range of 1 to tens of picoseconds, which corresponds to processing rates in the tens to hundreds of GHz. Thus, the rate at which the optical processor 140 can perform matrix multiplication operations is partially limited by the rate at which the optical input vectors can be generated. Modulators with bandwidths in the tens of GHz are readily available, and modulators with bandwidths exceeding 100 GHz are under development. Thus, for example, the modulation rate of the modulator array 144 can be in the range of 5 GHz, 8 GHz, or tens to hundreds of GHz. To maintain operation of the modulator array 144 at such a modulation rate, the integrated circuit of the controller 110 may be configured to output control signals for the DAC unit 130 at a rate greater than or equal to, for example, 5 GHz, 8 GHz, 10 GHz, 20 GHz, 25 GHz, 50 GHz, or 100 GHz.

[0441] Depending on the type of reconfigurable component implemented by the OMM cell 150, the reconfiguration rate of the OMM cell 150 can be significantly slower than the modulation rate. For example, the reconfigurable component of the OMM cell 150 can be of the thermo-optical type, which uses a microheater to adjust the temperature of the optical waveguide of the OMM cell 150, which in turn affects the phase of the optical signal within the OMM cell 150 and causes matrix multiplication. Due to the thermal time constant associated with the heating and cooling of the structure, the reconfiguration rate can be limited to a few 100 kHz to a few 10 MHz. As such, the modulator control signals used to control the modulator array 144 and the weight control signals used to reconfigure the OMM cell 150 may have significantly different speed requirements. In addition, the electrical characteristics of the modulator array 144 can be significantly different from the electrical characteristics of the reconfigurable component of the OMM cell 150.

[0442] To accommodate the different characteristics of the modulator control signal and the weight control signal, in some embodiments, the DAC unit 130 may include a first DAC subunit 132 and a second DAC subunit 134. The first DAC subunit 132 may be specifically configured to generate the modulator control signal, and the second DAC subunit 134 may be specifically configured to generate the weight control signal. For example, the modulation rate of the modulator array 144 may be 25 GHz, and the first DAC subunit 132 may have a per-channel output update rate of 25 gigasamples per second (GSPS) and a resolution of 8 bits or higher. The reconfiguration rate of the OMM unit 150 may be 1 MHz, and the second DAC subunit 134 may have an output update rate of 1 megasample per second (MSPS) and a resolution of 10 bits. Implementing separate first and second DAC subunits 132, 134 allows the DAC subunits to be independently optimized for their respective signals, which can reduce the overall power consumption, complexity, cost, or a combination thereof of the DAC unit 130. It is worth noting that although the first DAC subunit 132 and the second DAC subunit 134 are described as subcomponents of the DAC unit 130, generally speaking, the first DAC subunit 132 and the second DAC subunit 134 can be integrated on a common chip or can be implemented as separate chips.

[0443] Based on the different characteristics of the first DAC subunit 132 and the second DAC subunit 134, in some embodiments, the storage unit 120 may include a first storage subunit and a second storage subunit. The first storage subunit may be a memory dedicated to storing input data sets and digital input vectors, and may have an operating speed sufficient to support the modulation rate. The second storage subunit may be a memory dedicated to storing neural network weights, and may have an operating speed sufficient to support the reconfiguration rate of the OMM unit 150. In some embodiments, the first storage subunit may be implemented using SRAM, and the second storage subunit may be implemented using DRAM. In some embodiments, the first storage subunit and the second storage subunit may be implemented using DRAM. In some embodiments, the first storage unit may be implemented as part of the controller 110 or as a cache of the controller 110. In some embodiments, the first and second storage subunits may be implemented as different address spaces by a single physical memory device.

[0444] The OMM unit 150 outputs a light output vector of length N, which corresponds to the result of the N×N matrix multiplication of the light input vector and the neural network weights. The OMM unit 150 is coupled to a detection unit 146, which is configured to generate N output voltages corresponding to the N light signals of the light output vector. For example, the detection unit 146 may include an array of N photodetectors (which are configured to absorb light signals and generate photocurrents) and an array of N transimpedance amplifiers (which are configured to convert the photocurrents into output voltages). The bandwidths of the photodetectors and transimpedance amplifiers can be arranged based on the modulation rate of the modulator array 144. The photodetectors can be formed of various materials based on the wavelength of the detected light output vector. Examples of materials for the photodetectors include germanium, silicon-germanium alloys, and indium gallium arsenide (InGaAs).

[0445] Detection unit 146 is coupled to ADC unit 160. ADC unit 160 is configured to convert the N output voltages into N digital light outputs, which are quantized digital representations of the output voltages. For example, ADC unit 160 may be an N-channel ADC. Controller 110 can obtain N digital light outputs from ADC unit 160, corresponding to the light output vector of light matrix multiplication unit 150. Controller 110 can form a digital output vector of length N from the N digital light outputs, corresponding to the result of an N×N matrix multiplication of the length N input digital vector.

[0446] The various electronic components of the ANN computing system 100 can be integrated in various ways. For example, the controller 110 can be a dedicated integrated circuit manufactured on a semiconductor die. Other electronic components (such as the memory unit 120, the DAC unit 130, the ADC unit 160, or a combination thereof) can be monolithically integrated on the semiconductor die on which the controller 110 is manufactured. As another example, two or more electronic components can be integrated into a system-on-chip (SoC). In an embodiment of the SoC, the controller 110, the memory unit 120, the DAC unit 130, and the ADC unit 160 can be manufactured on corresponding dies, and the corresponding dies can be integrated on a common platform (e.g., an interposer) that provides electrical connections between the integrated components. Compared to a method of separately arranging and wiring components on a printed circuit board (PCB), this SoC approach can allow faster data transmission between the electronic components of the ANN computing system 100, thereby increasing the operating speed of the ANN computing system 100. In addition, the SoC approach can allow the use of different manufacturing technologies optimized for different electronic components, which can improve the performance of different components and reduce the overall cost of a monolithic integration approach. Although the integration of the controller 110, the memory unit 120, the DAC unit 130, and the ADC unit 160 has been described, in general, a subset of the components can be integrated, while other components are implemented as separate components for various reasons (e.g., performance or cost). For example, in some embodiments, the memory unit 120 can be integrated with the controller 110 as a functional block within the controller 110.

[0447] The various optical components of the ANN computing system 100 can also be integrated in various ways. Examples of optical components of the ANN computing system 100 include a laser unit 142, a modulator array 144, an OMM unit 150, and a photodetector of the detection unit 146. These optical components can be integrated in various ways to improve performance and / or reduce cost. For example, the laser unit 142, the modulator array 144, the OMM unit 150, and the photodetector can be monolithically integrated on a common semiconductor substrate as a photonic integrated circuit (PIC). On a photonic integrated circuit formed based on a compound semiconductor material system (e.g., a III-V compound semiconductor (e.g., indium phosphide (InP))), a laser, a modulator (e.g., an electro-absorption modulator), a waveguide, and a photodetector can be monolithically integrated on a single die. This monolithic integration approach can reduce the complexity of aligning the inputs and outputs of the various separate optical components, which may require alignment accuracy ranging from sub-micron to several microns. As another example, the laser source of laser unit 142 can be fabricated on a compound semiconductor die, while the optical power splitter of laser unit 142, modulator array 144, OMM unit 150, and the photodetector of detection unit 146 can be fabricated on a silicon die. PICs fabricated on silicon wafers (also known as silicon photonics) typically have greater integration density, higher lithographic resolution, and lower cost than III-V-based PICs. This greater integration density can be beneficial in the fabrication of OMM unit 150, as OMM unit 150 typically includes tens to hundreds of optical components, such as power splitters and phase shifters. Furthermore, the higher lithographic resolution of silicon photonics can reduce manufacturing variations in OMM unit 150, thereby improving the accuracy of OMM unit 150.

[0448] The ANN computing system 100 can be implemented in various form factors. For example, the ANN computing system 100 can be implemented as a co-processor that is plugged into a host computer. Such an ANN computing system 100 can have a form factor such as a PCI Express card and communicate with the host computer via a PCIe bus. The host computer can host multiple co-processor-type ANN computing systems 100 and connect them to the computer 102 via a network. This type of embodiment can be suitable for cloud data centers, where server racks can be dedicated to processing ANN computing requests received from other computers or servers. As another embodiment, the co-processor-type ANN computing system 100 can be directly plugged into the computer 102 that issues the ANN computing request.

[0449] In some embodiments, the ANN computing system 100 can be integrated into physical systems that require real-time ANN computing capabilities. For example, systems that rely heavily on real-time artificial intelligence tasks (such as self-driving vehicles, autonomous drones, object or facial recognition security cameras, and various Internet of Things (IoT) devices) can benefit from having the ANN computing system 100 directly integrated with other subsystems of such systems. The ANN computing system 100 with direct integration can implement real-time artificial intelligence in devices with poor or no network connectivity and enhance the reliability and availability of mission-critical artificial intelligence systems.

[0450] Although the DAC unit 130 and the ADC unit 160 are shown coupled to the controller 110, in some embodiments, the DAC unit 130, the ADC unit 160, or both may alternatively or additionally be coupled to the memory unit 120. For example, direct memory access (DMA) operations of the DAC unit 130 or the ADC unit 160 may reduce the computational burden on the controller 110 and reduce the latency of reading and writing to the memory unit 120, thereby further increasing the operating speed of the ANN computing unit 100.

[0451] Figure 2A A flow chart illustrating an example of a process 200 for performing ANN calculations is shown. The steps of process 200 may be performed by controller 110. In some embodiments, the respective steps of process 200 may be run in parallel, combined, looped, or in any order.

[0452] At step 210, an artificial neural network (ANN) computation request is received, including an input dataset and a first plurality of neural network weights. The input dataset includes a first digital input vector. The first digital input vector is a subset of the input dataset. For example, it may be a subregion of an image. The ANN computation request may be generated by various entities, such as computer 102. The computer may include one or more of various types of computing devices, such as a personal computer, a server computer, a vehicle computer, and a flight computer. The ANN computation request generally refers to an electrical signal that notifies or informs the ANN computing system 100 that an ANN computation is to be performed. In some embodiments, the ANN computation request may be divided into two or more signals. For example, a first signal may query the ANN computing system 100 to check whether the system 100 is ready to receive the input dataset and the first plurality of neural network weights. In response to a positive response from the system 100, the computer may transmit a second signal including the input dataset and the first plurality of neural network weights.

[0453] In step 220, the input dataset and the first plurality of neural network weights are stored. The controller 110 may store the input dataset and the first plurality of neural network weights in the storage unit 120. Storing the input dataset and the first plurality of neural network weights in the storage unit 120 may allow for flexibility in the operation of the ANN computing system 100, which may, for example, improve the overall performance of the system. For example, by retrieving desired portions of the input dataset from the storage unit 120, the input dataset may be divided into digital input vectors of a set size and format. Different portions of the input dataset may be processed in various orders or shuffled to allow for the execution of various types of ANN computations. For example, shuffling may allow for matrix multiplication to be performed using block matrix multiplication techniques when the input and output matrices are of different sizes. As another example, storing the input dataset and the first plurality of neural network weights in the storage unit 120 may allow for the queuing of multiple ANN computation requests by the ANN computing system 100, which may allow the ANN computing system 100 to maintain operation at its full speed without periods of inactivity.

[0454] In some embodiments, the input data set may be stored in a first storage subunit and the first plurality of neural network weights may be stored in a second storage subunit.

[0455] In step 230, a first plurality of modulator control signals are generated based on the first digital input vector, and a first plurality of weight control signals are generated based on the first plurality of neural network weights. The controller 110 may transmit a first DAC control signal to the DAC unit 130 to generate the first plurality of modulator control signals. The DAC unit 130 generates the first plurality of modulator control signals based on the first DAC control signal, and the modulator array 144 generates an optical input vector representing the first digital input vector.

[0456] The first DAC control signal may include a plurality of digital values ​​to be converted by DAC unit 130 into a first plurality of modulator control signals. The plurality of digital values ​​generally corresponds to a first digital input vector and may be related by various mathematical relationships or lookup tables. For example, the plurality of digital values ​​may be linearly proportional to the values ​​of the elements of the first digital input vector. As another example, the plurality of digital values ​​may be related to the elements of the first digital input vector by a lookup table configured to maintain a linear relationship between the digital input vector and the light input vector generated by modulator array 144.

[0457] The controller 110 may transmit the second DAC control signal to the DAC unit 130 to generate a first plurality of weight control signals. The DAC unit 130 generates a first plurality of weight control signals based on the second DAC control signal and reconfigures the OMM unit 150 according to the first plurality of weight control signals to implement a matrix corresponding to the first plurality of neural network weights.

[0458] The second DAC control signal may include a plurality of digital values ​​to be converted by DAC unit 130 into a first plurality of weight control signals. The plurality of digital values ​​generally corresponds to the first plurality of neural network weights and may be related by various mathematical relationships or lookup tables. For example, the plurality of digital values ​​may be linearly proportional to the first plurality of neural network weights. As another example, the plurality of digital values ​​may be calculated by performing various mathematical operations on the first plurality of neural network weights to generate weight control signals, which may configure OMM unit 150 to perform matrix multiplication corresponding to the first plurality of neural network weights.

[0459] In some embodiments, the first plurality of neural network weights representing matrix M can be decomposed into M=USV* via a singular value decomposition (SVD) method, where U is an M×M unitary matrix, S is an M×N diagonal matrix with non-negative real numbers on the diagonal, and V* is the complex conjugate of the N×N unitary matrix V. In this case, the first plurality of weight control signals may include a first plurality of OMM unit control signals corresponding to matrix V and a second plurality of OMM unit control signals corresponding to matrix S. Furthermore, OMM unit 150 may be configured to have a first OMM sub-unit configured to implement matrix V, a second OMM sub-unit configured to implement matrix S, and a third OMM sub-unit configured to implement matrix U, such that OMM unit 150 as a whole implements matrix M. SVD methods are further described in U.S. Patent Publication No. US2017 / 0351293A1, entitled “APPARATUS AND METHODS FOR OPTICAL NEURAL NETWORK,” which is incorporated herein by reference in its entirety.

[0460] In step 240, a first plurality of digital optical outputs corresponding to the optical output vectors of the optical matrix multiplication unit is obtained. The optical input vector generated by the modulator array 144 is processed by the OMM unit 150 and converted into an optical output vector. The optical output vector is detected by the detection unit 146 and converted into an electrical signal, which can be converted into a digital value by the ADC unit 160. The controller 110 can, for example, transmit a conversion request to the ADC unit 160 to initiate conversion of the voltage output by the detection unit 146 into a digital optical output. Once the conversion is complete, the ADC unit 160 can transmit the conversion result to the controller 110. Alternatively, the controller 110 can obtain the conversion result from the ADC unit 160. The controller 110 can form a digital output vector from the digital optical output, which corresponds to the result of the matrix multiplication of the input digital vector. For example, the digital optical output can be organized or concatenated to have a vector format.

[0461] In some embodiments, ADC unit 160 may be set or controlled to perform ADC conversion based on a DAC control signal issued by controller 110 to DAC unit 130. For example, ADC conversion may be set to start at a preset time after DAC unit 130 generates a modulation control signal. Such control of ADC conversion may simplify the operation of controller 110 and reduce the number of necessary control operations.

[0462] In step 250, a nonlinear transformation is performed on the first digital output vector to produce a first transformed digital output vector. The nodes or artificial neurons of the ANN operate by first performing a weighted sum of the signals received from the nodes of the previous layer, and then performing a nonlinear transformation ("activation") of the weighted sum to produce an output. Various types of ANNs can implement various types of differentiable nonlinear transformations. Examples of nonlinear transformation functions include a rectified linear unit (RELU) function, a sigmoid function, a hyperbolic tangent function, an X^2 function, and an |X| function. This nonlinear transformation is performed on the first digital output by the controller 110 to produce a first transformed digital output vector. In some embodiments, the nonlinear transformation may be performed by a dedicated digital integrated circuit within the controller 110. For example, the controller 110 may include one or more modules or circuit blocks that are particularly suitable for accelerating the calculation of one or more types of nonlinear transformations.

[0463] In step 260 , the first transformed digital output vector is stored. The controller 110 may store the first transformed digital output vector in the storage unit 120 . In the event that the input data set is divided into multiple digital input vectors, the first transformed digital output vector corresponds to an ANN calculation result for a portion of the input data set, such as the first digital input vector. Storing the first transformed digital output vector allows the ANN computing system 100 to perform and store additional calculations on other digital input vectors of the input data set for later aggregation into a single ANN output.

[0464] At step 270, an artificial neural network output generated based on the first transformed digital output vector is output. Controller 110 generates an ANN output that is the result of processing the input dataset using the ANN defined by the first plurality of neural network weights. Where the input dataset is divided into multiple digital input vectors, the generated ANN output is an aggregated output comprising the first transformed digital output, but may further include additional transformed digital outputs corresponding to other portions of the input dataset. Once the ANN output is generated, it is transmitted to the computer that initiated the ANN computation request (e.g., computer 102).

[0465] Various performance metrics can be defined for the ANN computing system 100 implementing process 200. Defining performance metrics can allow the performance of the ANN computing system 100 implementing the optical processor 140 to be compared with the performance of other systems for alternative ANN computing that implement an electronic matrix multiplication unit. In one aspect, the rate at which the ANN computing can be performed can be indicated in part by a first cycle period, which is defined as the time elapsed between step 220 of storing the input data set and the first plurality of neural network weights in the memory unit and step 260 of storing the first transformed digital output vector in the memory unit. Thus, the first cycle period includes the time taken to convert the electrical signal into an optical signal (e.g., step 230), perform the matrix multiplication in the optical domain, and convert the result back to the electrical domain (e.g., step 240). Both steps 220 and 260 involve storing data in the memory unit 120, which is a shared step between the ANN computing system 100 and a conventional ANN computing system without the optical processor 140. As such, measuring the first cycle period of the memory-to-memory transaction time may allow for a realistic or fair comparison of ANN computation throughput between the ANN computing system 100 and an ANN computing system without the optical processor 140 (e.g., a system implementing an electrical matrix multiplication unit).

[0466] Due to the rate at which the modulator array 144 can generate optical input vectors (e.g., at 25 GHz) and the processing rate of the OMM unit 150 (e.g., >100 GHz), the first cycle period of the ANN computing system 100 for performing a single ANN computation of a single digital input vector can be close to the inverse of the speed of the modulator array 144 (e.g., 40 ps). After accounting for delays associated with signal generation by the DAC unit 130 and ADC conversion by the ADC unit 160, the first cycle period can be, for example, less than or equal to 100 ps, ​​less than or equal to 200 ps, ​​less than or equal to 500 ps, ​​less than or equal to 1 ns, less than or equal to 2 ns, less than or equal to 5 ns, or less than or equal to 10 ns.

[0467] In comparison, the runtime for multiplication of an M×1 vector and an M×M matrix in an electronic matrix multiplication unit is typically proportional to M^2-1 processor clock cycles. For M=32, such a multiplication would take approximately 1024 cycles, which results in a runtime exceeding 300ns at a 3 GHz clock speed, which is several orders of magnitude slower than the first cycle period of the ANN computing system 100.

[0468] In some embodiments, process 200 further includes the step of generating a second plurality of modulator control signals based on the first transformed digital output vector. In some types of ANN calculations, a single digital input vector can be repeatedly propagated through or processed by the same ANN. An ANN that implements multi-pass processing can be referred to as a recurrent neural network (RNN). An RNN is a neural network in which the output of the network is recycled back to the input of the neural network during the (k)th pass through the neural network and is used as input during the (k+1)th pass. RNNs can have various applications in pattern recognition tasks, such as speech or handwriting recognition. Once the second plurality of modulator control signals are generated, process 200 can proceed from step 240 to step 260 to complete the second pass of the ANN for the first digital input vector. Generally speaking, depending on the characteristics of the RNN received in the ANN calculation request, the converted digital output can be repeatedly recycled into a digital input vector for a predetermined number of cycles.

[0469] In some embodiments, process 200 further includes the step of generating a second plurality of weight control signals based on a second plurality of neural network weights. In some cases, the artificial neural network computation request further includes a second plurality of neural network weights. Generally speaking, in addition to an input layer and an output layer, an ANN also has one or more hidden layers. For an ANN having two hidden layers, the second plurality of neural network weights may correspond to the connectivity between a first layer of the ANN and a second layer of the ANN. To process a first digital input vector through both hidden layers of the ANN, the first digital input vector may first be processed according to process 200 until step 260, where the result of processing the first digital input vector through the first hidden layer of the ANN is stored in memory unit 120. Controller 110 then reconfigures OMM unit 150 to perform matrix multiplication corresponding to the second plurality of neural network weights associated with the second hidden layer of the ANN. Once OMM unit 150 is reconfigured, process 200 may generate a plurality of modulator control signals based on the first transformed digital output vector, which generate an updated optical input vector corresponding to the output of the first hidden layer. The updated optical input vector is then processed by the reconfigured OMM unit 150, which corresponds to the second hidden layer of the ANN. Generally speaking, the steps described may be repeated until the digital input vector has been processed through all hidden layers of the ANN.

[0470] As described above, in some embodiments of the OMM unit 150, the reconfiguration rate of the OMM unit 150 may be significantly slower than the modulation rate of the modulator array 144. In this case, the throughput of the ANN computing system 100 may be adversely affected by the amount of time spent reconfiguring the OMM unit 150 during periods when ANN computations cannot be performed. To mitigate the impact of the relatively slow reconfiguration time of the OMM unit 150, batch processing techniques may be utilized, in which two or more digital input vectors are propagated through the OMM unit 150 without configuration changes, to amortize the reconfiguration time over a larger number of digital input vectors.

[0471] Figure 2B Shows the instructions Figure 2A 290 illustrates aspects of the processing 200 of the ANN. For an ANN with two hidden layers, instead of processing the first digital input vector through the first hidden layer, reconfiguring the OMM unit 150 for the second hidden layer, processing the first digital input vector through the reconfigured OMM unit 150, and repeating the same operation for the remaining digital input vectors, all digital input vectors of the input data set can first be processed by the OMM unit 150 configured for the first hidden layer (Configuration #1), as shown in the upper portion of the diagram 290. Once all digital input vectors have been processed by the OMM unit 150 with Configuration #1, the OMM unit 150 is reconfigured to Configuration #2, which corresponds to the second hidden layer of the ANN. This reconfiguration can be significantly slower than the rate at which the OMM unit 150 can process input vectors. Once the OMM unit 150 is reconfigured for the second hidden layer, the output vectors from the previous hidden layer can be processed in batches by the OMM unit 150. For large input data sets with tens or hundreds of thousands of numeric input vectors, the impact of reconfiguration time can be reduced by roughly the same factor, which can significantly reduce the portion of time the ANN computing system 100 spends in reconfiguration.

[0472] To implement batch processing, in some embodiments, process 200 further includes the steps of generating, via a DAC unit, a second plurality of modulator control signals based on a second digital input vector; obtaining, from an ADC unit, a second plurality of digital optical outputs corresponding to the optical output vector of the optical matrix multiplication unit, the second plurality of digital optical outputs forming a second digital output vector; performing a nonlinear transformation on the second digital output vector to generate a second transformed digital output vector; and storing the second transformed digital output vector in a memory unit. For example, generating the second plurality of modulator control signals may occur after step 260. Furthermore, in this case, the ANN output of step 270 is now based on the first transformed digital output vector and the second transformed digital output vector. The obtaining, executing, and storing steps are similar to steps 240 through 260.

[0473] Batch processing technology is one of the various technologies used to improve the throughput of the ANN computing system 100. Another technology for improving the throughput of the ANN computing system 100 is to process multiple digital input vectors in parallel by utilizing wavelength division multiplexing (WDM). WDM is a technology that simultaneously propagates multiple optical signals of different wavelengths through a common propagation channel (e.g., a waveguide of the OMM unit 150). Unlike electrical signals, optical signals of different wavelengths can propagate through a common channel without affecting other optical signals of different wavelengths on the same channel. In addition, optical signals can be added (multiplexed) or dropped (demultiplexed) from a common propagation channel using well-known structures such as optical multiplexers and demultiplexers.

[0474] In the context of the ANN computing system 100, multiple optical input vectors of different wavelengths can be independently generated, simultaneously propagated through the OMM unit 150, and independently detected to enhance the throughput of the ANN computing system 100. Figure 1F, a diagram illustrating an example of a wavelength division multiplexing (WDM) artificial neural network (ANN) computing system 104. Unless otherwise described, WDM ANN computing system 104 is similar to ANN computing system 100. To implement WDM technology, in some embodiments of ANN computing system 104, laser unit 142 is configured to generate multiple wavelengths, such as λ1, λ2, and λ3. The multiple wavelengths are preferably separated by a sufficiently large wavelength spacing to allow for easy multiplexing and demultiplexing onto a common propagation channel. For example, a wavelength spacing greater than 0.5 nm, 1.0 nm, 2.0 nm, 3.0 nm, or 5.0 nm can allow for simple multiplexing and demultiplexing. On the other hand, the range between the shortest and longest wavelengths of the multiple wavelengths ("WDM bandwidth") is preferably sufficiently small that the characteristics or performance of OMM unit 150 remain substantially the same across the multiple wavelengths. Optical components are typically dispersive, meaning that their optical properties vary with wavelength. For example, the power splitting ratio of the MZI can vary with wavelength. However, by designing the OMM unit 150 to have a sufficiently large operating wavelength window, and by restricting the wavelengths to be within the operating wavelength window, the optical output vector output by the OMM unit 150 at each wavelength can be a sufficiently accurate result of the matrix multiplication performed by the OMM unit 150. The operating wavelength window can be, for example, 1 nm, 2 nm, 3 nm, 4 nm, 5 nm, 10 nm, or 20 nm.

[0475] Figure 39A A diagram shows an example of a Mach-Zehnder modulator 3900 that can be used to modulate the amplitude of an optical signal. The Mach-Zehnder modulator 3900 includes two 1×2 port multimode interference couplers (MMI_1x2) 3902a and 3902b, two balanced arms 3904a and 3904b, and a phase shifter 3906 in one arm (or one phase shifter in each arm). When a voltage is applied to the phase shifter in one arm via a signal line 3908, a phase difference will exist between the two arms 3904a and 3904b, which will be converted into amplitude modulation. The 1×2 port multimode interference couplers 3902a and 3902b and the phase shifter 3906 are configured as broadband photonic components, and the optical path lengths of the two arms 3904a and 3904b are configured to be equal. This enables the Mach-Zehnder modulator 3900 to operate over a wide wavelength range.

[0476] Figure 39B Graph numeral 3910 shows a graph showing the effect of using a 1530 nm, 1550 nm, and 1570 nm wavelength on the Figure 39AGraph 3910 of the graph shows that the Mach-Zehnder modulator 3900 has similar intensity-voltage characteristics for different wavelengths in the range of 1530 nm to 1570 nm.

[0477] Return to reference Figure 1F , the modulator array 144 of the WDM ANN computing system 104 includes banks of optical modulators that are configured to generate a plurality of optical input vectors, each of the optical modulator banks corresponding to one of a plurality of wavelengths and generating a corresponding optical input vector having a corresponding wavelength. For example, for a system having an optical input vector of length 32 and 3 wavelengths (e.g., λ1, λ2, and λ3), the modulator array 144 may have 3 groups of 32 modulators each. In addition, the modulator array 144 also includes an optical multiplexer that is configured to combine the plurality of optical input vectors into a combined optical input vector comprising a plurality of wavelengths. For example, the optical multiplexer may combine the outputs of three modulator groups of three different wavelengths into a single propagation channel (e.g., a waveguide) for each element of the optical input vector. Thus, returning to the example above, the combined optical input vector will have 32 optical signals, each comprising 3 wavelengths.

[0478] In addition, the detection unit 146 of the WDM ANN calculation system 104 is further configured to demultiplex the multiple wavelengths and generate a plurality of demultiplexed output voltages. For example, the detection unit 146 may include a demultiplexer configured to demultiplex the three wavelengths in each of the 32 signals included in the multi-wavelength optical output vector and route the three single-wavelength optical output vectors to three sets of photodetectors coupled to three sets of transimpedance amplifiers.

[0479] Furthermore, the ADC unit 160 of the WDM ANN calculation system 104 includes an ADC group configured to convert the multiple demultiplexed output voltages of the detection unit 146. Each of the ADC groups corresponds to one of the multiple wavelengths and generates a corresponding digital demultiplexed optical output. For example, the ADC group can be coupled to a transimpedance amplifier group of the detection unit 146.

[0480] The controller 110 may implement a method similar to that of the process 200, but expanded to support multi-wavelength operation. For example, the method may include the steps of obtaining a plurality of digital demultiplexed optical outputs from the ADC unit 160, the plurality of digital demultiplexed optical outputs forming a plurality of first digital output vectors, wherein each of the plurality of first digital output vectors corresponds to one of the plurality of wavelengths; performing a nonlinear transformation on each of the plurality of first digital output vectors to generate a plurality of transformed first digital output vectors; and storing the plurality of transformed first digital output vectors in a storage unit.

[0481] In some cases, the ANN can be specifically designed and the digital input vector can be specifically formed so that the multi-wavelength light output vector can be detected without demultiplexing. In this case, the detection unit 146 can be a wavelength-insensitive detection unit that does not demultiplex the multiple wavelengths of the multi-wavelength light output vector. In this way, each photodetector of the detection unit 146 effectively adds the multiple wavelengths of the optical signal into a single photocurrent, and each voltage output by the detection unit 146 corresponds to the element-by-element sum of the matrix multiplication results of the multiple digital input vectors.

[0482] To date, nonlinear transformations of weighted sums performed as part of ANN computations have been performed in the digital domain by the controller 110. In some cases, the nonlinear transformations may be computationally intensive or power-consuming, significantly increasing the complexity of the controller 110 or limiting the performance of the ANN computing system 100 in terms of throughput or power efficiency. As such, in some embodiments of the ANN computing system, the nonlinear transformations may be performed in the analog domain by analog electronics.

[0483] Figure 3A FIG2 is a diagram illustrating an example of an ANN computing system 300. The ANN computing system 300 is similar to the ANN computing system 100, except that an analog nonlinear unit 310 is added. The analog nonlinear unit 310 is arranged between the detection unit 146 and the ADC unit 160. The analog nonlinear unit 310 is configured to receive the output voltage from the detection unit 146, apply a nonlinear transfer function, and output the converted output voltage to the ADC unit 160.

[0484] When the ADC unit 160 receives a voltage that has been nonlinearly transformed by the analog nonlinear unit 310, the controller 110 can obtain a converted digital output voltage corresponding to the converted output voltage from the ADC unit 160. Because the digital output voltage obtained from the ADC unit 160 has been nonlinearly transformed ("activated"), the nonlinear transformation step of the controller 110 can be omitted, thereby reducing the computational burden of the controller 110. Subsequently, the first converted voltage obtained directly from the ADC unit 160 can be stored in the storage unit 120 as a first converted digital output vector.

[0485] The analog nonlinear unit 310 can be implemented in various ways. For example, a high-gain amplifier in a feedback configuration, a comparator with an adjustable reference voltage, a nonlinear IV characteristic of a diode, a breakdown behavior of a diode, a nonlinear CV characteristic of a variable capacitor, or a nonlinear IV characteristic of a variable resistor can be used to implement the analog nonlinear unit 310.

[0486] The use of analog nonlinear units 310 can improve the performance of the ANN computing system 300, such as throughput or power efficiency, by reducing the number of steps performed in the digital domain. Moving the nonlinear transformation step out of the digital domain can allow for additional flexibility and improvements in the operation of the ANN computing system. For example, in a recurrent neural network, the output of the OMM unit 150 is activated and recycled back to the input of the OMM unit 150. The activation step is performed by the controller 110 in the ANN computing system 100, which requires digitizing the output voltage of the detection unit 146 each time it passes through the OMM unit 150. However, because the activation step is now performed before digitization by the ADC unit 160, the number of ADC conversions required in performing the recurrent neural network calculation can be reduced.

[0487] In some embodiments, the analog nonlinear unit 310 can be integrated into the ADC unit 160 as a nonlinear ADC unit. For example, the nonlinear ADC unit can be a linear ADC unit with a nonlinear lookup table that maps the linear digital output of the linear ADC unit to a desired nonlinearly transformed digital output.

[0488] Figure 3B A diagram illustrating an example of an example of an ANN computing system 302. The ANN computing system 302 is similar to Figure 3AThe system 300 is different in that it further includes an analog storage unit 320. The analog storage unit 320 is coupled to the DAC unit 130 (e.g., through the first DAC subunit 132), the modulator array 144, and the analog nonlinear unit 310. The analog storage unit 320 includes a multiplexer having a first input coupled to the DAC unit 130 and a second input coupled to the analog nonlinear unit 310. This allows the analog storage unit 320 to receive a signal from either the DAC unit 130 or the analog nonlinear unit 310. The analog storage unit 320 is configured to store an analog voltage and output the stored analog voltage.

[0489] The analog storage unit 320 can be implemented in various ways. For example, a capacitor array can be used as an analog voltage storage component. The capacitor of the analog storage unit 320 can be charged to the input voltage by a charging circuit. The storage of the input voltage can be controlled based on a control signal received from the controller 110. The capacitor can be electrically isolated from the surrounding environment to reduce charge leakage that causes undesirable capacitor discharge. Additionally (or alternatively), a feedback amplifier can be used to maintain the voltage stored on the capacitor. The stored voltage of the capacitor can be read out by a buffer amplifier, which allows the charge stored by the capacitor to be maintained while outputting the stored voltage. These aspects of the analog storage unit 320 can be similar to the operation of a sample and hold circuit. The buffer amplifier can implement the function of a modulator driver for driving the modulator array 144.

[0490] The operation of the ANN computing system 302 will now be described. A first plurality of modulator control signals output by the DAC unit 130 (e.g., by the first DAC subunit 132) are first input to the modulator array 144 via the analog storage unit 320. At this step, the analog storage unit 320 may simply pass or buffer the first plurality of modulator control signals. Based on the first plurality of modulator control signals, the modulator array 144 generates an optical input vector, which propagates through the OMM unit 150 and is detected by the detection unit 146. The output voltage of the detection unit 146 is nonlinearly transformed by the analog nonlinear unit 310. At this point, instead of being digitized by the ADC unit 160, the output voltage of the detection unit 146 is stored by the analog storage unit 320 and subsequently output to the modulator array 144 to be converted into the next optical input vector to be propagated through the OMM unit 150. This recurrent processing may be performed for a predetermined amount of time or a predetermined number of cycles under the control of the controller 110. Once the recursive process is completed for a given digital input vector, the converted output voltage of the analog non-linear unit 310 is converted by the ADC unit 160 .

[0491] The use of analog memory unit 320 can significantly reduce the number of ADC conversions during recurrent neural network computations, for example, down to a single ADC conversion for each RNN computation of a given digital input vector. Each ADC conversion takes time and consumes a certain amount of energy. As a result, the throughput of RNN computations of ANN computing system 302 can be higher than the throughput of RNN computations of ANN computing system 100.

[0492] The execution of the recurrent neural network computation can be controlled by controlling the analog storage unit 320. For example, the controller can control the analog storage unit 320 to store a voltage at a specific time and output the stored voltage at a different time. In this way, the signal loop from the analog storage unit 320 to the modulator array 144 through the analog nonlinear unit 310 and back to the analog storage unit 320 can be controlled by the controller 110 controlling the storage and readout of the analog storage unit 320.

[0493] Thus, in some embodiments, the controller 110 of the ANN computing system 302 may perform the following steps: based on generating a first plurality of modulator control signals and a first plurality of weight control signals, storing a plurality of conversion output voltages of the analog nonlinear unit through an analog storage unit; outputting the stored conversion output voltages through the analog storage unit; obtaining a second plurality of conversion digital output voltages from the ADC unit, the second plurality of conversion digital output voltages forming a second transformed digital output vector; and storing the second transformed digital output vector in the storage unit.

[0494] The input data set processed by the ANN computing system typically includes data with a resolution greater than 1 bit. For example, a typical pixel of a grayscale digital image may have a resolution of 8 bits, i.e., 256 different levels. One way to represent and process this data in the optical domain is to encode the 256 different intensity levels of the pixel as 256 different power levels of the optical signal input to the OMM unit 150. The optical signal is analog in nature and is therefore susceptible to noise and detection errors. Return to reference Figure 1A In order to maintain 8-bit resolution of the digital input vector throughout the ANN computing system 100 and generate a true 8-bit digital light output at the output of the ADC unit 160, each part of the signal chain can preferably be designed to reproduce and maintain 8-bit resolution.

[0495] For example, the DAC unit 130 may preferably be designed to support conversion of an 8-bit digital input vector into a modulator control signal having at least 8 bits of resolution, so that the modulator array 144 can generate an 8-bit optical input vector that faithfully represents the digital input vector. Generally speaking, the modulator control signal may need to have additional resolution exceeding 8 bits of the digital input vector to compensate for the nonlinear response of the modulator array 144. Furthermore, the internal configuration of the OMM unit 150 may preferably be sufficiently stable to ensure that the value of the optical output vector is not corrupted by any fluctuations in the configuration of the OMM unit 150. For example, the temperature of the OMM unit 150 may need to be stable within 5 degrees, 2 degrees, 1 degree, or 0.1 degrees. Furthermore, the detection unit 146 may preferably have sufficiently low noise to not corrupt the 8-bit resolution of the optical output vector, and the ADC unit 160 may preferably be designed to support digitization of an analog voltage having at least 8 bits of resolution.

[0496] The power consumption and design complexity of various electronic components generally increase with bit resolution, operating speed, and bandwidth. For example, as a first-order approximation, the power consumption of the ADC unit 160 can be scaled linearly with the sampling rate, and the scaling factor is 2^N, where N is the bit resolution of the conversion result. In addition, the design considerations of the DAC unit 130 and the ADC unit 160 generally result in a trade-off between sampling rate and bit resolution. As such, in some cases, it may be desirable for the ANN computing system to operate internally at a lower bit resolution than the resolution of the input data set while maintaining the resolution of the ANN computing output.

[0497] Reference Figure 4A , a diagram illustrating an example of an artificial neural network (ANN) computing system 400 having a 1-bit internal resolution is shown. The ANN computing system 400 is similar to the ANN computing system 100, except that the DAC unit 130 is now replaced by a driver unit 430, and the ADC unit 160 is now replaced by a comparator unit 460.

[0498] The driver unit 430 is configured to generate a 1-bit modulator control signal and a multi-bit weight control signal. For example, the driver circuit of the driver unit 430 can directly receive a binary digital output from the controller 110 and condition the binary signal into a two-level voltage or current output suitable for driving the modulator array 144.

[0499] The comparator unit 460 is configured to convert the output voltage of the detection unit 146 into a digital 1-bit optical output. For example, the comparison circuit of the comparator unit 460 can receive a voltage from the detection unit 146, compare the voltage with a preset threshold voltage, and output a digital 0 or 1 when the received voltage is less than or greater than the preset threshold voltage, respectively.

[0500] Reference Figure 4B , which shows a mathematical representation of the operation of the ANN computing system 400. Figure 4B The operation of the ANN computing system 400 is described. For a given ANN computing to be performed by the ANN computing system 400, there is a corresponding numeric input vector V and neural network weight matrix U. In this embodiment, the input vector V is a vector of length 4 with elements V0 to V3, and the matrix U is a vector with weights U 00 To U 33 4×4 matrix. Each element of vector V has 4 bits of resolution. Each 4-bit vector element has bits 0 (bit0) to 3 (bit3) corresponding to positions 2^0 to 2^3, respectively. Thus, the decimal (base 10) value of the 4-bit vector element is calculated by the sum of 2^0*bit0+2^1*bit1+2^2*bit2+2^3*bit3. Therefore, as shown in the figure, the input vector V can be similarly decomposed into V by the controller 110. bit0 To V bit3 .

[0501] A particular ANN calculation can then be performed by performing a series of matrix multiplications of 1-bit vectors and then summing the corresponding matrix multiplication results. For example, the decomposed input vector V can be generated by generating a sequence of four 1-bit modulator control signals corresponding to four 1-bit input vectors through the driver unit 430. bit0 to V bit3 Each of them is multiplied by the matrix U. This in turn produces a sequence of four 1-bit optical input vectors, which are propagated through the OMM unit 150, which is configured to perform matrix multiplication of the matrix U through the driver unit 430. The controller 110 can then obtain a sequence of four digital 1-bit optical outputs corresponding to the sequence of four 1-bit modulator control signals from the comparator unit 460.

[0502] In the case of decomposing a 4-bit vector into four 1-bit vectors, each vector should be processed by the ANN computing system 400 at a processing speed four times faster than other ANN computing systems (e.g., system 100) can process a single 4-bit vector to maintain the same effective ANN computing throughput. This increased internal processing speed can be viewed as time-division multiplexing the four 1-bit vectors into a single timeslot for processing the 4-bit vector. The required increase in processing speed can be achieved at least in part by increasing the operating speed of the driver unit 430 and the comparator unit 460 relative to the DAC unit 130 and the ADC unit 160, because a reduction in the resolution of the signal conversion process generally results in an increase in the achievable signal conversion rate.

[0503] Although the signal conversion rate in 1-bit operation is increased by a factor of four, the power consumption generated can be significantly reduced relative to 4-bit operation. As described above, the power consumption of the signal conversion process generally scales exponentially with the bit resolution and linearly with the conversion rate. Thus, a 16-fold reduction in power per conversion may be due to a 4-fold reduction in bit resolution, followed by a 4-fold increase in power due to the increase in conversion rate. In summary, the ANN computing system 400 can achieve a 4-fold reduction in operating power over, for example, the ANN computing system 100, while maintaining the same effective ANN computing throughput.

[0504] Next, the controller 110 may construct a 4-bit digital output vector from the four digital 1-bit optical outputs by multiplying each digital 1-bit optical output by a corresponding weight 2^0 to 2^3. Once the 4-bit digital output vector is constructed, an ANN calculation may be performed on the constructed 4-bit digital output vector by performing a nonlinear transformation to generate a transformed 4-bit digital output vector. The transformed 4-bit digital output vector is then stored in the storage unit 120.

[0505] Alternatively (or additionally), in some embodiments, a nonlinear transformation can be performed on each of the four digital 1-bit optical outputs. For example, a step-function nonlinear function can be used for the nonlinear transformation. A converted 4-bit digital output vector can then be constructed from the nonlinearly transformed digital 1-bit optical output.

[0506] Although a single ANN computing system 400 has been shown and described, in general, Figure 1AThe ANN computing system 100 can be designed to implement functions similar to those of the ANN computing system 400. For example, the DAC unit 130 can include a 1-bit DAC subunit configured to generate a 1-bit modulator control signal, and the ADC unit 160 can be designed to have a 1-bit resolution. Such a 1-bit ADC can be similar to or effectively equivalent to a comparator.

[0507] Furthermore, while the operation of the ANN computing system has been described with an internal resolution of 1 bit, generally speaking, the internal resolution of the ANN computing system can be reduced to an intermediate level below the N-bit resolution of the input data set. For example, the internal resolution can be reduced to 2^Y bits, where Y is an integer greater than or equal to 0.

[0508] The embodiments of the subject matter and functional operations described in the present disclosure can be implemented in digital electronic circuits, or in computer software, firmware or hardware, including the structures disclosed in the present disclosure and their structural equivalents, or one or more combinations thereof. The embodiments of the subject matter described in the present disclosure can be implemented using one or more computer program instruction modules encoded on a computer-readable medium to be executed or control the operation of the data processing device by a data processing device. The computer-readable medium can be a manufactured product (such as a hard drive in a computer system or an optical disc sold through a retail channel) or an embedded system. The computer-readable medium can be separately obtained and then encoded using one or more modules of the computer program instructions, such as transmitting one or more modules of the computer program instructions via a wired or wireless network. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a storage device or a combination of one or more of them.

[0509] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that hold one or more modules, subroutines, or portions of code). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0510] The processes and logic flows described in this disclosure may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and apparatus may be implemented as, special purpose logic circuitry, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC).

[0511] Although the present disclosure includes many implementation details, these should not be interpreted as limitations on the scope of the present disclosure or the claims, but rather as descriptions of specific features of specific embodiments of the present disclosure. Certain features described in the present disclosure in the context of the respective embodiments may also be implemented in combination in a single embodiment. Relatively speaking, the various features described in the context of a single embodiment may also be implemented in multiple embodiments or in any suitable sub-combination, respectively. In addition, although the above features are described as acting on certain combinations and even initially claimed as such, in some cases, one or more features may be removed from the claimed combination, and the claimed combination may point to a sub-combination or a change in the sub-combination.

[0512] Similarly, although operations are described in a particular order in the accompanying drawings, this should not be understood as requiring that the operations be performed in the particular order shown or in sequence, or that all of the operations shown be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0513] Thus, certain embodiments of the present disclosure have been described. Other embodiments are within the scope of the following claims. Additionally, the actions recited in the claims can be performed in a different order and still achieve the desired results. For example, Figure 1A The optical matrix multiplication unit 150 includes an optical interferometer 154, which includes a plurality of interconnected Mach-Zehnder interferometers. In some embodiments, the optical interferometer can be implemented using a one-dimensional, two-dimensional, or three-dimensional passive diffractive optical element that consumes almost no power. Compared to an optical interferometer including a Mach-Zehnder interferometer, if the number of inputs / outputs remains unchanged, the optical interferometer using the passive diffractive optical element can have a smaller size or can process a larger number of inputs / outputs for the same chip size. Compared to the Mach-Zehnder interferometer, the passive diffractive optical element can be manufactured at a lower cost.

[0514] Reference Figure 5 In some embodiments, the artificial neural network computing system 500 includes a controller 110, a storage unit 120, a DAC unit 506, an optical processor 504, and an ADC unit 160. The storage unit 120 and the ADC unit 160 are similar to Figure 1A The optical processor 504 is configured to perform matrix calculations using optical components. In the system 500, the weights of the two-dimensional optical matrix multiplication unit 502 are fixed. The DAC unit 506 is similar to Figure 1A The first DAC subunit 132 of the system 100 .

[0515] In an example operation of the ANN computing system 500, the computer 102 may issue an artificial neural network computing request to the ANN computing system 500. The ANN computing request may include an input data set to be processed by the provided ANN. The controller 110 receives the ANN computing request and stores the input data set in the storage unit 120.

[0516] In some embodiments, a hybrid approach is used where a portion of the optical matrix multiplication unit 150 includes a Mach-Zehnder interferometer and another portion of the optical matrix multiplication unit 150 includes a passive diffraction component.

[0517] The internal operation of the ANN computing system 500 will now be described. The optical processor 504 includes a laser unit 142, a modulator array 144, a detection unit 146, and an optical matrix multiplication (OMM) unit 502. The laser unit 142, the modulator array 144, and the detection unit 146 are similar to Figure 1A In this example, the two-dimensional OMM unit 502 includes a two-dimensional diffractive optical component and can be implemented as a passive integrated silicon photonic chip. The two-dimensional optical matrix multiplication unit 502 can be configured to implement a diffractive neural network and can perform matrix multiplication with almost zero power consumption.

[0518] The optical processor 504 operates by encoding a digital input vector of length N onto an optical input vector of length N and propagating the optical input vector through the two-dimensional OMM unit 502. The two-dimensional OMM unit 502 receives the optical input vector of length N and performs an N×N matrix multiplication on the received optical input vector in the optical domain. The N×N matrix multiplication performed by the two-dimensional OMM unit 502 is determined by the internal configuration of the two-dimensional OMM unit 502. The internal configuration of the two-dimensional OMM unit 502 includes the size, position, and geometry of the diffractive optical components, as well as the doping of impurities (if any).

[0519] The two-dimensional OMM unit 502 can be implemented in various ways. Figure 6 A diagram illustrates an example of a two-dimensional optical multimeter (OMM) cell 502 using a two-dimensional array of diffractive components. The two-dimensional OMM cell 502 may include an array of input waveguides 602 to receive an optical input vector, a two-dimensional optical interferometer cell 600 in optical communication with the array of input waveguides 602, and an array of output waveguides 604 in optical communication with the optical interferometer cell 600. The optical interferometer cell 600 includes a plurality of diffractive optical components and performs a transformation (e.g., a linear transformation) of the optical input vector into a second optical signal array. The array of output waveguides 604 guides the second optical signal array output by the optical interferometer cell 600. At least one input waveguide in the array of input waveguides 602 is in optical communication with each output waveguide in the array of output waveguides 604 via the optical interferometer cell 600. For example, for an optical input vector of length N, the two-dimensional OMM cell 502 may include N input waveguides 602 and N output waveguides 604.

[0520] In some embodiments, the optical interference unit 600 includes a substrate having diffraction components arranged in two dimensions (e.g., in a 2D array). For example, a plurality of circular holes can be drilled or etched in the substrate. The size of these holes can be comparable in magnitude to the size of the wavelength of the input light, so that the light is diffracted by the holes (or the structure defining the holes). For example, the size of the holes can be in the range of 100 nm to 2 μm. The holes can have the same or different sizes. The holes can also have other cross-sectional shapes, such as triangular, square, rectangular, hexagonal, or irregular shapes. The substrate can be made of a material that is transparent or translucent to the input light, for example, having a transmission rate of 1% to 99% relative to the input light. For example, the substrate can be made of silicon, silicon oxide, silicon nitride, quartz, a crystal (e.g., lithium niobate (LiNbO3)), a III-V material (e.g., gallium arsenide or indium phosphide), an erbium modified semiconductor, or a polymer.

[0521] In some embodiments, a holographic method can be used to form a two-dimensional diffractive optical element in a substrate. The substrate can be made of glass, crystal, or photorefractive material.

[0522] When designing a two-dimensional OMM unit 502, the size and position of the diffractive elements are considered in two dimensions (e.g., the X and Y directions), without considering the relative position of the diffractive elements in the third dimension (e.g., the Z direction). Each diffractive element can be a three-dimensional structure formed in the substrate, such as a hole of a certain depth, a column, or a stripe.

[0523] exist Figure 6 In the figure, the diffractive optical components are represented by circles. The diffractive optical components can also have other shapes, such as triangles, squares, rectangles, or irregular shapes. The diffractive optical components can have various sizes. The diffractive optical components do not have to be located on the grid points, and their positions can be changed. Figure 6 The figures in FIG are for illustration purposes only. Actual diffractive optical components may differ from those shown in the figures. Different arrangements of diffractive optical components may be used to implement different matrix calculations, such as different matrix multiplication functions.

[0524] An optimization process can be used to determine the configuration of the diffractive optical component. For example, the substrate can be divided into an array of pixels, and each pixel can be filled with the substrate material (no holes) or filled with air (holes). The configuration of the pixels can be modified iteratively, and for each configuration of the pixels, a simulation can be performed by passing light through the diffractive optical component and evaluating the output. After performing simulations of all possible configurations of the pixels, the configuration that provides the result closest to the desired matrix processing is selected as the diffractive optical component configuration of the two-dimensional OMM unit 502.

[0525] As another example, a diffractive component is initially configured as an array of holes. The position, size, and shape of the holes may vary slightly from their initial configuration. The parameters of each hole may be iteratively adjusted, and simulations may be performed to find the optimal configuration of the holes.

[0526] In some embodiments, a machine learning process is used to design a diffractive optical component. An analytical function is determined to determine how pixels affect input light to produce output light, and an optimization process (e.g., a gradient descent method) is used to determine the optimal configuration of the pixels.

[0527] In some embodiments, the two-dimensional OMM unit 502 can be implemented as a user-changeable component, and different two-dimensional OMM units 502 with different optical interference units 600 can be installed for different applications. For example, the system 500 can be configured as an optical character recognition system, and the optical interference units 600 can be configured to implement a neural network for performing optical character recognition. For example, a first OMM unit can have a first optical interference unit, which includes a passive diffractive optical component, configured to implement a first neural network for an optical character recognition engine for a first set of written languages ​​and fonts. A second OMM unit can have a second optical interference unit, which includes a passive diffractive optical component, configured to implement a second neural network for an optical character recognition engine for a second set of written languages ​​and fonts, etc. When a user wants to use the system 500 to apply optical character recognition to the first set of written languages ​​and fonts, the user can insert the first OMM unit into the system. When the user wants to use the system 500 to apply optical character recognition to the second set of written languages ​​and fonts, the user can swap out the first OMM unit and insert the second OMM unit into the system.

[0528] For example, system 500 can be configured as a speech recognition system, and optical interference unit 600 can be configured to implement a neural network for performing speech recognition. For example, a first OMM unit can have a first optical interference unit, which includes a passive diffractive optical component, which is configured to implement a first neural network for a speech recognition engine for a first spoken language. A second OMM unit can have a second optical interference unit, which includes a passive diffractive optical component, which is configured to implement a second neural network for a speech recognition engine for a second spoken language, and so on. When a user wants to use system 500 to recognize speech in a first spoken language, the user can insert the first OMM unit into the system. When the user wants to use system 500 to recognize speech in a second spoken language, the user can swap out the first OMM unit and insert the second OMM unit into the system.

[0529] For example, system 500 can be part of a control unit of an autonomous vehicle, and optical interference unit 600 can be configured to implement a neural network for performing road condition recognition. For example, a first optical interference unit (OMM) may include a first optical interference unit, which includes a passive diffractive optical component, and is configured to implement a first neural network for recognizing road conditions (including road signs) in the United States. A second OMM may include a second optical interference unit, which includes a passive diffractive optical component, and is configured to implement a second neural network for recognizing road conditions (including road signs) in Canada. A third OMM may include a third optical interference unit, which includes a passive diffractive optical component, and is configured to implement a third neural network for recognizing road conditions (including road signs) in Mexico, and so on. When the autonomous vehicle is used in the United States, the first OMM is inserted into the system. When the autonomous vehicle crosses the border into Canada, the first OMM is swapped out and the second OMM is inserted into the system. On the other hand, when the autonomous vehicle crosses the border into Mexico, the first OMM is swapped out and the third OMM is inserted into the system.

[0530] For example, system 500 can be used for genetic sequencing. DNA sequences can be classified using a convolutional neural network implemented using system 500 including a passive diffractive optical component. For example, system 500 can implement a neural network for distinguishing tumor types, predicting tumor grade, and predicting patient survival from gene expression patterns. For example, system 500 can implement a neural network for identifying a subset of genes or features that are most predictive of an analyzed characteristic. For example, system 500 can implement a neural network for predicting or inferring the expression levels of all genes from a profile of a subset of genes. For example, system 500 can implement a neural network for epigenomic analysis, such as predicting transcription factor binding sites, enhancer regions, and chromatin accessibility from a gene sequence. For example, system 500 can implement a neural network for capturing structure within a gene sequence.

[0531] For example, system 500 can be configured as a medical diagnostic system, and two-dimensional OMM units 502 can be configured to implement a neural network for analyzing physiological parameters to screen for diseases. For example, system 500 can be configured as a bacterial detection system, and two-dimensional OMM units 502 can be configured to implement a multiplication function for analyzing DNA sequences to detect certain bacterial strains.

[0532] In some embodiments, the two-dimensional OMM unit 502 includes a housing (e.g., a cartridge) that protects a substrate having a diffractive optical component. The housing supports an input interface coupled to the input waveguide 602 and an output interface coupled to the output waveguide 604. The input interface is configured to receive the output from the modulator array 144, and the output interface is configured to transmit the output of the two-dimensional OMM unit 502 to the detection unit 146. The two-dimensional OMM unit 502 can be designed as a module suitable for handling by ordinary consumers, allowing users to easily switch from one two-dimensional OMM unit 502 to another two-dimensional OMM unit 502. Machine learning technology improves over time. Users can upgrade the system 500 by swapping out the old two-dimensional OMM unit 502 and inserting a new upgraded version.

[0533] Similar to how optical compact discs can store digital information that can be retrieved by CD players, OMM units can store neural network configurations that can be used in optical processors. Just as optical compact discs are a low-cost medium for distributing digital information (including audio, video, and software programs) to consumers, OMM units can be a low-cost medium for distributing pre-configured neural networks or matrix processing functions (e.g., multiplication, convolution, or any other linear operation) to consumers.

[0534] In some embodiments, system 500 is an optical computing platform that is configured to operate with OMM units provided by different companies. This allows different companies to develop different passive optical neural networks for various applications. The passive optical neural network is sold to end users in standardized packaging, which can be installed in the optical computing platform to allow system 500 to perform various intelligent functions.

[0535] In some embodiments, the system may have a holder mechanism for supporting multiple two-dimensional OMM units 502, and may provide a mechanical handling mechanism for automatically swapping out two-dimensional OMM units 502. The system determines which two-dimensional OMM unit 502 is required for the current application, and uses the mechanical handling mechanism to automatically retrieve the appropriate OMM unit from the holder mechanism and insert it into the optical processor 504.

[0536] For a given size of optical chip, more passive diffraction components can be assembled on the substrate than when using active interferometers (e.g. Mach-Zehnder interferometers). Figure 1B The optical interferometer unit 154 in FIG. 1 can be configured to process 200×200 matrix multiplications, while the optical interferometer unit 600 having the same overall size and using passive diffraction components (each having a size of approximately 100 nm×100 nm) can be configured to process 5000×5000 matrix multiplications.

[0537] Passive diffractive optical components consume almost no power, so the two-dimensional OMM unit 502 can be used in low-power devices, such as battery-operated devices. The two-dimensional OMM unit 502 is suitable for edge computing. For example, the two-dimensional OMM unit 502 can be used in smart sensors, where the raw data from the sensor is processed using an optical processor using the two-dimensional OMM unit 502. The smart sensor can be configured to transmit the processed data to a central computer server, thereby reducing the amount of raw data transmitted to the central computer server. By placing intelligent processing functions on the smart sensor, faults and anomalies can be detected earlier and handled more efficiently. The two-dimensional OMM unit 502 is suitable for applications that need to process large matrix multiplications. The two-dimensional OMM unit 502 is suitable for applications where a neural network has been trained and the weights have been determined and do not need to be modified.

[0538] The substrate on which the diffractive optical element is formed can be planar or curved. Figure 6 In the example shown in FIG5 , input light enters the light interference unit 600 from the left side, and output light exits the light interference unit 600 from the right side (the terms "left," "right," "upper," and "lower" refer to the directions shown in the drawings). In some embodiments, the passive diffractive optical component can be configured so that some output light exits the light interference unit from the top or bottom, or any combination of the left, right, upper, and lower sides of the light interference unit 600. The substrate for the light interference unit 600 can have various shapes, such as square, rectangular, triangular, circular, or elliptical. The light interference unit 600 can include a reflective component or a mirror to redirect the propagation direction of light.

[0539] In some embodiments, the artificial neural network computing system 500 can be modified by adding an analog nonlinear unit 310 between the detection unit 146 and the ADC unit 160. The analog nonlinear unit 310 is configured to receive the output voltage from the detection unit 146, apply a nonlinear transfer function, and output the converted output voltage to the ADC unit 160. The controller 110 can obtain a converted digital output voltage corresponding to the converted output voltage from the ADC unit 160. Because the digital output voltage obtained from the ADC unit 160 has already been nonlinearly transformed ("activated"), the nonlinear transformation step of the controller 110 can be omitted, thereby reducing the computational burden of the controller 110. Subsequently, the first converted voltage obtained directly from the ADC unit 160 can be stored in the storage unit 120 as a first converted digital output vector.

[0540] The light interference unit can be realized using passive diffractive optical components arranged in three dimensions. Figure 7In some embodiments, the artificial neural network computing system 700 has an optical processor 702 that includes a three-dimensional OMM unit 708. The system 700 includes a storage unit 120 and an ADC unit 160, which are similar to Figure 5 The optical processor 702 is configured to perform matrix calculations using diffractive optical components arranged in three dimensions.

[0541] The optical processor 702 includes a laser unit 704 configured to output a two-dimensional beam array 714, and a two-dimensional modulator array 706 configured to modulate the two-dimensional beam array 714 to generate a modulated two-dimensional beam array 716. The optical processor 702 includes a three-dimensional optical matrix multiplication (OMM) unit 708 having a three-dimensional arrangement of diffractive optical components and configured to process the modulated two-dimensional beam array 716 and generate a two-dimensional output beam array 718. The optical processor 702 includes a detection unit 710 having a two-dimensional light sensor array to detect the two-dimensional output beam array 718. The ADC unit 160 converts the output of the detection unit 710 into a digital signal.

[0542] For example, the 3D OMM unit 708 can be implemented as a passive integrated silicon photonic rod or cube. The three-dimensional optical matrix multiplication unit 708 can be configured to implement a diffractive neuron network and can perform matrix multiplication with almost zero power consumption.

[0543] There are many ways to encode input data for use by the optical processor 702. For example, a digital input vector of length N×N can be encoded onto an optical input matrix of size N×N, which is propagated through the three-dimensional OMM unit 708. The three-dimensional OMM unit 708 performs an (N×N)×(N×N) matrix multiplication on the received optical input matrix in the optical domain. The (N×N)×(N×N) matrix multiplication performed by the three-dimensional OMM unit 708 is determined by the internal configuration of the three-dimensional OMM unit 708, which includes the size, position, and geometry of the diffractive optical components arranged in three dimensions, as well as the doping of impurities (if any).

[0544] The three-dimensional OMM unit 708 can be implemented in various ways. Figure 8A diagram illustrates an example of a three-dimensional OMM cell 708 using a three-dimensional arrangement of diffractive components. The three-dimensional OMM cell 708 may include an input waveguide matrix for receiving an optical input matrix 802, a three-dimensional optical interferometer 804 in optical communication with the input waveguide matrix, and an output waveguide matrix in optical communication with the optical interferometer 804 for providing an optical output matrix 806. The optical interferometer 804 includes a plurality of diffractive optical components and performs a conversion (e.g., a linear transformation) from an optical input (e.g., an N×N vector or matrix) to an optical output (e.g., an N×N vector or matrix). The output waveguide matrix guides the optical signals output by the optical interferometer 804. At least one input waveguide in the input waveguide matrix is ​​in optical communication with each output waveguide in the output waveguide matrix via the optical interferometer 804. For example, for an optical input vector of length N×N, the three-dimensional OMM cell 708 may include N×N input waveguides and N×N output waveguides.

[0545] In some embodiments, the optical interference unit 804 includes a substrate block having a diffraction component, and the diffraction component is arranged in three dimensions (for example, in a 3D matrix). For example, a plurality of holes can be drilled or etched in each of a plurality of substrate slices, and a plurality of substrate slices can be combined to form a substrate block. The size of these holes can be comparable in order of magnitude to the size of the wavelength of the input light, so that the light is diffracted by the holes (or the structure defining the holes). The holes can have the same or different sizes. The holes can also have other cross-sectional shapes, such as triangular, square, rectangular, hexagonal, or irregular shapes. In some embodiments, a holographic method can be used to form a three-dimensional diffraction optical component throughout the substrate block. The substrate can be made of a material that is transparent or translucent to the input light, for example, having a transmission rate of 1% to 99% relative to the input light.

[0546] When designing a three-dimensional OMM unit 708, the size and position of the diffractive optical assembly in the x, y, and z directions are considered. An optimization process can be used to determine the configuration of the diffractive optical assembly. For example, the substrate block can be divided into a three-dimensional matrix of pixels, and each pixel can be filled with the substrate material (no holes) or filled with air (holes). The configuration of the pixels can be modified iteratively, and for each configuration of pixels, simulations can be performed by passing light through the diffractive optical assembly and evaluating the output. After performing simulations of all possible pixel configurations, the configuration that provides the closest result to the desired matrix processing is selected as the diffractive optical assembly configuration for the three-dimensional OMM unit 708.

[0547] As another example, a diffractive component is initially configured as a three-dimensional matrix of holes. The position, size, and shape of the holes can vary slightly from their initial configuration. The parameters of each hole can be iteratively adjusted, and simulations can be performed to find the optimal configuration of the holes.

[0548] In some embodiments, a machine learning process is used to design a three-dimensional diffractive optical component. An analytical function is determined to determine how pixels affect input light, and a gradient descent method is used to determine the optimal configuration of the pixels.

[0549] In some embodiments, the three-dimensional OMM unit 708 can be implemented as a user-changeable component, and different three-dimensional OMM units 708 with different optical interferometer units 804 can be installed for different applications. For example, the system 700 can be configured as a medical diagnostic system, and the optical interferometer unit 804 can be configured to implement a neural network for analyzing physiological parameters to screen for diseases. For example, a first OMM unit may have a first optical interferometer unit, comprising a 3D passive diffractive optical component, configured to implement a first neural network for screening for a first set of diseases. A second OMM unit may have a second optical interferometer unit, comprising a 3D passive diffractive optical component, configured to implement a second neural network for screening for a second set of diseases, and so on. The first and second OMM units can be developed by different companies specializing in technology for screening for different diseases. When a user wants to use the system 700 to screen for the first set of diseases, the user can insert the first OMM unit into the system. When the user wants to use the system 700 to screen for the second set of diseases, the user can swap out the first OMM unit and insert the second OMM unit.

[0550] For example, system 700 may be configured as an optical character recognition system, and optical interferometry unit 804 may be configured to implement a neural network for performing optical character recognition. For example, system 700 may be configured as a speech recognition system, and optical interferometry unit 804 may be configured to implement a neural network for performing speech recognition. For example, system 700 may be part of a control unit of an autonomous vehicle, and optical interferometry unit 804 may be configured to implement a neural network for performing road condition recognition.

[0551] For example, system 700 can be used for gene sequencing. DNA sequences can be classified using a convolutional neural network implemented using system 700 including a passive diffractive optical component. For example, system 700 can implement a neural network for distinguishing tumor types, predicting tumor grade, and predicting patient survival from gene expression patterns. For example, system 700 can implement a neural network for identifying a subset of genes or features that are most predictive of the analyzed characteristics. For example, system 700 can implement a neural network for predicting or inferring the expression levels of all genes from a data graph of a subset of genes. For example, system 700 can implement a neural network for epigenomic analysis, such as predicting transcription factor binding sites, enhancer regions, and chromatin accessibility from gene sequences. For example, system 700 can implement a neural network for capturing structures within gene sequences. For example, system 700 can be configured as a bacterial detection system, and optical interferometry unit 804 can be configured to implement a multiplication function for analyzing DNA sequences to detect certain bacterial strains.

[0552] In some embodiments, the three-dimensional OMM unit 708 includes a housing (e.g., a cassette) that protects a substrate having a 3D diffractive optical component. The housing supports an input interface...

Claims

1. A computing system comprising: a first unit configured to generate a plurality of modulator control signals; Processor unit, including: a light source or port configured to provide a plurality of light outputs; a first set of optical modulators coupled to the light source or port and the first unit, the optical modulators in the first set of optical modulators being configured to generate an optical input vector by modulating the plurality of optical outputs provided by the light source or port based on digital input values ​​corresponding to a first set of modulator control signals in the plurality of modulator control signals, the optical input vector comprising a plurality of optical signals; and a matrix multiplication unit comprising a second set of optical modulators, wherein the matrix multiplication unit is coupled to the first unit and configured to transform the optical input vector into an analog output vector based on a plurality of digital weight values ​​corresponding to a second set of modulator control signals among the plurality of modulator control signals applied to the second set of optical modulators, wherein the matrix multiplication unit comprises: a plurality of replication modules, for each of at least two subsets of one or more optical signals of the optical input vector, the plurality of replication modules comprising a respective group of one or more replication modules configured to split the subset of the one or more optical signals into two or more replicas of the optical signals, at least one optical modulator of at least one of the first group of optical modulators or the second group of optical modulators being configured to modulate the optical signal based on a first modulator control signal of the plurality of modulator control signals, and the first unit being configured to shape the first modulator control signal to include a bandwidth enhancement associated with an amplitude variation associated with a corresponding variation in consecutive digital values ​​corresponding to the first modulator control signal.

2. The computing system of claim 1 , further comprising: a second unit coupled to the matrix multiplication unit, and the second unit is configured to convert the analog output vector into a digital output vector; as well as The controller, including an integrated circuit, is configured to perform operations including: receiving an artificial neural network computation request, the artificial neural network computation request including an input data set, the input data set including a first digital input vector; receiving a first plurality of neural network weights; and A first plurality of modulator control signals are generated, by the first unit, based on the first digital input vector, and a first plurality of weight control signals are generated based on the first plurality of neural network weights. The computing system of claim 1 , wherein the first unit comprises a digital-to-analog converter.

4. The computing system of claim 2 , further comprising a storage unit configured to store the data set and the plurality of neural network weights.

5. The computing system of claim 4, wherein the integrated circuit of the controller is further configured to perform operations including storing the input data set and the first plurality of neural network weights in the memory unit.

6. The computing system of claim 2, wherein the controller comprises an application specific integrated circuit, and Receiving the artificial neural network computation request includes receiving the artificial neural network computation request from a general purpose data processor.

7. The computing system of claim 2, wherein the first unit, the processor unit, the second unit, and the controller are arranged on at least one of a multi-chip module or an integrated circuit, and Receiving the artificial neural network computation request includes receiving the artificial neural network computation request from a second data processor, wherein the second data processor is external to the multi-chip module or the integrated circuit, the second data processor is coupled to the multi-chip module or the integrated circuit via a communication channel, and the processor unit is capable of processing data at a data rate that is at least one order of magnitude greater than a data rate of the communication channel.

8. The computing system according to any one of claims 2 to 7, wherein the first unit, the processor unit, the second unit, and the controller are used in a photoelectric processing cycle that is repeated in multiple iterations, and the photoelectric processing cycle comprises: (1) at least a first optical modulation operation based on at least one of the plurality of modulator control signals, and at least a second optical modulation operation based on at least one of the weight control signals, and (2) At least one of (a) an electrical summing operation or (b) an electrical storage operation.

9. The computing system of claim 8, wherein the photoelectric processing cycle includes an electrical storage operation, and the electrical storage operation is performed using a storage unit coupled to the controller, The operations performed by the controller further include storing the input data set and the first plurality of neural network weights in the storage unit.

10. The computing system of claim 8, wherein the optoelectronic processing cycle includes an electrical summation operation, and the electrical summation operation is performed using an electrical summation module within the matrix multiplication unit, The electrical summation module is configured to generate a current corresponding to an element of the analog output vector, the current representing the sum of the corresponding element of the optical input vector multiplied by the corresponding neural network weight.

11. The computing system of any one of claims 1 to 7, wherein the first modulator control signal comprises an analog signal associated with a plurality of predetermined amplitude levels, and each amplitude level is associated with a different corresponding digital value.

12. The computing system of claim 11, wherein the first modulator control signal comprises an analog signal associated with two predetermined amplitude levels, and each amplitude level is associated with a different corresponding binary value.

13. The computing system of claim 12, wherein the consecutive digital values ​​comprise consecutive binary values ​​in a series of binary values.

14. The computing system of claim 13 , wherein the controller is configured to shape the first modulator control signal to include a bandwidth enhancement for an initial portion of the second time interval by increasing a magnitude of an amplitude variation between a first predetermined amplitude level associated with the first time interval and a second predetermined amplitude level associated with the second time interval.

15. The computing system of claim 13, wherein the series of binary values ​​is used to determine an amplitude level of the first modulator control signal used to modulate the optical signal according to a non-return-to-zero modulation scheme.

16. The computing system of claim 13 , wherein the first unit is configured to shape the first modulator control signal to include bandwidth enhancement by pumping a current between a diode structure of a first modulator in the second group of optical modulators and a capacitor connected in series between the diode structure and a circuit providing the first modulator control signal, and wherein an amount of charge transferred by the pumping current is determined at least in part based on a voltage that is constant over a time period during which continuous digital values ​​are provided.

17. A computing device comprising: a plurality of optical waveguides coupled to a first set of optical amplitude modulators, wherein a set of a plurality of input values ​​is encoded on respective optical signals carried by the optical waveguides using the first set of optical amplitude modulators; a plurality of replication modules, and for each of the at least two subsets of the one or more optical signals, a corresponding set of the one or more replication modules is configured to split the subset of the one or more optical signals into two or more copies of the optical signals; a plurality of multiplication modules, each multiplication module including an optical amplitude modulator from the second set of optical amplitude modulators, and for each of the at least two copies of the first subset of the one or more optical signals, the corresponding multiplication module is configured to multiply the one or more optical signals in the first subset by one or more matrix element values ​​using the optical amplitude modulator from the second set of optical amplitude modulators; as well as One or more summing modules, and for the results of two or more multiplication modules, a corresponding one summing module is configured to generate an electrical signal, the electrical signal representing the sum of the results of the two or more multiplication modules; At least one optical amplitude modulator of at least one of the first group of optical amplitude modulators or the second group of optical amplitude modulators is configured to modulate the optical signal by the modulation value using a power that increases monotonically with respect to an absolute value of the modulation value.

18. The computing device of claim 17 , wherein at least one optical amplitude modulator of at least one of the first group of optical amplitude modulators or the second group of optical amplitude modulators comprises a coherence-sensitive optical amplitude modulator configured to modulate an optical signal by the modulation value based on interference between light waves, the light waves having a coherence length that is at least as long as a propagation distance through the coherence-sensitive optical amplitude modulator.

19. The computing device of claim 18 , wherein the coherent sensitive optical amplitude modulator comprises a Mach-Zehnder interferometer that distributes a light wave guided by an input optical waveguide to a first optical waveguide arm and a second optical waveguide arm of the Mach-Zehnder interferometer, wherein the first optical waveguide arm comprises an active phase shifter that generates a relative phase shift relative to a phase delay of the second optical waveguide arm, and wherein the Mach-Zehnder interferometer combines the light waves from the first optical waveguide arm and the second optical waveguide arm into at least one output optical waveguide.

20. The computing device of claim 19, wherein the power used to modulate the optical signal by the modulation value comprises power applied to an active phase shifter.

Images